[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 420382558 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002231] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.009132] LSM: Security Framework initializing [ 0.010048] Yama: becoming mindful. [ 0.011037] SELinux: Initializing. [ 0.012068] *** VALIDATE selinux *** [ 0.020114] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025541] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026166] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027123] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028121] *** VALIDATE tmpfs *** [ 0.029463] *** VALIDATE proc *** [ 0.030204] *** VALIDATE cgroup *** [ 0.031008] *** VALIDATE cgroup2 *** [ 0.032229] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033143] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035026] Spectre V2 : User space: Vulnerable [ 0.036009] Speculative Store Bypass: Vulnerable [ 0.039571] debug: unmapping init [mem 0xffffffffb4259000-0xffffffffb4260fff] [ 0.041165] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042731] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043023] ... version: 2 [ 0.044014] ... bit width: 48 [ 0.045014] ... generic registers: 4 [ 0.046011] ... value mask: 0000ffffffffffff [ 0.047010] ... max period: 00007fffffffffff [ 0.048008] ... fixed-purpose events: 3 [ 0.049010] ... event mask: 000000070000000f [ 0.050304] rcu: Hierarchical SRCU implementation. [ 0.052431] smp: Bringing up secondary CPUs ... [ 0.053484] x86: Booting SMP configuration: [ 0.054023] .... node #0, CPUs: #1 #2 #3 [ 0.057349] smp: Brought up 1 node, 4 CPUs [ 0.059014] smpboot: Max logical packages: 1 [ 0.060019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.148291] node 0 deferred pages initialised in 87ms [ 0.151442] devtmpfs: initialized [ 0.153325] x86/mm: Memory block size: 128MB [ 0.156965] gcov: version magic: 0x41383552 [ 0.159238] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.163150] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.166295] pinctrl core: initialized pinctrl subsystem [ 0.168207] [ 0.168773] ************************************************************* [ 0.171015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.173011] ** ** [ 0.174010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.176011] ** ** [ 0.178015] ** This means that this kernel is built to expose internal ** [ 0.181013] ** IOMMU data structures, which may compromise security on ** [ 0.183014] ** your system. ** [ 0.186013] ** ** [ 0.188013] ** If you see this message and you are not debugging the ** [ 0.190015] ** kernel, report this immediately to your vendor! ** [ 0.193016] ** ** [ 0.195011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.197013] ************************************************************* [ 0.199698] NET: Registered protocol family 16 [ 0.201449] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.204075] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.207061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.210101] cpuidle: using governor menu [ 0.211876] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.214683] PCI: Using configuration type 1 for base access [ 0.217131] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.227107] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.229015] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.231103] cryptd: max_cpu_qlen set to 1000 [ 0.234271] ACPI: Added _OSI(Module Device) [ 0.236019] ACPI: Added _OSI(Processor Device) [ 0.238015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.239010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.244723] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.249477] ACPI: Interpreter enabled [ 0.251062] ACPI: PM: (supports S0 S3 S4 S5) [ 0.252014] ACPI: Using IOAPIC for interrupt routing [ 0.254119] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.257419] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.267352] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.270064] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.273023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.275086] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.279953] acpiphp: Slot [2] registered [ 0.280097] acpiphp: Slot [5] registered [ 0.282119] acpiphp: Slot [6] registered [ 0.283092] acpiphp: Slot [3] registered [ 0.284076] acpiphp: Slot [4] registered [ 0.285083] acpiphp: Slot [7] registered [ 0.285955] acpiphp: Slot [8] registered [ 0.287074] acpiphp: Slot [9] registered [ 0.287953] acpiphp: Slot [10] registered [ 0.289074] acpiphp: Slot [11] registered [ 0.289945] acpiphp: Slot [12] registered [ 0.290057] acpiphp: Slot [13] registered [ 0.291042] acpiphp: Slot [14] registered [ 0.291871] acpiphp: Slot [15] registered [ 0.293071] acpiphp: Slot [16] registered [ 0.294075] acpiphp: Slot [17] registered [ 0.294913] acpiphp: Slot [18] registered [ 0.296070] acpiphp: Slot [19] registered [ 0.297020] acpiphp: Slot [20] registered [ 0.297859] acpiphp: Slot [21] registered [ 0.299113] acpiphp: Slot [22] registered [ 0.300041] acpiphp: Slot [23] registered [ 0.301099] acpiphp: Slot [24] registered [ 0.302100] acpiphp: Slot [25] registered [ 0.304071] acpiphp: Slot [26] registered [ 0.305111] acpiphp: Slot [27] registered [ 0.306062] acpiphp: Slot [28] registered [ 0.308068] acpiphp: Slot [29] registered [ 0.309098] acpiphp: Slot [30] registered [ 0.311109] acpiphp: Slot [31] registered [ 0.313079] PCI host bridge to bus 0000:00 [ 0.314017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.316021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.318022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.320024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.322023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.324024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.326225] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.329525] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.332321] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.339014] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.343042] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.346019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.348013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.349010] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.351545] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.353631] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.355042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.357829] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.361832] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.371018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.375015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.379421] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.384016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.391016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.407020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.415104] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.427019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.436017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.454022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.463721] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.465290] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.467287] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.470310] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.472163] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.477082] iommu: Default domain type: Passthrough [ 0.478371] SCSI subsystem initialized [ 0.480081] ACPI: bus type USB registered [ 0.481054] usbcore: registered new interface driver usbfs [ 0.482070] usbcore: registered new interface driver hub [ 0.484106] usbcore: registered new device driver usb [ 0.486191] pps_core: LinuxPPS API ver. 1 registered [ 0.488014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.494067] PTP clock support registered [ 0.496147] EDAC MC: Ver: 3.0.0 [ 0.497386] PCI: Using ACPI for IRQ routing [ 0.498777] NetLabel: Initializing [ 0.499011] NetLabel: domain hash size = 128 [ 0.500010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.501197] NetLabel: unlabeled traffic allowed by default [ 0.503193] vgaarb: loaded [ 0.505291] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.507012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.513335] clocksource: Switched to clocksource kvm-clock [ 0.623932] VFS: Disk quotas dquot_6.6.0 [ 0.625526] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.627983] *** VALIDATE ramfs *** [ 0.629095] *** VALIDATE hugetlbfs *** [ 0.630456] pnp: PnP ACPI init [ 0.632643] pnp: PnP ACPI: found 6 devices [ 0.649158] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.652396] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.654724] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.656970] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.659472] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.661980] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.664896] NET: Registered protocol family 2 [ 0.667361] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.672117] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.676021] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.681387] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.684694] TCP: Hash tables configured (established 65536 bind 65536) [ 0.687232] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.690109] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.693155] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.695653] NET: Registered protocol family 1 [ 0.697645] RPC: Registered named UNIX socket transport module. [ 0.699446] RPC: Registered udp transport module. [ 0.700879] RPC: Registered tcp transport module. [ 0.702743] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.705327] NET: Registered protocol family 44 [ 0.706994] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.708900] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.711315] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.713974] PCI: CLS 0 bytes, default 64 [ 0.715862] Unpacking initramfs... [ 2.095693] debug: unmapping init [mem 0xffff8ee4fcc64000-0xffff8ee4fffcffff] [ 2.099191] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.101511] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.104496] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.610086] Initialise system trusted keyrings [ 2.613155] Key type blacklist registered [ 2.615283] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.624590] zbud: loaded [ 2.628111] *** VALIDATE nfs *** [ 2.629474] *** VALIDATE nfs4 *** [ 2.631188] pstore: using deflate compression [ 2.635423] Platform Keyring initialized [ 2.743743] NET: Registered protocol family 38 [ 2.745665] Key type asymmetric registered [ 2.747395] Asymmetric key parser 'x509' registered [ 2.749538] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.752492] io scheduler mq-deadline registered [ 2.754507] io scheduler kyber registered [ 2.756405] io scheduler bfq registered [ 2.758437] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.761424] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.764494] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.767854] ACPI: Power Button [PWRF] [ 2.772953] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.777914] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.786419] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.811026] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.839016] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.844143] Non-volatile memory driver v1.3 [ 2.845914] Linux agpgart interface v0.103 [ 2.878679] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 2.881863] vda: detected capacity change from 0 to 74702848 [ 2.902961] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.906616] vdb: detected capacity change from 0 to 1073741824 [ 2.915646] libphy: Fixed MDIO Bus: probed [ 2.925135] usbcore: registered new interface driver usbserial_generic [ 2.927392] usbserial: USB Serial support registered for generic [ 2.929794] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.933937] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.935698] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.937937] mousedev: PS/2 mouse device common for all mice [ 2.940585] rtc_cmos 00:05: RTC can wake from S4 [ 2.943295] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.943371] rtc_cmos 00:05: registered as rtc0 [ 2.948539] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.951086] intel_pstate: CPU model not supported [ 2.953723] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.955390] hid: raw HID events driver (C) Jiri Kosina [ 2.959905] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.963751] usbcore: registered new interface driver usbhid [ 2.965892] usbhid: USB HID core driver [ 2.967540] drop_monitor: Initializing network drop monitor service [ 2.970089] Initializing XFRM netlink socket [ 2.972299] NET: Registered protocol family 10 [ 2.975684] Segment Routing with IPv6 [ 2.977255] NET: Registered protocol family 17 [ 2.979671] mpls_gso: MPLS GSO support [ 2.985639] RAS: Correctable Errors collector initialized. [ 2.987704] AVX version of gcm_enc/dec engaged. [ 2.989289] AES CTR mode by8 optimization enabled [ 3.055057] sched_clock: Marking stable (3055012962, 0)->(3935075895, -880062933) [ 3.058708] registered taskstats version 1 [ 3.060729] Loading compiled-in X.509 certificates [ 3.062652] zswap: loaded using pool lzo/zbud [ 3.088463] Key type big_key registered [ 3.100797] Key type encrypted registered [ 3.102376] ima: No TPM chip found, activating TPM-bypass! [ 3.104435] ima: Allocated hash algorithm: sha1 [ 3.106152] ima: No architecture policies found [ 3.107815] evm: Initialising EVM extended attributes: [ 3.109706] evm: security.selinux [ 3.110845] evm: security.ima [ 3.111854] evm: security.capability [ 3.113119] evm: HMAC attrs: 0x1 [ 3.115503] rtc_cmos 00:05: setting system clock to 2026-08-15 06:10:00 UTC (1786774200) [ 3.121720] debug: unmapping init [mem 0xffffffffb5203000-0xffffffffb53fffff] [ 3.125182] debug: unmapping init [mem 0xffffffffb3f82000-0xffffffffb4258fff] [ 3.135111] Write protecting the kernel read-only data: 28672k [ 3.138413] debug: unmapping init [mem 0xffffffffb2603000-0xffffffffb27fffff] [ 3.140660] debug: unmapping init [mem 0xffffffffb2f14000-0xffffffffb2ffffff] [ 3.173025] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.181809] systemd[1]: Detected virtualization kvm. [ 3.183762] systemd[1]: Detected architecture x86-64. [ 3.185776] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.213673] systemd[1]: No hostname configured. [ 3.215445] systemd[1]: Set hostname to . [ 3.217610] random: systemd: uninitialized urandom read (16 bytes read) [ 3.220102] systemd[1]: Initializing machine ID from random generator. [ 3.271138] random: ln: uninitialized urandom read (6 bytes read) [ 3.351094] random: systemd: uninitialized urandom read (16 bytes read) [ 3.353639] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.357358] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.360284] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.606743] device-mapper: uevent: version 1.0.3 [ 4.610808] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.288647] virtio_net virtio0 ens2: renamed from eth0 [ 6.420572] random: fast init done [ 6.619341] scsi host0: ata_piix [ 6.704659] scsi host1: ata_piix [ 6.706192] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 6.711893] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 11.621993] random: crng init done [ 11.623806] random: 7 urandom warning(s) missed due to ratelimiting [ 13.003630] dracut-initqueue[580]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 15.741919] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.655607] printk: systemd: 25 output lines suppressed due to ratelimiting [ 19.454914] SELinux: Disabled at runtime. [ 19.578710] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 19.594794] systemd[1]: Detected virtualization kvm. [ 19.603114] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.270184] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.281332] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.293865] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.308868] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.316485] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.331895] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.388707] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... [ OK ] Reached target Slices. [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ 22.108976] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 23.157153] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 23.793748] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 23.882631] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 24.397090] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 24.499853] EDAC sbridge: Ver: 1.1.2 [ 28.590376] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 29.506705] NFS: Registering the id_resolver key type [ 29.510847] Key type id_resolver registered [ 29.512761] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg229-client login: [ 79.045134] libcfs: loading out-of-tree module taints kernel. [ 79.195799] Key type ._llcrypt registered [ 79.247290] Key type .llcrypt registered [ 79.768107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 79.793307] alg: No test for adler32 (adler32-zlib) [ 80.024455] hrtimer: interrupt took 4478594 ns [ 81.432459] Lustre: Lustre: Build Version: 2.17.57_1_g05abe65 [ 82.501576] LNet: Added LNI 192.168.202.29@tcp [8/256/0/180] [ 84.304189] Key type lgssc registered [ 85.859638] Lustre: Echo OBD driver; http://www.lustre.org/ [ 204.723794] Lustre: Mounted lustre-client - version 2.17.57_1_g05abe65 [ 210.443912] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 223.875688] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing check_logdir /tmp/testlogs/ [ 228.184445] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing yml_node [ 230.373127] Lustre: lustre-OST0000-osc-ffff8ee54811e800: disconnect after 23s idle [ 232.621612] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 235.424877] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 238.765147] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 240.400814] Lustre: DEBUG MARKER: -----============= acceptance-small: recovery-small ============----- Sat Aug 15 02:13:56 EDT 2026 [ 254.587203] Lustre: DEBUG MARKER: excepting tests: 136 [ 256.364899] Lustre: DEBUG MARKER: === recovery-small: start setup 02:14:12 (1786774452) === [ 261.869883] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing check_config_client /mnt/lustre [ 278.711245] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 290.145452] Lustre: DEBUG MARKER: === recovery-small: finish setup 02:14:45 (1786774485) === [ 292.127980] Lustre: DEBUG MARKER: == recovery-small test 1: create, chmod, stat: drop req, drop rep ========================================================== 02:14:48 (1786774488) [ 308.704159] Lustre: 10012:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786774490/real 1786774490] req@ffff8ee546234a80 x1873568626388736/t0(0) o700->lustre-MDT0000-mdc-ffff8ee54811e800@192.168.202.129@tcp:30/10 lens 264/248 e 0 to 1 dl 1786774506 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 308.742517] Lustre: lustre-MDT0000-mdc-ffff8ee54811e800: Connection to lustre-MDT0000 (at 192.168.202.129@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 308.779813] Lustre: lustre-MDT0000-mdc-ffff8ee54811e800: Connection restored to 192.168.202.129@tcp (at 192.168.202.129@tcp) [ 327.139437] Lustre: 10033:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786774508/real 1786774508] req@ffff8ee546236300 x1873568626390528/t0(0) o36->lustre-MDT0000-mdc-ffff8ee54811e800@192.168.202.129@tcp:12/10 lens 520/576 e 0 to 1 dl 1786774524 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 327.170379] Lustre: lustre-MDT0000-mdc-ffff8ee54811e800: Connection to lustre-MDT0000 (at 192.168.202.129@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 327.209293] Lustre: lustre-MDT0000-mdc-ffff8ee54811e800: Connection restored to 192.168.202.129@tcp (at 192.168.202.129@tcp) [ 346.592243] Lustre: 10059:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786774527/real 1786774527] req@ffff8ee550b03100 x1873568626391808/t0(0) o101->lustre-MDT0000-mdc-ffff8ee54811e800@192.168.202.129@tcp:12/10 lens 576/1152 e 0 to 1 dl 1786774543 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'tchmod.0' uid:0 gid:0 projid:0 [ 346.613702] Lustre: lustre-MDT0000-mdc-ffff8ee54811e800: Connection to lustre-MDT0000 (at 192.168.202.129@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 346.652520] Lustre: lustre-MDT0000-mdc-ffff8ee54811e800: Connection restored to 192.168.202.129@tcp (at 192.168.202.129@tcp) [ 366.563418] Lustre: 10080:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786774547/real 1786774547] req@ffff8ee550f38e00 x1873568626393984/t0(0) o36->lustre-MDT0000-mdc-ffff8ee54811e800@192.168.202.129@tcp:12/10 lens 488/512 e 0 to 1 dl 1786774563 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'tchmod.0' uid:0 gid:0 projid:4294967295 [ 366.598733] Lustre: lustre-MDT0000-mdc-ffff8ee54811e800: Connection to lustre-MDT0000 (at 192.168.202.129@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 366.645785] Lustre: lustre-MDT0000-mdc-ffff8ee54811e800: Connection restored to 192.168.202.129@tcp (at 192.168.202.129@tcp) [ 385.504154] Lustre: 10106:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786774566/real 1786774566] req@ffff8ee550f38e00 x1873568626395264/t0(0) o34->lustre-MDT0000-mdc-ffff8ee54811e800@192.168.202.129@tcp:12/10 lens 472/728 e 0 to 1 dl 1786774582 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'statone.0' uid:0 gid:0 projid:0 [ 385.527765] Lustre: lustre-MDT0000-mdc-ffff8ee54811e800: Connection to lustre-MDT0000 (at 192.168.202.129@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 385.580962] Lustre: lustre-MDT0000-mdc-ffff8ee54811e800: Connection restored to 192.168.202.129@tcp (at 192.168.202.129@tcp) [ 404.448165] Lustre: 10127:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786774585/real 1786774585] req@ffff8ee550b02a00 x1873568626396544/t0(0) o34->lustre-MDT0000-mdc-ffff8ee54811e800@192.168.202.129@tcp:12/10 lens 472/728 e 0 to 1 dl 1786774601 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'statone.0' uid:0 gid:0 projid:0 [ 404.477564] Lustre: lustre-MDT0000-mdc-ffff8ee54811e800: Connection to lustre-MDT0000 (at 192.168.202.129@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 404.509870] Lustre: lustre-MDT0000-mdc-ffff8ee54811e800: Connection restored to 192.168.202.129@tcp (at 192.168.202.129@tcp) [ 411.593808] Lustre: DEBUG MARKER: == recovery-small test 4: open: drop req, drop rep ======= 02:16:48 (1786774608) [ 429.025661] Lustre: 10730:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786774610/real 1786774610] req@ffff8ee55890ca80 x1873568626398848/t0(0) o101->lustre-MDT0000-mdc-ffff8ee54811e800@192.168.202.129@tcp:12/10 lens 576/1152 e 0 to 1 dl 1786774626 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'cat.0' uid:0 gid:0 projid:0 [ 429.051814] Lustre: lustre-MDT0000-mdc-ffff8ee54811e800: Connection to lustre-MDT0000 (at 192.168.202.129@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 429.092197] Lustre: lustre-MDT0000-mdc-ffff8ee54811e800: Connection restored to 192.168.202.129@tcp (at 192.168.202.129@tcp) [ 455.038502] Lustre: DEBUG MARKER: == recovery-small test 5: rename: drop req, drop rep ===== 02:17:31 (1786774651) [ 472.544389] Lustre: 11352:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786774653/real 1786774653] req@ffff8ee550f39180 x1873568626404992/t0(0) o101->lustre-MDT0000-mdc-ffff8ee54811e800@192.168.202.129@tcp:12/10 lens 576/1152 e 0 to 1 dl 1786774669 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'mv.0' uid:0 gid:0 projid:0 [ 472.584955] Lustre: 11352:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 472.589167] Lustre: lustre-MDT0000-mdc-ffff8ee54811e800: Connection to lustre-MDT0000 (at 192.168.202.129@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 472.602079] Lustre: Skipped 1 previous similar message [ 472.627448] Lustre: lustre-MDT0000-mdc-ffff8ee54811e800: Connection restored to 192.168.202.129@tcp (at 192.168.202.129@tcp) [ 472.633159] Lustre: Skipped 1 previous similar message [ 500.500520] Lustre: DEBUG MARKER: == recovery-small test 6: link, unlink: drop req, drop rep ========================================================== 02:18:16 (1786774696) [ 517.088447] Lustre: lustre-OST0000-osc-ffff8ee54811e800: disconnect after 22s idle [ 517.093813] Lustre: Skipped 1 previous similar message [ 537.568330] Lustre: 12008:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786774718/real 1786774718] req@ffff8ee550b02680 x1873568626416000/t0(0) o36->lustre-MDT0000-mdc-ffff8ee54811e800@192.168.202.129@tcp:12/10 lens 512/440 e 0 to 1 dl 1786774734 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'link.0' uid:0 gid:0 projid:4294967295 [ 537.612290] Lustre: 12008:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 537.626818] Lustre: lustre-MDT0000-mdc-ffff8ee54811e800: Connection to lustre-MDT0000 (at 192.168.202.129@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 537.652723] Lustre: Skipped 2 previous similar messages [ 537.703272] Lustre: lustre-MDT0000-mdc-ffff8ee54811e800: Connection restored to 192.168.202.129@tcp (at 192.168.202.129@tcp) [ 537.716417] Lustre: Skipped 2 previous similar messages [ 581.372313] Lustre: DEBUG MARKER: == recovery-small test 8: touch: drop rep (bug 1423) ===== 02:19:37 (1786774777) [ 606.270619] Lustre: DEBUG MARKER: == recovery-small test 9: pause bulk on OST (bug 1420) === 02:20:02 (1786774802) [ 620.689125] Lustre: DEBUG MARKER: == recovery-small test 10a: finish request on server after client eviction (bug 1521) ========================================================== 02:20:16 (1786774816) [ 621.260419] Lustre: *** cfs_fail_loc=305, val=0*** [ 623.551323] Lustre: *** cfs_fail_loc=305, val=0*** [ 636.764246] Lustre: *** cfs_fail_loc=305, val=0*** [ 638.945200] Lustre: lustre-OST0000-osc-ffff8ee54811e800: disconnect after 21s idle [ 639.821517] Lustre: *** cfs_fail_loc=305, val=0*** [ 653.126459] Lustre: *** cfs_fail_loc=305, val=0*** [ 656.191926] Lustre: *** cfs_fail_loc=305, val=0*** [ 668.493817] Lustre: *** cfs_fail_loc=305, val=0*** [ 671.532747] Lustre: *** cfs_fail_loc=305, val=0*** [ 684.904239] Lustre: *** cfs_fail_loc=305, val=0*** [ 687.916059] Lustre: *** cfs_fail_loc=305, val=0*** [ 701.284236] Lustre: *** cfs_fail_loc=305, val=0*** [ 704.308752] Lustre: *** cfs_fail_loc=305, val=0*** [ 716.593227] Lustre: *** cfs_fail_loc=305, val=0*** [ 719.667147] Lustre: *** cfs_fail_loc=305, val=0*** [ 733.365773] LustreError: lustre-MDT0000-mdc-ffff8ee54811e800: operation ldlm_enqueue to node 192.168.202.129@tcp failed: rc = -107 [ 733.374117] Lustre: lustre-MDT0000-mdc-ffff8ee54811e800: Connection to lustre-MDT0000 (at 192.168.202.129@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 733.380075] Lustre: Skipped 3 previous similar messages [ 733.392715] LustreError: lustre-MDT0000-mdc-ffff8ee54811e800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 733.408701] Lustre: lustre-MDT0000-mdc-ffff8ee54811e800: Connection restored to 192.168.202.129@tcp (at 192.168.202.129@tcp) [ 733.424939] Lustre: Skipped 3 previous similar messages [ 736.250363] Lustre: lustre-OST0001-osc-ffff8ee54811e800: Connection to lustre-OST0001 (at 192.168.202.129@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 736.281522] LustreError: lustre-OST0001-osc-ffff8ee54811e800: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 736.301836] Lustre: lustre-OST0001-osc-ffff8ee54811e800: Connection restored to 192.168.202.129@tcp (at 192.168.202.129@tcp) [ 740.856667] Lustre: DEBUG MARKER: == recovery-small test 10b: re-send BL AST =============== 02:22:17 (1786774937) [ 756.704272] Lustre: lustre-OST0000-osc-ffff8ee54811e800: disconnect after 21s idle [ 765.201077] Lustre: DEBUG MARKER: == recovery-small test 10c: re-send BL AST vs reconnect race (LU-5569) ========================================================== 02:22:40 (1786774960) [ 773.863212] Lustre: DEBUG MARKER: == recovery-small test 10d: test failed blocking ast ===== 02:22:50 (1786774970) [ 776.115434] Lustre: Unmounted lustre-client [ 776.554054] Lustre: Mounted lustre-client - version 2.17.57_1_g05abe65 [ 778.214939] LustreError: lustre-OST0000-osc-ffff8ee5496da000: operation ost_statfs to node 192.168.202.129@tcp failed: rc = -107 [ 778.227362] LustreError: lustre-OST0000-osc-ffff8ee5496da000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 778.237222] Lustre: 2352:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.129@tcp:/lustre/fid: [0x200000403:0x1:0x0]/ may get corrupted (rc -108) [ 780.978140] Lustre: Unmounted lustre-client [ 787.256519] Lustre: DEBUG MARKER: == recovery-small test 10e: re-send BL AST vs reconnect race 2 ========================================================== 02:23:03 (1786774983) [ 788.982072] Lustre: DEBUG MARKER: SKIP: recovery-small test_10e need two clients [ 790.588252] Lustre: DEBUG MARKER: == recovery-small test 11: wake up a thread waiting for completion after eviction (b=2460) ========================================================== 02:23:06 (1786774986) [ 817.263515] Lustre: DEBUG MARKER: == recovery-small test 12: recover from timed out resend in ptlrpcd (b=2494) ========================================================== 02:23:33 (1786775013) [ 817.330467] Lustre: DEBUG MARKER: multiop /mnt/lustre/f12.recovery-small OS_c [ 834.529371] Lustre: 17635:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786775015/real 1786775015] req@ffff8ee550b03100 x1873568626484736/t0(0) o35->lustre-MDT0000-mdc-ffff8ee5496da000@192.168.202.129@tcp:23/10 lens 392/624 e 0 to 1 dl 1786775031 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 [ 834.564732] Lustre: 17635:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 861.547967] Lustre: DEBUG MARKER: == recovery-small test 13: mdc_readpage restart test (bug 1138) ========================================================== 02:24:17 (1786775057) [ 886.333465] Lustre: DEBUG MARKER: == recovery-small test 14: mdc_readpage resend test (bug 1138) ========================================================== 02:24:42 (1786775082) [ 894.555878] Lustre: DEBUG MARKER: == recovery-small test 15: failed open (-ENOMEM) ========= 02:24:50 (1786775090) [ 900.951566] Lustre: DEBUG MARKER: == recovery-small test 16: timeout bulk put, don't evict client (2732) ========================================================== 02:24:57 (1786775097) [ 946.620900] Lustre: DEBUG MARKER: == recovery-small test 17a: timeout bulk get, don't evict client (2732) ========================================================== 02:25:42 (1786775142) [ 1001.652878] Lustre: DEBUG MARKER: == recovery-small test 17b: timeout bulk get, dont evict client (3582) ========================================================== 02:26:37 (1786775197) [ 1003.743219] Lustre: DEBUG MARKER: SKIP: recovery-small test_17b Needs multiple clients [ 1005.836863] Lustre: DEBUG MARKER: == recovery-small test 18a: manual ost invalidate clears page cache immediately ========================================================== 02:26:41 (1786775201) [ 1006.484864] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 1006.555035] LustreError: lustre-OST0001-osc-ffff8ee5496da000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 1014.238910] Lustre: DEBUG MARKER: == recovery-small test 18b: eviction and reconnect clears page cache (2766) ========================================================== 02:26:50 (1786775210) [ 1020.408653] LustreError: lustre-OST0000-osc-ffff8ee5496da000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1044.959041] Lustre: DEBUG MARKER: == recovery-small test 18c: Dropped connect reply after eviction handing (14755) ========================================================== 02:27:21 (1786775241) [ 1048.325231] LustreError: lustre-OST0000-osc-ffff8ee5496da000: operation ost_statfs to node 192.168.202.129@tcp failed: rc = -107 [ 1057.785348] Lustre: Evicted from lustre-OST0000_UUID (at 192.168.202.129@tcp) after server handle changed from 0xf1d563bd77354df2 to 0xf1d563bd77354f42 [ 1057.795124] LustreError: lustre-OST0000-osc-ffff8ee5496da000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1065.911647] Lustre: DEBUG MARKER: == recovery-small test 19a: test expired_lock_main on mds (2867) ========================================================== 02:27:42 (1786775262) [ 1066.272295] Lustre: Mounted lustre-client - version 2.17.57_1_g05abe65 [ 1066.275252] Lustre: Skipped 1 previous similar message [ 1099.232164] Lustre: 2353:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786775280/real 1786775280] req@ffff8ee550b03100 x1873568626541696/t0(0) o103->lustre-MDT0000-mdc-ffff8ee5496da000@192.168.202.129@tcp:17/18 lens 328/224 e 0 to 1 dl 1786775296 ref 1 fl Rpc:XQr/202/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 [ 1099.284759] Lustre: 2353:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1169.917256] LustreError: lustre-MDT0000-mdc-ffff8ee5496da000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1173.567474] Lustre: Unmounted lustre-client [ 1179.765589] Lustre: DEBUG MARKER: == recovery-small test 19b: test expired_lock_main on ost (2867) ========================================================== 02:29:35 (1786775375) [ 1180.140175] Lustre: Mounted lustre-client - version 2.17.57_1_g05abe65 [ 1261.025706] Lustre: lustre-OST0001-osc-ffff8ee5496da000: Connection to lustre-OST0001 (at 192.168.202.129@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1261.041731] Lustre: Skipped 19 previous similar messages [ 1261.062918] Lustre: lustre-OST0001-osc-ffff8ee5496da000: Connection restored to 192.168.202.129@tcp (at 192.168.202.129@tcp) [ 1261.084439] Lustre: Skipped 16 previous similar messages [ 1287.687752] LustreError: lustre-OST0001-osc-ffff8ee5496da000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 1288.514758] Lustre: Unmounted lustre-client [ 1296.062758] Lustre: DEBUG MARKER: == recovery-small test 19c: check reconnect and lock resend do not trigger expired_lock_main ========================================================== 02:31:31 (1786775491) [ 1296.756238] Lustre: Mounted lustre-client - version 2.17.57_1_g05abe65 [ 1302.082511] Lustre: Unmounted lustre-client [ 1314.644479] Lustre: DEBUG MARKER: == recovery-small test 20a: ldlm_handle_enqueue error (should return error) ========================================================== 02:31:50 (1786775510) [ 1316.424147] LustreError: lustre-OST0000-osc-ffff8ee5496da000: operation ldlm_enqueue to node 192.168.202.129@tcp failed: rc = -12 [ 1322.260144] Lustre: DEBUG MARKER: == recovery-small test 20b: ldlm_handle_enqueue error (should return error) ========================================================== 02:31:58 (1786775518) [ 1323.860127] LustreError: lustre-OST0000-osc-ffff8ee5496da000: operation ldlm_enqueue to node 192.168.202.129@tcp failed: rc = -12 [ 1330.131889] Lustre: DEBUG MARKER: == recovery-small test 21a: drop close request while close and open are both in flight ========================================================== 02:32:06 (1786775526) [ 1358.220361] Lustre: DEBUG MARKER: == recovery-small test 21b: drop open request while close and open are both in flight ========================================================== 02:32:34 (1786775554) [ 1504.495693] Lustre: DEBUG MARKER: == recovery-small test 21c: drop both request while close and open are both in flight ========================================================== 02:35:00 (1786775700) [ 1535.364683] Lustre: DEBUG MARKER: == recovery-small test 21d: drop close reply while close and open are both in flight ========================================================== 02:35:31 (1786775731) [ 1563.225453] Lustre: DEBUG MARKER: == recovery-small test 21e: drop open reply while close and open are both in flight ========================================================== 02:35:59 (1786775759) [ 1700.832506] Lustre: 29635:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786775761/real 1786775761] req@ffff8ee550b02d80 x1873568626653568/t0(0) o36->lustre-MDT0000-mdc-ffff8ee5496da000@192.168.202.129@tcp:12/10 lens 488/512 e 0 to 1 dl 1786775898 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 1700.876297] Lustre: 29635:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 14 previous similar messages [ 1702.644539] Lustre: DEBUG MARKER: == recovery-small test 21f: drop both reply while close and open are both in flight ========================================================== 02:38:18 (1786775898) [ 1730.496984] Lustre: DEBUG MARKER: == recovery-small test 21g: drop open reply and close request while close and open are both in flight ========================================================== 02:38:46 (1786775926) [ 1754.867887] Lustre: DEBUG MARKER: == recovery-small test 21h: drop open request and close reply while close and open are both in flight ========================================================== 02:39:11 (1786775951) [ 1780.371989] Lustre: DEBUG MARKER: == recovery-small test 22: drop close request and do mknod ========================================================== 02:39:36 (1786775976) [ 1805.030514] Lustre: DEBUG MARKER: == recovery-small test 23: client hang when close a file after mds crash ========================================================== 02:40:01 (1786776001) [ 1834.400242] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 192.168.202.129@tcp) was lost; in progress operations using this service will fail [ 1834.431206] Lustre: Evicted from MGS (at 192.168.202.129@tcp) after server handle changed from 0xf1d563bd773544fa to 0xf1d563bd77356acb [ 1848.855328] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1850.714128] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1858.551909] Lustre: DEBUG MARKER: == recovery-small test 24a: fsync error (should return error) ========================================================== 02:40:54 (1786776054) [ 1860.766462] LustreError: lustre-OST0000-osc-ffff8ee5496da000: operation ost_write to node 192.168.202.129@tcp failed: rc = -107 [ 1860.803475] LustreError: lustre-OST0000-osc-ffff8ee5496da000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1860.829545] Lustre: 2355:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.129@tcp:/lustre/fid: [0x200000404:0x30:0x0]// may get corrupted (rc -5) [ 1860.858928] LustreError: 34127:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8ee5496da000: namespace resource [0x240000400:0x42:0x0].0x0 (ffff8ee548100600) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 1866.928675] Lustre: DEBUG MARKER: == recovery-small test 24b: test dirty page discard due to client eviction ========================================================== 02:41:03 (1786776063) [ 1868.472832] Lustre: lustre-OST0000-osc-ffff8ee5496da000: Connection to lustre-OST0000 (at 192.168.202.129@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1868.484969] Lustre: Skipped 13 previous similar messages [ 1868.494665] LustreError: lustre-OST0000-osc-ffff8ee5496da000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1868.508336] Lustre: 2354:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.129@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108) [ 1868.531738] Lustre: lustre-OST0000-osc-ffff8ee5496da000: Connection restored to 192.168.202.129@tcp (at 192.168.202.129@tcp) [ 1868.542339] Lustre: Skipped 13 previous similar messages [ 1874.102662] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 02:41:10 (1786776070) [ 1875.727771] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node [ 1877.717803] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 02:41:13 (1786776073) [ 1879.355489] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node [ 1880.977454] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 02:41:17 (1786776077) [ 1884.297095] LustreError: lustre-MDT0000-mdc-ffff8ee5496da000: operation mds_reint to node 192.168.202.129@tcp failed: rc = -19 [ 1884.312901] LustreError: Skipped 1 previous similar message [ 1904.102493] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 192.168.202.129@tcp) was lost; in progress operations using this service will fail [ 1904.125853] Lustre: Evicted from MGS (at 192.168.202.129@tcp) after server handle changed from 0xf1d563bd77356acb to 0xf1d563bd7735840f [ 1905.715778] Lustre: 15917:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.129@tcp: IR log lustre-cliir failed, not fatal: rc = -2 [ 1924.617168] BUG: spinlock recursion on CPU#1, ptlrpcd_rcv/2351 [ 1924.623392] lock: 0xffff8ee5497bb2e0, .magic: dead4ead, .owner: ptlrpcd_rcv/2351, .owner_cpu: 1 [ 1924.644303] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 1924.661728] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 1924.672120] Call Trace: [ 1924.680230] ? dump_stack+0xbb/0x10e [ 1924.686334] ? spin_dump.cold.3+0x32/0x65 [ 1924.693332] ? do_raw_spin_lock+0x10f/0x190 [ 1924.698884] ? _raw_spin_lock+0x21/0x30 [ 1924.702168] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 1924.708997] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 1924.717893] ? mdc_commit_open+0x91/0xb0 [mdc] [ 1924.722090] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 1924.727750] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 1924.731646] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 1924.735146] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 1924.740317] ? __next_timer_interrupt+0x160/0x160 [ 1924.743181] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 1924.746097] ? do_wait_intr+0xf0/0xf0 [ 1924.748128] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 1924.751259] ? kthread+0x1d1/0x200 [ 1924.752953] ? set_kthread_struct+0x70/0x70 [ 1924.755163] ? ret_from_fork+0x1f/0x30 [ 1952.052034] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 1952.055963] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 1952.056284] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 1952.056373] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 1952.056382] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 1952.056385] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 1952.056406] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 1952.056411] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 1952.056416] RAX: 0000000000003282 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 1952.056419] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 1952.056421] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 1952.056423] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 1952.056426] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 1952.056429] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 1952.056432] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1952.056435] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 1952.056443] Call Trace: [ 1952.056459] [ 1952.056468] ? show_regs.cold.9+0x22/0x2f [ 1952.056479] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 1952.056487] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 1952.056491] ? watchdog+0x40/0x40 [ 1952.056495] ? hrtimer_interrupt+0x128/0x2c0 [ 1952.056499] ? smp_apic_timer_interrupt+0xa6/0x270 [ 1952.056509] ? apic_timer_interrupt+0xf/0x20 [ 1952.056514] [ 1952.056516] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 1952.056521] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 1952.056526] do_raw_spin_lock+0x177/0x190 [ 1952.056530] _raw_spin_lock+0x21/0x30 [ 1952.056537] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 1952.057097] ? do_raw_spin_unlock+0x75/0x190 [ 1952.057104] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 1952.057402] ? do_wait_intr+0xf0/0xf0 [ 1952.057413] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 1952.062332] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 1952.062345] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 1952.062416] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 1952.062422] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 1952.062425] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 1952.062441] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 1952.062445] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 1952.062449] RAX: 0000000000004b5b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 1952.062451] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 1952.062453] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 1952.062455] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 1952.062458] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 1952.062462] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 1952.062464] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1952.062466] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 1952.062477] Call Trace: [ 1952.062483] [ 1952.062489] ? show_regs.cold.9+0x22/0x2f [ 1952.062499] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 1952.062505] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 1952.062510] ? watchdog+0x40/0x40 [ 1952.062515] ? hrtimer_interrupt+0x128/0x2c0 [ 1952.062519] ? smp_apic_timer_interrupt+0xa6/0x270 [ 1952.062544] ? apic_timer_interrupt+0xf/0x20 [ 1952.062549] [ 1952.062552] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 1952.062559] do_raw_spin_lock+0x177/0x190 [ 1952.062564] _raw_spin_lock+0x21/0x30 [ 1952.062571] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 1952.063310] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 1952.063317] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 1952.063377] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 1952.063382] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 1952.063387] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 1952.063659] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 1952.063668] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 1952.063672] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 1952.063676] RAX: 0000000000005507 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 1952.063678] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 1952.063681] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 1952.063683] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 1952.063685] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 1952.063688] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 1952.063691] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1952.063694] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 1952.063702] Call Trace: [ 1952.063707] [ 1952.063711] ? show_regs.cold.9+0x22/0x2f [ 1952.063720] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 1952.063728] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 1952.063732] ? watchdog+0x40/0x40 [ 1952.063736] ? hrtimer_interrupt+0x128/0x2c0 [ 1952.063741] ? smp_apic_timer_interrupt+0xa6/0x270 [ 1952.063750] ? apic_timer_interrupt+0xf/0x20 [ 1952.063755] [ 1952.063758] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 1952.063763] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 1952.063769] do_raw_spin_lock+0x177/0x190 [ 1952.063774] _raw_spin_lock+0x21/0x30 [ 1952.063779] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 1952.063988] process_one_work+0x2c8/0x700 [ 1952.063996] worker_thread+0x296/0x6e0 [ 1952.064000] ? rescuer_thread+0x570/0x570 [ 1952.064004] kthread+0x1d1/0x200 [ 1952.064008] ? set_kthread_struct+0x70/0x70 [ 1952.064012] ret_from_fork+0x1f/0x30 [ 1952.064101] ? do_raw_spin_unlock+0x75/0x190 [ 1952.064109] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 1952.064326] ? do_wait_intr+0xf0/0xf0 [ 1952.064335] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 1952.064532] mdc_close+0x53b/0xd70 [mdc] [ 1952.064575] lmv_close+0x18d/0x490 [lmv] [ 1952.064609] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 1952.064796] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 1952.065679] ll_md_real_close+0xa5/0x2a0 [lustre] [ 1952.065723] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 1952.065760] ll_md_close+0x274/0x800 [lustre] [ 1952.065797] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 1952.065806] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 1952.065809] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 1952.065818] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 1952.065822] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 1952.065827] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 1952.065829] RDX: 0000000000003b89 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 1952.065832] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 1952.065834] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 1952.065836] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 1952.065839] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 1952.065842] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1952.065845] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 1952.065854] Call Trace: [ 1952.065857] [ 1952.065860] ? show_regs.cold.9+0x22/0x2f [ 1952.065868] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 1952.065873] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 1952.065877] ? watchdog+0x40/0x40 [ 1952.065881] ? hrtimer_interrupt+0x128/0x2c0 [ 1952.065885] ? smp_apic_timer_interrupt+0xa6/0x270 [ 1952.065892] ? apic_timer_interrupt+0xf/0x20 [ 1952.065897] [ 1952.065900] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 1952.065905] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 1952.065910] do_raw_spin_lock+0x177/0x190 [ 1952.065914] _raw_spin_lock+0x21/0x30 [ 1952.065919] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 1952.066218] obd_mod_free+0x46/0x140 [ptlrpc] [ 1952.066304] ll_file_release+0x128/0xc50 [lustre] [ 1952.066851] mdc_commit_open+0x91/0xb0 [mdc] [ 1952.066889] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 1952.067093] __fput+0x10d/0x440 [ 1952.067103] ____fput+0x12/0x20 [ 1952.067108] task_work_run+0xc6/0x110 [ 1952.067120] exit_to_usermode_loop+0x1dd/0x1f0 [ 1952.067128] do_syscall_64+0x3d6/0x440 [ 1952.067133] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 1952.067211] RIP: 0033:0x7f151bee3cd8 [ 1952.067118] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 1952.067219] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 1952.067224] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 1952.067229] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 1952.067232] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 1952.067234] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 1952.067237] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 1952.067239] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 1952.070749] after_reply+0xe2c/0x1780 [ptlrpc] [ 1952.110566] mdc_reint+0x2d/0x110 [mdc] [ 1952.110767] mdc_unlink+0x658/0x870 [mdc] [ 1952.110795] lmv_unlink+0x34d/0xbd0 [lmv] [ 1952.110834] ll_unlink+0x24d/0x7f0 [lustre] [ 1952.110972] vfs_unlink+0x202/0x330 [ 1952.110983] do_unlinkat+0x2fa/0x460 [ 1952.110989] __x64_sys_unlink+0x37/0x50 [ 1952.110993] do_syscall_64+0xc1/0x440 [ 1952.111001] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 1952.111014] RIP: 0033:0x7f151bdee10b [ 1952.111024] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 1952.111031] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 1952.111037] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 1952.111040] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 1952.111042] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 1952.111045] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 1952.111047] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 1953.172183] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 1953.174576] ? __next_timer_interrupt+0x160/0x160 [ 1953.176562] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 1953.179578] ? do_wait_intr+0xf0/0xf0 [ 1953.181540] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 1953.183732] kthread+0x1d1/0x200 [ 1953.185315] ? set_kthread_struct+0x70/0x70 [ 1953.187680] ret_from_fork+0x1f/0x30 [ 1966.575748] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 35s! [ 1966.589648] Showing busy workqueues and worker pools: [ 1966.596538] workqueue events: flags=0x0 [ 1966.606511] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 1966.620900] pending: cpu_latency_work [ptlrpc] [ 1966.625663] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 1966.640815] pending: kfree_rcu_monitor [ 1966.647678] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 1966.670179] pending: nfs4_renew_state [nfsv4] [ 1966.672217] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 1966.690025] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 1966.715345] workqueue events_long: flags=0x0 [ 1966.729040] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 1966.749293] pending: lu_objects_flush [obdclass] [ 1966.751379] workqueue events_unbound: flags=0x2 [ 1966.769300] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/512 refcnt=3 [ 1966.783932] pending: flush_memcg_stats_dwork [ 1966.790363] workqueue events_power_efficient: flags=0x80 [ 1966.802698] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 1966.815591] pending: do_cache_clean, neigh_periodic_work [ 1966.828835] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 1966.844188] pending: neigh_periodic_work [ 1966.845787] workqueue mm_percpu_wq: flags=0x8 [ 1966.853920] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 1966.857145] pending: vmstat_update, lru_add_drain_per_cpu [ 1966.862511] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 1966.881928] pending: vmstat_update [ 1966.895737] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 1966.897750] pending: lru_add_drain_per_cpu [ 1966.912489] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 1966.916145] pending: lru_add_drain_per_cpu BAR(48) [ 1966.920701] workqueue writeback: flags=0x4a [ 1966.924302] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 1966.929183] pending: wb_workfn [ 1966.932509] workqueue xprtiod: flags=0xa [ 1966.933874] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 1966.937438] pending: xs_stream_data_receive_workfn [ 1966.940910] workqueue knbd0-recv: flags=0x1a [ 1966.943951] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 1966.949189] in-flight: 658:recv_work [ 1966.955655] workqueue ptlrpc_pinger: flags=0x2 [ 1966.958825] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 1966.966212] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 1966.975928] pool 8: cpus=0-3 flags=0x4 nice=0 hung=36s workers=4 idle: 32984 37 8 [ 1966.990388] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=1951s workers=2 idle: 661 [ 1980.055067] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 1980.056290] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 1980.057021] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 1980.057032] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 1980.057107] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 1980.057123] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 1980.057126] RIP: 0010:native_safe_halt+0x2e/0x50 [ 1980.057138] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 1980.057143] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 1980.057147] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 1980.057150] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 1980.057152] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 1980.057154] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 1980.057156] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 1980.057159] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 1980.057162] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1980.057165] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 1980.057174] Call Trace: [ 1980.057179] [ 1980.057184] ? show_regs.cold.9+0x22/0x2f [ 1980.057194] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 1980.057201] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 1980.057205] ? watchdog+0x40/0x40 [ 1980.057209] ? hrtimer_interrupt+0x128/0x2c0 [ 1980.057213] ? smp_apic_timer_interrupt+0xa6/0x270 [ 1980.057219] ? apic_timer_interrupt+0xf/0x20 [ 1980.057295] [ 1980.057298] ? native_safe_halt+0x2e/0x50 [ 1980.057302] kvm_wait+0xcd/0xe0 [ 1980.057312] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 1980.057322] do_raw_spin_lock+0x177/0x190 [ 1980.057327] _raw_spin_lock+0x21/0x30 [ 1980.057331] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 1980.057614] ? do_raw_spin_unlock+0x75/0x190 [ 1980.057619] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 1980.057848] ? do_wait_intr+0xf0/0xf0 [ 1980.057855] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 1980.058103] mdc_close+0x53b/0xd70 [mdc] [ 1980.058206] lmv_close+0x18d/0x490 [lmv] [ 1980.058232] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 1980.058385] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 1980.058500] ll_md_real_close+0xa5/0x2a0 [lustre] [ 1980.058605] ll_md_close+0x274/0x800 [lustre] [ 1980.058697] ll_file_release+0x128/0xc50 [lustre] [ 1980.058784] __fput+0x10d/0x440 [ 1980.058799] ____fput+0x12/0x20 [ 1980.058804] task_work_run+0xc6/0x110 [ 1980.058814] exit_to_usermode_loop+0x1dd/0x1f0 [ 1980.058822] do_syscall_64+0x3d6/0x440 [ 1980.058827] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 1980.058835] RIP: 0033:0x7f151bee3cd8 [ 1980.058842] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 1980.058846] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 1980.058851] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 1980.058853] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 1980.058856] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 1980.058858] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 1980.058860] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 1980.059543] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 1980.059604] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 1980.059610] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 1980.059613] RIP: 0010:native_safe_halt+0x2e/0x50 [ 1980.059622] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 1980.059626] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 1980.059630] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 1980.059632] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 1980.059634] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 1980.059636] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 1980.059639] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 1980.059641] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 1980.059644] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1980.059646] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 1980.059652] Call Trace: [ 1980.059655] [ 1980.059657] ? show_regs.cold.9+0x22/0x2f [ 1980.059665] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 1980.059670] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 1980.059674] ? watchdog+0x40/0x40 [ 1980.059678] ? hrtimer_interrupt+0x128/0x2c0 [ 1980.059682] ? smp_apic_timer_interrupt+0xa6/0x270 [ 1980.059687] ? apic_timer_interrupt+0xf/0x20 [ 1980.059691] [ 1980.059694] ? native_safe_halt+0x2e/0x50 [ 1980.059698] kvm_wait+0xcd/0xe0 [ 1980.059703] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 1980.059709] do_raw_spin_lock+0x177/0x190 [ 1980.059713] _raw_spin_lock+0x21/0x30 [ 1980.059717] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 1980.060229] obd_mod_free+0x46/0x140 [ptlrpc] [ 1980.060533] mdc_commit_open+0x91/0xb0 [mdc] [ 1980.060572] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 1980.060774] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 1980.065142] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 1980.065231] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 1980.065294] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 1980.065301] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 1980.065587] RIP: 0010:native_safe_halt+0x2e/0x50 [ 1980.065596] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 1980.065601] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 1980.065605] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 1980.065608] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 1980.065610] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 1980.065613] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 1980.065615] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 1980.065619] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 1980.065622] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1980.065624] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 1980.065631] Call Trace: [ 1980.065636] [ 1980.065641] ? show_regs.cold.9+0x22/0x2f [ 1980.065651] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 1980.065658] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 1980.065663] ? watchdog+0x40/0x40 [ 1980.065667] ? hrtimer_interrupt+0x128/0x2c0 [ 1980.065672] ? smp_apic_timer_interrupt+0xa6/0x270 [ 1980.065678] ? apic_timer_interrupt+0xf/0x20 [ 1980.065683] [ 1980.065686] ? native_safe_halt+0x2e/0x50 [ 1980.065690] kvm_wait+0xcd/0xe0 [ 1980.065698] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 1980.065705] do_raw_spin_lock+0x177/0x190 [ 1980.065710] _raw_spin_lock+0x21/0x30 [ 1980.065716] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 1980.071592] after_reply+0xe2c/0x1780 [ptlrpc] [ 1980.071877] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 1980.093969] process_one_work+0x2c8/0x700 [ 1980.097608] ? __next_timer_interrupt+0x160/0x160 [ 1980.100420] worker_thread+0x296/0x6e0 [ 1980.101997] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 1980.108359] ? rescuer_thread+0x570/0x570 [ 1980.111452] ? do_wait_intr+0xf0/0xf0 [ 1980.113736] kthread+0x1d1/0x200 [ 1980.117378] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 1980.120254] ? set_kthread_struct+0x70/0x70 [ 1980.120270] ret_from_fork+0x1f/0x30 [ 1980.555360] kthread+0x1d1/0x200 [ 1980.556737] ? set_kthread_struct+0x70/0x70 [ 1980.558711] ret_from_fork+0x1f/0x30 [ 1984.649675] rcu: INFO: rcu_sched self-detected stall on CPU [ 1984.655720] rcu: 1-...!: (34325 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 1984.664662] (t=60002 jiffies g=249657 q=2512) [ 1984.666264] rcu: rcu_sched kthread starved for 54954 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 1984.685541] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 1984.694730] rcu: RCU grace-period kthread stack dump: [ 1984.704591] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 1984.717263] Call Trace: [ 1984.718082] __schedule+0x351/0xcb0 [ 1984.719358] schedule+0xc0/0x180 [ 1984.732718] schedule_timeout+0xb4/0x190 [ 1984.738763] ? do_raw_spin_unlock+0x75/0x190 [ 1984.745364] ? __next_timer_interrupt+0x160/0x160 [ 1984.747509] ? prepare_to_swait_event+0xf3/0x140 [ 1984.759525] rcu_gp_kthread+0x8dd/0x15b0 [ 1984.764644] ? force_qs_rnp+0x2c0/0x2c0 [ 1984.766561] kthread+0x1d1/0x200 [ 1984.774366] ? set_kthread_struct+0x70/0x70 [ 1984.778572] ret_from_fork+0x1f/0x30 [ 1984.787433] rcu: Stack dump where RCU GP kthread last ran: [ 1984.790226] NMI backtrace for cpu 1 [ 1984.791449] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 1984.830639] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 1984.836743] Call Trace: [ 1984.837810] [ 1984.841212] ? dump_stack+0xbb/0x10e [ 1984.843315] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 1984.848585] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 1984.851372] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 1984.860578] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 1984.870522] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 1984.874429] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 1984.883000] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 1984.893198] ? update_process_times+0x6d/0xc0 [ 1984.894771] ? tick_sched_handle.isra.18+0x27/0x90 [ 1984.908057] ? tick_sched_timer+0x53/0xb0 [ 1984.910567] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 1984.919400] ? tick_sched_handle.isra.18+0x90/0x90 [ 1984.920982] ? hrtimer_interrupt+0x128/0x2c0 [ 1984.928238] ? smp_apic_timer_interrupt+0xa6/0x270 [ 1984.939619] ? apic_timer_interrupt+0xf/0x20 [ 1984.944650] [ 1984.949733] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 1984.955035] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 1984.961434] ? do_raw_spin_lock+0x177/0x190 [ 1984.963688] ? _raw_spin_lock+0x21/0x30 [ 1984.965604] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 1984.972157] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 1984.978963] ? mdc_commit_open+0x91/0xb0 [mdc] [ 1984.980726] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 1984.984691] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 1984.990433] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 1984.995690] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 1984.998442] ? __next_timer_interrupt+0x160/0x160 [ 1984.999916] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 1985.007642] ? do_wait_intr+0xf0/0xf0 [ 1985.013479] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 1985.020163] ? kthread+0x1d1/0x200 [ 1985.022323] ? set_kthread_struct+0x70/0x70 [ 1985.024962] ? ret_from_fork+0x1f/0x30 [ 1985.028375] NMI backtrace for cpu 1 [ 1985.029737] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 1985.038268] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 1985.049104] Call Trace: [ 1985.050344] [ 1985.051290] ? dump_stack+0xbb/0x10e [ 1985.052706] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 1985.055394] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 1985.058725] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 1985.060721] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 1985.062715] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 1985.064667] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 1985.066106] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 1985.067624] ? update_process_times+0x6d/0xc0 [ 1985.069454] ? tick_sched_handle.isra.18+0x27/0x90 [ 1985.071478] ? tick_sched_timer+0x53/0xb0 [ 1985.073554] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 1985.075434] ? tick_sched_handle.isra.18+0x90/0x90 [ 1985.077524] ? hrtimer_interrupt+0x128/0x2c0 [ 1985.079084] ? smp_apic_timer_interrupt+0xa6/0x270 [ 1985.081047] ? apic_timer_interrupt+0xf/0x20 [ 1985.082781] [ 1985.083640] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 1985.085726] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 1985.087773] ? do_raw_spin_lock+0x177/0x190 [ 1985.089352] ? _raw_spin_lock+0x21/0x30 [ 1985.090716] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 1985.092557] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 1985.094253] ? mdc_commit_open+0x91/0xb0 [mdc] [ 1985.095770] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 1985.097996] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 1985.100053] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 1985.101635] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 1985.103756] ? __next_timer_interrupt+0x160/0x160 [ 1985.105346] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 1985.107315] ? do_wait_intr+0xf0/0xf0 [ 1985.108583] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 1985.110609] ? kthread+0x1d1/0x200 [ 1985.111640] ? set_kthread_struct+0x70/0x70 [ 1985.113298] ? ret_from_fork+0x1f/0x30 [ 1992.052661] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 1992.056545] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 1992.077418] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 1992.082858] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 1992.086610] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 1992.089485] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 1992.098036] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 1992.101614] RAX: 0000000000006b0d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 1992.104914] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 1992.107963] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 1992.111071] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 1992.114372] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 1992.117478] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 1992.121068] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1992.123589] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 1992.126670] Call Trace: [ 1992.127781] [ 1992.128596] ? show_regs.cold.9+0x22/0x2f [ 1992.130371] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 1992.132177] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 1992.134069] ? watchdog+0x40/0x40 [ 1992.135302] ? hrtimer_interrupt+0x128/0x2c0 [ 1992.137347] ? smp_apic_timer_interrupt+0xa6/0x270 [ 1992.139412] ? apic_timer_interrupt+0xf/0x20 [ 1992.141464] [ 1992.142266] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 1992.144617] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 1992.146908] do_raw_spin_lock+0x177/0x190 [ 1992.149286] _raw_spin_lock+0x21/0x30 [ 1992.150681] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 1992.153078] ? do_raw_spin_unlock+0x75/0x190 [ 1992.155287] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 1992.157689] ? do_wait_intr+0xf0/0xf0 [ 1992.159477] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 1992.161887] mdc_reint+0x2d/0x110 [mdc] [ 1992.163445] mdc_unlink+0x658/0x870 [mdc] [ 1992.165600] lmv_unlink+0x34d/0xbd0 [lmv] [ 1992.167817] ll_unlink+0x24d/0x7f0 [lustre] [ 1992.169945] vfs_unlink+0x202/0x330 [ 1992.171559] do_unlinkat+0x2fa/0x460 [ 1992.173404] __x64_sys_unlink+0x37/0x50 [ 1992.174858] do_syscall_64+0xc1/0x440 [ 1992.176312] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 1992.179685] RIP: 0033:0x7f151bdee10b [ 1992.181590] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 1992.188271] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 1992.191397] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 1992.195122] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 1992.200865] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 1992.204515] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 1992.208725] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 1997.280121] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 66s! [ 1997.284452] Showing busy workqueues and worker pools: [ 1997.287293] workqueue events: flags=0x0 [ 1997.289441] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 1997.293173] pending: cpu_latency_work [ptlrpc] [ 1997.296344] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 1997.299958] pending: kfree_rcu_monitor [ 1997.301955] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 1997.305814] pending: nfs4_renew_state [nfsv4] [ 1997.308567] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 1997.312268] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 1997.317398] workqueue events_long: flags=0x0 [ 1997.319981] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 1997.323367] pending: lu_objects_flush [obdclass] [ 1997.326391] workqueue events_unbound: flags=0x2 [ 1997.328783] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/512 refcnt=3 [ 1997.332387] pending: flush_memcg_stats_dwork [ 1997.335057] workqueue events_power_efficient: flags=0x80 [ 1997.337846] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 1997.341318] pending: do_cache_clean, neigh_periodic_work [ 1997.344131] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 1997.347306] pending: neigh_periodic_work [ 1997.349441] workqueue mm_percpu_wq: flags=0x8 [ 1997.351559] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 1997.354919] pending: vmstat_update, lru_add_drain_per_cpu [ 1997.357731] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 1997.361289] pending: vmstat_update [ 1997.363519] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 1997.366245] pending: lru_add_drain_per_cpu [ 1997.368305] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 1997.370698] pending: lru_add_drain_per_cpu BAR(48) [ 1997.372742] workqueue writeback: flags=0x4a [ 1997.374153] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 1997.376609] pending: wb_workfn [ 1997.377953] workqueue xprtiod: flags=0xa [ 1997.379929] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 1997.385593] pending: xs_stream_data_receive_workfn [ 1997.391097] workqueue knbd0-recv: flags=0x1a [ 1997.393615] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 1997.398331] in-flight: 658:recv_work [ 1997.400740] workqueue ptlrpc_pinger: flags=0x2 [ 1997.406668] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 1997.409632] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 1997.413516] pool 8: cpus=0-3 flags=0x4 nice=0 hung=66s workers=4 idle: 32984 37 8 [ 1997.417371] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=1982s workers=2 idle: 661 [ 2008.056502] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 2008.060339] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 2008.062230] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) [ 2008.073153] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2008.073425] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2008.073436] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2008.073439] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2008.073711] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2008.073725] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2008.073731] RAX: 0000000000007d86 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2008.073733] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 2008.073739] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2008.073741] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 2008.073743] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 2008.073747] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 2008.073753] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2008.073755] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 2008.073762] Call Trace: [ 2008.073786] [ 2008.073796] ? show_regs.cold.9+0x22/0x2f [ 2008.073808] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2008.073817] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2008.073822] ? watchdog+0x40/0x40 [ 2008.073826] ? hrtimer_interrupt+0x128/0x2c0 [ 2008.073830] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2008.073839] ? apic_timer_interrupt+0xf/0x20 [ 2008.073847] [ 2008.073850] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2008.073858] do_raw_spin_lock+0x177/0x190 [ 2008.073869] _raw_spin_lock+0x21/0x30 [ 2008.073941] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2008.092168] ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2008.092254] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2008.092261] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2008.092267] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2008.092563] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2008.092573] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2008.092577] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2008.093950] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2008.093960] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 2008.093963] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 2008.093965] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 2008.093967] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 2008.093970] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 2008.093973] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2008.093975] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2008.093983] Call Trace: [ 2008.093990] [ 2008.093995] ? show_regs.cold.9+0x22/0x2f [ 2008.094006] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2008.094013] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2008.094018] ? watchdog+0x40/0x40 [ 2008.094022] ? hrtimer_interrupt+0x128/0x2c0 [ 2008.094026] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2008.094035] ? apic_timer_interrupt+0xf/0x20 [ 2008.094039] [ 2008.094042] ? native_safe_halt+0x2e/0x50 [ 2008.094047] kvm_wait+0xcd/0xe0 [ 2008.094054] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2008.094064] do_raw_spin_lock+0x177/0x190 [ 2008.094068] _raw_spin_lock+0x21/0x30 [ 2008.094072] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2008.119027] ? do_raw_spin_unlock+0x75/0x190 [ 2008.119049] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2008.119476] ? do_wait_intr+0xf0/0xf0 [ 2008.119485] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2008.130295] process_one_work+0x2c8/0x700 [ 2008.135314] mdc_close+0x53b/0xd70 [mdc] [ 2008.135369] lmv_close+0x18d/0x490 [lmv] [ 2008.135400] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2008.136390] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2008.136739] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2008.136826] ll_md_close+0x274/0x800 [lustre] [ 2008.136986] ll_file_release+0x128/0xc50 [lustre] [ 2008.137066] __fput+0x10d/0x440 [ 2008.137076] ____fput+0x12/0x20 [ 2008.137080] task_work_run+0xc6/0x110 [ 2008.137088] exit_to_usermode_loop+0x1dd/0x1f0 [ 2008.137095] do_syscall_64+0x3d6/0x440 [ 2008.137099] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2008.137109] RIP: 0033:0x7f151bee3cd8 [ 2008.141824] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2008.158492] worker_thread+0x296/0x6e0 [ 2008.158511] ? rescuer_thread+0x570/0x570 [ 2008.158515] kthread+0x1d1/0x200 [ 2008.158520] ? set_kthread_struct+0x70/0x70 [ 2008.158524] ret_from_fork+0x1f/0x30 [ 2008.484393] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2008.492551] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 2008.496762] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 2008.500855] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2008.504401] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2008.508097] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2012.055041] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 2012.059065] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2012.077885] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2012.082238] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2012.085810] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2012.097026] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2012.130273] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2012.138315] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2012.142465] RDX: 00000000000050da RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 2012.166905] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 2012.172895] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 2012.180446] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2012.186930] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 2012.191988] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2012.195852] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2012.199220] Call Trace: [ 2012.199903] [ 2012.200498] ? show_regs.cold.9+0x22/0x2f [ 2012.204356] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2012.207010] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2012.211693] ? watchdog+0x40/0x40 [ 2012.213317] ? hrtimer_interrupt+0x128/0x2c0 [ 2012.216815] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2012.220857] ? apic_timer_interrupt+0xf/0x20 [ 2012.223955] [ 2012.225339] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2012.228952] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2012.234888] do_raw_spin_lock+0x177/0x190 [ 2012.236359] _raw_spin_lock+0x21/0x30 [ 2012.237402] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2012.242474] obd_mod_free+0x46/0x140 [ptlrpc] [ 2012.245358] mdc_commit_open+0x91/0xb0 [mdc] [ 2012.248150] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2012.257114] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2012.259008] after_reply+0xe2c/0x1780 [ptlrpc] [ 2012.266320] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2012.272304] ? __next_timer_interrupt+0x160/0x160 [ 2012.278599] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2012.284313] ? do_wait_intr+0xf0/0xf0 [ 2012.290394] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2012.296775] kthread+0x1d1/0x200 [ 2012.305322] ? set_kthread_struct+0x70/0x70 [ 2012.308744] ret_from_fork+0x1f/0x30 [ 2024.052743] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 2024.062347] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2024.105227] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2024.111424] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2024.118742] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2024.121168] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2024.129777] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2024.134819] RAX: 00000000000035f1 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2024.138236] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 2024.141229] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 2024.144493] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 2024.147852] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 2024.151609] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 2024.158976] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2024.161802] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 2024.173830] Call Trace: [ 2024.174678] [ 2024.177071] ? show_regs.cold.9+0x22/0x2f [ 2024.179277] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2024.185517] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2024.188901] ? watchdog+0x40/0x40 [ 2024.196133] ? hrtimer_interrupt+0x128/0x2c0 [ 2024.197695] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2024.202851] ? apic_timer_interrupt+0xf/0x20 [ 2024.210241] [ 2024.212812] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2024.220926] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2024.223658] do_raw_spin_lock+0x177/0x190 [ 2024.226386] _raw_spin_lock+0x21/0x30 [ 2024.228330] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2024.233072] ? do_raw_spin_unlock+0x75/0x190 [ 2024.234178] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2024.244015] ? do_wait_intr+0xf0/0xf0 [ 2024.252304] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2024.260350] mdc_reint+0x2d/0x110 [mdc] [ 2024.267059] mdc_unlink+0x658/0x870 [mdc] [ 2024.275748] lmv_unlink+0x34d/0xbd0 [lmv] [ 2024.277622] ll_unlink+0x24d/0x7f0 [lustre] [ 2024.290747] vfs_unlink+0x202/0x330 [ 2024.293286] do_unlinkat+0x2fa/0x460 [ 2024.295041] __x64_sys_unlink+0x37/0x50 [ 2024.299446] do_syscall_64+0xc1/0x440 [ 2024.300749] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2024.303862] RIP: 0033:0x7f151bdee10b [ 2024.306368] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2024.325383] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2024.332344] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 2024.335651] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 2024.343076] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2024.351832] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2024.359174] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2027.490750] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 96s! [ 2027.496880] Showing busy workqueues and worker pools: [ 2027.506211] workqueue events: flags=0x0 [ 2027.507507] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2027.513110] pending: cpu_latency_work [ptlrpc] [ 2027.515073] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2027.519932] pending: kfree_rcu_monitor [ 2027.527563] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2027.538959] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 2027.548703] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2027.559939] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2027.565244] workqueue events_long: flags=0x0 [ 2027.570661] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2027.581717] pending: lu_objects_flush [obdclass] [ 2027.585213] workqueue events_unbound: flags=0x2 [ 2027.588822] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/512 refcnt=3 [ 2027.593634] pending: flush_memcg_stats_dwork [ 2027.596831] workqueue events_power_efficient: flags=0x80 [ 2027.601066] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2027.604732] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 2027.616988] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2027.620305] pending: neigh_periodic_work [ 2027.624051] workqueue mm_percpu_wq: flags=0x8 [ 2027.627919] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2027.631781] pending: vmstat_update, lru_add_drain_per_cpu [ 2027.633884] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2027.639338] pending: vmstat_update [ 2027.643228] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2027.650773] pending: lru_add_drain_per_cpu [ 2027.656516] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2027.665065] pending: lru_add_drain_per_cpu BAR(48) [ 2027.668858] workqueue writeback: flags=0x4a [ 2027.672806] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2027.676609] pending: wb_workfn [ 2027.678883] workqueue xprtiod: flags=0xa [ 2027.683037] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2027.687923] pending: xs_stream_data_receive_workfn [ 2027.690256] workqueue ipv6_addrconf: flags=0x40008 [ 2027.692220] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2027.695219] pending: addrconf_verify_work [ 2027.697280] workqueue knbd0-recv: flags=0x1a [ 2027.698924] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2027.701381] in-flight: 658:recv_work [ 2027.706194] workqueue ptlrpc_pinger: flags=0x2 [ 2027.710327] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2027.715348] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2027.722393] pool 8: cpus=0-3 flags=0x4 nice=0 hung=97s workers=4 idle: 32984 37 8 [ 2027.730265] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2012s workers=2 idle: 661 [ 2036.057104] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 2036.057092] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 2036.061594] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2036.061908] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2036.061916] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2036.061918] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2036.061946] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2036.061950] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2036.061954] RAX: 0000000000007b85 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2036.061957] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 2036.061959] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2036.061961] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 2036.061963] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 2036.061966] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 2036.061969] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2036.061971] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 2036.061979] Call Trace: [ 2036.061995] [ 2036.062002] ? show_regs.cold.9+0x22/0x2f [ 2036.062014] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2036.062021] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2036.062026] ? watchdog+0x40/0x40 [ 2036.062031] ? hrtimer_interrupt+0x128/0x2c0 [ 2036.062036] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2036.062045] ? apic_timer_interrupt+0xf/0x20 [ 2036.062050] [ 2036.062052] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2036.062059] do_raw_spin_lock+0x177/0x190 [ 2036.062063] _raw_spin_lock+0x21/0x30 [ 2036.062068] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2036.062496] ? do_raw_spin_unlock+0x75/0x190 [ 2036.062503] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2036.062831] ? do_wait_intr+0xf0/0xf0 [ 2036.062839] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2036.063029] mdc_close+0x53b/0xd70 [mdc] [ 2036.063068] lmv_close+0x18d/0x490 [lmv] [ 2036.063098] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2036.063255] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2036.063827] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2036.063933] ll_md_close+0x274/0x800 [lustre] [ 2036.064030] ll_file_release+0x128/0xc50 [lustre] [ 2036.076424] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2036.076518] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2036.076526] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2036.076587] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2036.077386] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2036.077400] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2036.077409] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2036.077413] RAX: 0000000000007d83 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2036.077416] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 2036.077418] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 2036.077421] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 2036.077423] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 2036.077427] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 2036.077430] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2036.077432] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2036.077441] Call Trace: [ 2036.077459] [ 2036.077465] ? show_regs.cold.9+0x22/0x2f [ 2036.077475] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2036.077481] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2036.077486] ? watchdog+0x40/0x40 [ 2036.077490] ? hrtimer_interrupt+0x128/0x2c0 [ 2036.077494] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2036.077502] ? apic_timer_interrupt+0xf/0x20 [ 2036.077506] [ 2036.077509] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2036.077514] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2036.077519] do_raw_spin_lock+0x177/0x190 [ 2036.077524] _raw_spin_lock+0x21/0x30 [ 2036.077528] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2036.077970] process_one_work+0x2c8/0x700 [ 2036.077977] worker_thread+0x296/0x6e0 [ 2036.077981] ? rescuer_thread+0x570/0x570 [ 2036.077985] kthread+0x1d1/0x200 [ 2036.077990] ? set_kthread_struct+0x70/0x70 [ 2036.077993] ret_from_fork+0x1f/0x30 [ 2036.366504] __fput+0x10d/0x440 [ 2036.367568] ____fput+0x12/0x20 [ 2036.371576] task_work_run+0xc6/0x110 [ 2036.373851] exit_to_usermode_loop+0x1dd/0x1f0 [ 2036.375340] do_syscall_64+0x3d6/0x440 [ 2036.378552] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2036.381404] RIP: 0033:0x7f151bee3cd8 [ 2036.383042] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2036.391422] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2036.394482] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 2036.397193] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 2036.403853] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2036.407218] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2036.409986] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2040.055176] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 2040.057979] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2040.090123] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2040.097458] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2040.102085] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2040.104811] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2040.123177] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2040.127208] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2040.134163] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 2040.140687] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 2040.145082] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 2040.150631] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2040.157649] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 2040.173941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2040.181650] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2040.184007] Call Trace: [ 2040.191827] [ 2040.193653] ? show_regs.cold.9+0x22/0x2f [ 2040.195600] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2040.201164] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2040.213832] ? watchdog+0x40/0x40 [ 2040.223608] ? hrtimer_interrupt+0x128/0x2c0 [ 2040.235376] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2040.237616] ? apic_timer_interrupt+0xf/0x20 [ 2040.243249] [ 2040.246630] ? native_safe_halt+0x2e/0x50 [ 2040.250068] kvm_wait+0xcd/0xe0 [ 2040.257186] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2040.267660] do_raw_spin_lock+0x177/0x190 [ 2040.272243] _raw_spin_lock+0x21/0x30 [ 2040.274080] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2040.277971] obd_mod_free+0x46/0x140 [ptlrpc] [ 2040.284324] mdc_commit_open+0x91/0xb0 [mdc] [ 2040.285365] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2040.297983] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2040.304071] after_reply+0xe2c/0x1780 [ptlrpc] [ 2040.309075] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2040.313254] ? __next_timer_interrupt+0x160/0x160 [ 2040.325064] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2040.335147] ? do_wait_intr+0xf0/0xf0 [ 2040.339236] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2040.344334] kthread+0x1d1/0x200 [ 2040.348158] ? set_kthread_struct+0x70/0x70 [ 2040.353105] ret_from_fork+0x1f/0x30 [ 2052.052027] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 2052.060078] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2052.103150] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2052.110666] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2052.119083] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2052.121429] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2052.132875] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2052.136302] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2052.140147] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 2052.145112] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 2052.149304] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 2052.155478] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 2052.158215] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 2052.166381] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2052.177702] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 2052.184258] Call Trace: [ 2052.187360] [ 2052.190028] ? show_regs.cold.9+0x22/0x2f [ 2052.192687] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2052.196575] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2052.200335] ? watchdog+0x40/0x40 [ 2052.202965] ? hrtimer_interrupt+0x128/0x2c0 [ 2052.207357] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2052.211121] ? apic_timer_interrupt+0xf/0x20 [ 2052.214733] [ 2052.217678] ? native_safe_halt+0x2e/0x50 [ 2052.219745] kvm_wait+0xcd/0xe0 [ 2052.222701] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2052.226499] do_raw_spin_lock+0x177/0x190 [ 2052.229667] _raw_spin_lock+0x21/0x30 [ 2052.232351] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2052.235983] ? do_raw_spin_unlock+0x75/0x190 [ 2052.240245] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2052.244103] ? do_wait_intr+0xf0/0xf0 [ 2052.246369] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2052.252726] mdc_reint+0x2d/0x110 [mdc] [ 2052.257248] mdc_unlink+0x658/0x870 [mdc] [ 2052.260129] lmv_unlink+0x34d/0xbd0 [lmv] [ 2052.266978] ll_unlink+0x24d/0x7f0 [lustre] [ 2052.268995] vfs_unlink+0x202/0x330 [ 2052.270098] do_unlinkat+0x2fa/0x460 [ 2052.271120] __x64_sys_unlink+0x37/0x50 [ 2052.272936] do_syscall_64+0xc1/0x440 [ 2052.274536] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2052.276715] RIP: 0033:0x7f151bdee10b [ 2052.278462] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2052.286626] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2052.290072] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 2052.293337] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 2052.296815] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2052.300341] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2052.303522] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2058.210495] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 127s! [ 2058.216477] Showing busy workqueues and worker pools: [ 2058.219465] workqueue events: flags=0x0 [ 2058.223914] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2058.232292] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2058.241983] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2058.250812] pending: kfree_rcu_monitor [ 2058.253638] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2058.255083] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 2058.261130] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2058.263969] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2058.269403] workqueue events_long: flags=0x0 [ 2058.270770] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2058.274555] pending: lu_objects_flush [obdclass] [ 2058.281454] workqueue events_unbound: flags=0x2 [ 2058.284146] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/512 refcnt=5 [ 2058.287953] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn [ 2058.290048] workqueue events_power_efficient: flags=0x80 [ 2058.295764] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2058.306811] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 2058.313601] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2058.321807] pending: neigh_periodic_work [ 2058.325711] workqueue mm_percpu_wq: flags=0x8 [ 2058.329080] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2058.332657] pending: vmstat_update, lru_add_drain_per_cpu [ 2058.335042] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2058.339064] pending: vmstat_update [ 2058.342916] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2058.346082] pending: lru_add_drain_per_cpu [ 2058.348847] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2058.352274] pending: lru_add_drain_per_cpu BAR(48) [ 2058.356698] workqueue writeback: flags=0x4a [ 2058.358764] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2058.361246] pending: wb_workfn [ 2058.363772] workqueue xprtiod: flags=0xa [ 2058.367300] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2058.369947] pending: xs_stream_data_receive_workfn [ 2058.375472] workqueue ipv6_addrconf: flags=0x40008 [ 2058.377055] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2058.379961] pending: addrconf_verify_work [ 2058.385778] workqueue knbd0-recv: flags=0x1a [ 2058.388180] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2058.390218] in-flight: 658:recv_work [ 2058.392268] workqueue ptlrpc_pinger: flags=0x2 [ 2058.393952] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2058.399661] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2058.407564] pool 8: cpus=0-3 flags=0x4 nice=0 hung=127s workers=4 idle: 32984 37 8 [ 2058.418206] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2043s workers=2 idle: 661 [ 2064.058164] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 2064.060419] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 2064.063876] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2064.063956] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2064.063964] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2064.063970] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2064.064264] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2064.064275] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2064.064280] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2064.064284] RAX: 0000000000007990 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2064.064287] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 2064.064288] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 2064.064290] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 2064.064292] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 2064.064295] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 2064.064297] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2064.064300] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2064.064308] Call Trace: [ 2064.064314] [ 2064.064320] ? show_regs.cold.9+0x22/0x2f [ 2064.064329] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2064.064336] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2064.064341] ? watchdog+0x40/0x40 [ 2064.064345] ? hrtimer_interrupt+0x128/0x2c0 [ 2064.064349] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2064.064357] ? apic_timer_interrupt+0xf/0x20 [ 2064.064362] [ 2064.064365] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2064.064370] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2064.064375] do_raw_spin_lock+0x177/0x190 [ 2064.064379] _raw_spin_lock+0x21/0x30 [ 2064.064383] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2064.064587] process_one_work+0x2c8/0x700 [ 2064.064595] worker_thread+0x296/0x6e0 [ 2064.064600] ? rescuer_thread+0x570/0x570 [ 2064.064603] kthread+0x1d1/0x200 [ 2064.064607] ? set_kthread_struct+0x70/0x70 [ 2064.064610] ret_from_fork+0x1f/0x30 [ 2064.238902] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2064.262028] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2064.267546] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2064.271936] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2064.275263] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2064.284705] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2064.288780] RAX: 0000000000007d89 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2064.292005] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 2064.296935] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2064.301758] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 2064.307916] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 2064.313136] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 2064.317008] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2064.318980] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 2064.328799] Call Trace: [ 2064.331004] [ 2064.332872] ? show_regs.cold.9+0x22/0x2f [ 2064.334953] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2064.337030] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2064.339033] ? watchdog+0x40/0x40 [ 2064.340553] ? hrtimer_interrupt+0x128/0x2c0 [ 2064.341777] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2064.345544] ? apic_timer_interrupt+0xf/0x20 [ 2064.348546] [ 2064.349563] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2064.351700] do_raw_spin_lock+0x177/0x190 [ 2064.354112] _raw_spin_lock+0x21/0x30 [ 2064.355640] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2064.362389] ? do_raw_spin_unlock+0x75/0x190 [ 2064.364579] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2064.367423] ? do_wait_intr+0xf0/0xf0 [ 2064.369509] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2064.372391] mdc_close+0x53b/0xd70 [mdc] [ 2064.373936] lmv_close+0x18d/0x490 [lmv] [ 2064.376141] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2064.382635] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2064.388420] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2064.392677] ll_md_close+0x274/0x800 [lustre] [ 2064.397260] ll_file_release+0x128/0xc50 [lustre] [ 2064.399946] __fput+0x10d/0x440 [ 2064.401482] ____fput+0x12/0x20 [ 2064.406954] task_work_run+0xc6/0x110 [ 2064.410487] exit_to_usermode_loop+0x1dd/0x1f0 [ 2064.412861] do_syscall_64+0x3d6/0x440 [ 2064.414826] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2064.417237] RIP: 0033:0x7f151bee3cd8 [ 2064.419056] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2064.428124] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2064.431939] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 2064.435200] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 2064.438932] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2064.442479] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2064.446721] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2068.059734] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 2068.065053] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2068.100592] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2068.106720] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2068.115824] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2068.120529] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2068.138531] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2068.147187] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2068.155419] RDX: 0000000000007b08 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 2068.163396] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 2068.168894] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 2068.176447] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2068.181600] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 2068.191739] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2068.200954] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2068.213467] Call Trace: [ 2068.214952] [ 2068.216398] ? show_regs.cold.9+0x22/0x2f [ 2068.218359] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2068.220550] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2068.222971] ? watchdog+0x40/0x40 [ 2068.225945] ? hrtimer_interrupt+0x128/0x2c0 [ 2068.228963] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2068.232168] ? apic_timer_interrupt+0xf/0x20 [ 2068.236703] [ 2068.237511] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2068.242842] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2068.250913] do_raw_spin_lock+0x177/0x190 [ 2068.253875] _raw_spin_lock+0x21/0x30 [ 2068.255095] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2068.259408] obd_mod_free+0x46/0x140 [ptlrpc] [ 2068.264031] mdc_commit_open+0x91/0xb0 [mdc] [ 2068.267586] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2068.275095] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2068.284672] after_reply+0xe2c/0x1780 [ptlrpc] [ 2068.286921] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2068.290508] ? __next_timer_interrupt+0x160/0x160 [ 2068.292872] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2068.296826] ? do_wait_intr+0xf0/0xf0 [ 2068.299629] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2068.301654] kthread+0x1d1/0x200 [ 2068.304530] ? set_kthread_struct+0x70/0x70 [ 2068.307161] ret_from_fork+0x1f/0x30 [ 2084.052032] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 2084.054879] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2084.112624] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2084.125829] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2084.133359] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2084.138290] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2084.162401] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2084.167692] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2084.176193] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 2084.180532] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 2084.187595] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 2084.200203] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 2084.208315] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 2084.216841] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2084.229337] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 2084.236975] Call Trace: [ 2084.238501] [ 2084.240022] ? show_regs.cold.9+0x22/0x2f [ 2084.242809] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2084.248906] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2084.255989] ? watchdog+0x40/0x40 [ 2084.258570] ? hrtimer_interrupt+0x128/0x2c0 [ 2084.263640] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2084.270393] ? apic_timer_interrupt+0xf/0x20 [ 2084.276859] [ 2084.278514] ? native_safe_halt+0x2e/0x50 [ 2084.281150] kvm_wait+0xcd/0xe0 [ 2084.282479] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2084.285816] do_raw_spin_lock+0x177/0x190 [ 2084.288001] _raw_spin_lock+0x21/0x30 [ 2084.289212] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2084.296615] ? do_raw_spin_unlock+0x75/0x190 [ 2084.297778] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2084.303019] ? do_wait_intr+0xf0/0xf0 [ 2084.305214] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2084.307731] mdc_reint+0x2d/0x110 [mdc] [ 2084.309470] mdc_unlink+0x658/0x870 [mdc] [ 2084.311506] lmv_unlink+0x34d/0xbd0 [lmv] [ 2084.314174] ll_unlink+0x24d/0x7f0 [lustre] [ 2084.316553] vfs_unlink+0x202/0x330 [ 2084.318523] do_unlinkat+0x2fa/0x460 [ 2084.320711] __x64_sys_unlink+0x37/0x50 [ 2084.322940] do_syscall_64+0xc1/0x440 [ 2084.332559] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2084.335264] RIP: 0033:0x7f151bdee10b [ 2084.336714] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2084.354959] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2084.359087] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 2084.370201] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 2084.376748] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2084.381441] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2084.384749] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2088.934536] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 158s! [ 2088.938750] Showing busy workqueues and worker pools: [ 2088.941113] workqueue events: flags=0x0 [ 2088.943393] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2088.949321] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2088.961920] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2088.966780] pending: kfree_rcu_monitor [ 2088.973562] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2088.978668] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 2088.989729] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2088.999389] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2089.007750] workqueue events_long: flags=0x0 [ 2089.011765] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2089.019407] pending: lu_objects_flush [obdclass] [ 2089.023439] workqueue events_unbound: flags=0x2 [ 2089.032374] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/512 refcnt=5 [ 2089.039813] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn [ 2089.050047] workqueue events_power_efficient: flags=0x80 [ 2089.053766] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2089.061391] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 2089.068328] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2089.074735] pending: neigh_periodic_work [ 2089.080837] workqueue mm_percpu_wq: flags=0x8 [ 2089.083772] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2089.090168] pending: vmstat_update, lru_add_drain_per_cpu [ 2089.093115] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2089.103037] pending: vmstat_update [ 2089.105477] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2089.112035] pending: lru_add_drain_per_cpu [ 2089.113788] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2089.118132] pending: lru_add_drain_per_cpu BAR(48) [ 2089.125122] workqueue writeback: flags=0x4a [ 2089.127718] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2089.133065] pending: wb_workfn [ 2089.139730] workqueue xprtiod: flags=0xa [ 2089.143681] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2089.146115] pending: xs_stream_data_receive_workfn [ 2089.148496] workqueue ipv6_addrconf: flags=0x40008 [ 2089.151504] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2089.154787] pending: addrconf_verify_work [ 2089.157748] workqueue knbd0-recv: flags=0x1a [ 2089.160607] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2089.165739] in-flight: 658:recv_work [ 2089.168673] workqueue ptlrpc_pinger: flags=0x2 [ 2089.171582] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2089.176398] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2089.178992] pool 8: cpus=0-3 flags=0x4 nice=0 hung=158s workers=4 idle: 32984 37 8 [ 2089.182917] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2073s workers=2 idle: 661 [ 2092.056029] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2092.059078] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2092.066112] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 2092.077133] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2092.081597] Modules linked in: lustre(O) [ 2092.086495] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2092.086510] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2092.087137] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2092.087151] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2092.087155] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2092.087160] RAX: 00000000000023e2 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2092.087162] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 2092.087164] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 2092.087166] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 2092.087169] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 2092.087172] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 2092.087175] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2092.087177] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2092.087187] Call Trace: [ 2092.087193] [ 2092.087197] ? show_regs.cold.9+0x22/0x2f [ 2092.087206] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2092.087212] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2092.087216] ? watchdog+0x40/0x40 [ 2092.087220] ? hrtimer_interrupt+0x128/0x2c0 [ 2092.087224] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2092.087233] ? apic_timer_interrupt+0xf/0x20 [ 2092.087237] [ 2092.087240] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2092.087245] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2092.087251] do_raw_spin_lock+0x177/0x190 [ 2092.087255] _raw_spin_lock+0x21/0x30 [ 2092.087260] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2092.087651] process_one_work+0x2c8/0x700 [ 2092.087659] worker_thread+0x296/0x6e0 [ 2092.087664] ? rescuer_thread+0x570/0x570 [ 2092.087667] kthread+0x1d1/0x200 [ 2092.087673] ? set_kthread_struct+0x70/0x70 [ 2092.087677] ret_from_fork+0x1f/0x30 [ 2092.238290] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2092.263940] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2092.269590] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2092.278143] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2092.281707] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2092.295287] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2092.300051] RAX: 00000000000079c1 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2092.303721] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 2092.306578] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2092.316253] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 2092.325994] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 2092.332443] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 2092.341935] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2092.345114] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 2092.352579] Call Trace: [ 2092.354587] [ 2092.355581] ? show_regs.cold.9+0x22/0x2f [ 2092.357080] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2092.359635] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2092.361467] ? watchdog+0x40/0x40 [ 2092.367129] ? hrtimer_interrupt+0x128/0x2c0 [ 2092.369046] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2092.371130] ? apic_timer_interrupt+0xf/0x20 [ 2092.372648] [ 2092.378372] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2092.382829] do_raw_spin_lock+0x177/0x190 [ 2092.384879] _raw_spin_lock+0x21/0x30 [ 2092.386253] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2092.389399] ? do_raw_spin_unlock+0x75/0x190 [ 2092.391239] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2092.393732] ? do_wait_intr+0xf0/0xf0 [ 2092.399741] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2092.402670] mdc_close+0x53b/0xd70 [mdc] [ 2092.404538] lmv_close+0x18d/0x490 [lmv] [ 2092.406647] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2092.412582] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2092.416296] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2092.421133] ll_md_close+0x274/0x800 [lustre] [ 2092.423166] ll_file_release+0x128/0xc50 [lustre] [ 2092.425376] __fput+0x10d/0x440 [ 2092.426348] ____fput+0x12/0x20 [ 2092.427446] task_work_run+0xc6/0x110 [ 2092.429038] exit_to_usermode_loop+0x1dd/0x1f0 [ 2092.430746] do_syscall_64+0x3d6/0x440 [ 2092.432117] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2092.435236] RIP: 0033:0x7f151bee3cd8 [ 2092.440325] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2092.448400] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2092.454722] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 2092.458317] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 2092.461485] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2092.465242] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2092.470310] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2096.055016] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 2096.057917] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2096.090384] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2096.095981] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2096.100334] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2096.102919] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2096.112984] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2096.116967] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2096.121650] RDX: 0000000000002b10 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 2096.126293] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 2096.133364] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 2096.137622] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2096.141585] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 2096.146358] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2096.150342] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2096.156946] Call Trace: [ 2096.158642] [ 2096.160712] ? show_regs.cold.9+0x22/0x2f [ 2096.165647] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2096.169363] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2096.174584] ? watchdog+0x40/0x40 [ 2096.177874] ? hrtimer_interrupt+0x128/0x2c0 [ 2096.179509] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2096.180830] ? apic_timer_interrupt+0xf/0x20 [ 2096.187272] [ 2096.190652] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2096.193897] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2096.196656] do_raw_spin_lock+0x177/0x190 [ 2096.198028] _raw_spin_lock+0x21/0x30 [ 2096.200383] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2096.206646] obd_mod_free+0x46/0x140 [ptlrpc] [ 2096.211596] mdc_commit_open+0x91/0xb0 [mdc] [ 2096.216000] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2096.219466] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2096.226399] after_reply+0xe2c/0x1780 [ptlrpc] [ 2096.232896] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2096.238282] ? __next_timer_interrupt+0x160/0x160 [ 2096.240020] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2096.245652] ? do_wait_intr+0xf0/0xf0 [ 2096.247914] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2096.251609] kthread+0x1d1/0x200 [ 2096.255602] ? set_kthread_struct+0x70/0x70 [ 2096.258008] ret_from_fork+0x1f/0x30 [ 2116.053313] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 2116.058499] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2116.106347] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2116.115837] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2116.119467] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2116.125430] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2116.139222] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2116.146489] RAX: 00000000000061b2 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2116.152834] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 2116.158876] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 2116.163197] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 2116.175678] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 2116.182941] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 2116.197385] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2116.205028] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 2116.212924] Call Trace: [ 2116.217322] [ 2116.218363] ? show_regs.cold.9+0x22/0x2f [ 2116.220807] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2116.230239] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2116.231826] ? watchdog+0x40/0x40 [ 2116.232868] ? hrtimer_interrupt+0x128/0x2c0 [ 2116.235931] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2116.239698] ? apic_timer_interrupt+0xf/0x20 [ 2116.245120] [ 2116.246088] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2116.253153] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2116.262165] do_raw_spin_lock+0x177/0x190 [ 2116.265160] _raw_spin_lock+0x21/0x30 [ 2116.267177] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2116.270594] ? do_raw_spin_unlock+0x75/0x190 [ 2116.273008] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2116.276909] ? do_wait_intr+0xf0/0xf0 [ 2116.279234] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2116.283227] mdc_reint+0x2d/0x110 [mdc] [ 2116.285041] mdc_unlink+0x658/0x870 [mdc] [ 2116.287497] lmv_unlink+0x34d/0xbd0 [lmv] [ 2116.291363] ll_unlink+0x24d/0x7f0 [lustre] [ 2116.295660] vfs_unlink+0x202/0x330 [ 2116.297770] do_unlinkat+0x2fa/0x460 [ 2116.299509] __x64_sys_unlink+0x37/0x50 [ 2116.302228] do_syscall_64+0xc1/0x440 [ 2116.305120] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2116.307494] RIP: 0033:0x7f151bdee10b [ 2116.312067] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2116.326299] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2116.328457] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 2116.331402] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 2116.339983] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2116.346646] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2116.352620] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2119.648039] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 189s! [ 2119.650469] Showing busy workqueues and worker pools: [ 2119.652145] workqueue events: flags=0x0 [ 2119.653683] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2119.655246] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2119.656961] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2119.659421] pending: kfree_rcu_monitor [ 2119.660724] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2119.662372] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 2119.665560] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2119.668151] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2119.672620] workqueue events_long: flags=0x0 [ 2119.673729] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2119.675967] pending: lu_objects_flush [obdclass] [ 2119.677558] workqueue events_unbound: flags=0x2 [ 2119.678702] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/512 refcnt=5 [ 2119.680388] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn [ 2119.682141] workqueue events_power_efficient: flags=0x80 [ 2119.683772] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2119.685402] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 2119.687151] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2119.689348] pending: neigh_periodic_work [ 2119.691145] workqueue mm_percpu_wq: flags=0x8 [ 2119.692731] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2119.696877] pending: vmstat_update, lru_add_drain_per_cpu [ 2119.698817] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2119.700785] pending: vmstat_update [ 2119.701700] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2119.703282] pending: lru_add_drain_per_cpu [ 2119.704419] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2119.706095] pending: lru_add_drain_per_cpu BAR(48) [ 2119.707409] workqueue writeback: flags=0x4a [ 2119.708432] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2119.710016] pending: wb_workfn [ 2119.710847] workqueue xprtiod: flags=0xa [ 2119.711805] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2119.713266] pending: xs_stream_data_receive_workfn [ 2119.714568] workqueue ipv6_addrconf: flags=0x40008 [ 2119.715741] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2119.717306] pending: addrconf_verify_work [ 2119.718342] workqueue knbd0-recv: flags=0x1a [ 2119.719367] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2119.720963] in-flight: 658:recv_work [ 2119.721920] workqueue ptlrpc_pinger: flags=0x2 [ 2119.723037] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2119.724436] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2119.726168] pool 8: cpus=0-3 flags=0x4 nice=0 hung=189s workers=4 idle: 32984 37 8 [ 2119.727994] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2104s workers=2 idle: 661 [ 2120.056098] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2120.058911] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 2120.061782] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) [ 2120.065760] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2120.065815] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2120.065872] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2120.065875] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2120.065887] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2120.065890] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 2120.065893] RAX: 0000000000007e98 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2120.065895] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 2120.065896] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2120.065898] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 2120.065899] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 2120.065902] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 2120.065904] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2120.065905] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 2120.065912] Call Trace: [ 2120.065916] [ 2120.065921] ? show_regs.cold.9+0x22/0x2f [ 2120.065928] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2120.065933] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2120.065936] ? watchdog+0x40/0x40 [ 2120.065938] ? hrtimer_interrupt+0x128/0x2c0 [ 2120.065941] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2120.065948] ? apic_timer_interrupt+0xf/0x20 [ 2120.065951] [ 2120.065953] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2120.065957] do_raw_spin_lock+0x177/0x190 [ 2120.065960] _raw_spin_lock+0x21/0x30 [ 2120.065964] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2120.068003] ? do_raw_spin_unlock+0x75/0x190 [ 2120.068012] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2120.068921] ? do_wait_intr+0xf0/0xf0 [ 2120.068928] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2120.087732] ptlrpc(O) [ 2120.112403] mdc_close+0x53b/0xd70 [mdc] [ 2120.112461] lmv_close+0x18d/0x490 [lmv] [ 2120.112559] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2120.112744] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2120.112858] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2120.112945] ll_md_close+0x274/0x800 [lustre] [ 2120.113136] ll_file_release+0x128/0xc50 [lustre] [ 2120.113225] __fput+0x10d/0x440 [ 2120.113236] ____fput+0x12/0x20 [ 2120.113241] task_work_run+0xc6/0x110 [ 2120.113249] exit_to_usermode_loop+0x1dd/0x1f0 [ 2120.113256] do_syscall_64+0x3d6/0x440 [ 2120.113261] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2120.113271] RIP: 0033:0x7f151bee3cd8 [ 2120.113279] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2120.113283] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2120.113289] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 2120.113291] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 2120.113294] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2120.113296] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2120.113298] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2120.265381] obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2120.287386] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2120.293160] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2120.298433] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2120.302038] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2120.304713] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2120.318879] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2120.326023] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2120.331732] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 2120.337412] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 2120.341212] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 2120.345103] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 2120.349855] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 2120.354066] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2120.360477] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2120.366960] Call Trace: [ 2120.371132] [ 2120.374060] ? show_regs.cold.9+0x22/0x2f [ 2120.381027] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2120.384223] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2120.386095] ? watchdog+0x40/0x40 [ 2120.390083] ? hrtimer_interrupt+0x128/0x2c0 [ 2120.394752] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2120.396477] ? apic_timer_interrupt+0xf/0x20 [ 2120.398745] [ 2120.399636] ? native_safe_halt+0x2e/0x50 [ 2120.401152] kvm_wait+0xcd/0xe0 [ 2120.402683] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2120.404690] do_raw_spin_lock+0x177/0x190 [ 2120.406417] _raw_spin_lock+0x21/0x30 [ 2120.407928] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2120.410369] process_one_work+0x2c8/0x700 [ 2120.411942] worker_thread+0x296/0x6e0 [ 2120.413200] ? rescuer_thread+0x570/0x570 [ 2120.415351] kthread+0x1d1/0x200 [ 2120.416734] ? set_kthread_struct+0x70/0x70 [ 2120.418438] ret_from_fork+0x1f/0x30 [ 2124.055009] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 2124.062236] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2124.108908] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2124.118103] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2124.124500] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2124.131116] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2124.144060] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2124.151355] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2124.156404] RDX: 0000000000005057 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 2124.160266] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 2124.164810] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 2124.167354] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2124.171872] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 2124.174920] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2124.177622] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2124.182040] Call Trace: [ 2124.182915] [ 2124.183944] ? show_regs.cold.9+0x22/0x2f [ 2124.186067] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2124.188371] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2124.190553] ? watchdog+0x40/0x40 [ 2124.191825] ? hrtimer_interrupt+0x128/0x2c0 [ 2124.195062] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2124.197967] ? apic_timer_interrupt+0xf/0x20 [ 2124.200508] [ 2124.201577] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2124.204680] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2124.210804] do_raw_spin_lock+0x177/0x190 [ 2124.216046] _raw_spin_lock+0x21/0x30 [ 2124.218787] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2124.222270] obd_mod_free+0x46/0x140 [ptlrpc] [ 2124.228693] mdc_commit_open+0x91/0xb0 [mdc] [ 2124.231301] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2124.234952] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2124.240322] after_reply+0xe2c/0x1780 [ptlrpc] [ 2124.249336] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2124.259991] ? __next_timer_interrupt+0x160/0x160 [ 2124.265044] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2124.272324] ? do_wait_intr+0xf0/0xf0 [ 2124.274351] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2124.277518] kthread+0x1d1/0x200 [ 2124.278748] ? set_kthread_struct+0x70/0x70 [ 2124.282759] ret_from_fork+0x1f/0x30 [ 2144.055305] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 2144.059657] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2144.108640] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2144.130907] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2144.139089] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2144.144703] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2144.159800] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2144.164847] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2144.168927] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 2144.173884] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 2144.177597] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 2144.181747] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 2144.186863] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 2144.192179] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2144.195631] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 2144.201843] Call Trace: [ 2144.202877] [ 2144.204463] ? show_regs.cold.9+0x22/0x2f [ 2144.225744] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2144.230111] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2144.244991] ? watchdog+0x40/0x40 [ 2144.247107] ? hrtimer_interrupt+0x128/0x2c0 [ 2144.250387] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2144.257105] ? apic_timer_interrupt+0xf/0x20 [ 2144.258996] [ 2144.262544] ? native_safe_halt+0x2e/0x50 [ 2144.265870] kvm_wait+0xcd/0xe0 [ 2144.270078] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2144.273965] do_raw_spin_lock+0x177/0x190 [ 2144.276935] _raw_spin_lock+0x21/0x30 [ 2144.279971] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2144.282422] ? do_raw_spin_unlock+0x75/0x190 [ 2144.285773] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2144.289529] ? do_wait_intr+0xf0/0xf0 [ 2144.291216] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2144.295206] mdc_reint+0x2d/0x110 [mdc] [ 2144.297721] mdc_unlink+0x658/0x870 [mdc] [ 2144.300105] lmv_unlink+0x34d/0xbd0 [lmv] [ 2144.302139] ll_unlink+0x24d/0x7f0 [lustre] [ 2144.303769] vfs_unlink+0x202/0x330 [ 2144.306438] do_unlinkat+0x2fa/0x460 [ 2144.307968] __x64_sys_unlink+0x37/0x50 [ 2144.312412] do_syscall_64+0xc1/0x440 [ 2144.315812] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2144.320814] RIP: 0033:0x7f151bdee10b [ 2144.330829] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2144.367743] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2144.378949] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 2144.384841] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 2144.406905] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2144.419851] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2144.434522] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2148.056019] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2148.059289] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2148.064831] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 2148.119709] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2148.119724] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2148.119730] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2148.120200] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2148.120211] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2148.120215] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2148.120220] RAX: 0000000000002785 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2148.120222] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 2148.120224] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 2148.120227] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 2148.120229] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 2148.120232] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 2148.120235] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2148.120237] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2148.120246] Call Trace: [ 2148.120252] [ 2148.120256] ? show_regs.cold.9+0x22/0x2f [ 2148.120266] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2148.120283] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2148.120287] ? watchdog+0x40/0x40 [ 2148.120292] ? hrtimer_interrupt+0x128/0x2c0 [ 2148.120295] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2148.127946] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) [ 2148.138146] ? apic_timer_interrupt+0xf/0x20 [ 2148.138168] [ 2148.138173] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2148.138179] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2148.138197] do_raw_spin_lock+0x177/0x190 [ 2148.138206] _raw_spin_lock+0x21/0x30 [ 2148.138211] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2148.138633] process_one_work+0x2c8/0x700 [ 2148.138642] worker_thread+0x296/0x6e0 [ 2148.138646] ? rescuer_thread+0x570/0x570 [ 2148.138650] kthread+0x1d1/0x200 [ 2148.138654] ? set_kthread_struct+0x70/0x70 [ 2148.138658] ret_from_fork+0x1f/0x30 [ 2148.350144] lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2148.367877] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2148.371790] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2148.375709] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2148.379980] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2148.392124] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2148.395919] RAX: 0000000000003893 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2148.399600] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 2148.402785] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2148.408476] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 2148.412739] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 2148.417176] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 2148.423112] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2148.431480] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 2148.437586] Call Trace: [ 2148.440073] [ 2148.441570] ? show_regs.cold.9+0x22/0x2f [ 2148.444739] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2148.447618] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2148.452120] ? watchdog+0x40/0x40 [ 2148.459536] ? hrtimer_interrupt+0x128/0x2c0 [ 2148.461945] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2148.471589] ? apic_timer_interrupt+0xf/0x20 [ 2148.477651] [ 2148.480348] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2148.482182] do_raw_spin_lock+0x177/0x190 [ 2148.484164] _raw_spin_lock+0x21/0x30 [ 2148.485690] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2148.489053] ? do_raw_spin_unlock+0x75/0x190 [ 2148.490448] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2148.496863] ? do_wait_intr+0xf0/0xf0 [ 2148.497846] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2148.500646] mdc_close+0x53b/0xd70 [mdc] [ 2148.504328] lmv_close+0x18d/0x490 [lmv] [ 2148.505274] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2148.509839] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2148.511586] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2148.515164] ll_md_close+0x274/0x800 [lustre] [ 2148.516276] ll_file_release+0x128/0xc50 [lustre] [ 2148.518045] __fput+0x10d/0x440 [ 2148.519460] ____fput+0x12/0x20 [ 2148.520775] task_work_run+0xc6/0x110 [ 2148.522132] exit_to_usermode_loop+0x1dd/0x1f0 [ 2148.523777] do_syscall_64+0x3d6/0x440 [ 2148.525497] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2148.527754] RIP: 0033:0x7f151bee3cd8 [ 2148.529627] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2148.537522] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2148.540652] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 2148.543844] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 2148.546938] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2148.550548] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2148.553934] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2149.858044] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 219s! [ 2149.862789] Showing busy workqueues and worker pools: [ 2149.864524] workqueue events: flags=0x0 [ 2149.867801] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2149.870764] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2149.880414] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2149.883078] pending: kfree_rcu_monitor [ 2149.887716] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2149.891316] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 2149.907900] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2149.915393] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2149.927456] workqueue events_long: flags=0x0 [ 2149.930187] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2149.939143] pending: lu_objects_flush [obdclass] [ 2149.947826] workqueue events_unbound: flags=0x2 [ 2149.952646] pwq 8: cpus=0-3 flags=0x4 nice=0 active=4/512 refcnt=6 [ 2149.955886] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2149.960842] workqueue events_power_efficient: flags=0x80 [ 2149.964419] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2149.966637] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 2149.971575] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2149.974965] pending: neigh_periodic_work [ 2149.977723] workqueue mm_percpu_wq: flags=0x8 [ 2149.982521] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2149.989837] pending: vmstat_update, lru_add_drain_per_cpu [ 2149.996478] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2150.004413] pending: vmstat_update [ 2150.006656] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2150.009871] pending: lru_add_drain_per_cpu [ 2150.012405] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2150.015698] pending: lru_add_drain_per_cpu BAR(48) [ 2150.018071] workqueue writeback: flags=0x4a [ 2150.022758] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2150.029738] pending: wb_workfn [ 2150.033584] workqueue xprtiod: flags=0xa [ 2150.038211] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2150.043696] pending: xs_stream_data_receive_workfn [ 2150.049192] workqueue ipv6_addrconf: flags=0x40008 [ 2150.052461] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2150.059643] pending: addrconf_verify_work [ 2150.062585] workqueue knbd0-recv: flags=0x1a [ 2150.064896] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2150.070306] in-flight: 658:recv_work [ 2150.072335] workqueue ptlrpc_pinger: flags=0x2 [ 2150.074511] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2150.081789] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2150.088728] pool 8: cpus=0-3 flags=0x4 nice=0 hung=219s workers=4 idle: 32984 37 8 [ 2150.097072] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2134s workers=2 idle: 661 [ 2152.055080] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 2152.058825] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2152.084362] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2152.090998] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2152.096146] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2152.100008] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2152.109042] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2152.113166] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2152.117376] RDX: 0000000000001e4b RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 2152.122200] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 2152.125232] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 2152.130366] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2152.133012] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 2152.142296] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2152.151422] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2152.155856] Call Trace: [ 2152.157101] [ 2152.162199] ? show_regs.cold.9+0x22/0x2f [ 2152.163684] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2152.170004] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2152.175140] ? watchdog+0x40/0x40 [ 2152.177233] ? hrtimer_interrupt+0x128/0x2c0 [ 2152.180306] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2152.185267] ? apic_timer_interrupt+0xf/0x20 [ 2152.189966] [ 2152.191045] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2152.196250] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2152.199982] do_raw_spin_lock+0x177/0x190 [ 2152.203057] _raw_spin_lock+0x21/0x30 [ 2152.208165] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2152.215061] obd_mod_free+0x46/0x140 [ptlrpc] [ 2152.220381] mdc_commit_open+0x91/0xb0 [mdc] [ 2152.225120] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2152.232016] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2152.236850] after_reply+0xe2c/0x1780 [ptlrpc] [ 2152.243353] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2152.254330] ? __next_timer_interrupt+0x160/0x160 [ 2152.261870] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2152.264849] ? do_wait_intr+0xf0/0xf0 [ 2152.266094] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2152.275680] kthread+0x1d1/0x200 [ 2152.277968] ? set_kthread_struct+0x70/0x70 [ 2152.282890] ret_from_fork+0x1f/0x30 [ 2164.652073] rcu: INFO: rcu_sched self-detected stall on CPU [ 2164.655770] rcu: 1-...!: (147431 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 2164.662181] (t=240005 jiffies g=249657 q=2547) [ 2164.664918] rcu: rcu_sched kthread starved for 234957 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 2164.671720] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 2164.677689] rcu: RCU grace-period kthread stack dump: [ 2164.681042] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 2164.687092] Call Trace: [ 2164.693887] __schedule+0x351/0xcb0 [ 2164.696910] schedule+0xc0/0x180 [ 2164.699279] schedule_timeout+0xb4/0x190 [ 2164.701846] ? do_raw_spin_unlock+0x75/0x190 [ 2164.704247] ? __next_timer_interrupt+0x160/0x160 [ 2164.707123] ? prepare_to_swait_event+0xf3/0x140 [ 2164.710321] rcu_gp_kthread+0x8dd/0x15b0 [ 2164.712896] ? force_qs_rnp+0x2c0/0x2c0 [ 2164.715595] kthread+0x1d1/0x200 [ 2164.717741] ? set_kthread_struct+0x70/0x70 [ 2164.721424] ret_from_fork+0x1f/0x30 [ 2164.723634] rcu: Stack dump where RCU GP kthread last ran: [ 2164.726517] Sending NMI from CPU 1 to CPUs 3: [ 2164.729925] NMI backtrace for cpu 3 skipped: idling at native_safe_halt+0x2e/0x50 [ 2164.734228] NMI backtrace for cpu 1 [ 2164.737380] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2164.745839] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2164.754577] Call Trace: [ 2164.756320] [ 2164.758577] ? dump_stack+0xbb/0x10e [ 2164.760571] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2164.765573] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2164.769176] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2164.773712] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2164.779365] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2164.784812] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 2164.787520] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 2164.791748] ? update_process_times+0x6d/0xc0 [ 2164.795910] ? tick_sched_handle.isra.18+0x27/0x90 [ 2164.798987] ? tick_sched_timer+0x53/0xb0 [ 2164.801397] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2164.804510] ? tick_sched_handle.isra.18+0x90/0x90 [ 2164.807857] ? hrtimer_interrupt+0x128/0x2c0 [ 2164.810455] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2164.813193] ? apic_timer_interrupt+0xf/0x20 [ 2164.816929] [ 2164.818383] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2164.822896] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2164.826749] ? do_raw_spin_lock+0x177/0x190 [ 2164.830530] ? _raw_spin_lock+0x21/0x30 [ 2164.833151] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2164.837837] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2164.842689] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2164.859423] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2164.870420] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2164.881527] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2164.887009] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2164.895052] ? __next_timer_interrupt+0x160/0x160 [ 2164.897854] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2164.901550] ? do_wait_intr+0xf0/0xf0 [ 2164.906205] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2164.910620] ? kthread+0x1d1/0x200 [ 2164.913020] ? set_kthread_struct+0x70/0x70 [ 2164.928859] ? ret_from_fork+0x1f/0x30 [ 2176.052014] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 2176.055743] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2176.067741] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2176.106499] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2176.106591] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2176.106595] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2176.106615] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2176.106619] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2176.106625] RAX: 00000000000015e8 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2176.106627] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 2176.106629] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 2176.106632] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 2176.106634] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 2176.106637] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 2176.106640] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2176.106642] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 2176.106649] Call Trace: [ 2176.106656] [ 2176.106662] ? show_regs.cold.9+0x22/0x2f [ 2176.106725] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2176.106733] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2176.106737] ? watchdog+0x40/0x40 [ 2176.106742] ? hrtimer_interrupt+0x128/0x2c0 [ 2176.106746] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2176.106754] ? apic_timer_interrupt+0xf/0x20 [ 2176.106759] [ 2176.106762] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2176.106767] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2176.106771] do_raw_spin_lock+0x177/0x190 [ 2176.106775] _raw_spin_lock+0x21/0x30 [ 2176.106783] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2176.107130] ? do_raw_spin_unlock+0x75/0x190 [ 2176.107136] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2176.107459] ? do_wait_intr+0xf0/0xf0 [ 2176.107467] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2176.107731] mdc_reint+0x2d/0x110 [mdc] [ 2176.107767] mdc_unlink+0x658/0x870 [mdc] [ 2176.107792] lmv_unlink+0x34d/0xbd0 [lmv] [ 2176.107820] ll_unlink+0x24d/0x7f0 [lustre] [ 2176.107936] vfs_unlink+0x202/0x330 [ 2176.107944] do_unlinkat+0x2fa/0x460 [ 2176.107950] __x64_sys_unlink+0x37/0x50 [ 2176.108148] do_syscall_64+0xc1/0x440 [ 2176.108156] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2176.108164] RIP: 0033:0x7f151bdee10b [ 2176.108170] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2176.108174] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2176.108183] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 2176.108186] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 2176.108188] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2176.108190] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2176.108192] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2176.294167] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2176.326798] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2176.332196] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2176.336455] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2176.339428] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2176.342366] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2176.351293] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2176.354770] RAX: 000000000000631c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2176.358716] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 2176.362646] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 2176.366063] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 2176.369567] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 2176.373385] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 2176.378765] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2176.383126] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2176.387419] Call Trace: [ 2176.388897] [ 2176.389995] ? show_regs.cold.9+0x22/0x2f [ 2176.392385] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2176.395024] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2176.397855] ? watchdog+0x40/0x40 [ 2176.399624] ? hrtimer_interrupt+0x128/0x2c0 [ 2176.401754] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2176.405198] ? apic_timer_interrupt+0xf/0x20 [ 2176.407049] [ 2176.408014] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2176.411209] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2176.415507] do_raw_spin_lock+0x177/0x190 [ 2176.418013] _raw_spin_lock+0x21/0x30 [ 2176.420144] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2176.423520] process_one_work+0x2c8/0x700 [ 2176.425960] worker_thread+0x296/0x6e0 [ 2176.428177] ? rescuer_thread+0x570/0x570 [ 2176.430401] kthread+0x1d1/0x200 [ 2176.432050] ? set_kthread_struct+0x70/0x70 [ 2176.434181] ret_from_fork+0x1f/0x30 [ 2180.581024] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 249s! [ 2180.589514] Showing busy workqueues and worker pools: [ 2180.594438] workqueue events: flags=0x0 [ 2180.597625] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2180.609996] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2180.614552] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2180.619370] pending: kfree_rcu_monitor [ 2180.621040] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2180.630160] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 2180.637390] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2180.641135] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2180.644596] workqueue events_long: flags=0x0 [ 2180.646598] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2180.649001] pending: lu_objects_flush [obdclass] [ 2180.653210] workqueue events_unbound: flags=0x2 [ 2180.656493] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2180.660809] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2180.667083] workqueue events_power_efficient: flags=0x80 [ 2180.670632] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2180.674925] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 2180.684099] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2180.692211] pending: neigh_periodic_work [ 2180.695417] workqueue mm_percpu_wq: flags=0x8 [ 2180.699102] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2180.706608] pending: vmstat_update, lru_add_drain_per_cpu [ 2180.711452] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2180.714882] pending: vmstat_update [ 2180.716678] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2180.719401] pending: lru_add_drain_per_cpu [ 2180.721568] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2180.724731] pending: lru_add_drain_per_cpu BAR(48) [ 2180.727236] workqueue writeback: flags=0x4a [ 2180.729346] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2180.732409] pending: wb_workfn [ 2180.734528] workqueue xprtiod: flags=0xa [ 2180.736426] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2180.739566] pending: xs_stream_data_receive_workfn [ 2180.742134] workqueue ipv6_addrconf: flags=0x40008 [ 2180.745446] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2180.753799] pending: addrconf_verify_work [ 2180.758478] workqueue knbd0-recv: flags=0x1a [ 2180.760144] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2180.764747] in-flight: 658:recv_work [ 2180.766767] workqueue ptlrpc_pinger: flags=0x2 [ 2180.769620] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2180.775889] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2180.779459] pool 8: cpus=0-3 flags=0x4 nice=0 hung=250s workers=4 idle: 32984 37 8 [ 2180.783800] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2165s workers=2 idle: 661 [ 2192.057029] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 2192.057514] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 2192.060365] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2192.060450] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2192.060457] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2192.060460] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2192.060474] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2192.060478] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2192.060483] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2192.060485] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 2192.060487] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2192.060490] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 2192.060492] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 2192.060495] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 2192.060498] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2192.060501] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 2192.060510] Call Trace: [ 2192.060515] [ 2192.060521] ? show_regs.cold.9+0x22/0x2f [ 2192.060532] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2192.060539] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2192.060544] ? watchdog+0x40/0x40 [ 2192.060548] ? hrtimer_interrupt+0x128/0x2c0 [ 2192.060552] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2192.060559] ? apic_timer_interrupt+0xf/0x20 [ 2192.060564] [ 2192.060567] ? native_safe_halt+0x2e/0x50 [ 2192.060570] kvm_wait+0xcd/0xe0 [ 2192.060579] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2192.060588] do_raw_spin_lock+0x177/0x190 [ 2192.060592] _raw_spin_lock+0x21/0x30 [ 2192.060597] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2192.060984] ? do_raw_spin_unlock+0x75/0x190 [ 2192.060990] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2192.061193] ? do_wait_intr+0xf0/0xf0 [ 2192.061200] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2192.061400] mdc_close+0x53b/0xd70 [mdc] [ 2192.061438] lmv_close+0x18d/0x490 [lmv] [ 2192.061466] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2192.061641] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2192.081709] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2192.081864] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2192.081877] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2192.081879] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2192.081894] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2192.081898] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2192.081903] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2192.081905] RDX: 00000000000039d8 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 2192.081908] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 2192.081910] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 2192.081912] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2192.081915] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 2192.081918] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2192.081921] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2192.081929] Call Trace: [ 2192.081944] [ 2192.081950] ? show_regs.cold.9+0x22/0x2f [ 2192.081959] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2192.081965] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2192.081969] ? watchdog+0x40/0x40 [ 2192.081973] ? hrtimer_interrupt+0x128/0x2c0 [ 2192.081976] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2192.081985] ? apic_timer_interrupt+0xf/0x20 [ 2192.081989] [ 2192.081992] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2192.081997] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2192.082001] do_raw_spin_lock+0x177/0x190 [ 2192.082005] _raw_spin_lock+0x21/0x30 [ 2192.082016] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2192.087324] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2192.087460] ll_md_close+0x274/0x800 [lustre] [ 2192.087558] ll_file_release+0x128/0xc50 [lustre] [ 2192.087648] __fput+0x10d/0x440 [ 2192.087657] ____fput+0x12/0x20 [ 2192.087661] task_work_run+0xc6/0x110 [ 2192.087669] exit_to_usermode_loop+0x1dd/0x1f0 [ 2192.087676] do_syscall_64+0x3d6/0x440 [ 2192.087681] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2192.087690] RIP: 0033:0x7f151bee3cd8 [ 2192.087698] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2192.087702] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2192.087708] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 2192.087710] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 2192.087712] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2192.087715] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2192.087717] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2192.502320] obd_mod_free+0x46/0x140 [ptlrpc] [ 2192.505129] mdc_commit_open+0x91/0xb0 [mdc] [ 2192.508515] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2192.511612] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2192.520507] after_reply+0xe2c/0x1780 [ptlrpc] [ 2192.524672] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2192.533174] ? __next_timer_interrupt+0x160/0x160 [ 2192.537744] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2192.545991] ? do_wait_intr+0xf0/0xf0 [ 2192.548058] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2192.552719] kthread+0x1d1/0x200 [ 2192.554383] ? set_kthread_struct+0x70/0x70 [ 2192.559130] ret_from_fork+0x1f/0x30 [ 2204.062357] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2204.070280] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2204.129517] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2204.142944] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2204.166519] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2204.173878] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2204.188701] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2204.209679] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2204.214453] RAX: 00000000000017c4 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2204.217950] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 2204.222262] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 2204.227652] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 2204.231972] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 2204.235482] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 2204.240141] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2204.243007] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2204.246385] Call Trace: [ 2204.247663] [ 2204.248488] ? show_regs.cold.9+0x22/0x2f [ 2204.250372] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2204.253888] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2204.257465] ? watchdog+0x40/0x40 [ 2204.260971] ? hrtimer_interrupt+0x128/0x2c0 [ 2204.264076] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2204.265598] ? apic_timer_interrupt+0xf/0x20 [ 2204.269440] [ 2204.271306] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2204.274951] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2204.278099] do_raw_spin_lock+0x177/0x190 [ 2204.279867] _raw_spin_lock+0x21/0x30 [ 2204.285415] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2204.290379] process_one_work+0x2c8/0x700 [ 2204.294742] worker_thread+0x296/0x6e0 [ 2204.296510] ? rescuer_thread+0x570/0x570 [ 2204.299975] kthread+0x1d1/0x200 [ 2204.304652] ? set_kthread_struct+0x70/0x70 [ 2204.309033] ret_from_fork+0x1f/0x30 [ 2208.052244] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 2208.063043] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2208.110918] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2208.123314] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2208.132897] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2208.137488] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2208.147448] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2208.153303] RAX: 0000000000001f77 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2208.161552] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 2208.170220] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 2208.172986] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 2208.175998] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 2208.178937] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 2208.182671] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2208.184993] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 2208.187799] Call Trace: [ 2208.188799] [ 2208.189863] ? show_regs.cold.9+0x22/0x2f [ 2208.191436] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2208.193869] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2208.195615] ? watchdog+0x40/0x40 [ 2208.196933] ? hrtimer_interrupt+0x128/0x2c0 [ 2208.198791] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2208.200916] ? apic_timer_interrupt+0xf/0x20 [ 2208.202628] [ 2208.203506] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2208.205907] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2208.207989] do_raw_spin_lock+0x177/0x190 [ 2208.210210] _raw_spin_lock+0x21/0x30 [ 2208.211849] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2208.214189] ? do_raw_spin_unlock+0x75/0x190 [ 2208.215866] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2208.218125] ? do_wait_intr+0xf0/0xf0 [ 2208.219528] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2208.224484] mdc_reint+0x2d/0x110 [mdc] [ 2208.226437] mdc_unlink+0x658/0x870 [mdc] [ 2208.228309] lmv_unlink+0x34d/0xbd0 [lmv] [ 2208.230237] ll_unlink+0x24d/0x7f0 [lustre] [ 2208.232015] vfs_unlink+0x202/0x330 [ 2208.233609] do_unlinkat+0x2fa/0x460 [ 2208.234874] __x64_sys_unlink+0x37/0x50 [ 2208.236401] do_syscall_64+0xc1/0x440 [ 2208.237911] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2208.240167] RIP: 0033:0x7f151bdee10b [ 2208.241625] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2208.250862] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2208.254910] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 2208.258139] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 2208.261390] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2208.264449] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2208.267611] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2211.305276] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 280s! [ 2211.312080] Showing busy workqueues and worker pools: [ 2211.321015] workqueue events: flags=0x0 [ 2211.325286] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2211.333882] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2211.336327] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2211.343317] pending: kfree_rcu_monitor [ 2211.346121] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2211.354941] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 2211.361083] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2211.364940] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2211.368909] workqueue events_long: flags=0x0 [ 2211.370962] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2211.374303] pending: lu_objects_flush [obdclass] [ 2211.377735] workqueue events_unbound: flags=0x2 [ 2211.380296] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2211.385390] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2211.394624] workqueue events_power_efficient: flags=0x80 [ 2211.397083] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2211.404957] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 2211.410854] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2211.417151] pending: neigh_periodic_work [ 2211.422430] workqueue mm_percpu_wq: flags=0x8 [ 2211.425963] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2211.431153] pending: vmstat_update, lru_add_drain_per_cpu [ 2211.435158] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2211.442704] pending: vmstat_update [ 2211.444990] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2211.450064] pending: lru_add_drain_per_cpu [ 2211.453254] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2211.457793] pending: lru_add_drain_per_cpu BAR(48) [ 2211.461090] workqueue writeback: flags=0x4a [ 2211.465722] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2211.471883] pending: wb_workfn [ 2211.475943] workqueue xprtiod: flags=0xa [ 2211.477815] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2211.480729] pending: xs_stream_data_receive_workfn [ 2211.483135] workqueue ipv6_addrconf: flags=0x40008 [ 2211.485359] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2211.488272] pending: addrconf_verify_work [ 2211.494391] workqueue knbd0-recv: flags=0x1a [ 2211.501062] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2211.508226] in-flight: 658:recv_work [ 2211.515969] workqueue ptlrpc_pinger: flags=0x2 [ 2211.519186] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2211.522920] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2211.531979] pool 8: cpus=0-3 flags=0x4 nice=0 hung=280s workers=4 idle: 32984 37 8 [ 2211.538728] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2196s workers=2 idle: 661 [ 2220.056575] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 2220.057723] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 2220.060127] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2220.060216] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2220.060224] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2220.060230] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2220.060246] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2220.060251] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2220.060256] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2220.060258] RDX: 0000000000007778 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 2220.060261] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 2220.060262] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 2220.060265] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2220.060268] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 2220.060271] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2220.060273] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2220.060281] Call Trace: [ 2220.060291] [ 2220.060297] ? show_regs.cold.9+0x22/0x2f [ 2220.060306] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2220.060312] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2220.060316] ? watchdog+0x40/0x40 [ 2220.060324] ? hrtimer_interrupt+0x128/0x2c0 [ 2220.060328] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2220.060338] ? apic_timer_interrupt+0xf/0x20 [ 2220.060343] [ 2220.060346] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2220.060350] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2220.060354] do_raw_spin_lock+0x177/0x190 [ 2220.060359] _raw_spin_lock+0x21/0x30 [ 2220.060365] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2220.060701] obd_mod_free+0x46/0x140 [ptlrpc] [ 2220.061010] mdc_commit_open+0x91/0xb0 [mdc] [ 2220.061107] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2220.065115] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2220.066385] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2220.066398] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2220.066401] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2220.066416] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2220.066421] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2220.066425] RAX: 0000000000007709 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2220.066427] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 2220.066429] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2220.066431] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 2220.066434] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 2220.066437] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 2220.066440] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2220.066442] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 2220.066450] Call Trace: [ 2220.066456] [ 2220.066461] ? show_regs.cold.9+0x22/0x2f [ 2220.066471] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2220.066478] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2220.066483] ? watchdog+0x40/0x40 [ 2220.066486] ? hrtimer_interrupt+0x128/0x2c0 [ 2220.066491] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2220.066499] ? apic_timer_interrupt+0xf/0x20 [ 2220.066504] [ 2220.066506] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2220.066513] do_raw_spin_lock+0x177/0x190 [ 2220.066519] _raw_spin_lock+0x21/0x30 [ 2220.066524] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2220.096625] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2220.097712] after_reply+0xe2c/0x1780 [ptlrpc] [ 2220.097995] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2220.098926] ? __next_timer_interrupt+0x160/0x160 [ 2220.098937] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2220.099159] ? do_wait_intr+0xf0/0xf0 [ 2220.099168] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2220.099366] kthread+0x1d1/0x200 [ 2220.099371] ? set_kthread_struct+0x70/0x70 [ 2220.099375] ret_from_fork+0x1f/0x30 [ 2220.369023] ? do_raw_spin_unlock+0x75/0x190 [ 2220.370933] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2220.373260] ? do_wait_intr+0xf0/0xf0 [ 2220.374919] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2220.380390] mdc_close+0x53b/0xd70 [mdc] [ 2220.382413] lmv_close+0x18d/0x490 [lmv] [ 2220.385908] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2220.391459] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2220.394203] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2220.396364] ll_md_close+0x274/0x800 [lustre] [ 2220.398485] ll_file_release+0x128/0xc50 [lustre] [ 2220.400639] __fput+0x10d/0x440 [ 2220.402020] ____fput+0x12/0x20 [ 2220.403244] task_work_run+0xc6/0x110 [ 2220.404623] exit_to_usermode_loop+0x1dd/0x1f0 [ 2220.405743] do_syscall_64+0x3d6/0x440 [ 2220.408719] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2220.412355] RIP: 0033:0x7f151bee3cd8 [ 2220.414190] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2220.422969] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2220.426144] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 2220.428389] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 2220.431208] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2220.434328] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2220.436872] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2232.056305] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2232.064896] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2232.102879] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2232.110313] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2232.115050] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2232.120331] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2232.123154] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2232.134649] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2232.147400] RAX: 0000000000005324 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2232.151445] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 2232.160001] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 2232.166781] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 2232.178910] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 2232.189256] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 2232.196302] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2232.204921] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2232.210021] Call Trace: [ 2232.211402] [ 2232.212103] ? show_regs.cold.9+0x22/0x2f [ 2232.213536] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2232.221355] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2232.231875] ? watchdog+0x40/0x40 [ 2232.234110] ? hrtimer_interrupt+0x128/0x2c0 [ 2232.236040] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2232.238764] ? apic_timer_interrupt+0xf/0x20 [ 2232.242945] [ 2232.243918] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2232.247864] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2232.256164] do_raw_spin_lock+0x177/0x190 [ 2232.262980] _raw_spin_lock+0x21/0x30 [ 2232.270600] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2232.278352] process_one_work+0x2c8/0x700 [ 2232.292021] worker_thread+0x296/0x6e0 [ 2232.293267] ? rescuer_thread+0x570/0x570 [ 2232.294576] kthread+0x1d1/0x200 [ 2232.301115] ? set_kthread_struct+0x70/0x70 [ 2232.304336] ret_from_fork+0x1f/0x30 [ 2236.052062] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 2236.055340] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2236.096295] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2236.104424] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2236.110966] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2236.114353] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2236.130196] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2236.140165] RAX: 0000000000006b0c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2236.144503] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 2236.147684] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 2236.151797] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 2236.155740] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 2236.164172] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 2236.173969] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2236.182180] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 2236.187128] Call Trace: [ 2236.192083] [ 2236.194973] ? show_regs.cold.9+0x22/0x2f [ 2236.199166] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2236.204094] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2236.207243] ? watchdog+0x40/0x40 [ 2236.208796] ? hrtimer_interrupt+0x128/0x2c0 [ 2236.210239] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2236.216924] ? apic_timer_interrupt+0xf/0x20 [ 2236.218323] [ 2236.223225] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2236.228316] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2236.233776] do_raw_spin_lock+0x177/0x190 [ 2236.237180] _raw_spin_lock+0x21/0x30 [ 2236.239207] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2236.243280] ? do_raw_spin_unlock+0x75/0x190 [ 2236.251042] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2236.261558] ? do_wait_intr+0xf0/0xf0 [ 2236.265226] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2236.267228] mdc_reint+0x2d/0x110 [mdc] [ 2236.270420] mdc_unlink+0x658/0x870 [mdc] [ 2236.272917] lmv_unlink+0x34d/0xbd0 [lmv] [ 2236.276260] ll_unlink+0x24d/0x7f0 [lustre] [ 2236.285205] vfs_unlink+0x202/0x330 [ 2236.288055] do_unlinkat+0x2fa/0x460 [ 2236.291608] __x64_sys_unlink+0x37/0x50 [ 2236.301110] do_syscall_64+0xc1/0x440 [ 2236.306607] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2236.311419] RIP: 0033:0x7f151bdee10b [ 2236.317226] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2236.343491] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2236.352142] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 2236.362195] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 2236.369228] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2236.377536] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2236.385202] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2242.017339] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 311s! [ 2242.027791] Showing busy workqueues and worker pools: [ 2242.034852] workqueue events: flags=0x0 [ 2242.040865] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2242.049534] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2242.060758] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2242.078965] pending: kfree_rcu_monitor [ 2242.082853] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2242.091780] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 2242.105609] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2242.108024] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2242.118793] workqueue events_long: flags=0x0 [ 2242.126690] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2242.137832] pending: lu_objects_flush [obdclass] [ 2242.144236] workqueue events_unbound: flags=0x2 [ 2242.153413] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2242.160784] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2242.171651] workqueue events_power_efficient: flags=0x80 [ 2242.174199] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2242.177491] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 2242.190173] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2242.194423] pending: neigh_periodic_work [ 2242.206619] workqueue mm_percpu_wq: flags=0x8 [ 2242.211986] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2242.218167] pending: vmstat_update, lru_add_drain_per_cpu [ 2242.230606] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2242.242135] pending: vmstat_update [ 2242.243428] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2242.249423] pending: lru_add_drain_per_cpu [ 2242.253809] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2242.259065] pending: lru_add_drain_per_cpu BAR(48) [ 2242.273167] workqueue writeback: flags=0x4a [ 2242.279968] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2242.289429] pending: wb_workfn [ 2242.292715] workqueue xprtiod: flags=0xa [ 2242.294068] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2242.305478] pending: xs_stream_data_receive_workfn, xprt_autoclose [ 2242.311546] workqueue ipv6_addrconf: flags=0x40008 [ 2242.313791] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2242.316573] pending: addrconf_verify_work [ 2242.318826] workqueue knbd0-recv: flags=0x1a [ 2242.325530] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2242.329281] in-flight: 658:recv_work [ 2242.331951] workqueue ptlrpc_pinger: flags=0x2 [ 2242.333538] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2242.344866] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2242.352732] pool 8: cpus=0-3 flags=0x4 nice=0 hung=311s workers=4 idle: 32984 37 8 [ 2242.359763] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2227s workers=2 idle: 661 [ 2248.055082] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 2248.057141] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 2248.058530] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2248.058707] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2248.058714] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2248.058717] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2248.058730] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2248.058735] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2248.058739] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 2248.058741] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 2248.058743] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 2248.058746] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 2248.058748] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2248.058751] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 2248.058754] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2248.058756] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2248.058766] Call Trace: [ 2248.058771] [ 2248.058778] ? show_regs.cold.9+0x22/0x2f [ 2248.058790] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2248.058797] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2248.058801] ? watchdog+0x40/0x40 [ 2248.058805] ? hrtimer_interrupt+0x128/0x2c0 [ 2248.058808] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2248.058816] ? apic_timer_interrupt+0xf/0x20 [ 2248.058820] [ 2248.058823] ? native_safe_halt+0x2e/0x50 [ 2248.058827] kvm_wait+0xcd/0xe0 [ 2248.058834] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2248.058842] do_raw_spin_lock+0x177/0x190 [ 2248.058846] _raw_spin_lock+0x21/0x30 [ 2248.058851] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2248.067119] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2248.067198] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2248.067206] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2248.067209] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2248.067222] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2248.067226] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2248.067231] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2248.067233] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 2248.067235] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2248.067237] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 2248.067240] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 2248.067243] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 2248.067246] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2248.067248] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 2248.067436] Call Trace: [ 2248.091055] obd_mod_free+0x46/0x140 [ptlrpc] [ 2248.102207] [ 2248.109779] mdc_commit_open+0x91/0xb0 [mdc] [ 2248.109830] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2248.122844] ? show_regs.cold.9+0x22/0x2f [ 2248.122869] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2248.122875] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2248.122879] ? watchdog+0x40/0x40 [ 2248.122883] ? hrtimer_interrupt+0x128/0x2c0 [ 2248.122887] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2248.122895] ? apic_timer_interrupt+0xf/0x20 [ 2248.122900] [ 2248.122905] ? native_safe_halt+0x2e/0x50 [ 2248.122910] kvm_wait+0xcd/0xe0 [ 2248.122919] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2248.122929] do_raw_spin_lock+0x177/0x190 [ 2248.122933] _raw_spin_lock+0x21/0x30 [ 2248.122937] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2248.141465] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2248.150997] ? do_raw_spin_unlock+0x75/0x190 [ 2248.151015] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2248.158125] after_reply+0xe2c/0x1780 [ptlrpc] [ 2248.160568] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2248.160944] ? __next_timer_interrupt+0x160/0x160 [ 2248.160954] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2248.161226] ? do_wait_intr+0xf0/0xf0 [ 2248.161234] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2248.170827] ? do_wait_intr+0xf0/0xf0 [ 2248.170849] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2248.171102] mdc_close+0x53b/0xd70 [mdc] [ 2248.171140] lmv_close+0x18d/0x490 [lmv] [ 2248.171172] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2248.171325] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2248.178952] kthread+0x1d1/0x200 [ 2248.178970] ? set_kthread_struct+0x70/0x70 [ 2248.178975] ret_from_fork+0x1f/0x30 [ 2248.477070] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2248.486584] ll_md_close+0x274/0x800 [lustre] [ 2248.496761] ll_file_release+0x128/0xc50 [lustre] [ 2248.499888] __fput+0x10d/0x440 [ 2248.501634] ____fput+0x12/0x20 [ 2248.503444] task_work_run+0xc6/0x110 [ 2248.505065] exit_to_usermode_loop+0x1dd/0x1f0 [ 2248.507483] do_syscall_64+0x3d6/0x440 [ 2248.509901] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2248.512416] RIP: 0033:0x7f151bee3cd8 [ 2248.517253] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2248.532421] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2248.536368] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 2248.542575] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 2248.551707] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2248.558905] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2248.569936] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2260.056118] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2260.059137] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2260.096216] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2260.100974] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2260.104691] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2260.107297] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2260.109232] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2260.116057] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2260.118924] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2260.121518] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 2260.124002] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 2260.126890] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 2260.129995] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 2260.132882] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 2260.136003] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2260.138130] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2260.140782] Call Trace: [ 2260.141677] [ 2260.142466] ? show_regs.cold.9+0x22/0x2f [ 2260.144324] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2260.145877] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2260.147809] ? watchdog+0x40/0x40 [ 2260.149065] ? hrtimer_interrupt+0x128/0x2c0 [ 2260.150690] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2260.153028] ? apic_timer_interrupt+0xf/0x20 [ 2260.154591] [ 2260.155669] ? native_safe_halt+0x2e/0x50 [ 2260.157129] kvm_wait+0xcd/0xe0 [ 2260.158555] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2260.160886] do_raw_spin_lock+0x177/0x190 [ 2260.162401] _raw_spin_lock+0x21/0x30 [ 2260.163946] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2260.165897] process_one_work+0x2c8/0x700 [ 2260.167643] worker_thread+0x296/0x6e0 [ 2260.168943] ? rescuer_thread+0x570/0x570 [ 2260.170501] kthread+0x1d1/0x200 [ 2260.172017] ? set_kthread_struct+0x70/0x70 [ 2260.173828] ret_from_fork+0x1f/0x30 [ 2268.052183] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 2268.060848] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2268.088573] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2268.106301] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2268.112560] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2268.117816] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2268.129392] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2268.133340] RAX: 0000000000001e9b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2268.137110] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 2268.141133] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 2268.149583] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 2268.154935] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 2268.162973] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 2268.170546] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2268.176298] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 2268.179869] Call Trace: [ 2268.180749] [ 2268.181832] ? show_regs.cold.9+0x22/0x2f [ 2268.183625] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2268.184973] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2268.187401] ? watchdog+0x40/0x40 [ 2268.189304] ? hrtimer_interrupt+0x128/0x2c0 [ 2268.191962] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2268.194756] ? apic_timer_interrupt+0xf/0x20 [ 2268.201202] [ 2268.202581] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2268.206450] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2268.208574] do_raw_spin_lock+0x177/0x190 [ 2268.212442] _raw_spin_lock+0x21/0x30 [ 2268.216526] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2268.220119] ? do_raw_spin_unlock+0x75/0x190 [ 2268.222660] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2268.226508] ? do_wait_intr+0xf0/0xf0 [ 2268.234171] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2268.239670] mdc_reint+0x2d/0x110 [mdc] [ 2268.243134] mdc_unlink+0x658/0x870 [mdc] [ 2268.246038] lmv_unlink+0x34d/0xbd0 [lmv] [ 2268.247615] ll_unlink+0x24d/0x7f0 [lustre] [ 2268.250300] vfs_unlink+0x202/0x330 [ 2268.261579] do_unlinkat+0x2fa/0x460 [ 2268.263497] __x64_sys_unlink+0x37/0x50 [ 2268.264591] do_syscall_64+0xc1/0x440 [ 2268.268248] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2268.271419] RIP: 0033:0x7f151bdee10b [ 2268.273280] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2268.290715] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2268.300502] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 2268.311466] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 2268.318542] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2268.323872] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2268.328597] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2272.746194] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 342s! [ 2272.753305] Showing busy workqueues and worker pools: [ 2272.768323] workqueue events: flags=0x0 [ 2272.775777] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2272.778284] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2272.788491] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2272.793716] pending: kfree_rcu_monitor [ 2272.797979] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2272.800810] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 2272.808532] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2272.814262] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2272.818527] workqueue events_long: flags=0x0 [ 2272.824257] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2272.827949] pending: lu_objects_flush [obdclass] [ 2272.830899] workqueue events_unbound: flags=0x2 [ 2272.834863] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2272.836803] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2272.855578] workqueue events_power_efficient: flags=0x80 [ 2272.858206] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2272.864870] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 2272.869873] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2272.876748] pending: neigh_periodic_work [ 2272.880334] workqueue mm_percpu_wq: flags=0x8 [ 2272.882612] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2272.887646] pending: vmstat_update, lru_add_drain_per_cpu [ 2272.895590] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2272.902903] pending: vmstat_update [ 2272.906523] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2272.909441] pending: lru_add_drain_per_cpu [ 2272.913159] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2272.920953] pending: lru_add_drain_per_cpu BAR(48) [ 2272.926346] workqueue writeback: flags=0x4a [ 2272.929965] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2272.940485] pending: wb_workfn [ 2272.945195] workqueue xprtiod: flags=0xa [ 2272.947342] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2272.950821] pending: xs_stream_data_receive_workfn, xprt_autoclose [ 2272.957694] workqueue ipv6_addrconf: flags=0x40008 [ 2272.962894] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2272.966666] pending: addrconf_verify_work [ 2272.971066] workqueue knbd0-recv: flags=0x1a [ 2272.973518] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2272.979217] in-flight: 658:recv_work [ 2272.981518] workqueue ptlrpc_pinger: flags=0x2 [ 2272.983943] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2272.992789] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2272.994888] pool 8: cpus=0-3 flags=0x4 nice=0 hung=342s workers=4 idle: 32984 37 8 [ 2273.006535] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2257s workers=2 idle: 661 [ 2276.055193] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 2276.057927] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 2276.060471] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2276.060685] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2276.060693] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2276.060696] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2276.060712] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2276.060716] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2276.060721] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 2276.060723] RDX: 0000000000006be8 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 2276.060725] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 2276.060727] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 2276.060730] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2276.060733] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 2276.060735] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2276.060737] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2276.060746] Call Trace: [ 2276.060751] [ 2276.060757] ? show_regs.cold.9+0x22/0x2f [ 2276.060767] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2276.060773] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2276.060778] ? watchdog+0x40/0x40 [ 2276.060782] ? hrtimer_interrupt+0x128/0x2c0 [ 2276.060786] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2276.060794] ? apic_timer_interrupt+0xf/0x20 [ 2276.060799] [ 2276.060801] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2276.060806] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2276.060810] do_raw_spin_lock+0x177/0x190 [ 2276.060815] _raw_spin_lock+0x21/0x30 [ 2276.060821] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2276.061169] obd_mod_free+0x46/0x140 [ptlrpc] [ 2276.061405] mdc_commit_open+0x91/0xb0 [mdc] [ 2276.061445] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2276.061792] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2276.062098] after_reply+0xe2c/0x1780 [ptlrpc] [ 2276.062355] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2276.062574] ? __next_timer_interrupt+0x160/0x160 [ 2276.062584] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2276.062903] ? do_wait_intr+0xf0/0xf0 [ 2276.062912] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2276.063215] kthread+0x1d1/0x200 [ 2276.063220] ? set_kthread_struct+0x70/0x70 [ 2276.063224] ret_from_fork+0x1f/0x30 [ 2276.333387] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2276.364038] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2276.376682] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2276.381580] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2276.384784] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2276.401249] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2276.411260] RAX: 0000000000000906 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2276.415609] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 2276.426724] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2276.432697] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 2276.436210] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 2276.441175] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 2276.452395] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2276.457558] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 2276.464535] Call Trace: [ 2276.467216] [ 2276.470328] ? show_regs.cold.9+0x22/0x2f [ 2276.472607] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2276.476743] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2276.479584] ? watchdog+0x40/0x40 [ 2276.480728] ? hrtimer_interrupt+0x128/0x2c0 [ 2276.484530] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2276.493795] ? apic_timer_interrupt+0xf/0x20 [ 2276.497867] [ 2276.501480] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2276.509939] do_raw_spin_lock+0x177/0x190 [ 2276.511306] _raw_spin_lock+0x21/0x30 [ 2276.521571] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2276.523707] ? do_raw_spin_unlock+0x75/0x190 [ 2276.540265] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2276.545345] ? do_wait_intr+0xf0/0xf0 [ 2276.546926] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2276.556666] mdc_close+0x53b/0xd70 [mdc] [ 2276.558104] lmv_close+0x18d/0x490 [lmv] [ 2276.568585] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2276.570779] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2276.574374] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2276.582723] ll_md_close+0x274/0x800 [lustre] [ 2276.591803] ll_file_release+0x128/0xc50 [lustre] [ 2276.603877] __fput+0x10d/0x440 [ 2276.606935] ____fput+0x12/0x20 [ 2276.609163] task_work_run+0xc6/0x110 [ 2276.611733] exit_to_usermode_loop+0x1dd/0x1f0 [ 2276.614804] do_syscall_64+0x3d6/0x440 [ 2276.616146] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2276.630137] RIP: 0033:0x7f151bee3cd8 [ 2276.631451] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2276.672673] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2276.683489] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 2276.687931] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 2276.708543] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2276.715606] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2276.735690] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2288.060294] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2288.071141] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2288.113375] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2288.130123] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2288.142532] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2288.151713] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2288.162429] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2288.174812] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2288.182035] RAX: 0000000000006833 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2288.184479] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 2288.190727] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 2288.201454] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 2288.206650] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 2288.215691] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 2288.221387] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2288.225540] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2288.230046] Call Trace: [ 2288.230943] [ 2288.231766] ? show_regs.cold.9+0x22/0x2f [ 2288.233464] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2288.235389] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2288.237414] ? watchdog+0x40/0x40 [ 2288.238387] ? hrtimer_interrupt+0x128/0x2c0 [ 2288.240571] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2288.242559] ? apic_timer_interrupt+0xf/0x20 [ 2288.245152] [ 2288.245799] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2288.247448] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2288.252161] do_raw_spin_lock+0x177/0x190 [ 2288.254066] _raw_spin_lock+0x21/0x30 [ 2288.255800] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2288.258590] process_one_work+0x2c8/0x700 [ 2288.261173] worker_thread+0x296/0x6e0 [ 2288.262781] ? rescuer_thread+0x570/0x570 [ 2288.265228] kthread+0x1d1/0x200 [ 2288.266410] ? set_kthread_struct+0x70/0x70 [ 2288.268338] ret_from_fork+0x1f/0x30 [ 2300.053966] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 2300.067323] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2300.107593] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2300.113947] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2300.121202] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2300.128232] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2300.144944] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2300.149065] RAX: 0000000000000904 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2300.152729] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 2300.160099] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 2300.167584] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 2300.171185] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 2300.178325] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 2300.183068] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2300.186414] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 2300.190058] Call Trace: [ 2300.191527] [ 2300.192669] ? show_regs.cold.9+0x22/0x2f [ 2300.195705] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2300.198297] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2300.200475] ? watchdog+0x40/0x40 [ 2300.202331] ? hrtimer_interrupt+0x128/0x2c0 [ 2300.204471] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2300.208790] ? apic_timer_interrupt+0xf/0x20 [ 2300.211687] [ 2300.213819] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2300.216578] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2300.220087] do_raw_spin_lock+0x177/0x190 [ 2300.222352] _raw_spin_lock+0x21/0x30 [ 2300.224972] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2300.234132] ? do_raw_spin_unlock+0x75/0x190 [ 2300.239377] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2300.248335] ? do_wait_intr+0xf0/0xf0 [ 2300.252415] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2300.256048] mdc_reint+0x2d/0x110 [mdc] [ 2300.258760] mdc_unlink+0x658/0x870 [mdc] [ 2300.261672] lmv_unlink+0x34d/0xbd0 [lmv] [ 2300.264986] ll_unlink+0x24d/0x7f0 [lustre] [ 2300.269052] vfs_unlink+0x202/0x330 [ 2300.271579] do_unlinkat+0x2fa/0x460 [ 2300.274683] __x64_sys_unlink+0x37/0x50 [ 2300.277070] do_syscall_64+0xc1/0x440 [ 2300.283089] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2300.287301] RIP: 0033:0x7f151bdee10b [ 2300.289162] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2300.306453] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2300.310961] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 2300.314465] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 2300.318118] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2300.322010] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2300.325531] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2303.466064] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 372s! [ 2303.469572] Showing busy workqueues and worker pools: [ 2303.471730] workqueue events: flags=0x0 [ 2303.473615] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2303.476902] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2303.480172] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2303.483181] pending: kfree_rcu_monitor [ 2303.485158] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2303.488312] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 2303.492863] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2303.496590] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2303.500410] workqueue events_long: flags=0x0 [ 2303.502486] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2303.505279] pending: lu_objects_flush [obdclass] [ 2303.508230] workqueue events_unbound: flags=0x2 [ 2303.511071] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2303.514106] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2303.519504] workqueue events_power_efficient: flags=0x80 [ 2303.522403] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2303.526108] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 2303.529870] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2303.533103] pending: neigh_periodic_work [ 2303.535563] workqueue mm_percpu_wq: flags=0x8 [ 2303.537798] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2303.541109] pending: vmstat_update, lru_add_drain_per_cpu [ 2303.544154] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2303.548222] pending: vmstat_update [ 2303.550108] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2303.553536] pending: lru_add_drain_per_cpu [ 2303.555593] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2303.558689] pending: lru_add_drain_per_cpu BAR(48) [ 2303.562189] workqueue writeback: flags=0x4a [ 2303.564936] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2303.568170] pending: wb_workfn [ 2303.570525] workqueue xprtiod: flags=0xa [ 2303.573127] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2303.576445] pending: xs_stream_data_receive_workfn, xprt_autoclose [ 2303.580455] workqueue ipv6_addrconf: flags=0x40008 [ 2303.582824] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2303.587266] pending: addrconf_verify_work [ 2303.589846] workqueue knbd0-recv: flags=0x1a [ 2303.594964] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2303.603421] in-flight: 658:recv_work [ 2303.606904] workqueue ptlrpc_pinger: flags=0x2 [ 2303.608895] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2303.613358] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2303.618782] pool 8: cpus=0-3 flags=0x4 nice=0 hung=373s workers=4 idle: 32984 37 8 [ 2303.623233] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2288s workers=2 idle: 661 [ 2304.057487] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 2304.058758] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 2304.060446] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2304.060523] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2304.060531] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2304.060534] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2304.060548] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2304.060552] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2304.060556] RAX: 0000000000003b84 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2304.060558] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 2304.060560] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2304.060562] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 2304.060564] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 2304.060567] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 2304.060570] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2304.060572] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 2304.060581] Call Trace: [ 2304.060587] [ 2304.060593] ? show_regs.cold.9+0x22/0x2f [ 2304.060602] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2304.060609] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2304.060613] ? watchdog+0x40/0x40 [ 2304.060618] ? hrtimer_interrupt+0x128/0x2c0 [ 2304.060622] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2304.060630] ? apic_timer_interrupt+0xf/0x20 [ 2304.060635] [ 2304.060638] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2304.060645] do_raw_spin_lock+0x177/0x190 [ 2304.060650] _raw_spin_lock+0x21/0x30 [ 2304.060655] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2304.070771] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash [ 2304.096150] ? do_raw_spin_unlock+0x75/0x190 [ 2304.096173] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2304.096521] ? do_wait_intr+0xf0/0xf0 [ 2304.096529] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2304.096908] mdc_close+0x53b/0xd70 [mdc] [ 2304.115086] dm_log dm_mod sha512_ssse3 sha512_generic [ 2304.115105] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2304.115116] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2304.115118] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2304.115133] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2304.115137] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2304.115142] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 2304.115144] RDX: 0000000000000a13 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 2304.115147] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 2304.115149] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 2304.115151] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2304.115155] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 2304.115157] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2304.115160] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2304.115167] Call Trace: [ 2304.115172] [ 2304.115178] ? show_regs.cold.9+0x22/0x2f [ 2304.115187] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2304.115194] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2304.115198] ? watchdog+0x40/0x40 [ 2304.115497] ? hrtimer_interrupt+0x128/0x2c0 [ 2304.115504] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2304.115512] ? apic_timer_interrupt+0xf/0x20 [ 2304.115517] [ 2304.115520] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2304.115525] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2304.115529] do_raw_spin_lock+0x177/0x190 [ 2304.115534] _raw_spin_lock+0x21/0x30 [ 2304.115542] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2304.116055] obd_mod_free+0x46/0x140 [ptlrpc] [ 2304.126399] lmv_close+0x18d/0x490 [lmv] [ 2304.126450] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2304.127334] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2304.127445] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2304.127532] ll_md_close+0x274/0x800 [lustre] [ 2304.127615] ll_file_release+0x128/0xc50 [lustre] [ 2304.128581] __fput+0x10d/0x440 [ 2304.128592] ____fput+0x12/0x20 [ 2304.128598] task_work_run+0xc6/0x110 [ 2304.128606] exit_to_usermode_loop+0x1dd/0x1f0 [ 2304.128612] do_syscall_64+0x3d6/0x440 [ 2304.128617] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2304.128626] RIP: 0033:0x7f151bee3cd8 [ 2304.128635] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2304.128639] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2304.128644] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 2304.128647] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 2304.128649] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2304.128651] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2304.129369] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2304.714504] mdc_commit_open+0x91/0xb0 [mdc] [ 2304.717556] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2304.722011] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2304.729542] after_reply+0xe2c/0x1780 [ptlrpc] [ 2304.734926] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2304.742101] ? __next_timer_interrupt+0x160/0x160 [ 2304.746343] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2304.750110] ? do_wait_intr+0xf0/0xf0 [ 2304.756913] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2304.761329] kthread+0x1d1/0x200 [ 2304.762520] ? set_kthread_struct+0x70/0x70 [ 2304.770880] ret_from_fork+0x1f/0x30 [ 2316.062420] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 2316.074219] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2316.142875] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2316.160826] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2316.179099] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2316.186421] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2316.190701] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2316.207100] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2316.214118] RAX: 0000000000003152 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2316.221738] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 2316.231026] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 2316.239343] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 2316.251854] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 2316.256988] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 2316.263274] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2316.274426] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2316.283843] Call Trace: [ 2316.285400] [ 2316.288263] ? show_regs.cold.9+0x22/0x2f [ 2316.292798] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2316.295753] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2316.300590] ? watchdog+0x40/0x40 [ 2316.302451] ? hrtimer_interrupt+0x128/0x2c0 [ 2316.308812] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2316.310529] ? apic_timer_interrupt+0xf/0x20 [ 2316.316017] [ 2316.318068] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2316.325322] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2316.327314] do_raw_spin_lock+0x177/0x190 [ 2316.340684] _raw_spin_lock+0x21/0x30 [ 2316.345161] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2316.352374] process_one_work+0x2c8/0x700 [ 2316.353655] worker_thread+0x296/0x6e0 [ 2316.362880] ? rescuer_thread+0x570/0x570 [ 2316.368893] kthread+0x1d1/0x200 [ 2316.374353] ? set_kthread_struct+0x70/0x70 [ 2316.378837] ret_from_fork+0x1f/0x30 [ 2328.052092] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 2328.054499] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2328.076225] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2328.079604] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2328.082048] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2328.083212] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2328.090801] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2328.093653] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2328.096592] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 2328.099433] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 2328.103129] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 2328.106142] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 2328.110007] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 2328.113542] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2328.115963] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 2328.118885] Call Trace: [ 2328.119810] [ 2328.120608] ? show_regs.cold.9+0x22/0x2f [ 2328.122091] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2328.123784] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2328.125485] ? watchdog+0x40/0x40 [ 2328.126739] ? hrtimer_interrupt+0x128/0x2c0 [ 2328.128389] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2328.130410] ? apic_timer_interrupt+0xf/0x20 [ 2328.132069] [ 2328.132741] ? native_safe_halt+0x2e/0x50 [ 2328.134654] kvm_wait+0xcd/0xe0 [ 2328.136171] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2328.137816] do_raw_spin_lock+0x177/0x190 [ 2328.139429] _raw_spin_lock+0x21/0x30 [ 2328.140590] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2328.142728] ? do_raw_spin_unlock+0x75/0x190 [ 2328.144235] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2328.146303] ? do_wait_intr+0xf0/0xf0 [ 2328.147571] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2328.149708] mdc_reint+0x2d/0x110 [mdc] [ 2328.151277] mdc_unlink+0x658/0x870 [mdc] [ 2328.152774] lmv_unlink+0x34d/0xbd0 [lmv] [ 2328.154155] ll_unlink+0x24d/0x7f0 [lustre] [ 2328.155809] vfs_unlink+0x202/0x330 [ 2328.157306] do_unlinkat+0x2fa/0x460 [ 2328.158469] __x64_sys_unlink+0x37/0x50 [ 2328.160106] do_syscall_64+0xc1/0x440 [ 2328.161680] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2328.163885] RIP: 0033:0x7f151bdee10b [ 2328.165164] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2328.172196] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2328.175147] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 2328.178030] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 2328.180600] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2328.185386] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2328.191130] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2332.055424] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 2332.058561] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 2332.059208] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2332.059301] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2332.059311] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2332.059314] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2332.059329] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2332.059338] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2332.059342] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 2332.059344] RDX: 000000000000486a RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 2332.059346] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 2332.059348] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 2332.059350] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2332.059356] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 2332.059362] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2332.059364] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2332.059371] Call Trace: [ 2332.059376] [ 2332.059383] ? show_regs.cold.9+0x22/0x2f [ 2332.059397] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2332.059403] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2332.059407] ? watchdog+0x40/0x40 [ 2332.059411] ? hrtimer_interrupt+0x128/0x2c0 [ 2332.059415] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2332.059423] ? apic_timer_interrupt+0xf/0x20 [ 2332.059428] [ 2332.059430] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2332.059435] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2332.059439] do_raw_spin_lock+0x177/0x190 [ 2332.059444] _raw_spin_lock+0x21/0x30 [ 2332.059450] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2332.059851] obd_mod_free+0x46/0x140 [ptlrpc] [ 2332.060074] mdc_commit_open+0x91/0xb0 [mdc] [ 2332.060112] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2332.060529] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2332.060854] after_reply+0xe2c/0x1780 [ptlrpc] [ 2332.061065] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2332.065020] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2332.065108] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2332.065115] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2332.065118] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2332.065132] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2332.065136] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2332.065140] RAX: 0000000000007db0 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2332.065143] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 2332.065145] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2332.065147] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 2332.065149] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 2332.065158] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 2332.065161] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2332.065163] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 2332.065171] Call Trace: [ 2332.065176] [ 2332.065182] ? show_regs.cold.9+0x22/0x2f [ 2332.065199] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2332.065205] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2332.065213] ? watchdog+0x40/0x40 [ 2332.065220] ? hrtimer_interrupt+0x128/0x2c0 [ 2332.065224] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2332.065232] ? apic_timer_interrupt+0xf/0x20 [ 2332.065236] [ 2332.065238] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2332.065248] do_raw_spin_lock+0x177/0x190 [ 2332.065253] _raw_spin_lock+0x21/0x30 [ 2332.065259] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2332.065588] ? do_raw_spin_unlock+0x75/0x190 [ 2332.065594] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2332.065813] ? do_wait_intr+0xf0/0xf0 [ 2332.065822] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2332.066032] mdc_close+0x53b/0xd70 [mdc] [ 2332.066077] lmv_close+0x18d/0x490 [lmv] [ 2332.066105] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2332.066294] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2332.066431] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2332.066511] ll_md_close+0x274/0x800 [lustre] [ 2332.066618] ll_file_release+0x128/0xc50 [lustre] [ 2332.066907] __fput+0x10d/0x440 [ 2332.066914] ____fput+0x12/0x20 [ 2332.066919] task_work_run+0xc6/0x110 [ 2332.066926] exit_to_usermode_loop+0x1dd/0x1f0 [ 2332.066932] do_syscall_64+0x3d6/0x440 [ 2332.066937] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2332.066944] RIP: 0033:0x7f151bee3cd8 [ 2332.066957] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2332.066960] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2332.066965] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 2332.066968] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 2332.066974] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2332.066975] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2332.066977] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2332.365609] ? __next_timer_interrupt+0x160/0x160 [ 2332.368892] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2332.370949] ? do_wait_intr+0xf0/0xf0 [ 2332.373831] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2332.378300] kthread+0x1d1/0x200 [ 2332.379592] ? set_kthread_struct+0x70/0x70 [ 2332.380661] ret_from_fork+0x1f/0x30 [ 2333.665323] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 403s! [ 2333.670964] Showing busy workqueues and worker pools: [ 2333.673504] workqueue events: flags=0x0 [ 2333.677249] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2333.679987] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2333.685858] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2333.689532] pending: kfree_rcu_monitor [ 2333.691496] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2333.694512] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 2333.703704] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2333.707338] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2333.713054] workqueue events_long: flags=0x0 [ 2333.715244] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2333.718331] pending: lu_objects_flush [obdclass] [ 2333.723043] workqueue events_unbound: flags=0x2 [ 2333.727616] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2333.730207] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2333.739717] workqueue events_power_efficient: flags=0x80 [ 2333.742355] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2333.745707] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 2333.750813] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2333.756040] pending: neigh_periodic_work [ 2333.757979] workqueue mm_percpu_wq: flags=0x8 [ 2333.761202] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2333.771192] pending: vmstat_update, lru_add_drain_per_cpu [ 2333.777288] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2333.780204] pending: vmstat_update [ 2333.783688] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2333.790079] pending: lru_add_drain_per_cpu [ 2333.795261] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2333.798977] pending: lru_add_drain_per_cpu BAR(48) [ 2333.802254] workqueue writeback: flags=0x4a [ 2333.806018] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2333.809792] pending: wb_workfn [ 2333.811639] workqueue xprtiod: flags=0xa [ 2333.814345] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2333.820940] pending: xs_stream_data_receive_workfn, xprt_autoclose [ 2333.831271] workqueue ipv6_addrconf: flags=0x40008 [ 2333.837553] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2333.840090] pending: addrconf_verify_work [ 2333.845037] workqueue knbd0-recv: flags=0x1a [ 2333.848219] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2333.853306] in-flight: 658:recv_work [ 2333.857010] workqueue ptlrpc_pinger: flags=0x2 [ 2333.860327] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2333.867300] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2333.871874] pool 8: cpus=0-3 flags=0x4 nice=0 hung=403s workers=4 idle: 32984 37 8 [ 2333.883013] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2318s workers=2 idle: 661 [ 2344.056027] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 2344.060080] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2344.080501] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2344.085473] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2344.098069] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2344.114115] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2344.115909] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2344.135282] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2344.138896] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2344.142031] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 2344.145249] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 2344.148387] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 2344.151514] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 2344.154823] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 2344.158204] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2344.160797] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2344.164055] Call Trace: [ 2344.165150] [ 2344.166078] ? show_regs.cold.9+0x22/0x2f [ 2344.167900] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2344.169982] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2344.171803] ? watchdog+0x40/0x40 [ 2344.172650] ? hrtimer_interrupt+0x128/0x2c0 [ 2344.174936] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2344.177017] ? apic_timer_interrupt+0xf/0x20 [ 2344.178962] [ 2344.179924] ? native_safe_halt+0x2e/0x50 [ 2344.181591] kvm_wait+0xcd/0xe0 [ 2344.182999] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2344.185201] do_raw_spin_lock+0x177/0x190 [ 2344.186944] _raw_spin_lock+0x21/0x30 [ 2344.188802] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2344.190594] process_one_work+0x2c8/0x700 [ 2344.192103] worker_thread+0x296/0x6e0 [ 2344.194560] ? rescuer_thread+0x570/0x570 [ 2344.196394] kthread+0x1d1/0x200 [ 2344.197929] ? set_kthread_struct+0x70/0x70 [ 2344.199906] ret_from_fork+0x1f/0x30 [ 2344.655150] rcu: INFO: rcu_sched self-detected stall on CPU [ 2344.657628] rcu: 1-...!: (254069 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 2344.660706] (t=420008 jiffies g=249657 q=2584) [ 2344.662258] rcu: rcu_sched kthread starved for 414960 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 2344.667333] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 2344.672200] rcu: RCU grace-period kthread stack dump: [ 2344.674834] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 2344.678549] Call Trace: [ 2344.679421] __schedule+0x351/0xcb0 [ 2344.681235] schedule+0xc0/0x180 [ 2344.682383] schedule_timeout+0xb4/0x190 [ 2344.684193] ? do_raw_spin_unlock+0x75/0x190 [ 2344.685707] ? __next_timer_interrupt+0x160/0x160 [ 2344.687801] ? prepare_to_swait_event+0xf3/0x140 [ 2344.689634] rcu_gp_kthread+0x8dd/0x15b0 [ 2344.691428] ? force_qs_rnp+0x2c0/0x2c0 [ 2344.692962] kthread+0x1d1/0x200 [ 2344.694442] ? set_kthread_struct+0x70/0x70 [ 2344.696665] ret_from_fork+0x1f/0x30 [ 2344.698160] rcu: Stack dump where RCU GP kthread last ran: [ 2344.700271] NMI backtrace for cpu 1 [ 2344.701799] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2344.706315] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2344.710363] Call Trace: [ 2344.711503] [ 2344.712247] ? dump_stack+0xbb/0x10e [ 2344.715117] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2344.717258] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2344.719277] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2344.721390] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2344.724127] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2344.726185] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 2344.728612] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 2344.732151] ? update_process_times+0x6d/0xc0 [ 2344.733771] ? tick_sched_handle.isra.18+0x27/0x90 [ 2344.735657] ? tick_sched_timer+0x53/0xb0 [ 2344.737167] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2344.739083] ? tick_sched_handle.isra.18+0x90/0x90 [ 2344.740977] ? hrtimer_interrupt+0x128/0x2c0 [ 2344.742766] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2344.744693] ? apic_timer_interrupt+0xf/0x20 [ 2344.746530] [ 2344.747276] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2344.749761] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2344.751641] ? do_raw_spin_lock+0x177/0x190 [ 2344.753459] ? _raw_spin_lock+0x21/0x30 [ 2344.755003] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2344.757416] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2344.759723] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2344.761612] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2344.764094] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2344.767014] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2344.769362] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2344.771672] ? __next_timer_interrupt+0x160/0x160 [ 2344.773962] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2344.775884] ? do_wait_intr+0xf0/0xf0 [ 2344.777804] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2344.780030] ? kthread+0x1d1/0x200 [ 2344.781838] ? set_kthread_struct+0x70/0x70 [ 2344.783784] ? ret_from_fork+0x1f/0x30 [ 2344.785210] NMI backtrace for cpu 1 [ 2344.786381] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2344.791143] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2344.795066] Call Trace: [ 2344.796096] [ 2344.797347] ? dump_stack+0xbb/0x10e [ 2344.798630] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2344.800438] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2344.802389] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2344.804083] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2344.805499] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2344.807398] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 2344.808961] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 2344.810805] ? update_process_times+0x6d/0xc0 [ 2344.812557] ? tick_sched_handle.isra.18+0x27/0x90 [ 2344.814520] ? tick_sched_timer+0x53/0xb0 [ 2344.815987] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2344.817896] ? tick_sched_handle.isra.18+0x90/0x90 [ 2344.819822] ? hrtimer_interrupt+0x128/0x2c0 [ 2344.821918] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2344.823993] ? apic_timer_interrupt+0xf/0x20 [ 2344.826183] [ 2344.826967] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2344.828854] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2344.831385] ? do_raw_spin_lock+0x177/0x190 [ 2344.833175] ? _raw_spin_lock+0x21/0x30 [ 2344.834727] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2344.837280] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2344.839561] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2344.841971] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2344.844915] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2344.848316] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2344.850839] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2344.853361] ? __next_timer_interrupt+0x160/0x160 [ 2344.855530] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2344.857547] ? do_wait_intr+0xf0/0xf0 [ 2344.858664] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2344.861134] ? kthread+0x1d1/0x200 [ 2344.862862] ? set_kthread_struct+0x70/0x70 [ 2344.864395] ? ret_from_fork+0x1f/0x30 [ 2360.054235] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 2360.057007] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2360.058652] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 2360.108118] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2360.108137] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2360.108141] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2360.108162] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2360.108417] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2360.108425] RAX: 0000000000000c0b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2360.108427] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 2360.108429] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 2360.108431] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 2360.108433] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 2360.108436] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 2360.108439] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2360.108442] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 2360.108451] Call Trace: [ 2360.108458] [ 2360.108467] ? show_regs.cold.9+0x22/0x2f [ 2360.108479] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2360.108487] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2360.108492] ? watchdog+0x40/0x40 [ 2360.108496] ? hrtimer_interrupt+0x128/0x2c0 [ 2360.108500] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2360.108510] ? apic_timer_interrupt+0xf/0x20 [ 2360.108514] [ 2360.108517] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2360.108521] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2360.108526] do_raw_spin_lock+0x177/0x190 [ 2360.108531] _raw_spin_lock+0x21/0x30 [ 2360.108541] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2360.109616] ? do_raw_spin_unlock+0x75/0x190 [ 2360.109624] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2360.110135] ? do_wait_intr+0xf0/0xf0 [ 2360.110144] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2360.110438] mdc_reint+0x2d/0x110 [mdc] [ 2360.110483] mdc_unlink+0x658/0x870 [mdc] [ 2360.110510] lmv_unlink+0x34d/0xbd0 [lmv] [ 2360.110550] ll_unlink+0x24d/0x7f0 [lustre] [ 2360.110850] vfs_unlink+0x202/0x330 [ 2360.110861] do_unlinkat+0x2fa/0x460 [ 2360.110866] __x64_sys_unlink+0x37/0x50 [ 2360.110869] do_syscall_64+0xc1/0x440 [ 2360.110880] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2360.110892] RIP: 0033:0x7f151bdee10b [ 2360.111148] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2360.111152] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2360.111159] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 2360.111161] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 2360.111163] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2360.111236] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2360.126368] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2360.126452] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2360.126460] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2360.126464] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2360.126479] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2360.126484] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2360.126489] RAX: 0000000000003f0b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2360.126491] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 2360.126494] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2360.126496] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 2360.126498] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 2360.126502] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 2360.126505] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2360.126508] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 2360.126517] Call Trace: [ 2360.126523] [ 2360.126528] ? show_regs.cold.9+0x22/0x2f [ 2360.126539] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2360.126546] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2360.126697] ? watchdog+0x40/0x40 [ 2360.126702] ? hrtimer_interrupt+0x128/0x2c0 [ 2360.126706] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2360.126714] ? apic_timer_interrupt+0xf/0x20 [ 2360.126719] [ 2360.126722] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2360.126729] do_raw_spin_lock+0x177/0x190 [ 2360.126733] _raw_spin_lock+0x21/0x30 [ 2360.126737] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2360.127835] ? do_raw_spin_unlock+0x75/0x190 [ 2360.127843] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2360.128096] ? do_wait_intr+0xf0/0xf0 [ 2360.128105] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2360.128398] mdc_close+0x53b/0xd70 [mdc] [ 2360.128445] lmv_close+0x18d/0x490 [lmv] [ 2360.128484] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2360.129851] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2360.146888] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2360.679484] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2360.684537] ll_md_close+0x274/0x800 [lustre] [ 2360.692150] ll_file_release+0x128/0xc50 [lustre] [ 2360.696529] __fput+0x10d/0x440 [ 2360.697727] ____fput+0x12/0x20 [ 2360.698949] task_work_run+0xc6/0x110 [ 2360.707465] exit_to_usermode_loop+0x1dd/0x1f0 [ 2360.711313] do_syscall_64+0x3d6/0x440 [ 2360.717589] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2360.724746] RIP: 0033:0x7f151bee3cd8 [ 2360.726027] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2360.746903] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2360.751361] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 2360.761362] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 2360.765725] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2360.770074] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2360.773873] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2364.385049] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 433s! [ 2364.390768] Showing busy workqueues and worker pools: [ 2364.398177] workqueue events: flags=0x0 [ 2364.405708] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2364.410637] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2364.418115] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2364.423977] pending: kfree_rcu_monitor [ 2364.427387] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2364.434610] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 2364.440833] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2364.448045] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2364.450711] workqueue events_long: flags=0x0 [ 2364.456245] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2364.460906] pending: lu_objects_flush [obdclass] [ 2364.465422] workqueue events_unbound: flags=0x2 [ 2364.468189] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2364.472840] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2364.484854] workqueue events_power_efficient: flags=0x80 [ 2364.491308] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2364.494751] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 2364.503376] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2364.505469] pending: neigh_periodic_work [ 2364.510455] workqueue mm_percpu_wq: flags=0x8 [ 2364.516479] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2364.521383] pending: vmstat_update, lru_add_drain_per_cpu [ 2364.526422] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2364.531079] pending: vmstat_update [ 2364.536900] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2364.543603] pending: lru_add_drain_per_cpu [ 2364.548251] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2364.555483] pending: lru_add_drain_per_cpu BAR(48) [ 2364.558833] workqueue writeback: flags=0x4a [ 2364.560672] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2364.570411] pending: wb_workfn [ 2364.572597] workqueue xprtiod: flags=0xa [ 2364.574945] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 2364.578421] pending: xs_stream_data_receive_workfn, xprt_autoclose [ 2364.586329] workqueue ipv6_addrconf: flags=0x40008 [ 2364.590375] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2364.598370] pending: addrconf_verify_work [ 2364.604618] workqueue knbd0-recv: flags=0x1a [ 2364.608318] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2364.612355] in-flight: 658:recv_work [ 2364.615206] workqueue ptlrpc_pinger: flags=0x2 [ 2364.618179] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2364.625799] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2364.633447] pool 8: cpus=0-3 flags=0x4 nice=0 hung=434s workers=4 idle: 32984 37 8 [ 2364.644780] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2349s workers=2 idle: 661 [ 2372.055020] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 2372.059418] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2372.060265] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 2372.086308] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2372.093572] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2372.093664] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2372.093670] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2372.093678] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2372.093951] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2372.093961] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2372.093965] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2372.093969] RAX: 00000000000058e1 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2372.093972] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 2372.093974] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 2372.093976] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 2372.093979] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 2372.093982] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 2372.093985] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2372.093987] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2372.093997] Call Trace: [ 2372.094002] [ 2372.094006] ? show_regs.cold.9+0x22/0x2f [ 2372.094016] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2372.094022] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2372.094026] ? watchdog+0x40/0x40 [ 2372.094030] ? hrtimer_interrupt+0x128/0x2c0 [ 2372.094034] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2372.094043] ? apic_timer_interrupt+0xf/0x20 [ 2372.094047] [ 2372.094049] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2372.094054] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2372.094060] do_raw_spin_lock+0x177/0x190 [ 2372.094064] _raw_spin_lock+0x21/0x30 [ 2372.094069] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2372.094297] process_one_work+0x2c8/0x700 [ 2372.094304] worker_thread+0x296/0x6e0 [ 2372.094308] ? rescuer_thread+0x570/0x570 [ 2372.094312] kthread+0x1d1/0x200 [ 2372.094316] ? set_kthread_struct+0x70/0x70 [ 2372.094320] ret_from_fork+0x1f/0x30 [ 2372.252931] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2372.257560] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2372.262545] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2372.279281] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2372.286790] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 2372.299175] RDX: 0000000000000520 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 2372.304916] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 2372.314360] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 2372.320917] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2372.325265] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 2372.329609] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2372.332780] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2372.342343] Call Trace: [ 2372.347248] [ 2372.348743] ? show_regs.cold.9+0x22/0x2f [ 2372.352511] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2372.356560] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2372.363910] ? watchdog+0x40/0x40 [ 2372.367444] ? hrtimer_interrupt+0x128/0x2c0 [ 2372.372602] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2372.380558] ? apic_timer_interrupt+0xf/0x20 [ 2372.384913] [ 2372.386576] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2372.391821] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2372.395544] do_raw_spin_lock+0x177/0x190 [ 2372.399099] _raw_spin_lock+0x21/0x30 [ 2372.403010] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2372.408482] obd_mod_free+0x46/0x140 [ptlrpc] [ 2372.412103] mdc_commit_open+0x91/0xb0 [mdc] [ 2372.416510] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2372.423993] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2372.431011] after_reply+0xe2c/0x1780 [ptlrpc] [ 2372.438059] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2372.448376] ? __next_timer_interrupt+0x160/0x160 [ 2372.452007] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2372.460503] ? do_wait_intr+0xf0/0xf0 [ 2372.466147] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2372.473192] kthread+0x1d1/0x200 [ 2372.477524] ? set_kthread_struct+0x70/0x70 [ 2372.488256] ret_from_fork+0x1f/0x30 [ 2388.065503] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 2388.073205] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2388.153368] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2388.170391] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2388.186339] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2388.199650] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2388.224594] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2388.231717] RAX: 0000000000003c12 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2388.252481] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 2388.264861] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2388.273499] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 2388.277125] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 2388.280163] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 2388.284997] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2388.292546] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 2388.295654] Call Trace: [ 2388.298307] [ 2388.305096] ? show_regs.cold.9+0x22/0x2f [ 2388.313894] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2388.325741] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2388.327349] ? watchdog+0x40/0x40 [ 2388.328154] ? hrtimer_interrupt+0x128/0x2c0 [ 2388.336564] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2388.344602] ? apic_timer_interrupt+0xf/0x20 [ 2388.351357] [ 2388.356026] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2388.365814] do_raw_spin_lock+0x177/0x190 [ 2388.368807] _raw_spin_lock+0x21/0x30 [ 2388.381443] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2388.392911] ? do_raw_spin_unlock+0x75/0x190 [ 2388.394004] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2388.408257] ? do_wait_intr+0xf0/0xf0 [ 2388.421244] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2388.428342] mdc_close+0x53b/0xd70 [mdc] [ 2388.436402] lmv_close+0x18d/0x490 [lmv] [ 2388.447290] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2388.456436] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2388.468020] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2388.478890] ll_md_close+0x274/0x800 [lustre] [ 2388.492073] ll_file_release+0x128/0xc50 [lustre] [ 2388.500865] __fput+0x10d/0x440 [ 2388.506663] ____fput+0x12/0x20 [ 2388.511129] task_work_run+0xc6/0x110 [ 2388.518306] exit_to_usermode_loop+0x1dd/0x1f0 [ 2388.537445] do_syscall_64+0x3d6/0x440 [ 2388.543434] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2388.558285] RIP: 0033:0x7f151bee3cd8 [ 2388.570244] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2388.598693] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2388.604708] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 2388.608570] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 2388.615545] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2388.622706] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2388.625841] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2392.063793] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 2392.087718] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2392.158764] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2392.193347] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2392.200771] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2392.209703] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2392.246904] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2392.256134] RAX: 0000000000005708 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2392.261642] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 2392.271528] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 2392.283192] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 2392.285510] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 2392.311785] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 2392.329902] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2392.344420] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 2392.362601] Call Trace: [ 2392.363544] [ 2392.364239] ? show_regs.cold.9+0x22/0x2f [ 2392.368762] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2392.374032] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2392.378806] ? watchdog+0x40/0x40 [ 2392.380919] ? hrtimer_interrupt+0x128/0x2c0 [ 2392.383338] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2392.386631] ? apic_timer_interrupt+0xf/0x20 [ 2392.389632] [ 2392.391019] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2392.394572] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2392.397742] do_raw_spin_lock+0x177/0x190 [ 2392.400432] _raw_spin_lock+0x21/0x30 [ 2392.402457] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2392.406588] ? do_raw_spin_unlock+0x75/0x190 [ 2392.408453] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2392.412043] ? do_wait_intr+0xf0/0xf0 [ 2392.414421] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2392.417459] mdc_reint+0x2d/0x110 [mdc] [ 2392.419384] mdc_unlink+0x658/0x870 [mdc] [ 2392.423462] lmv_unlink+0x34d/0xbd0 [lmv] [ 2392.428448] ll_unlink+0x24d/0x7f0 [lustre] [ 2392.433429] vfs_unlink+0x202/0x330 [ 2392.436053] do_unlinkat+0x2fa/0x460 [ 2392.438109] __x64_sys_unlink+0x37/0x50 [ 2392.442366] do_syscall_64+0xc1/0x440 [ 2392.445532] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2392.449821] RIP: 0033:0x7f151bdee10b [ 2392.454382] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2392.471078] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2392.482812] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 2392.488643] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 2392.491444] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2392.501758] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2392.508098] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2395.107361] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 464s! [ 2395.120654] Showing busy workqueues and worker pools: [ 2395.124595] workqueue events: flags=0x0 [ 2395.127324] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2395.129729] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2395.143482] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2395.147402] pending: kfree_rcu_monitor [ 2395.150859] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2395.157759] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 2395.163599] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2395.178579] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2395.189688] workqueue events_long: flags=0x0 [ 2395.197025] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2395.199470] pending: lu_objects_flush [obdclass] [ 2395.216038] workqueue events_unbound: flags=0x2 [ 2395.218391] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2395.225486] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2395.244626] workqueue events_power_efficient: flags=0x80 [ 2395.249517] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2395.256766] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 2395.261304] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2395.266556] pending: neigh_periodic_work [ 2395.268414] workqueue mm_percpu_wq: flags=0x8 [ 2395.274805] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2395.283112] pending: vmstat_update, lru_add_drain_per_cpu [ 2395.287490] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2395.292747] pending: vmstat_update [ 2395.306629] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2395.318587] pending: lru_add_drain_per_cpu [ 2395.320393] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2395.326345] pending: lru_add_drain_per_cpu BAR(48) [ 2395.331108] workqueue writeback: flags=0x4a [ 2395.334512] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2395.339047] pending: wb_workfn [ 2395.340612] workqueue xprtiod: flags=0xa [ 2395.342699] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2395.350545] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 2395.356255] workqueue ipv6_addrconf: flags=0x40008 [ 2395.358556] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2395.362510] pending: addrconf_verify_work [ 2395.364528] workqueue knbd0-recv: flags=0x1a [ 2395.366408] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2395.369240] in-flight: 658:recv_work [ 2395.371239] workqueue ptlrpc_pinger: flags=0x2 [ 2395.375694] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2395.379247] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2395.385137] pool 8: cpus=0-3 flags=0x4 nice=0 hung=464s workers=4 idle: 32984 37 8 [ 2395.390243] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2380s workers=2 idle: 661 [ 2400.055971] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 2400.056061] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 2400.059232] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2400.059311] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2400.059318] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2400.059321] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2400.059393] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2400.059397] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2400.059402] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 2400.059404] RDX: 00000000000051a0 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 2400.059406] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 2400.059408] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 2400.059411] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2400.059414] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 2400.059417] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2400.059419] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2400.059426] Call Trace: [ 2400.059431] [ 2400.059436] ? show_regs.cold.9+0x22/0x2f [ 2400.059444] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2400.059450] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2400.059454] ? watchdog+0x40/0x40 [ 2400.059458] ? hrtimer_interrupt+0x128/0x2c0 [ 2400.059462] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2400.059470] ? apic_timer_interrupt+0xf/0x20 [ 2400.059474] [ 2400.059477] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2400.059481] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2400.059485] do_raw_spin_lock+0x177/0x190 [ 2400.059489] _raw_spin_lock+0x21/0x30 [ 2400.059494] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2400.059958] obd_mod_free+0x46/0x140 [ptlrpc] [ 2400.060161] mdc_commit_open+0x91/0xb0 [mdc] [ 2400.060200] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2400.062271] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2400.062334] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2400.062339] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2400.062344] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2400.083528] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2400.083903] after_reply+0xe2c/0x1780 [ptlrpc] [ 2400.084117] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2400.084309] ? __next_timer_interrupt+0x160/0x160 [ 2400.084319] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2400.084519] ? do_wait_intr+0xf0/0xf0 [ 2400.084528] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2400.088087] [ 2400.088095] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2400.088111] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2400.088115] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2400.088122] RAX: 0000000000003e0b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2400.088125] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 2400.088127] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 2400.088129] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 2400.088131] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 2400.088134] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 2400.088136] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2400.088138] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2400.088145] Call Trace: [ 2400.088150] [ 2400.088158] ? show_regs.cold.9+0x22/0x2f [ 2400.088167] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2400.088172] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2400.088176] ? watchdog+0x40/0x40 [ 2400.088180] ? hrtimer_interrupt+0x128/0x2c0 [ 2400.088183] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2400.088190] ? apic_timer_interrupt+0xf/0x20 [ 2400.088194] [ 2400.088196] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2400.088200] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2400.088206] do_raw_spin_lock+0x177/0x190 [ 2400.088212] _raw_spin_lock+0x21/0x30 [ 2400.088217] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2400.088502] process_one_work+0x2c8/0x700 [ 2400.088508] worker_thread+0x296/0x6e0 [ 2400.088512] ? rescuer_thread+0x570/0x570 [ 2400.088515] kthread+0x1d1/0x200 [ 2400.088519] ? set_kthread_struct+0x70/0x70 [ 2400.088522] ret_from_fork+0x1f/0x30 [ 2400.248350] kthread+0x1d1/0x200 [ 2400.249350] ? set_kthread_struct+0x70/0x70 [ 2400.250575] ret_from_fork+0x1f/0x30 [ 2416.058844] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 2416.062081] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2416.086900] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2416.091914] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2416.095495] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2416.097751] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2416.105906] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2416.108770] RAX: 0000000000000c84 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2416.111739] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 2416.114775] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2416.117851] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 2416.120950] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 2416.124449] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 2416.128323] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2416.130864] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 2416.133671] Call Trace: [ 2416.134815] [ 2416.135666] ? show_regs.cold.9+0x22/0x2f [ 2416.137495] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2416.139713] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2416.141506] ? watchdog+0x40/0x40 [ 2416.142753] ? hrtimer_interrupt+0x128/0x2c0 [ 2416.145261] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2416.146900] ? apic_timer_interrupt+0xf/0x20 [ 2416.148714] [ 2416.149944] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2416.152359] do_raw_spin_lock+0x177/0x190 [ 2416.154176] _raw_spin_lock+0x21/0x30 [ 2416.155857] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2416.158277] ? do_raw_spin_unlock+0x75/0x190 [ 2416.161900] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2416.166440] ? do_wait_intr+0xf0/0xf0 [ 2416.169361] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2416.174306] mdc_close+0x53b/0xd70 [mdc] [ 2416.177407] lmv_close+0x18d/0x490 [lmv] [ 2416.180481] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2416.184783] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2416.187286] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2416.189210] ll_md_close+0x274/0x800 [lustre] [ 2416.191095] ll_file_release+0x128/0xc50 [lustre] [ 2416.193327] __fput+0x10d/0x440 [ 2416.194897] ____fput+0x12/0x20 [ 2416.196250] task_work_run+0xc6/0x110 [ 2416.208260] exit_to_usermode_loop+0x1dd/0x1f0 [ 2416.210930] do_syscall_64+0x3d6/0x440 [ 2416.214910] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2416.219641] RIP: 0033:0x7f151bee3cd8 [ 2416.222691] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2416.232192] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2416.235570] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 2416.244291] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 2416.248873] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2416.252547] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2416.256327] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2420.056221] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 2420.060377] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2420.080220] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2420.085070] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2420.089072] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2420.091270] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2420.098336] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2420.100969] RAX: 0000000000005c16 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2420.102803] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 2420.105316] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 2420.107743] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 2420.109490] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 2420.111108] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 2420.113243] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2420.114537] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 2420.116520] Call Trace: [ 2420.117100] [ 2420.117533] ? show_regs.cold.9+0x22/0x2f [ 2420.118425] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2420.119483] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2420.120787] ? watchdog+0x40/0x40 [ 2420.121534] ? hrtimer_interrupt+0x128/0x2c0 [ 2420.122521] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2420.123690] ? apic_timer_interrupt+0xf/0x20 [ 2420.124573] [ 2420.125100] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2420.126373] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2420.127449] do_raw_spin_lock+0x177/0x190 [ 2420.128308] _raw_spin_lock+0x21/0x30 [ 2420.129159] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2420.130483] ? do_raw_spin_unlock+0x75/0x190 [ 2420.131435] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2420.132704] ? do_wait_intr+0xf0/0xf0 [ 2420.133756] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2420.135027] mdc_reint+0x2d/0x110 [mdc] [ 2420.135942] mdc_unlink+0x658/0x870 [mdc] [ 2420.136830] lmv_unlink+0x34d/0xbd0 [lmv] [ 2420.137607] ll_unlink+0x24d/0x7f0 [lustre] [ 2420.138556] vfs_unlink+0x202/0x330 [ 2420.139304] do_unlinkat+0x2fa/0x460 [ 2420.140143] __x64_sys_unlink+0x37/0x50 [ 2420.141028] do_syscall_64+0xc1/0x440 [ 2420.141868] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2420.143398] RIP: 0033:0x7f151bdee10b [ 2420.144388] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2420.148820] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2420.150397] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 2420.152077] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 2420.153733] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2420.155214] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2420.156754] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2425.824961] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 495s! [ 2425.829759] Showing busy workqueues and worker pools: [ 2425.832756] workqueue events: flags=0x0 [ 2425.835094] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2425.840336] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2425.845266] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2425.850004] pending: kfree_rcu_monitor [ 2425.852613] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2425.857033] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 2425.863322] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2425.867727] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2425.871397] workqueue events_long: flags=0x0 [ 2425.873108] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2425.876626] pending: lu_objects_flush [obdclass] [ 2425.879389] workqueue events_unbound: flags=0x2 [ 2425.881951] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2425.884658] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2425.890511] workqueue events_power_efficient: flags=0x80 [ 2425.893255] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2425.896680] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 2425.902351] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2425.913396] pending: neigh_periodic_work [ 2425.918729] workqueue mm_percpu_wq: flags=0x8 [ 2425.920906] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2425.924249] pending: vmstat_update, lru_add_drain_per_cpu [ 2425.927641] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2425.930441] pending: vmstat_update [ 2425.932053] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2425.934721] pending: lru_add_drain_per_cpu [ 2425.936449] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2425.938972] pending: lru_add_drain_per_cpu BAR(48) [ 2425.940964] workqueue writeback: flags=0x4a [ 2425.942170] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2425.944272] pending: wb_workfn [ 2425.945857] workqueue xprtiod: flags=0xa [ 2425.947735] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2425.954215] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 2425.958110] workqueue ipv6_addrconf: flags=0x40008 [ 2425.962216] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2425.965410] pending: addrconf_verify_work [ 2425.969752] workqueue knbd0-recv: flags=0x1a [ 2425.971203] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2425.974195] in-flight: 658:recv_work [ 2425.979675] workqueue ptlrpc_pinger: flags=0x2 [ 2425.983034] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2425.989464] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2425.993548] pool 8: cpus=0-3 flags=0x4 nice=0 hung=495s workers=4 idle: 32984 37 8 [ 2426.003347] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2410s workers=2 idle: 661 [ 2428.055019] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 2428.058826] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2428.059341] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 2428.086906] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2428.086919] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2428.086922] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2428.086936] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2428.086940] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2428.086945] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 2428.086947] RDX: 0000000000007737 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 2428.086949] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 2428.086955] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 2428.086957] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2428.086960] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 2428.086971] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2428.086974] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2428.086982] Call Trace: [ 2428.086989] [ 2428.086995] ? show_regs.cold.9+0x22/0x2f [ 2428.087005] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2428.087022] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2428.087026] ? watchdog+0x40/0x40 [ 2428.087031] ? hrtimer_interrupt+0x128/0x2c0 [ 2428.087035] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2428.087107] ? apic_timer_interrupt+0xf/0x20 [ 2428.087112] [ 2428.087115] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2428.087120] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2428.087125] do_raw_spin_lock+0x177/0x190 [ 2428.087129] _raw_spin_lock+0x21/0x30 [ 2428.087137] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2428.087403] obd_mod_free+0x46/0x140 [ptlrpc] [ 2428.087726] mdc_commit_open+0x91/0xb0 [mdc] [ 2428.087768] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2428.087972] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2428.088300] after_reply+0xe2c/0x1780 [ptlrpc] [ 2428.088496] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2428.088840] ? __next_timer_interrupt+0x160/0x160 [ 2428.088850] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2428.089290] ? do_wait_intr+0xf0/0xf0 [ 2428.089300] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2428.089503] kthread+0x1d1/0x200 [ 2428.089888] ? set_kthread_struct+0x70/0x70 [ 2428.089892] ret_from_fork+0x1f/0x30 [ 2428.358105] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2428.385141] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2428.390229] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2428.395401] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2428.398923] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2428.414412] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2428.430817] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2428.434732] RAX: 0000000000007e88 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2428.446384] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 2428.457884] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 2428.467441] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 2428.473354] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 2428.479482] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 2428.488760] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2428.494910] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2428.501200] Call Trace: [ 2428.502080] [ 2428.504288] ? show_regs.cold.9+0x22/0x2f [ 2428.506044] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2428.509269] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2428.511453] ? watchdog+0x40/0x40 [ 2428.514427] ? hrtimer_interrupt+0x128/0x2c0 [ 2428.516389] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2428.518780] ? apic_timer_interrupt+0xf/0x20 [ 2428.524118] [ 2428.524792] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2428.526389] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2428.533499] do_raw_spin_lock+0x177/0x190 [ 2428.536059] _raw_spin_lock+0x21/0x30 [ 2428.539422] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2428.547967] process_one_work+0x2c8/0x700 [ 2428.549295] worker_thread+0x296/0x6e0 [ 2428.550492] ? rescuer_thread+0x570/0x570 [ 2428.562794] kthread+0x1d1/0x200 [ 2428.564002] ? set_kthread_struct+0x70/0x70 [ 2428.569286] ret_from_fork+0x1f/0x30 [ 2444.059398] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 2444.065051] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2444.130583] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2444.144584] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2444.151509] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2444.155464] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2444.174654] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2444.179188] RAX: 000000000000682a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2444.187201] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 2444.197463] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2444.201180] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 2444.205248] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 2444.209590] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 2444.214469] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2444.218045] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 2444.220376] Call Trace: [ 2444.228180] [ 2444.230184] ? show_regs.cold.9+0x22/0x2f [ 2444.231317] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2444.235016] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2444.238471] ? watchdog+0x40/0x40 [ 2444.239729] ? hrtimer_interrupt+0x128/0x2c0 [ 2444.241250] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2444.245893] ? apic_timer_interrupt+0xf/0x20 [ 2444.251816] [ 2444.253223] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2444.258757] do_raw_spin_lock+0x177/0x190 [ 2444.260280] _raw_spin_lock+0x21/0x30 [ 2444.266086] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2444.270774] ? do_raw_spin_unlock+0x75/0x190 [ 2444.274170] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2444.276783] ? do_wait_intr+0xf0/0xf0 [ 2444.279074] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2444.286214] mdc_close+0x53b/0xd70 [mdc] [ 2444.291162] lmv_close+0x18d/0x490 [lmv] [ 2444.295599] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2444.308443] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2444.313143] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2444.316248] ll_md_close+0x274/0x800 [lustre] [ 2444.319347] ll_file_release+0x128/0xc50 [lustre] [ 2444.324412] __fput+0x10d/0x440 [ 2444.328440] ____fput+0x12/0x20 [ 2444.336045] task_work_run+0xc6/0x110 [ 2444.343430] exit_to_usermode_loop+0x1dd/0x1f0 [ 2444.344747] do_syscall_64+0x3d6/0x440 [ 2444.346828] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2444.348914] RIP: 0033:0x7f151bee3cd8 [ 2444.353769] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2444.363691] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2444.366802] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 2444.370084] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 2444.374954] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2444.379802] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2444.389390] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2452.058667] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 2452.070430] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2452.100429] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2452.108028] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2452.116014] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2452.120164] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2452.132949] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2452.139391] RAX: 0000000000007f15 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2452.143086] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 2452.146495] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 2452.150954] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 2452.154535] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 2452.161826] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 2452.166670] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2452.169575] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 2452.171949] Call Trace: [ 2452.173348] [ 2452.174137] ? show_regs.cold.9+0x22/0x2f [ 2452.176128] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2452.178529] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2452.180758] ? watchdog+0x40/0x40 [ 2452.182387] ? hrtimer_interrupt+0x128/0x2c0 [ 2452.184528] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2452.186603] ? apic_timer_interrupt+0xf/0x20 [ 2452.188517] [ 2452.189914] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2452.192041] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2452.194586] do_raw_spin_lock+0x177/0x190 [ 2452.196235] _raw_spin_lock+0x21/0x30 [ 2452.197731] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2452.200348] ? do_raw_spin_unlock+0x75/0x190 [ 2452.202457] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2452.205047] ? do_wait_intr+0xf0/0xf0 [ 2452.206741] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2452.209359] mdc_reint+0x2d/0x110 [mdc] [ 2452.211572] mdc_unlink+0x658/0x870 [mdc] [ 2452.213540] lmv_unlink+0x34d/0xbd0 [lmv] [ 2452.215983] ll_unlink+0x24d/0x7f0 [lustre] [ 2452.217994] vfs_unlink+0x202/0x330 [ 2452.219573] do_unlinkat+0x2fa/0x460 [ 2452.221152] __x64_sys_unlink+0x37/0x50 [ 2452.223476] do_syscall_64+0xc1/0x440 [ 2452.225099] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2452.227619] RIP: 0033:0x7f151bdee10b [ 2452.229241] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2452.237722] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2452.240314] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 2452.244173] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 2452.248578] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2452.251726] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2452.256384] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2456.032457] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 525s! [ 2456.039794] Showing busy workqueues and worker pools: [ 2456.044285] workqueue events: flags=0x0 [ 2456.050456] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2456.057287] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 2456.057341] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2456.057349] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2456.057431] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2456.057436] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2456.057443] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2456.057731] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2456.057742] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2456.057746] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2456.057751] RAX: 0000000000007f1d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2456.057753] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 2456.057755] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 2456.057757] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 2456.057759] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 2456.057762] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 2456.057765] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2456.057767] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2456.057775] Call Trace: [ 2456.057780] [ 2456.057785] ? show_regs.cold.9+0x22/0x2f [ 2456.057795] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2456.057801] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2456.057806] ? watchdog+0x40/0x40 [ 2456.057810] ? hrtimer_interrupt+0x128/0x2c0 [ 2456.057814] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2456.057822] ? apic_timer_interrupt+0xf/0x20 [ 2456.057827] [ 2456.057830] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2456.057834] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2456.057839] do_raw_spin_lock+0x177/0x190 [ 2456.057844] _raw_spin_lock+0x21/0x30 [ 2456.057849] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2456.058056] process_one_work+0x2c8/0x700 [ 2456.058064] worker_thread+0x296/0x6e0 [ 2456.058068] ? rescuer_thread+0x570/0x570 [ 2456.058071] kthread+0x1d1/0x200 [ 2456.058075] ? set_kthread_struct+0x70/0x70 [ 2456.058079] ret_from_fork+0x1f/0x30 [ 2456.060164] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2456.060500] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2456.060511] pending: kfree_rcu_monitor [ 2456.060520] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2456.060527] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 2456.060696] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2456.060705] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2456.060938] workqueue events_long: flags=0x0 [ 2456.060941] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2456.060948] pending: lu_objects_flush [obdclass] [ 2456.061150] workqueue events_unbound: flags=0x2 [ 2456.061154] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2456.061163] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2456.061185] workqueue events_power_efficient: flags=0x80 [ 2456.061187] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2456.061193] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 2456.061213] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2456.061219] pending: neigh_periodic_work [ 2456.061227] workqueue mm_percpu_wq: flags=0x8 [ 2456.061229] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2456.061236] pending: vmstat_update, lru_add_drain_per_cpu [ 2456.061245] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2456.061251] pending: vmstat_update [ 2456.061254] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2456.061260] pending: lru_add_drain_per_cpu [ 2456.061265] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2456.061270] pending: lru_add_drain_per_cpu BAR(48) [ 2456.061280] workqueue writeback: flags=0x4a [ 2456.061281] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2456.061286] pending: wb_workfn [ 2456.061298] workqueue xprtiod: flags=0xa [ 2456.061300] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2456.061306] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 2456.061324] workqueue ipv6_addrconf: flags=0x40008 [ 2456.061326] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2456.061332] pending: addrconf_verify_work [ 2456.061344] workqueue knbd0-recv: flags=0x1a [ 2456.061346] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2456.061352] in-flight: 658:recv_work [ 2456.061362] workqueue ptlrpc_pinger: flags=0x2 [ 2456.061364] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2456.061369] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2456.061599] pool 8: cpus=0-3 flags=0x4 nice=0 hung=525s workers=4 idle: 32984 37 8 [ 2456.061611] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2440s workers=2 idle: 661 [ 2456.524829] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2456.568066] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2456.575787] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2456.584063] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2456.588646] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2456.603169] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2456.608144] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 2456.616291] RDX: 000000000000521b RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 2456.622421] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 2456.633587] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 2456.640705] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2456.645793] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 2456.651917] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2456.657393] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2456.660729] Call Trace: [ 2456.662180] [ 2456.664054] ? show_regs.cold.9+0x22/0x2f [ 2456.668937] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2456.674726] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2456.676027] ? watchdog+0x40/0x40 [ 2456.677210] ? hrtimer_interrupt+0x128/0x2c0 [ 2456.680398] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2456.683516] ? apic_timer_interrupt+0xf/0x20 [ 2456.688547] [ 2456.689750] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2456.698403] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2456.705458] do_raw_spin_lock+0x177/0x190 [ 2456.709531] _raw_spin_lock+0x21/0x30 [ 2456.714192] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2456.721827] obd_mod_free+0x46/0x140 [ptlrpc] [ 2456.723871] mdc_commit_open+0x91/0xb0 [mdc] [ 2456.736571] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2456.745032] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2456.750942] after_reply+0xe2c/0x1780 [ptlrpc] [ 2456.759673] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2456.764032] ? __next_timer_interrupt+0x160/0x160 [ 2456.766668] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2456.768816] ? do_wait_intr+0xf0/0xf0 [ 2456.770927] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2456.773460] kthread+0x1d1/0x200 [ 2456.774643] ? set_kthread_struct+0x70/0x70 [ 2456.775866] ret_from_fork+0x1f/0x30 [ 2472.057366] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 2472.061729] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2472.108919] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2472.120490] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2472.139597] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2472.150010] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2472.180047] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2472.201582] RAX: 0000000000002c9f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2472.207207] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 2472.209580] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2472.214096] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 2472.222734] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 2472.232495] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 2472.241940] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2472.253246] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 2472.265762] Call Trace: [ 2472.266676] [ 2472.277340] ? show_regs.cold.9+0x22/0x2f [ 2472.278550] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2472.291926] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2472.294825] ? watchdog+0x40/0x40 [ 2472.301654] ? hrtimer_interrupt+0x128/0x2c0 [ 2472.308468] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2472.314004] ? apic_timer_interrupt+0xf/0x20 [ 2472.318897] [ 2472.325549] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2472.329495] do_raw_spin_lock+0x177/0x190 [ 2472.332520] _raw_spin_lock+0x21/0x30 [ 2472.334663] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2472.342451] ? do_raw_spin_unlock+0x75/0x190 [ 2472.347479] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2472.351618] ? do_wait_intr+0xf0/0xf0 [ 2472.354719] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2472.357713] mdc_close+0x53b/0xd70 [mdc] [ 2472.359888] lmv_close+0x18d/0x490 [lmv] [ 2472.361335] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2472.363479] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2472.366488] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2472.372486] ll_md_close+0x274/0x800 [lustre] [ 2472.375596] ll_file_release+0x128/0xc50 [lustre] [ 2472.379746] __fput+0x10d/0x440 [ 2472.382631] ____fput+0x12/0x20 [ 2472.384121] task_work_run+0xc6/0x110 [ 2472.385772] exit_to_usermode_loop+0x1dd/0x1f0 [ 2472.391588] do_syscall_64+0x3d6/0x440 [ 2472.392937] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2472.396374] RIP: 0033:0x7f151bee3cd8 [ 2472.397529] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2472.415008] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2472.420264] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 2472.424993] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 2472.429247] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2472.433610] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2472.436717] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2484.053747] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 2484.057337] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 2484.060346] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2484.060358] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2484.060434] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2484.060440] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2484.060446] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2484.060719] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2484.060730] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2484.060734] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2484.060738] RAX: 0000000000007984 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2484.060741] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 2484.060744] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 2484.060746] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 2484.060748] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 2484.060751] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 2484.060754] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2484.060756] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2484.060764] Call Trace: [ 2484.060770] [ 2484.060776] ? show_regs.cold.9+0x22/0x2f [ 2484.060785] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2484.060793] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2484.060797] ? watchdog+0x40/0x40 [ 2484.060801] ? hrtimer_interrupt+0x128/0x2c0 [ 2484.060805] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2484.060814] ? apic_timer_interrupt+0xf/0x20 [ 2484.060819] [ 2484.060822] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2484.060826] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2484.060831] do_raw_spin_lock+0x177/0x190 [ 2484.060837] _raw_spin_lock+0x21/0x30 [ 2484.060841] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2484.065153] process_one_work+0x2c8/0x700 [ 2484.065163] worker_thread+0x296/0x6e0 [ 2484.065168] ? rescuer_thread+0x570/0x570 [ 2484.065172] kthread+0x1d1/0x200 [ 2484.065176] ? set_kthread_struct+0x70/0x70 [ 2484.065180] ret_from_fork+0x1f/0x30 [ 2484.065197] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2484.065274] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2484.065280] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2484.065283] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2484.065294] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2484.065298] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2484.065302] RAX: 0000000000007b41 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2484.065304] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 2484.065306] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 2484.065308] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 2484.065310] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 2484.065313] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 2484.065316] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2484.065319] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 2484.065328] Call Trace: [ 2484.065332] [ 2484.065336] ? show_regs.cold.9+0x22/0x2f [ 2484.065346] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2484.065352] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2484.065357] ? watchdog+0x40/0x40 [ 2484.065361] ? hrtimer_interrupt+0x128/0x2c0 [ 2484.065365] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2484.065371] ? apic_timer_interrupt+0xf/0x20 [ 2484.065375] [ 2484.065379] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2484.065384] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2484.065388] do_raw_spin_lock+0x177/0x190 [ 2484.065392] _raw_spin_lock+0x21/0x30 [ 2484.065397] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2484.065635] ? do_raw_spin_unlock+0x75/0x190 [ 2484.065641] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2484.065836] ? do_wait_intr+0xf0/0xf0 [ 2484.065844] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2484.081121] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2484.081198] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2484.081206] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2484.081209] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2484.081224] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2484.081229] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2484.081233] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 2484.081236] RDX: 0000000000007cbb RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 2484.081238] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 2484.081241] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 2484.081243] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2484.081247] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 2484.081250] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2484.081253] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2484.081261] Call Trace: [ 2484.081267] [ 2484.081273] ? show_regs.cold.9+0x22/0x2f [ 2484.081282] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2484.081289] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2484.081293] ? watchdog+0x40/0x40 [ 2484.081296] ? hrtimer_interrupt+0x128/0x2c0 [ 2484.081300] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2484.081308] ? apic_timer_interrupt+0xf/0x20 [ 2484.081312] [ 2484.081315] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2484.081320] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2484.081324] do_raw_spin_lock+0x177/0x190 [ 2484.081328] _raw_spin_lock+0x21/0x30 [ 2484.081334] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2484.081630] obd_mod_free+0x46/0x140 [ptlrpc] [ 2484.081840] mdc_commit_open+0x91/0xb0 [mdc] [ 2484.081878] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2484.082091] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2484.082319] after_reply+0xe2c/0x1780 [ptlrpc] [ 2484.091420] mdc_reint+0x2d/0x110 [mdc] [ 2484.091470] mdc_unlink+0x658/0x870 [mdc] [ 2484.091495] lmv_unlink+0x34d/0xbd0 [lmv] [ 2484.091525] ll_unlink+0x24d/0x7f0 [lustre] [ 2484.091640] vfs_unlink+0x202/0x330 [ 2484.091649] do_unlinkat+0x2fa/0x460 [ 2484.091654] __x64_sys_unlink+0x37/0x50 [ 2484.091658] do_syscall_64+0xc1/0x440 [ 2484.091665] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2484.091674] RIP: 0033:0x7f151bdee10b [ 2484.091681] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2484.091685] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2484.091691] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 2484.091693] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 2484.091695] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2484.091697] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2484.091700] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2484.906903] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2484.909191] ? __next_timer_interrupt+0x160/0x160 [ 2484.910861] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2484.912673] ? do_wait_intr+0xf0/0xf0 [ 2484.914146] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2484.916194] kthread+0x1d1/0x200 [ 2484.917372] ? set_kthread_struct+0x70/0x70 [ 2484.918836] ret_from_fork+0x1f/0x30 [ 2486.241756] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 555s! [ 2486.251678] Showing busy workqueues and worker pools: [ 2486.254022] workqueue events: flags=0x0 [ 2486.255451] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2486.264649] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2486.272589] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2486.277631] pending: kfree_rcu_monitor [ 2486.281090] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2486.285424] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 2486.303533] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2486.315164] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2486.328631] workqueue events_long: flags=0x0 [ 2486.334650] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2486.339237] pending: lu_objects_flush [obdclass] [ 2486.345322] workqueue events_unbound: flags=0x2 [ 2486.353222] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2486.364573] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2486.389847] workqueue events_power_efficient: flags=0x80 [ 2486.391948] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2486.401772] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 2486.408161] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2486.414582] pending: neigh_periodic_work [ 2486.418598] workqueue mm_percpu_wq: flags=0x8 [ 2486.422419] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2486.430253] pending: vmstat_update, lru_add_drain_per_cpu [ 2486.433571] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2486.437303] pending: vmstat_update [ 2486.439523] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2486.442884] pending: lru_add_drain_per_cpu [ 2486.445832] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2486.454237] pending: lru_add_drain_per_cpu BAR(48) [ 2486.459622] workqueue writeback: flags=0x4a [ 2486.461150] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2486.464189] pending: wb_workfn [ 2486.465764] workqueue xprtiod: flags=0xa [ 2486.467687] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2486.470560] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 2486.477207] workqueue ipv6_addrconf: flags=0x40008 [ 2486.480562] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2486.490030] pending: addrconf_verify_work [ 2486.492731] workqueue knbd0-recv: flags=0x1a [ 2486.499209] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2486.508167] in-flight: 658:recv_work [ 2486.510755] workqueue ptlrpc_pinger: flags=0x2 [ 2486.513108] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2486.517364] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2486.521657] pool 8: cpus=0-3 flags=0x4 nice=0 hung=555s workers=4 idle: 32984 37 8 [ 2486.532848] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2471s workers=2 idle: 661 [ 2500.057420] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 2500.062519] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2500.097227] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2500.105018] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2500.113752] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2500.121897] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2500.148914] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2500.159979] RAX: 000000000000748c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2500.168654] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 2500.177644] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2500.189244] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 2500.197573] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 2500.204050] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 2500.211118] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2500.221475] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 2500.229474] Call Trace: [ 2500.231176] [ 2500.233319] ? show_regs.cold.9+0x22/0x2f [ 2500.238384] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2500.240738] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2500.250458] ? watchdog+0x40/0x40 [ 2500.256353] ? hrtimer_interrupt+0x128/0x2c0 [ 2500.258788] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2500.263566] ? apic_timer_interrupt+0xf/0x20 [ 2500.267542] [ 2500.271443] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2500.274032] do_raw_spin_lock+0x177/0x190 [ 2500.276137] _raw_spin_lock+0x21/0x30 [ 2500.277891] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2500.280674] ? do_raw_spin_unlock+0x75/0x190 [ 2500.284724] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2500.289250] ? do_wait_intr+0xf0/0xf0 [ 2500.290768] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2500.292639] mdc_close+0x53b/0xd70 [mdc] [ 2500.294207] lmv_close+0x18d/0x490 [lmv] [ 2500.296805] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2500.302206] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2500.308705] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2500.314468] ll_md_close+0x274/0x800 [lustre] [ 2500.318600] ll_file_release+0x128/0xc50 [lustre] [ 2500.321216] __fput+0x10d/0x440 [ 2500.325532] ____fput+0x12/0x20 [ 2500.331370] task_work_run+0xc6/0x110 [ 2500.335641] exit_to_usermode_loop+0x1dd/0x1f0 [ 2500.340540] do_syscall_64+0x3d6/0x440 [ 2500.343003] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2500.348162] RIP: 0033:0x7f151bee3cd8 [ 2500.349495] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2500.360841] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2500.369662] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 2500.376878] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 2500.385014] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2500.388579] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2500.398097] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2512.052039] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 2512.055010] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 2512.056138] Modules linked in: lustre(O) [ 2512.057139] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2512.057147] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2512.057211] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2512.057217] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2512.057258] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2512.057536] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2512.057559] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2512.057563] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2512.057567] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2512.057569] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 2512.057571] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 2512.057573] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 2512.057575] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 2512.057579] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 2512.057581] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2512.057583] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2512.057591] Call Trace: [ 2512.057618] [ 2512.057623] ? show_regs.cold.9+0x22/0x2f [ 2512.057632] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2512.057638] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2512.057642] ? watchdog+0x40/0x40 [ 2512.057647] ? hrtimer_interrupt+0x128/0x2c0 [ 2512.057650] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2512.057657] ? apic_timer_interrupt+0xf/0x20 [ 2512.057661] [ 2512.057664] ? native_safe_halt+0x2e/0x50 [ 2512.057668] kvm_wait+0xcd/0xe0 [ 2512.057675] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2512.057683] do_raw_spin_lock+0x177/0x190 [ 2512.057689] _raw_spin_lock+0x21/0x30 [ 2512.057693] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2512.057915] process_one_work+0x2c8/0x700 [ 2512.057921] worker_thread+0x296/0x6e0 [ 2512.057926] ? rescuer_thread+0x570/0x570 [ 2512.057930] kthread+0x1d1/0x200 [ 2512.057935] ? set_kthread_struct+0x70/0x70 [ 2512.058026] ret_from_fork+0x1f/0x30 [ 2512.066352] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2512.066531] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2512.066538] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2512.066540] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2512.066554] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2512.066559] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2512.066563] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 2512.066566] RDX: 00000000000000ff RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 2512.066568] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 2512.066570] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 2512.066572] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2512.066575] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 2512.066578] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2512.066580] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2512.068210] Call Trace: [ 2512.070339] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2512.070549] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2512.070556] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2512.070559] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2512.070571] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2512.070575] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2512.070580] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2512.070582] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 2512.070584] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 2512.070585] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 2512.070587] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 2512.070591] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 2512.070594] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2512.070596] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 2512.070604] Call Trace: [ 2512.070609] [ 2512.070615] ? show_regs.cold.9+0x22/0x2f [ 2512.070628] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2512.070635] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2512.070640] ? watchdog+0x40/0x40 [ 2512.070644] ? hrtimer_interrupt+0x128/0x2c0 [ 2512.070648] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2512.070655] ? apic_timer_interrupt+0xf/0x20 [ 2512.070659] [ 2512.070662] ? native_safe_halt+0x2e/0x50 [ 2512.070665] kvm_wait+0xcd/0xe0 [ 2512.070673] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2512.070680] do_raw_spin_lock+0x177/0x190 [ 2512.070685] _raw_spin_lock+0x21/0x30 [ 2512.070691] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2512.070987] ? do_raw_spin_unlock+0x75/0x190 [ 2512.070997] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2512.071395] ? do_wait_intr+0xf0/0xf0 [ 2512.071404] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2512.071600] mdc_reint+0x2d/0x110 [mdc] [ 2512.071636] mdc_unlink+0x658/0x870 [mdc] [ 2512.071667] lmv_unlink+0x34d/0xbd0 [lmv] [ 2512.071873] ll_unlink+0x24d/0x7f0 [lustre] [ 2512.072008] vfs_unlink+0x202/0x330 [ 2512.072017] do_unlinkat+0x2fa/0x460 [ 2512.072023] __x64_sys_unlink+0x37/0x50 [ 2512.072026] do_syscall_64+0xc1/0x440 [ 2512.072046] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2512.072053] RIP: 0033:0x7f151bdee10b [ 2512.072060] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2512.072064] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2512.072068] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 2512.072070] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 2512.072072] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2512.072074] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2512.072076] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2512.482864] [ 2512.483950] ? show_regs.cold.9+0x22/0x2f [ 2512.486088] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2512.487638] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2512.489177] ? watchdog+0x40/0x40 [ 2512.490452] ? hrtimer_interrupt+0x128/0x2c0 [ 2512.491968] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2512.493567] ? apic_timer_interrupt+0xf/0x20 [ 2512.495303] [ 2512.496334] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2512.498951] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2512.501341] do_raw_spin_lock+0x177/0x190 [ 2512.503159] _raw_spin_lock+0x21/0x30 [ 2512.504955] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2512.507362] obd_mod_free+0x46/0x140 [ptlrpc] [ 2512.514626] mdc_commit_open+0x91/0xb0 [mdc] [ 2512.517023] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2512.519908] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2512.522979] after_reply+0xe2c/0x1780 [ptlrpc] [ 2512.526166] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2512.529347] ? __next_timer_interrupt+0x160/0x160 [ 2512.531562] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2512.533904] ? do_wait_intr+0xf0/0xf0 [ 2512.535668] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2512.538340] kthread+0x1d1/0x200 [ 2512.540482] ? set_kthread_struct+0x70/0x70 [ 2512.542624] ret_from_fork+0x1f/0x30 [ 2516.960186] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 586s! [ 2516.963629] Showing busy workqueues and worker pools: [ 2516.967401] workqueue events: flags=0x0 [ 2516.969819] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2516.974968] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2516.977884] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2516.982888] pending: kfree_rcu_monitor [ 2516.985874] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2516.990453] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 2516.997183] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2517.001592] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2517.006050] workqueue events_long: flags=0x0 [ 2517.012167] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2517.020413] pending: lu_objects_flush [obdclass] [ 2517.027223] workqueue events_unbound: flags=0x2 [ 2517.030813] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2517.034087] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2517.044377] workqueue events_power_efficient: flags=0x80 [ 2517.047874] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2517.051960] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 2517.058387] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2517.063393] pending: neigh_periodic_work [ 2517.065944] workqueue mm_percpu_wq: flags=0x8 [ 2517.068843] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2517.072468] pending: vmstat_update, lru_add_drain_per_cpu [ 2517.075700] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2517.078315] pending: vmstat_update [ 2517.082578] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2517.086907] pending: lru_add_drain_per_cpu [ 2517.089859] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2517.095991] pending: lru_add_drain_per_cpu BAR(48) [ 2517.099209] workqueue writeback: flags=0x4a [ 2517.100508] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2517.105239] pending: wb_workfn [ 2517.107589] workqueue xprtiod: flags=0xa [ 2517.109053] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2517.110932] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 2517.117466] workqueue ipv6_addrconf: flags=0x40008 [ 2517.119363] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2517.121552] pending: addrconf_verify_work [ 2517.123622] workqueue knbd0-recv: flags=0x1a [ 2517.125038] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2517.127261] in-flight: 658:recv_work [ 2517.129072] workqueue ptlrpc_pinger: flags=0x2 [ 2517.130974] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2517.133925] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2517.137164] pool 8: cpus=0-3 flags=0x4 nice=0 hung=586s workers=4 idle: 32984 37 8 [ 2517.140209] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2501s workers=2 idle: 661 [ 2524.658151] rcu: INFO: rcu_sched self-detected stall on CPU [ 2524.665825] rcu: 1-...!: (356026 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 2524.679472] (t=600011 jiffies g=249657 q=2648) [ 2524.683946] rcu: rcu_sched kthread starved for 594963 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 2524.695694] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 2524.705597] rcu: RCU grace-period kthread stack dump: [ 2524.706860] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 2524.719332] Call Trace: [ 2524.720458] __schedule+0x351/0xcb0 [ 2524.722061] schedule+0xc0/0x180 [ 2524.724675] schedule_timeout+0xb4/0x190 [ 2524.727363] ? do_raw_spin_unlock+0x75/0x190 [ 2524.731010] ? __next_timer_interrupt+0x160/0x160 [ 2524.733761] ? prepare_to_swait_event+0xf3/0x140 [ 2524.736354] rcu_gp_kthread+0x8dd/0x15b0 [ 2524.738669] ? force_qs_rnp+0x2c0/0x2c0 [ 2524.741854] kthread+0x1d1/0x200 [ 2524.743457] ? set_kthread_struct+0x70/0x70 [ 2524.745802] ret_from_fork+0x1f/0x30 [ 2524.748190] rcu: Stack dump where RCU GP kthread last ran: [ 2524.751255] Sending NMI from CPU 1 to CPUs 2: [ 2524.756809] NMI backtrace for cpu 2 [ 2524.756814] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2524.756816] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2524.756818] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2524.756820] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2524.756822] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2524.756824] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000212 [ 2524.756825] RAX: 0000000000007c14 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2524.756826] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 2524.756827] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 2524.756828] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 2524.756830] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 2524.756831] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 2524.756832] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2524.756834] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2524.756835] Call Trace: [ 2524.756836] [ 2524.756836] ? show_regs.cold.9+0x22/0x2f [ 2524.756837] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 2524.756838] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 2524.756839] ? nmi_handle+0xbd/0x210 [ 2524.756841] ? default_do_nmi+0x4b/0x1b0 [ 2524.756841] ? do_nmi+0x280/0x380 [ 2524.756842] ? end_repeat_nmi+0x16/0x1f [ 2524.756844] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2524.756845] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2524.756846] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2524.756847] [ 2524.756847] do_raw_spin_lock+0x177/0x190 [ 2524.756848] _raw_spin_lock+0x21/0x30 [ 2524.756849] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2524.756850] process_one_work+0x2c8/0x700 [ 2524.756851] worker_thread+0x296/0x6e0 [ 2524.756852] ? rescuer_thread+0x570/0x570 [ 2524.756852] kthread+0x1d1/0x200 [ 2524.756853] ? set_kthread_struct+0x70/0x70 [ 2524.756854] ret_from_fork+0x1f/0x30 [ 2524.769684] NMI backtrace for cpu 1 [ 2524.985044] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2524.989587] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2524.998333] Call Trace: [ 2524.999213] [ 2524.999977] ? dump_stack+0xbb/0x10e [ 2525.001217] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2525.006469] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2525.009883] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2525.011562] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2525.014068] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2525.018450] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 2525.022633] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 2525.026861] ? update_process_times+0x6d/0xc0 [ 2525.029081] ? tick_sched_handle.isra.18+0x27/0x90 [ 2525.030524] ? tick_sched_timer+0x53/0xb0 [ 2525.034905] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2525.037763] ? tick_sched_handle.isra.18+0x90/0x90 [ 2525.040404] ? hrtimer_interrupt+0x128/0x2c0 [ 2525.042771] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2525.045805] ? apic_timer_interrupt+0xf/0x20 [ 2525.048187] [ 2525.049717] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2525.051479] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2525.053810] ? do_raw_spin_lock+0x177/0x190 [ 2525.055079] ? _raw_spin_lock+0x21/0x30 [ 2525.056291] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2525.058945] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2525.060631] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2525.062279] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2525.063888] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2525.070349] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2525.073154] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2525.076039] ? __next_timer_interrupt+0x160/0x160 [ 2525.078151] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2525.080479] ? do_wait_intr+0xf0/0xf0 [ 2525.084040] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2525.086516] ? kthread+0x1d1/0x200 [ 2525.087713] ? set_kthread_struct+0x70/0x70 [ 2525.089424] ? ret_from_fork+0x1f/0x30 [ 2528.057291] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 2528.063271] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2528.099179] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2528.104968] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2528.112340] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2528.115071] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2528.142783] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2528.146719] RAX: 0000000000003691 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2528.149116] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 2528.152390] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2528.156349] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 2528.162781] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 2528.168446] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 2528.173303] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2528.179731] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 2528.186703] Call Trace: [ 2528.188083] [ 2528.189854] ? show_regs.cold.9+0x22/0x2f [ 2528.192225] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2528.196069] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2528.197841] ? watchdog+0x40/0x40 [ 2528.199388] ? hrtimer_interrupt+0x128/0x2c0 [ 2528.201611] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2528.205728] ? apic_timer_interrupt+0xf/0x20 [ 2528.209233] [ 2528.210118] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2528.212232] do_raw_spin_lock+0x177/0x190 [ 2528.217654] _raw_spin_lock+0x21/0x30 [ 2528.220694] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2528.224340] ? do_raw_spin_unlock+0x75/0x190 [ 2528.226310] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2528.231363] ? do_wait_intr+0xf0/0xf0 [ 2528.232490] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2528.239725] mdc_close+0x53b/0xd70 [mdc] [ 2528.241627] lmv_close+0x18d/0x490 [lmv] [ 2528.245126] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2528.249427] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2528.253275] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2528.261123] ll_md_close+0x274/0x800 [lustre] [ 2528.263151] ll_file_release+0x128/0xc50 [lustre] [ 2528.265218] __fput+0x10d/0x440 [ 2528.269140] ____fput+0x12/0x20 [ 2528.270337] task_work_run+0xc6/0x110 [ 2528.271755] exit_to_usermode_loop+0x1dd/0x1f0 [ 2528.273403] do_syscall_64+0x3d6/0x440 [ 2528.275030] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2528.276881] RIP: 0033:0x7f151bee3cd8 [ 2528.280451] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2528.295632] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2528.303327] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 2528.310031] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 2528.313338] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2528.315397] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2528.324212] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2544.055682] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 2544.060686] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2544.107590] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2544.123767] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2544.132300] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2544.140681] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2544.155286] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2544.164960] RAX: 000000000000630d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2544.181737] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 2544.195657] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 2544.205339] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 2544.214696] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 2544.229151] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 2544.241546] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2544.249234] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 2544.261077] Call Trace: [ 2544.261897] [ 2544.262570] ? show_regs.cold.9+0x22/0x2f [ 2544.263774] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2544.271328] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2544.276227] ? watchdog+0x40/0x40 [ 2544.279452] ? hrtimer_interrupt+0x128/0x2c0 [ 2544.284416] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2544.292636] ? apic_timer_interrupt+0xf/0x20 [ 2544.302727] [ 2544.305158] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2544.312008] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2544.321136] do_raw_spin_lock+0x177/0x190 [ 2544.328295] _raw_spin_lock+0x21/0x30 [ 2544.334701] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2544.342843] ? do_raw_spin_unlock+0x75/0x190 [ 2544.349121] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2544.350965] ? do_wait_intr+0xf0/0xf0 [ 2544.358274] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2544.364074] mdc_reint+0x2d/0x110 [mdc] [ 2544.368906] mdc_unlink+0x658/0x870 [mdc] [ 2544.374551] lmv_unlink+0x34d/0xbd0 [lmv] [ 2544.379283] ll_unlink+0x24d/0x7f0 [lustre] [ 2544.383968] vfs_unlink+0x202/0x330 [ 2544.386487] do_unlinkat+0x2fa/0x460 [ 2544.394415] __x64_sys_unlink+0x37/0x50 [ 2544.399520] do_syscall_64+0xc1/0x440 [ 2544.404331] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2544.416696] RIP: 0033:0x7f151bdee10b [ 2544.418066] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2544.431210] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2544.441863] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 2544.453215] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 2544.460895] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2544.469738] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2544.476779] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2547.170523] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 616s! [ 2547.176009] Showing busy workqueues and worker pools: [ 2547.178132] workqueue events: flags=0x0 [ 2547.181942] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2547.195880] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2547.204119] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2547.212089] pending: kfree_rcu_monitor [ 2547.219081] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2547.230556] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 2547.236667] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2547.242438] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2547.247716] workqueue events_long: flags=0x0 [ 2547.249636] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2547.258822] pending: lu_objects_flush [obdclass] [ 2547.262621] workqueue events_unbound: flags=0x2 [ 2547.266848] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2547.271570] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2547.278962] workqueue events_power_efficient: flags=0x80 [ 2547.281677] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2547.284883] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 2547.287908] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2547.291249] pending: neigh_periodic_work [ 2547.293562] workqueue mm_percpu_wq: flags=0x8 [ 2547.299308] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2547.304120] pending: vmstat_update, lru_add_drain_per_cpu [ 2547.308899] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2547.315648] pending: vmstat_update [ 2547.324588] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2547.329303] pending: lru_add_drain_per_cpu [ 2547.332500] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2547.335098] pending: lru_add_drain_per_cpu BAR(48) [ 2547.337243] workqueue writeback: flags=0x4a [ 2547.338566] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2547.344434] pending: wb_workfn [ 2547.346086] workqueue xprtiod: flags=0xa [ 2547.347930] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2547.350686] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 2547.356904] workqueue ipv6_addrconf: flags=0x40008 [ 2547.358613] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2547.362294] pending: addrconf_verify_work [ 2547.368221] workqueue knbd0-recv: flags=0x1a [ 2547.369785] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2547.376814] in-flight: 658:recv_work [ 2547.378202] workqueue ptlrpc_pinger: flags=0x2 [ 2547.381469] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2547.388440] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2547.391583] pool 8: cpus=0-3 flags=0x4 nice=0 hung=616s workers=4 idle: 32984 37 8 [ 2547.398963] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2532s workers=2 idle: 661 [ 2552.056016] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2552.056373] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 2552.058882] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2552.059041] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2552.059048] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2552.059054] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2552.064094] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2552.064178] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2552.064185] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2552.064187] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2552.064200] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2552.064204] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2552.064208] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 2552.064210] RDX: 000000000000319a RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 2552.064212] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 2552.064214] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 2552.064216] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2552.064219] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 2552.064222] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2552.064223] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2552.064230] Call Trace: [ 2552.064234] [ 2552.064240] ? show_regs.cold.9+0x22/0x2f [ 2552.064248] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2552.064253] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2552.064256] ? watchdog+0x40/0x40 [ 2552.064260] ? hrtimer_interrupt+0x128/0x2c0 [ 2552.064263] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2552.064270] ? apic_timer_interrupt+0xf/0x20 [ 2552.064275] [ 2552.064278] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2552.064282] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2552.064286] do_raw_spin_lock+0x177/0x190 [ 2552.064295] _raw_spin_lock+0x21/0x30 [ 2552.064301] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2552.064830] obd_mod_free+0x46/0x140 [ptlrpc] [ 2552.065124] mdc_commit_open+0x91/0xb0 [mdc] [ 2552.065250] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2552.065727] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2552.065944] after_reply+0xe2c/0x1780 [ptlrpc] [ 2552.066300] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2552.066676] ? __next_timer_interrupt+0x160/0x160 [ 2552.066686] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2552.088493] [ 2552.088502] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2552.088519] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2552.088524] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2552.088529] RAX: 0000000000002c95 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2552.088531] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 2552.088533] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 2552.088536] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 2552.088539] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 2552.088542] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 2552.088554] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2552.088556] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2552.088564] Call Trace: [ 2552.088569] [ 2552.088575] ? show_regs.cold.9+0x22/0x2f [ 2552.088586] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2552.088592] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2552.088596] ? watchdog+0x40/0x40 [ 2552.088605] ? hrtimer_interrupt+0x128/0x2c0 [ 2552.088610] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2552.088619] ? apic_timer_interrupt+0xf/0x20 [ 2552.088623] [ 2552.088628] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2552.088634] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2552.088641] do_raw_spin_lock+0x177/0x190 [ 2552.088645] _raw_spin_lock+0x21/0x30 [ 2552.088650] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2552.089124] process_one_work+0x2c8/0x700 [ 2552.089131] worker_thread+0x296/0x6e0 [ 2552.089135] ? rescuer_thread+0x570/0x570 [ 2552.089139] kthread+0x1d1/0x200 [ 2552.089144] ? set_kthread_struct+0x70/0x70 [ 2552.089147] ret_from_fork+0x1f/0x30 [ 2552.275852] ? do_wait_intr+0xf0/0xf0 [ 2552.281146] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2552.284471] kthread+0x1d1/0x200 [ 2552.286889] ? set_kthread_struct+0x70/0x70 [ 2552.294224] ret_from_fork+0x1f/0x30 [ 2556.057095] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 2556.059949] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2556.082620] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2556.088991] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2556.092978] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2556.098769] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2556.110142] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2556.113275] RAX: 00000000000060c4 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2556.116613] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 2556.120260] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2556.123964] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 2556.127866] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 2556.132631] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 2556.136848] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2556.139913] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 2556.142707] Call Trace: [ 2556.143557] [ 2556.144499] ? show_regs.cold.9+0x22/0x2f [ 2556.146069] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2556.148503] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2556.149909] ? watchdog+0x40/0x40 [ 2556.150999] ? hrtimer_interrupt+0x128/0x2c0 [ 2556.154567] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2556.157006] ? apic_timer_interrupt+0xf/0x20 [ 2556.159171] [ 2556.159891] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2556.162090] do_raw_spin_lock+0x177/0x190 [ 2556.164119] _raw_spin_lock+0x21/0x30 [ 2556.165856] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2556.168105] ? do_raw_spin_unlock+0x75/0x190 [ 2556.169813] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2556.172471] ? do_wait_intr+0xf0/0xf0 [ 2556.174130] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2556.176587] mdc_close+0x53b/0xd70 [mdc] [ 2556.178436] lmv_close+0x18d/0x490 [lmv] [ 2556.180129] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2556.182576] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2556.186831] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2556.191397] ll_md_close+0x274/0x800 [lustre] [ 2556.195160] ll_file_release+0x128/0xc50 [lustre] [ 2556.198090] __fput+0x10d/0x440 [ 2556.199961] ____fput+0x12/0x20 [ 2556.201077] task_work_run+0xc6/0x110 [ 2556.203132] exit_to_usermode_loop+0x1dd/0x1f0 [ 2556.205597] do_syscall_64+0x3d6/0x440 [ 2556.207947] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2556.210948] RIP: 0033:0x7f151bee3cd8 [ 2556.212773] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2556.223198] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2556.230370] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 2556.243932] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 2556.249467] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2556.257219] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2556.261466] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2572.052131] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 2572.061921] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2572.106143] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2572.117558] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2572.125959] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2572.130005] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2572.143973] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2572.148385] RAX: 000000000000768a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2572.152315] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 2572.155731] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 2572.158984] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 2572.165093] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 2572.171667] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 2572.179668] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2572.183307] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 2572.190540] Call Trace: [ 2572.193360] [ 2572.195656] ? show_regs.cold.9+0x22/0x2f [ 2572.199692] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2572.201220] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2572.213951] ? watchdog+0x40/0x40 [ 2572.215126] ? hrtimer_interrupt+0x128/0x2c0 [ 2572.220001] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2572.223604] ? apic_timer_interrupt+0xf/0x20 [ 2572.227351] [ 2572.228126] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2572.231897] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2572.234613] do_raw_spin_lock+0x177/0x190 [ 2572.237595] _raw_spin_lock+0x21/0x30 [ 2572.240918] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2572.245501] ? do_raw_spin_unlock+0x75/0x190 [ 2572.248556] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2572.251856] ? do_wait_intr+0xf0/0xf0 [ 2572.253171] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2572.255942] mdc_reint+0x2d/0x110 [mdc] [ 2572.257926] mdc_unlink+0x658/0x870 [mdc] [ 2572.259957] lmv_unlink+0x34d/0xbd0 [lmv] [ 2572.262402] ll_unlink+0x24d/0x7f0 [lustre] [ 2572.263964] vfs_unlink+0x202/0x330 [ 2572.265089] do_unlinkat+0x2fa/0x460 [ 2572.266945] __x64_sys_unlink+0x37/0x50 [ 2572.269330] do_syscall_64+0xc1/0x440 [ 2572.271227] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2572.273561] RIP: 0033:0x7f151bdee10b [ 2572.278062] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2572.291960] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2572.299542] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 2572.303736] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 2572.307551] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2572.311352] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2572.314916] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2577.888250] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 647s! [ 2577.894209] Showing busy workqueues and worker pools: [ 2577.904120] workqueue events: flags=0x0 [ 2577.914803] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2577.925786] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2577.938820] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2577.944510] pending: kfree_rcu_monitor [ 2577.946767] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2577.951791] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 2577.963771] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2577.970659] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2577.980457] workqueue events_long: flags=0x0 [ 2577.982056] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2577.990946] pending: lu_objects_flush [obdclass] [ 2577.997988] workqueue events_unbound: flags=0x2 [ 2578.000745] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2578.008577] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2578.019457] workqueue events_power_efficient: flags=0x80 [ 2578.024823] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2578.032884] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 2578.038748] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2578.047174] pending: neigh_periodic_work [ 2578.051530] workqueue mm_percpu_wq: flags=0x8 [ 2578.057957] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2578.072143] pending: vmstat_update, lru_add_drain_per_cpu [ 2578.081837] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2578.087449] pending: vmstat_update [ 2578.094361] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2578.100003] pending: lru_add_drain_per_cpu [ 2578.101691] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2578.107878] pending: lru_add_drain_per_cpu BAR(48) [ 2578.115622] workqueue writeback: flags=0x4a [ 2578.122726] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2578.135773] pending: wb_workfn [ 2578.145014] workqueue xprtiod: flags=0xa [ 2578.147542] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2578.158794] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 2578.168828] workqueue ipv6_addrconf: flags=0x40008 [ 2578.174635] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2578.187555] pending: addrconf_verify_work [ 2578.194392] workqueue knbd0-recv: flags=0x1a [ 2578.200093] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2578.211291] in-flight: 658:recv_work [ 2578.218528] workqueue ptlrpc_pinger: flags=0x2 [ 2578.224738] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2578.228670] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2578.232797] pool 8: cpus=0-3 flags=0x4 nice=0 hung=647s workers=4 idle: 32984 37 8 [ 2578.240849] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2562s workers=2 idle: 661 [ 2580.055191] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 2580.056048] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2580.060272] Modules linked in: lustre(O) [ 2580.064568] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2580.064647] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2580.066640] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2580.066721] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2580.066729] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2580.066732] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2580.066748] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2580.066752] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2580.066757] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 2580.066760] RDX: 00000000000002e7 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 2580.066762] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 2580.066764] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 2580.066767] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2580.066770] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 2580.066773] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2580.066776] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2580.066784] Call Trace: [ 2580.066789] [ 2580.066795] ? show_regs.cold.9+0x22/0x2f [ 2580.066805] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2580.066812] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2580.066817] ? watchdog+0x40/0x40 [ 2580.066822] ? hrtimer_interrupt+0x128/0x2c0 [ 2580.066825] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2580.066835] ? apic_timer_interrupt+0xf/0x20 [ 2580.066840] [ 2580.066843] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2580.066848] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2580.066852] do_raw_spin_lock+0x177/0x190 [ 2580.066857] _raw_spin_lock+0x21/0x30 [ 2580.066862] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2580.067157] obd_mod_free+0x46/0x140 [ptlrpc] [ 2580.067365] mdc_commit_open+0x91/0xb0 [mdc] [ 2580.067404] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2580.067598] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2580.067798] after_reply+0xe2c/0x1780 [ptlrpc] [ 2580.067994] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2580.068203] ? __next_timer_interrupt+0x160/0x160 [ 2580.068213] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2580.068410] ? do_wait_intr+0xf0/0xf0 [ 2580.068419] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2580.068767] kthread+0x1d1/0x200 [ 2580.068773] ? set_kthread_struct+0x70/0x70 [ 2580.068776] ret_from_fork+0x1f/0x30 [ 2580.407681] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2580.415683] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2580.418282] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2580.420399] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2580.427268] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2580.429709] RAX: 0000000000004384 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2580.432312] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 2580.434697] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 2580.437646] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 2580.440192] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 2580.443550] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 2580.445753] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2580.450216] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2580.452708] Call Trace: [ 2580.453650] [ 2580.454547] ? show_regs.cold.9+0x22/0x2f [ 2580.459011] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2580.461630] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2580.466342] ? watchdog+0x40/0x40 [ 2580.467245] ? hrtimer_interrupt+0x128/0x2c0 [ 2580.468471] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2580.480897] ? apic_timer_interrupt+0xf/0x20 [ 2580.486236] [ 2580.487280] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2580.489855] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2580.494392] do_raw_spin_lock+0x177/0x190 [ 2580.498966] _raw_spin_lock+0x21/0x30 [ 2580.502885] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2580.509639] process_one_work+0x2c8/0x700 [ 2580.516340] worker_thread+0x296/0x6e0 [ 2580.521416] ? rescuer_thread+0x570/0x570 [ 2580.528290] kthread+0x1d1/0x200 [ 2580.531669] ? set_kthread_struct+0x70/0x70 [ 2580.536637] ret_from_fork+0x1f/0x30 [ 2584.057219] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 2584.061363] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2584.090304] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2584.096808] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2584.101720] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 2584.105970] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 2584.115011] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 2584.119361] RAX: 0000000000003c64 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2584.123296] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 2584.127036] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2584.130739] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 2584.134549] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 2584.138984] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 2584.143830] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2584.147443] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 2584.150936] Call Trace: [ 2584.152139] [ 2584.152906] ? show_regs.cold.9+0x22/0x2f [ 2584.154873] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2584.157706] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2584.160517] ? watchdog+0x40/0x40 [ 2584.162670] ? hrtimer_interrupt+0x128/0x2c0 [ 2584.164640] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2584.168152] ? apic_timer_interrupt+0xf/0x20 [ 2584.170503] [ 2584.171874] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 2584.174193] do_raw_spin_lock+0x177/0x190 [ 2584.176273] _raw_spin_lock+0x21/0x30 [ 2584.178880] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2584.183945] ? do_raw_spin_unlock+0x75/0x190 [ 2584.185953] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2584.187970] ? do_wait_intr+0xf0/0xf0 [ 2584.189322] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2584.192440] mdc_close+0x53b/0xd70 [mdc] [ 2584.194418] lmv_close+0x18d/0x490 [lmv] [ 2584.196838] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2584.200500] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2584.203163] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2584.205996] ll_md_close+0x274/0x800 [lustre] [ 2584.208202] ll_file_release+0x128/0xc50 [lustre] [ 2584.210917] __fput+0x10d/0x440 [ 2584.212547] ____fput+0x12/0x20 [ 2584.213990] task_work_run+0xc6/0x110 [ 2584.215538] exit_to_usermode_loop+0x1dd/0x1f0 [ 2584.217311] do_syscall_64+0x3d6/0x440 [ 2584.219332] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2584.221669] RIP: 0033:0x7f151bee3cd8 [ 2584.223562] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2584.233024] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2584.235534] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 2584.238553] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 2584.242184] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2584.245863] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2584.249169] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2604.052080] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 2604.055682] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2604.096216] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2604.102009] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2604.105663] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2604.110363] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2604.133791] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2604.139695] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2604.145527] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 2604.154269] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 2604.162381] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 2604.168039] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 2604.175187] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 2604.183243] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2604.191919] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 2604.200540] Call Trace: [ 2604.202061] [ 2604.207036] ? show_regs.cold.9+0x22/0x2f [ 2604.209192] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2604.213435] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2604.220294] ? watchdog+0x40/0x40 [ 2604.226098] ? hrtimer_interrupt+0x128/0x2c0 [ 2604.231517] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2604.241318] ? apic_timer_interrupt+0xf/0x20 [ 2604.250359] [ 2604.252984] ? native_safe_halt+0x2e/0x50 [ 2604.254515] kvm_wait+0xcd/0xe0 [ 2604.257634] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2604.263480] do_raw_spin_lock+0x177/0x190 [ 2604.269696] _raw_spin_lock+0x21/0x30 [ 2604.274658] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2604.277648] ? do_raw_spin_unlock+0x75/0x190 [ 2604.281212] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2604.294129] ? do_wait_intr+0xf0/0xf0 [ 2604.298598] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2604.307580] mdc_reint+0x2d/0x110 [mdc] [ 2604.313221] mdc_unlink+0x658/0x870 [mdc] [ 2604.315602] lmv_unlink+0x34d/0xbd0 [lmv] [ 2604.317767] ll_unlink+0x24d/0x7f0 [lustre] [ 2604.321649] vfs_unlink+0x202/0x330 [ 2604.323951] do_unlinkat+0x2fa/0x460 [ 2604.328193] __x64_sys_unlink+0x37/0x50 [ 2604.333414] do_syscall_64+0xc1/0x440 [ 2604.337149] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2604.344074] RIP: 0033:0x7f151bdee10b [ 2604.346101] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2604.362490] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2604.373209] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 2604.377665] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 2604.383634] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2604.390780] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2604.398017] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2608.055933] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 2608.056400] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2608.059981] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2608.060064] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2608.060072] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2608.060075] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2608.060090] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2608.060093] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2608.060098] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 2608.060100] RDX: 0000000000001516 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 2608.060102] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 2608.060105] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 2608.060108] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2608.060111] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 2608.060114] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2608.060116] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2608.060124] Call Trace: [ 2608.060130] [ 2608.060135] ? show_regs.cold.9+0x22/0x2f [ 2608.060145] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2608.060151] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2608.060156] ? watchdog+0x40/0x40 [ 2608.060160] ? hrtimer_interrupt+0x128/0x2c0 [ 2608.060164] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2608.060172] ? apic_timer_interrupt+0xf/0x20 [ 2608.060177] [ 2608.060180] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2608.060185] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2608.060190] do_raw_spin_lock+0x177/0x190 [ 2608.060194] _raw_spin_lock+0x21/0x30 [ 2608.060200] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2608.064375] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2608.064451] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2608.064458] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2608.064464] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2608.065308] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2608.065324] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2608.065328] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2608.065333] RAX: 0000000000005be2 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2608.065335] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 2608.065338] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 2608.065340] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 2608.065343] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 2608.065346] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 2608.065349] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2608.065352] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2608.065361] Call Trace: [ 2608.065366] [ 2608.065370] ? show_regs.cold.9+0x22/0x2f [ 2608.065379] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2608.065386] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2608.065390] ? watchdog+0x40/0x40 [ 2608.065395] ? hrtimer_interrupt+0x128/0x2c0 [ 2608.065399] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2608.065409] ? apic_timer_interrupt+0xf/0x20 [ 2608.097573] obd_mod_free+0x46/0x140 [ptlrpc] [ 2608.103253] [ 2608.103264] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2608.103276] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2608.103285] do_raw_spin_lock+0x177/0x190 [ 2608.103292] _raw_spin_lock+0x21/0x30 [ 2608.103299] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2608.103644] process_one_work+0x2c8/0x700 [ 2608.103652] worker_thread+0x296/0x6e0 [ 2608.103657] ? rescuer_thread+0x570/0x570 [ 2608.103660] kthread+0x1d1/0x200 [ 2608.103665] ? set_kthread_struct+0x70/0x70 [ 2608.103669] ret_from_fork+0x1f/0x30 [ 2608.244927] mdc_commit_open+0x91/0xb0 [mdc] [ 2608.246486] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2608.248799] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2608.251174] after_reply+0xe2c/0x1780 [ptlrpc] [ 2608.253181] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2608.255217] ? __next_timer_interrupt+0x160/0x160 [ 2608.256870] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2608.258580] ? do_wait_intr+0xf0/0xf0 [ 2608.259930] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2608.261841] kthread+0x1d1/0x200 [ 2608.263014] ? set_kthread_struct+0x70/0x70 [ 2608.264578] ret_from_fork+0x1f/0x30 [ 2608.609043] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 678s! [ 2608.611346] Showing busy workqueues and worker pools: [ 2608.613179] workqueue events: flags=0x0 [ 2608.614352] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2608.616455] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2608.618820] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2608.621094] pending: kfree_rcu_monitor [ 2608.622372] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2608.624555] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 2608.627632] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2608.630064] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2608.632515] workqueue events_long: flags=0x0 [ 2608.634101] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2608.636269] pending: lu_objects_flush [obdclass] [ 2608.638426] workqueue events_unbound: flags=0x2 [ 2608.640087] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2608.642134] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2608.645342] workqueue events_power_efficient: flags=0x80 [ 2608.646786] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2608.649434] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 2608.652228] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2608.654357] pending: neigh_periodic_work [ 2608.655693] workqueue mm_percpu_wq: flags=0x8 [ 2608.657279] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2608.659431] pending: vmstat_update, lru_add_drain_per_cpu [ 2608.661401] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2608.663521] pending: vmstat_update [ 2608.665018] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2608.667197] pending: lru_add_drain_per_cpu [ 2608.668558] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2608.671117] pending: lru_add_drain_per_cpu BAR(48) [ 2608.672627] workqueue writeback: flags=0x4a [ 2608.674164] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2608.676173] pending: wb_workfn [ 2608.677282] workqueue xprtiod: flags=0xa [ 2608.678442] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2608.680436] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 2608.683186] workqueue ipv6_addrconf: flags=0x40008 [ 2608.684598] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2608.686487] pending: addrconf_verify_work [ 2608.687631] workqueue knbd0-recv: flags=0x1a [ 2608.689397] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2608.691460] in-flight: 658:recv_work [ 2608.692746] workqueue ptlrpc_pinger: flags=0x2 [ 2608.694296] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2608.696315] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2608.698489] pool 8: cpus=0-3 flags=0x4 nice=0 hung=678s workers=4 idle: 32984 37 8 [ 2608.702273] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2593s workers=2 idle: 661 [ 2612.057712] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 2612.061669] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2612.108157] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2612.118788] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2612.126003] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2612.131255] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2612.145433] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2612.151118] RAX: 0000000000007f85 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2612.158542] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 2612.169180] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2612.174146] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 2612.178190] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 2612.188455] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 2612.200347] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2612.208463] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 2612.215121] Call Trace: [ 2612.216027] [ 2612.222364] ? show_regs.cold.9+0x22/0x2f [ 2612.223586] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2612.225216] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2612.242367] ? watchdog+0x40/0x40 [ 2612.246957] ? hrtimer_interrupt+0x128/0x2c0 [ 2612.254372] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2612.259301] ? apic_timer_interrupt+0xf/0x20 [ 2612.262306] [ 2612.263583] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2612.266389] do_raw_spin_lock+0x177/0x190 [ 2612.269169] _raw_spin_lock+0x21/0x30 [ 2612.270430] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2612.272868] ? do_raw_spin_unlock+0x75/0x190 [ 2612.274333] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2612.276783] ? do_wait_intr+0xf0/0xf0 [ 2612.278192] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2612.279963] mdc_close+0x53b/0xd70 [mdc] [ 2612.282699] lmv_close+0x18d/0x490 [lmv] [ 2612.285789] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2612.288979] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2612.296450] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2612.304624] ll_md_close+0x274/0x800 [lustre] [ 2612.308607] ll_file_release+0x128/0xc50 [lustre] [ 2612.312943] __fput+0x10d/0x440 [ 2612.315147] ____fput+0x12/0x20 [ 2612.317022] task_work_run+0xc6/0x110 [ 2612.318281] exit_to_usermode_loop+0x1dd/0x1f0 [ 2612.324325] do_syscall_64+0x3d6/0x440 [ 2612.328415] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2612.334734] RIP: 0033:0x7f151bee3cd8 [ 2612.340417] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2612.360297] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2612.374346] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 2612.381063] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 2612.386711] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2612.392898] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2612.402244] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2636.052764] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 2636.056572] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 2636.057045] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 2636.057053] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2636.057136] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2636.057141] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2636.057144] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2636.057159] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2636.057164] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2636.057168] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 2636.057171] RDX: 0000000000006fa5 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 2636.057173] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 2636.057176] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 2636.057178] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2636.057182] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 2636.057186] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2636.057188] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2636.057195] Call Trace: [ 2636.057206] [ 2636.057212] ? show_regs.cold.9+0x22/0x2f [ 2636.057222] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2636.057229] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2636.057234] ? watchdog+0x40/0x40 [ 2636.057238] ? hrtimer_interrupt+0x128/0x2c0 [ 2636.057242] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2636.057251] ? apic_timer_interrupt+0xf/0x20 [ 2636.057255] [ 2636.057258] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2636.057263] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2636.057268] do_raw_spin_lock+0x177/0x190 [ 2636.057272] _raw_spin_lock+0x21/0x30 [ 2636.057279] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2636.057576] obd_mod_free+0x46/0x140 [ptlrpc] [ 2636.057913] mdc_commit_open+0x91/0xb0 [mdc] [ 2636.057948] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2636.058143] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2636.058338] after_reply+0xe2c/0x1780 [ptlrpc] [ 2636.058993] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2636.059060] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2636.059065] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2636.059068] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2636.059079] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2636.059083] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2636.059087] RAX: 0000000000005585 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2636.059090] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 2636.059092] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 2636.059094] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 2636.059096] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 2636.059099] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 2636.059101] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2636.059103] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 2636.059110] Call Trace: [ 2636.059114] [ 2636.059117] ? show_regs.cold.9+0x22/0x2f [ 2636.059125] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2636.059131] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2636.059135] ? watchdog+0x40/0x40 [ 2636.059139] ? hrtimer_interrupt+0x128/0x2c0 [ 2636.059143] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2636.059152] ? apic_timer_interrupt+0xf/0x20 [ 2636.058536] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2636.065501] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2636.065558] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2636.065564] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2636.065570] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2636.067608] ? __next_timer_interrupt+0x160/0x160 [ 2636.067811] [ 2636.067815] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2636.067825] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2636.067829] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2636.067834] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2636.067842] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 2636.067844] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 2636.067846] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 2636.067849] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 2636.067852] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 2636.067855] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2636.067857] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2636.067865] Call Trace: [ 2636.067869] [ 2636.067877] ? show_regs.cold.9+0x22/0x2f [ 2636.067886] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2636.067621] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2636.068538] ? do_wait_intr+0xf0/0xf0 [ 2636.068547] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2636.069953] kthread+0x1d1/0x200 [ 2636.071275] ? set_kthread_struct+0x70/0x70 [ 2636.071280] ret_from_fork+0x1f/0x30 [ 2636.080697] [ 2636.080712] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2636.080725] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2636.080730] do_raw_spin_lock+0x177/0x190 [ 2636.080736] _raw_spin_lock+0x21/0x30 [ 2636.080747] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2636.103123] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2636.103151] ? watchdog+0x40/0x40 [ 2636.103159] ? hrtimer_interrupt+0x128/0x2c0 [ 2636.103164] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2636.103178] ? apic_timer_interrupt+0xf/0x20 [ 2636.103297] [ 2636.103303] ? native_safe_halt+0x2e/0x50 [ 2636.103309] kvm_wait+0xcd/0xe0 [ 2636.103318] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2636.103329] do_raw_spin_lock+0x177/0x190 [ 2636.103333] _raw_spin_lock+0x21/0x30 [ 2636.103338] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2636.103681] process_one_work+0x2c8/0x700 [ 2636.103687] worker_thread+0x296/0x6e0 [ 2636.103692] ? rescuer_thread+0x570/0x570 [ 2636.103696] kthread+0x1d1/0x200 [ 2636.103700] ? set_kthread_struct+0x70/0x70 [ 2636.103704] ret_from_fork+0x1f/0x30 [ 2636.776925] ? do_raw_spin_unlock+0x75/0x190 [ 2636.785859] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2636.793648] ? do_wait_intr+0xf0/0xf0 [ 2636.802869] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2636.807301] mdc_reint+0x2d/0x110 [mdc] [ 2636.811915] mdc_unlink+0x658/0x870 [mdc] [ 2636.816871] lmv_unlink+0x34d/0xbd0 [lmv] [ 2636.828346] ll_unlink+0x24d/0x7f0 [lustre] [ 2636.832433] vfs_unlink+0x202/0x330 [ 2636.833664] do_unlinkat+0x2fa/0x460 [ 2636.839775] __x64_sys_unlink+0x37/0x50 [ 2636.844303] do_syscall_64+0xc1/0x440 [ 2636.847290] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2636.858790] RIP: 0033:0x7f151bdee10b [ 2636.860770] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2636.891628] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2636.899269] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 2636.910740] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 2636.922747] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2636.928030] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2636.932585] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2638.818166] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 708s! [ 2638.828479] Showing busy workqueues and worker pools: [ 2638.834825] workqueue events: flags=0x0 [ 2638.838787] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2638.848322] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2638.853760] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2638.863770] pending: kfree_rcu_monitor [ 2638.870459] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2638.876168] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 2638.882527] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2638.887798] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2638.892853] workqueue events_long: flags=0x0 [ 2638.895499] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2638.899310] pending: lu_objects_flush [obdclass] [ 2638.901454] workqueue events_unbound: flags=0x2 [ 2638.904246] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2638.907070] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2638.912615] workqueue events_power_efficient: flags=0x80 [ 2638.915310] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2638.919680] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 2638.924117] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2638.928384] pending: neigh_periodic_work [ 2638.930042] workqueue mm_percpu_wq: flags=0x8 [ 2638.933340] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2638.937698] pending: vmstat_update, lru_add_drain_per_cpu [ 2638.942089] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2638.953270] pending: vmstat_update [ 2638.957198] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2638.965672] pending: lru_add_drain_per_cpu [ 2638.970319] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2638.975698] pending: lru_add_drain_per_cpu BAR(48) [ 2638.980680] workqueue writeback: flags=0x4a [ 2638.984520] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2638.994926] pending: wb_workfn [ 2639.000686] workqueue xprtiod: flags=0xa [ 2639.002940] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2639.011124] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 2639.018150] workqueue ipv6_addrconf: flags=0x40008 [ 2639.021855] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2639.026218] pending: addrconf_verify_work [ 2639.031368] workqueue knbd0-recv: flags=0x1a [ 2639.033868] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2639.036698] in-flight: 658:recv_work [ 2639.038706] workqueue ptlrpc_pinger: flags=0x2 [ 2639.041081] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2639.043964] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2639.047047] pool 8: cpus=0-3 flags=0x4 nice=0 hung=708s workers=4 idle: 32984 37 8 [ 2639.051936] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2623s workers=2 idle: 661 [ 2640.058479] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 2640.069003] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2640.123931] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2640.134561] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2640.140524] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2640.145623] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2640.156893] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2640.160553] RAX: 000000000000670a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2640.163099] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 2640.166722] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2640.169819] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 2640.174539] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 2640.177043] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 2640.180779] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2640.187679] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 2640.191127] Call Trace: [ 2640.192307] [ 2640.193638] ? show_regs.cold.9+0x22/0x2f [ 2640.195736] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2640.198342] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2640.200382] ? watchdog+0x40/0x40 [ 2640.202659] ? hrtimer_interrupt+0x128/0x2c0 [ 2640.204565] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2640.206804] ? apic_timer_interrupt+0xf/0x20 [ 2640.208886] [ 2640.210154] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2640.212707] do_raw_spin_lock+0x177/0x190 [ 2640.214569] _raw_spin_lock+0x21/0x30 [ 2640.216203] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2640.219163] ? do_raw_spin_unlock+0x75/0x190 [ 2640.221651] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2640.224222] ? do_wait_intr+0xf0/0xf0 [ 2640.226081] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2640.228638] mdc_close+0x53b/0xd70 [mdc] [ 2640.230445] lmv_close+0x18d/0x490 [lmv] [ 2640.232353] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2640.235046] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2640.238171] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2640.240576] ll_md_close+0x274/0x800 [lustre] [ 2640.242828] ll_file_release+0x128/0xc50 [lustre] [ 2640.245238] __fput+0x10d/0x440 [ 2640.246786] ____fput+0x12/0x20 [ 2640.248454] task_work_run+0xc6/0x110 [ 2640.250186] exit_to_usermode_loop+0x1dd/0x1f0 [ 2640.253236] do_syscall_64+0x3d6/0x440 [ 2640.255398] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2640.258095] RIP: 0033:0x7f151bee3cd8 [ 2640.260065] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2640.270398] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2640.274592] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 2640.278331] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 2640.282313] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2640.286742] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2640.289877] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2664.053634] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 2664.055075] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 2664.056166] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 2664.056174] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2664.056256] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2664.056262] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2664.056268] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2664.056560] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2664.056571] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2664.056575] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2664.056579] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2664.056582] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 2664.056585] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 2664.056602] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 2664.056604] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 2664.056608] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 2664.056612] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2664.056615] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2664.056625] Call Trace: [ 2664.056631] [ 2664.056637] ? show_regs.cold.9+0x22/0x2f [ 2664.056647] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2664.056655] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2664.056659] ? watchdog+0x40/0x40 [ 2664.056664] ? hrtimer_interrupt+0x128/0x2c0 [ 2664.056669] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2664.056677] ? apic_timer_interrupt+0xf/0x20 [ 2664.056682] [ 2664.056685] ? native_safe_halt+0x2e/0x50 [ 2664.056689] kvm_wait+0xcd/0xe0 [ 2664.056707] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2664.056717] do_raw_spin_lock+0x177/0x190 [ 2664.056723] _raw_spin_lock+0x21/0x30 [ 2664.056728] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2664.056931] process_one_work+0x2c8/0x700 [ 2664.056939] worker_thread+0x296/0x6e0 [ 2664.056944] ? rescuer_thread+0x570/0x570 [ 2664.056948] kthread+0x1d1/0x200 [ 2664.056954] ? set_kthread_struct+0x70/0x70 [ 2664.056959] ret_from_fork+0x1f/0x30 [ 2664.060544] Modules linked in: [ 2664.064007] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2664.064087] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2664.064095] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2664.064098] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2664.064111] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2664.064115] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2664.064120] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 2664.064122] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 2664.064124] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 2664.064126] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 2664.064129] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2664.064132] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 2664.064135] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2664.064137] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2664.064147] Call Trace: [ 2664.064152] [ 2664.064158] ? show_regs.cold.9+0x22/0x2f [ 2664.064167] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2664.064174] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2664.064178] ? watchdog+0x40/0x40 [ 2664.064183] ? hrtimer_interrupt+0x128/0x2c0 [ 2664.064187] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2664.074872] lustre(O) [ 2664.096589] ? apic_timer_interrupt+0xf/0x20 [ 2664.096608] [ 2664.096613] ? native_safe_halt+0x2e/0x50 [ 2664.096617] kvm_wait+0xcd/0xe0 [ 2664.096625] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2664.096633] do_raw_spin_lock+0x177/0x190 [ 2664.096637] _raw_spin_lock+0x21/0x30 [ 2664.096644] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2664.097002] obd_mod_free+0x46/0x140 [ptlrpc] [ 2664.106773] osp(O) [ 2664.110539] mdc_commit_open+0x91/0xb0 [mdc] [ 2664.117603] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2664.117677] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2664.117685] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2664.117687] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2664.117700] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2664.117705] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2664.117710] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2664.117712] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 2664.117714] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 2664.117716] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 2664.117719] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 2664.117722] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 2664.117725] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2664.117728] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 2664.117737] Call Trace: [ 2664.117743] [ 2664.117749] ? show_regs.cold.9+0x22/0x2f [ 2664.117759] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2664.117766] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2664.117770] ? watchdog+0x40/0x40 [ 2664.117774] ? hrtimer_interrupt+0x128/0x2c0 [ 2664.117779] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2664.122454] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2664.122734] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2664.123545] after_reply+0xe2c/0x1780 [ptlrpc] [ 2664.123747] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2664.137017] ? apic_timer_interrupt+0xf/0x20 [ 2664.137036] [ 2664.137042] ? native_safe_halt+0x2e/0x50 [ 2664.137047] kvm_wait+0xcd/0xe0 [ 2664.137055] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2664.137063] do_raw_spin_lock+0x177/0x190 [ 2664.137067] _raw_spin_lock+0x21/0x30 [ 2664.137074] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2664.137410] ? do_raw_spin_unlock+0x75/0x190 [ 2664.137416] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2664.137620] ? do_wait_intr+0xf0/0xf0 [ 2664.137626] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2664.137913] mdc_reint+0x2d/0x110 [mdc] [ 2664.137951] mdc_unlink+0x658/0x870 [mdc] [ 2664.138044] lmv_unlink+0x34d/0xbd0 [lmv] [ 2664.138075] ll_unlink+0x24d/0x7f0 [lustre] [ 2664.138182] vfs_unlink+0x202/0x330 [ 2664.138190] do_unlinkat+0x2fa/0x460 [ 2664.138194] __x64_sys_unlink+0x37/0x50 [ 2664.138197] do_syscall_64+0xc1/0x440 [ 2664.138205] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2664.138211] RIP: 0033:0x7f151bdee10b [ 2664.138277] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2664.138282] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2664.138287] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 2664.138289] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 2664.138291] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2664.138292] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2664.138294] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2664.526298] ? __next_timer_interrupt+0x160/0x160 [ 2664.527873] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2664.529783] ? do_wait_intr+0xf0/0xf0 [ 2664.532508] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2664.536111] kthread+0x1d1/0x200 [ 2664.537883] ? set_kthread_struct+0x70/0x70 [ 2664.540609] ret_from_fork+0x1f/0x30 [ 2668.058096] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 2668.064607] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2668.100433] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2668.104513] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2668.108391] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2668.110652] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2668.119131] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2668.121998] RAX: 0000000000007287 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2668.130155] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 2668.133373] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2668.139091] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 2668.145091] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 2668.148904] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 2668.158922] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2668.163814] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 2668.167559] Call Trace: [ 2668.169473] [ 2668.170498] ? show_regs.cold.9+0x22/0x2f [ 2668.172627] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2668.175041] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2668.177106] ? watchdog+0x40/0x40 [ 2668.178534] ? hrtimer_interrupt+0x128/0x2c0 [ 2668.179949] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2668.182688] ? apic_timer_interrupt+0xf/0x20 [ 2668.184614] [ 2668.185558] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2668.188482] do_raw_spin_lock+0x177/0x190 [ 2668.189885] _raw_spin_lock+0x21/0x30 [ 2668.191518] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2668.193853] ? do_raw_spin_unlock+0x75/0x190 [ 2668.195566] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2668.197456] ? do_wait_intr+0xf0/0xf0 [ 2668.198850] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2668.201811] mdc_close+0x53b/0xd70 [mdc] [ 2668.203445] lmv_close+0x18d/0x490 [lmv] [ 2668.205464] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2668.208056] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2668.210851] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2668.213108] ll_md_close+0x274/0x800 [lustre] [ 2668.215796] ll_file_release+0x128/0xc50 [lustre] [ 2668.218611] __fput+0x10d/0x440 [ 2668.220062] ____fput+0x12/0x20 [ 2668.221724] task_work_run+0xc6/0x110 [ 2668.223521] exit_to_usermode_loop+0x1dd/0x1f0 [ 2668.225383] do_syscall_64+0x3d6/0x440 [ 2668.226650] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2668.228770] RIP: 0033:0x7f151bee3cd8 [ 2668.230112] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2668.238684] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2668.242759] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 2668.248436] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 2668.251112] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2668.254486] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2668.256932] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2669.538280] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 738s! [ 2669.541647] Showing busy workqueues and worker pools: [ 2669.547308] workqueue events: flags=0x0 [ 2669.549778] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2669.553699] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2669.557835] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2669.560849] pending: kfree_rcu_monitor [ 2669.562954] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2669.566823] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 2669.572244] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2669.575381] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2669.579011] workqueue events_long: flags=0x0 [ 2669.580909] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2669.584143] pending: lu_objects_flush [obdclass] [ 2669.588780] workqueue events_unbound: flags=0x2 [ 2669.594451] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2669.598063] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2669.605035] workqueue events_power_efficient: flags=0x80 [ 2669.607120] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2669.609520] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 2669.611865] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2669.614600] pending: neigh_periodic_work [ 2669.615826] workqueue mm_percpu_wq: flags=0x8 [ 2669.617403] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2669.619288] pending: vmstat_update, lru_add_drain_per_cpu [ 2669.620712] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2669.622708] pending: vmstat_update [ 2669.623865] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2669.625637] pending: lru_add_drain_per_cpu [ 2669.627288] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2669.629722] pending: lru_add_drain_per_cpu BAR(48) [ 2669.632077] workqueue writeback: flags=0x4a [ 2669.634277] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2669.647025] pending: wb_workfn [ 2669.649796] workqueue xprtiod: flags=0xa [ 2669.652976] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2669.656164] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 2669.660042] workqueue ipv6_addrconf: flags=0x40008 [ 2669.662090] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2669.666883] pending: addrconf_verify_work [ 2669.669032] workqueue knbd0-recv: flags=0x1a [ 2669.670916] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2669.679479] in-flight: 658:recv_work [ 2669.685220] workqueue ptlrpc_pinger: flags=0x2 [ 2669.687454] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2669.690589] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2669.693348] pool 8: cpus=0-3 flags=0x4 nice=0 hung=739s workers=4 idle: 32984 37 8 [ 2669.696976] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2654s workers=2 idle: 661 [ 2692.056046] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 2692.056078] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 2692.060878] Modules linked in: [ 2692.071431] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2692.071512] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2692.071520] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2692.071523] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2692.071539] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2692.071542] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2692.071547] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 2692.071549] RDX: 0000000000000995 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 2692.071551] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 2692.071553] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 2692.071555] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2692.071558] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 2692.071561] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2692.071563] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2692.071572] Call Trace: [ 2692.071578] [ 2692.071584] ? show_regs.cold.9+0x22/0x2f [ 2692.071593] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2692.071600] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2692.071605] ? watchdog+0x40/0x40 [ 2692.071609] ? hrtimer_interrupt+0x128/0x2c0 [ 2692.071613] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2692.071624] ? apic_timer_interrupt+0xf/0x20 [ 2692.071628] [ 2692.071631] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2692.071636] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2692.071640] do_raw_spin_lock+0x177/0x190 [ 2692.071645] _raw_spin_lock+0x21/0x30 [ 2692.071801] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2692.073769] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2692.073842] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2692.073848] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2692.073855] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2692.074444] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2692.074456] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2692.074460] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 2692.074464] RAX: 0000000000004689 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2692.074467] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 2692.074469] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 2692.074471] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 2692.074474] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 2692.074477] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 2692.074479] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2692.074482] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2692.074490] Call Trace: [ 2692.074495] [ 2692.074499] ? show_regs.cold.9+0x22/0x2f [ 2692.074509] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2692.074515] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2692.074519] ? watchdog+0x40/0x40 [ 2692.074524] ? hrtimer_interrupt+0x128/0x2c0 [ 2692.074528] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2692.074536] ? apic_timer_interrupt+0xf/0x20 [ 2692.074541] [ 2692.074543] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2692.074549] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2692.074554] do_raw_spin_lock+0x177/0x190 [ 2692.074558] _raw_spin_lock+0x21/0x30 [ 2692.074563] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2692.075046] process_one_work+0x2c8/0x700 [ 2692.075054] worker_thread+0x296/0x6e0 [ 2692.075058] ? rescuer_thread+0x570/0x570 [ 2692.075062] kthread+0x1d1/0x200 [ 2692.075067] ? set_kthread_struct+0x70/0x70 [ 2692.075070] ret_from_fork+0x1f/0x30 [ 2692.365564] obd_mod_free+0x46/0x140 [ptlrpc] [ 2692.368469] mdc_commit_open+0x91/0xb0 [mdc] [ 2692.371568] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2692.374512] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2692.377248] after_reply+0xe2c/0x1780 [ptlrpc] [ 2692.379853] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2692.382805] ? __next_timer_interrupt+0x160/0x160 [ 2692.385821] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2692.388059] ? do_wait_intr+0xf0/0xf0 [ 2692.389852] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2692.392464] kthread+0x1d1/0x200 [ 2692.393924] ? set_kthread_struct+0x70/0x70 [ 2692.395865] ret_from_fork+0x1f/0x30 [ 2696.057259] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 2696.060945] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2696.062310] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 2696.076273] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2696.076286] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2696.076289] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2696.076304] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2696.076307] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2696.076313] RAX: 0000000000007f87 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2696.076315] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 2696.076317] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2696.076319] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 2696.076322] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 2696.076325] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 2696.076328] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2696.076330] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 2696.076340] Call Trace: [ 2696.076346] [ 2696.076352] ? show_regs.cold.9+0x22/0x2f [ 2696.076362] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2696.076369] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2696.076374] ? watchdog+0x40/0x40 [ 2696.076378] ? hrtimer_interrupt+0x128/0x2c0 [ 2696.076382] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2696.076391] ? apic_timer_interrupt+0xf/0x20 [ 2696.076396] [ 2696.076398] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2696.076406] do_raw_spin_lock+0x177/0x190 [ 2696.076411] _raw_spin_lock+0x21/0x30 [ 2696.076415] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2696.076676] ? do_raw_spin_unlock+0x75/0x190 [ 2696.076683] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2696.076883] ? do_wait_intr+0xf0/0xf0 [ 2696.076891] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2696.077102] mdc_close+0x53b/0xd70 [mdc] [ 2696.077143] lmv_close+0x18d/0x490 [lmv] [ 2696.077174] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2696.077337] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2696.077448] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2696.077528] ll_md_close+0x274/0x800 [lustre] [ 2696.077611] ll_file_release+0x128/0xc50 [lustre] [ 2696.077694] __fput+0x10d/0x440 [ 2696.077704] ____fput+0x12/0x20 [ 2696.077708] task_work_run+0xc6/0x110 [ 2696.077714] exit_to_usermode_loop+0x1dd/0x1f0 [ 2696.077721] do_syscall_64+0x3d6/0x440 [ 2696.077724] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2696.077731] RIP: 0033:0x7f151bee3cd8 [ 2696.077737] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2696.077741] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2696.077745] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 2696.077747] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 2696.077749] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2696.077751] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2696.077752] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2696.378872] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2696.403582] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2696.408744] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2696.412148] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2696.417177] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2696.426332] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2696.428625] RAX: 0000000000007d89 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2696.434332] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 2696.438413] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 2696.441691] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 2696.445629] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 2696.448264] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 2696.450851] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2696.455204] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 2696.459007] Call Trace: [ 2696.459739] [ 2696.460743] ? show_regs.cold.9+0x22/0x2f [ 2696.461777] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2696.464607] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2696.467569] ? watchdog+0x40/0x40 [ 2696.472192] ? hrtimer_interrupt+0x128/0x2c0 [ 2696.473740] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2696.479228] ? apic_timer_interrupt+0xf/0x20 [ 2696.481476] [ 2696.483942] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2696.485874] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2696.489901] do_raw_spin_lock+0x177/0x190 [ 2696.492324] _raw_spin_lock+0x21/0x30 [ 2696.494009] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2696.495862] ? do_raw_spin_unlock+0x75/0x190 [ 2696.499271] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2696.503258] ? do_wait_intr+0xf0/0xf0 [ 2696.504853] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2696.509788] mdc_reint+0x2d/0x110 [mdc] [ 2696.512457] mdc_unlink+0x658/0x870 [mdc] [ 2696.516367] lmv_unlink+0x34d/0xbd0 [lmv] [ 2696.518330] ll_unlink+0x24d/0x7f0 [lustre] [ 2696.523225] vfs_unlink+0x202/0x330 [ 2696.524810] do_unlinkat+0x2fa/0x460 [ 2696.528356] __x64_sys_unlink+0x37/0x50 [ 2696.530593] do_syscall_64+0xc1/0x440 [ 2696.531600] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2696.532811] RIP: 0033:0x7f151bdee10b [ 2696.535711] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2696.551059] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2696.556629] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 2696.562682] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 2696.568454] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2696.573436] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2696.577225] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2699.746621] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 769s! [ 2699.764278] Showing busy workqueues and worker pools: [ 2699.772724] workqueue events: flags=0x0 [ 2699.777986] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2699.780717] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2699.788162] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2699.796143] pending: kfree_rcu_monitor [ 2699.801660] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2699.811217] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 2699.846228] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2699.866306] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2699.868924] workqueue events_long: flags=0x0 [ 2699.882161] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2699.893638] pending: lu_objects_flush [obdclass] [ 2699.903766] workqueue events_unbound: flags=0x2 [ 2699.911035] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2699.918594] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2699.926319] workqueue events_power_efficient: flags=0x80 [ 2699.931414] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2699.936554] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 2699.941803] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2699.946014] pending: neigh_periodic_work [ 2699.949105] workqueue mm_percpu_wq: flags=0x8 [ 2699.952472] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2699.961171] pending: vmstat_update, lru_add_drain_per_cpu [ 2699.965318] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2699.968759] pending: vmstat_update [ 2699.970318] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2699.975787] pending: lru_add_drain_per_cpu [ 2699.979338] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2699.985191] pending: lru_add_drain_per_cpu BAR(48) [ 2699.989309] workqueue writeback: flags=0x4a [ 2699.991683] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2699.998250] pending: wb_workfn [ 2700.000678] workqueue xprtiod: flags=0xa [ 2700.004694] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2700.011783] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 2700.018671] workqueue ipv6_addrconf: flags=0x40008 [ 2700.023749] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2700.034211] pending: addrconf_verify_work [ 2700.040208] workqueue knbd0-recv: flags=0x1a [ 2700.045581] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2700.050649] in-flight: 658:recv_work [ 2700.059551] workqueue ptlrpc_pinger: flags=0x2 [ 2700.063568] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2700.072246] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2700.080788] pool 8: cpus=0-3 flags=0x4 nice=0 hung=769s workers=4 idle: 32984 37 8 [ 2700.087588] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2684s workers=2 idle: 661 [ 2704.661219] rcu: INFO: rcu_sched self-detected stall on CPU [ 2704.676168] rcu: 1-...!: (470785 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 2704.690839] (t=780014 jiffies g=249657 q=2703) [ 2704.701804] rcu: rcu_sched kthread starved for 774966 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 2704.712726] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 2704.718774] rcu: RCU grace-period kthread stack dump: [ 2704.721788] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 2704.728253] Call Trace: [ 2704.730473] __schedule+0x351/0xcb0 [ 2704.736450] schedule+0xc0/0x180 [ 2704.740024] schedule_timeout+0xb4/0x190 [ 2704.744894] ? do_raw_spin_unlock+0x75/0x190 [ 2704.749183] ? __next_timer_interrupt+0x160/0x160 [ 2704.753333] ? prepare_to_swait_event+0xf3/0x140 [ 2704.756524] rcu_gp_kthread+0x8dd/0x15b0 [ 2704.759159] ? force_qs_rnp+0x2c0/0x2c0 [ 2704.761538] kthread+0x1d1/0x200 [ 2704.764311] ? set_kthread_struct+0x70/0x70 [ 2704.767318] ret_from_fork+0x1f/0x30 [ 2704.773370] rcu: Stack dump where RCU GP kthread last ran: [ 2704.780379] Sending NMI from CPU 1 to CPUs 3: [ 2704.786758] INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 3.106 msecs [ 2704.789891] NMI backtrace for cpu 3 [ 2704.789899] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2704.789901] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2704.789902] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2704.789903] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2704.789905] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 [ 2704.789908] RAX: 0000000000007a0d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2704.789910] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 2704.789911] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2704.789912] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 2704.789913] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 2704.789915] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 2704.789916] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2704.789917] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 2704.789918] Call Trace: [ 2704.789919] [ 2704.789920] ? show_regs.cold.9+0x22/0x2f [ 2704.789921] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 2704.789922] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 2704.789923] ? nmi_handle+0xbd/0x210 [ 2704.789923] ? default_do_nmi+0x4b/0x1b0 [ 2704.789924] ? do_nmi+0x280/0x380 [ 2704.789925] ? end_repeat_nmi+0x16/0x1f [ 2704.789926] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2704.789927] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2704.789928] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2704.789929] [ 2704.789930] do_raw_spin_lock+0x177/0x190 [ 2704.789931] _raw_spin_lock+0x21/0x30 [ 2704.789932] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2704.789933] ? do_raw_spin_unlock+0x75/0x190 [ 2704.789934] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2704.789935] ? do_wait_intr+0xf0/0xf0 [ 2704.789936] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2704.789936] mdc_close+0x53b/0xd70 [mdc] [ 2704.789937] lmv_close+0x18d/0x490 [lmv] [ 2704.789938] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2704.789939] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2704.789941] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2704.789941] ll_md_close+0x274/0x800 [lustre] [ 2704.789942] ll_file_release+0x128/0xc50 [lustre] [ 2704.789943] __fput+0x10d/0x440 [ 2704.789944] ____fput+0x12/0x20 [ 2704.789945] task_work_run+0xc6/0x110 [ 2704.789946] exit_to_usermode_loop+0x1dd/0x1f0 [ 2704.789947] do_syscall_64+0x3d6/0x440 [ 2704.789948] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2704.789949] RIP: 0033:0x7f151bee3cd8 [ 2704.789950] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2704.789952] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2704.789954] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 2704.789955] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 2704.789965] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2704.789967] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2704.789968] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2704.789974] NMI backtrace for cpu 1 [ 2705.299844] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2705.316947] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2705.326185] Call Trace: [ 2705.327479] [ 2705.330237] ? dump_stack+0xbb/0x10e [ 2705.331516] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2705.340723] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2705.349460] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2705.360030] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2705.372223] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2705.377705] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 2705.387408] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 2705.401282] ? update_process_times+0x6d/0xc0 [ 2705.404447] ? tick_sched_handle.isra.18+0x27/0x90 [ 2705.410167] ? tick_sched_timer+0x53/0xb0 [ 2705.411511] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2705.418255] ? tick_sched_handle.isra.18+0x90/0x90 [ 2705.424426] ? hrtimer_interrupt+0x128/0x2c0 [ 2705.435326] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2705.445565] ? apic_timer_interrupt+0xf/0x20 [ 2705.450191] [ 2705.451226] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2705.453391] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2705.456411] ? do_raw_spin_lock+0x177/0x190 [ 2705.458443] ? _raw_spin_lock+0x21/0x30 [ 2705.460392] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2705.463460] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2705.465084] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2705.466294] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2705.479565] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2705.483383] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2705.490852] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2705.494531] ? __next_timer_interrupt+0x160/0x160 [ 2705.496917] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2705.499337] ? do_wait_intr+0xf0/0xf0 [ 2705.500796] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2705.504780] ? kthread+0x1d1/0x200 [ 2705.506732] ? set_kthread_struct+0x70/0x70 [ 2705.508279] ? ret_from_fork+0x1f/0x30 [ 2720.056028] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 2720.062816] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2720.091854] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2720.100988] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2720.107109] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2720.109093] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2720.114144] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2720.121171] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2720.123568] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2720.128606] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 2720.131234] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 2720.133879] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 2720.142413] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 2720.144064] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 2720.149012] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2720.151290] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2720.155248] Call Trace: [ 2720.157471] [ 2720.160189] ? show_regs.cold.9+0x22/0x2f [ 2720.169465] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2720.172224] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2720.174994] ? watchdog+0x40/0x40 [ 2720.176611] ? hrtimer_interrupt+0x128/0x2c0 [ 2720.178503] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2720.180610] ? apic_timer_interrupt+0xf/0x20 [ 2720.182215] [ 2720.183971] ? native_safe_halt+0x2e/0x50 [ 2720.187676] kvm_wait+0xcd/0xe0 [ 2720.189607] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2720.193133] do_raw_spin_lock+0x177/0x190 [ 2720.195601] _raw_spin_lock+0x21/0x30 [ 2720.201169] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2720.203514] process_one_work+0x2c8/0x700 [ 2720.204773] worker_thread+0x296/0x6e0 [ 2720.206659] ? rescuer_thread+0x570/0x570 [ 2720.208584] kthread+0x1d1/0x200 [ 2720.210033] ? set_kthread_struct+0x70/0x70 [ 2720.211918] ret_from_fork+0x1f/0x30 [ 2728.056176] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 2728.064149] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2728.104717] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2728.116530] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2728.121662] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2728.124956] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2728.143997] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 2728.150133] RAX: 0000000000004889 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2728.158729] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 2728.162136] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 2728.166721] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 2728.170099] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 2728.172137] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 2728.177496] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2728.181523] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 2728.186383] Call Trace: [ 2728.187241] [ 2728.189326] ? show_regs.cold.9+0x22/0x2f [ 2728.192846] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2728.195622] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2728.197587] ? watchdog+0x40/0x40 [ 2728.199550] ? hrtimer_interrupt+0x128/0x2c0 [ 2728.201988] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2728.203608] ? apic_timer_interrupt+0xf/0x20 [ 2728.206134] [ 2728.206820] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2728.208470] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2728.211547] do_raw_spin_lock+0x177/0x190 [ 2728.213346] _raw_spin_lock+0x21/0x30 [ 2728.214778] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2728.218328] ? do_raw_spin_unlock+0x75/0x190 [ 2728.219673] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2728.221875] ? do_wait_intr+0xf0/0xf0 [ 2728.223386] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2728.227072] mdc_reint+0x2d/0x110 [mdc] [ 2728.230707] mdc_unlink+0x658/0x870 [mdc] [ 2728.233285] lmv_unlink+0x34d/0xbd0 [lmv] [ 2728.239024] ll_unlink+0x24d/0x7f0 [lustre] [ 2728.241888] vfs_unlink+0x202/0x330 [ 2728.243496] do_unlinkat+0x2fa/0x460 [ 2728.249439] __x64_sys_unlink+0x37/0x50 [ 2728.251770] do_syscall_64+0xc1/0x440 [ 2728.253714] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2728.256718] RIP: 0033:0x7f151bdee10b [ 2728.260573] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2728.274504] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2728.278121] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 2728.284089] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 2728.288110] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2728.292053] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2728.295687] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2730.470095] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 799s! [ 2730.479426] Showing busy workqueues and worker pools: [ 2730.485596] workqueue events: flags=0x0 [ 2730.492526] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2730.507325] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2730.509717] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2730.524211] pending: kfree_rcu_monitor [ 2730.534113] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2730.543662] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 2730.559423] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2730.568580] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2730.578797] workqueue events_long: flags=0x0 [ 2730.580726] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2730.583929] pending: lu_objects_flush [obdclass] [ 2730.586728] workqueue events_unbound: flags=0x2 [ 2730.588469] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2730.591788] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2730.596883] workqueue events_power_efficient: flags=0x80 [ 2730.598698] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2730.602406] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 2730.605873] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2730.608516] pending: neigh_periodic_work [ 2730.610181] workqueue mm_percpu_wq: flags=0x8 [ 2730.612595] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2730.617163] pending: vmstat_update, lru_add_drain_per_cpu [ 2730.622967] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2730.627534] pending: vmstat_update [ 2730.636403] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2730.640525] pending: lru_add_drain_per_cpu [ 2730.644494] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2730.651776] pending: lru_add_drain_per_cpu BAR(48) [ 2730.653590] workqueue writeback: flags=0x4a [ 2730.655216] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2730.658032] pending: wb_workfn [ 2730.659914] workqueue xprtiod: flags=0xa [ 2730.661904] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2730.664978] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 2730.667934] workqueue ipv6_addrconf: flags=0x40008 [ 2730.676530] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2730.685612] pending: addrconf_verify_work [ 2730.687191] workqueue knbd0-recv: flags=0x1a [ 2730.698392] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2730.704485] in-flight: 658:recv_work [ 2730.706984] workqueue ptlrpc_pinger: flags=0x2 [ 2730.713549] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2730.718412] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2730.723336] pool 8: cpus=0-3 flags=0x4 nice=0 hung=800s workers=4 idle: 32984 37 8 [ 2730.728768] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2715s workers=2 idle: 661 [ 2732.056149] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 2732.057076] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 2732.063531] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2732.063613] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2732.066847] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2732.066923] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2732.066930] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2732.066932] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2732.066945] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2732.066949] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2732.066953] RAX: 0000000000000ddb RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2732.066955] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 2732.066957] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2732.066959] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 2732.066962] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 2732.066965] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 2732.066967] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2732.066969] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 2732.066977] Call Trace: [ 2732.066981] [ 2732.066986] ? show_regs.cold.9+0x22/0x2f [ 2732.066994] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2732.067000] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2732.067004] ? watchdog+0x40/0x40 [ 2732.067008] ? hrtimer_interrupt+0x128/0x2c0 [ 2732.067012] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2732.067021] ? apic_timer_interrupt+0xf/0x20 [ 2732.067025] [ 2732.067028] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2732.067034] do_raw_spin_lock+0x177/0x190 [ 2732.067039] _raw_spin_lock+0x21/0x30 [ 2732.067044] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2732.067312] ? do_raw_spin_unlock+0x75/0x190 [ 2732.067318] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2732.104101] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2732.110139] ? do_wait_intr+0xf0/0xf0 [ 2732.140534] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2732.145716] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2732.146163] mdc_close+0x53b/0xd70 [mdc] [ 2732.146204] lmv_close+0x18d/0x490 [lmv] [ 2732.146237] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2732.146482] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2732.146749] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2732.146830] ll_md_close+0x274/0x800 [lustre] [ 2732.146917] ll_file_release+0x128/0xc50 [lustre] [ 2732.146998] __fput+0x10d/0x440 [ 2732.147008] ____fput+0x12/0x20 [ 2732.147012] task_work_run+0xc6/0x110 [ 2732.147019] exit_to_usermode_loop+0x1dd/0x1f0 [ 2732.147026] do_syscall_64+0x3d6/0x440 [ 2732.147031] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2732.147039] RIP: 0033:0x7f151bee3cd8 [ 2732.147046] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2732.147050] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2732.147055] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 2732.147058] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 2732.147060] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2732.147062] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2732.147064] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2732.438293] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2732.460644] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2732.465527] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 2732.471868] RDX: 0000000000003d6c RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 2732.477291] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 2732.481135] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 2732.485808] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2732.494142] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 2732.500086] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2732.507136] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2732.515630] Call Trace: [ 2732.519055] [ 2732.519976] ? show_regs.cold.9+0x22/0x2f [ 2732.523489] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2732.525543] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2732.529249] ? watchdog+0x40/0x40 [ 2732.531247] ? hrtimer_interrupt+0x128/0x2c0 [ 2732.536516] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2732.539479] ? apic_timer_interrupt+0xf/0x20 [ 2732.542344] [ 2732.543576] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2732.553424] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2732.554746] do_raw_spin_lock+0x177/0x190 [ 2732.555674] _raw_spin_lock+0x21/0x30 [ 2732.558990] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2732.564187] obd_mod_free+0x46/0x140 [ptlrpc] [ 2732.565921] mdc_commit_open+0x91/0xb0 [mdc] [ 2732.567330] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2732.572264] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2732.575675] after_reply+0xe2c/0x1780 [ptlrpc] [ 2732.581528] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2732.588972] ? __next_timer_interrupt+0x160/0x160 [ 2732.595023] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2732.598849] ? do_wait_intr+0xf0/0xf0 [ 2732.602515] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2732.607997] kthread+0x1d1/0x200 [ 2732.613006] ? set_kthread_struct+0x70/0x70 [ 2732.615058] ret_from_fork+0x1f/0x30 [ 2748.056031] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 2748.065066] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2748.099480] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2748.109933] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2748.119196] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2748.124609] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2748.131061] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2748.144346] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2748.149071] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2748.156036] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 2748.161472] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 2748.165200] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 2748.172951] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 2748.177780] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 2748.186790] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2748.193128] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2748.204109] Call Trace: [ 2748.206950] [ 2748.213896] ? show_regs.cold.9+0x22/0x2f [ 2748.219227] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2748.234849] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2748.239154] ? watchdog+0x40/0x40 [ 2748.242874] ? hrtimer_interrupt+0x128/0x2c0 [ 2748.249156] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2748.252564] ? apic_timer_interrupt+0xf/0x20 [ 2748.258582] [ 2748.260777] ? native_safe_halt+0x2e/0x50 [ 2748.263182] kvm_wait+0xcd/0xe0 [ 2748.264890] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2748.271444] do_raw_spin_lock+0x177/0x190 [ 2748.277914] _raw_spin_lock+0x21/0x30 [ 2748.282753] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2748.289761] process_one_work+0x2c8/0x700 [ 2748.297241] worker_thread+0x296/0x6e0 [ 2748.298644] ? rescuer_thread+0x570/0x570 [ 2748.309602] kthread+0x1d1/0x200 [ 2748.311601] ? set_kthread_struct+0x70/0x70 [ 2748.317725] ret_from_fork+0x1f/0x30 [ 2756.052024] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 2756.054693] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2756.099247] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2756.108526] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2756.113458] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2756.115591] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2756.127545] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2756.130876] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2756.135006] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 2756.144589] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 2756.158285] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 2756.170598] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 2756.174173] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 2756.194075] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2756.205512] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 2756.211533] Call Trace: [ 2756.213484] [ 2756.215121] ? show_regs.cold.9+0x22/0x2f [ 2756.226488] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2756.233119] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2756.234734] ? watchdog+0x40/0x40 [ 2756.244775] ? hrtimer_interrupt+0x128/0x2c0 [ 2756.252137] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2756.256806] ? apic_timer_interrupt+0xf/0x20 [ 2756.267489] [ 2756.269020] ? native_safe_halt+0x2e/0x50 [ 2756.275304] kvm_wait+0xcd/0xe0 [ 2756.277131] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2756.280864] do_raw_spin_lock+0x177/0x190 [ 2756.288911] _raw_spin_lock+0x21/0x30 [ 2756.293945] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2756.304972] ? do_raw_spin_unlock+0x75/0x190 [ 2756.307389] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2756.314496] ? do_wait_intr+0xf0/0xf0 [ 2756.316677] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2756.324142] mdc_reint+0x2d/0x110 [mdc] [ 2756.329093] mdc_unlink+0x658/0x870 [mdc] [ 2756.332524] lmv_unlink+0x34d/0xbd0 [lmv] [ 2756.337242] ll_unlink+0x24d/0x7f0 [lustre] [ 2756.343265] vfs_unlink+0x202/0x330 [ 2756.350086] do_unlinkat+0x2fa/0x460 [ 2756.357078] __x64_sys_unlink+0x37/0x50 [ 2756.370467] do_syscall_64+0xc1/0x440 [ 2756.372731] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2756.380884] RIP: 0033:0x7f151bdee10b [ 2756.383890] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2756.403768] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2756.415273] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 2756.424282] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 2756.432567] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2756.444177] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2756.446750] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2760.057524] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 2760.058134] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 2760.062134] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2760.062219] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2760.062226] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2760.062229] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2760.062245] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2760.062486] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2760.062494] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2760.062497] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 2760.062499] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2760.062501] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 2760.062503] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 2760.062506] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 2760.062508] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2760.062511] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 2760.062519] Call Trace: [ 2760.062525] [ 2760.062542] ? show_regs.cold.9+0x22/0x2f [ 2760.062552] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2760.065134] Modules linked in: lustre(O) [ 2760.082340] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2760.087085] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) [ 2760.090133] ? watchdog+0x40/0x40 [ 2760.091798] mdc(O) [ 2760.097150] ? hrtimer_interrupt+0x128/0x2c0 [ 2760.100225] lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs [ 2760.102147] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2760.104761] ata_generic crct10dif_pclmul crc32_pclmul [ 2760.106931] ? apic_timer_interrupt+0xf/0x20 [ 2760.106950] [ 2760.106955] ? native_safe_halt+0x2e/0x50 [ 2760.106960] kvm_wait+0xcd/0xe0 [ 2760.110127] crc32c_intel [ 2760.112554] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2760.112581] do_raw_spin_lock+0x177/0x190 [ 2760.112586] _raw_spin_lock+0x21/0x30 [ 2760.112594] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2760.115598] ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod [ 2760.117534] ? do_raw_spin_unlock+0x75/0x190 [ 2760.117552] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2760.117829] ? do_wait_intr+0xf0/0xf0 [ 2760.117837] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2760.118042] mdc_close+0x53b/0xd70 [mdc] [ 2760.118084] lmv_close+0x18d/0x490 [lmv] [ 2760.118114] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2760.118279] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2760.118408] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2760.118502] ll_md_close+0x274/0x800 [lustre] [ 2760.118590] ll_file_release+0x128/0xc50 [lustre] [ 2760.118676] __fput+0x10d/0x440 [ 2760.118685] ____fput+0x12/0x20 [ 2760.118689] task_work_run+0xc6/0x110 [ 2760.118696] exit_to_usermode_loop+0x1dd/0x1f0 [ 2760.118704] do_syscall_64+0x3d6/0x440 [ 2760.118709] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2760.118718] RIP: 0033:0x7f151bee3cd8 [ 2760.118727] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2760.118731] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2760.118737] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 2760.118739] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 2760.118742] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2760.118744] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2760.118746] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2760.224855] sha512_ssse3 sha512_generic [ 2760.226099] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2760.230465] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2760.234475] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2760.236574] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2760.244263] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2760.248125] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 2760.251168] RDX: 0000000000000d9b RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 2760.253911] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 2760.256875] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 2760.260100] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2760.262693] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 2760.265015] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2760.266933] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2760.269458] Call Trace: [ 2760.270156] [ 2760.270805] ? show_regs.cold.9+0x22/0x2f [ 2760.272060] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2760.273807] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2760.275670] ? watchdog+0x40/0x40 [ 2760.277475] ? hrtimer_interrupt+0x128/0x2c0 [ 2760.279590] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2760.281142] ? apic_timer_interrupt+0xf/0x20 [ 2760.283167] [ 2760.284084] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2760.286825] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2760.288490] do_raw_spin_lock+0x177/0x190 [ 2760.289852] _raw_spin_lock+0x21/0x30 [ 2760.290859] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2760.293822] obd_mod_free+0x46/0x140 [ptlrpc] [ 2760.295854] mdc_commit_open+0x91/0xb0 [mdc] [ 2760.297169] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2760.299012] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2760.301698] after_reply+0xe2c/0x1780 [ptlrpc] [ 2760.304679] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2760.307658] ? __next_timer_interrupt+0x160/0x160 [ 2760.310746] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2760.313658] ? do_wait_intr+0xf0/0xf0 [ 2760.315041] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2760.317689] kthread+0x1d1/0x200 [ 2760.319798] ? set_kthread_struct+0x70/0x70 [ 2760.321502] ret_from_fork+0x1f/0x30 [ 2761.186548] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 830s! [ 2761.196953] Showing busy workqueues and worker pools: [ 2761.201063] workqueue events: flags=0x0 [ 2761.203114] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2761.207739] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2761.216166] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2761.224709] pending: kfree_rcu_monitor [ 2761.230209] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2761.238307] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 2761.244457] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2761.248926] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2761.257511] workqueue events_long: flags=0x0 [ 2761.263421] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2761.266782] pending: lu_objects_flush [obdclass] [ 2761.272600] workqueue events_unbound: flags=0x2 [ 2761.278945] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2761.282440] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2761.294315] workqueue events_power_efficient: flags=0x80 [ 2761.296383] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2761.300510] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 2761.308164] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2761.314814] pending: neigh_periodic_work [ 2761.317025] workqueue mm_percpu_wq: flags=0x8 [ 2761.320733] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2761.326666] pending: vmstat_update, lru_add_drain_per_cpu [ 2761.333639] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2761.339201] pending: vmstat_update [ 2761.344185] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2761.348156] pending: lru_add_drain_per_cpu [ 2761.352963] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2761.357632] pending: lru_add_drain_per_cpu BAR(48) [ 2761.359641] workqueue writeback: flags=0x4a [ 2761.365619] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2761.374213] pending: wb_workfn [ 2761.376117] workqueue xprtiod: flags=0xa [ 2761.382468] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2761.389031] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 2761.399071] workqueue ipv6_addrconf: flags=0x40008 [ 2761.401105] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2761.404606] pending: addrconf_verify_work [ 2761.406671] workqueue knbd0-recv: flags=0x1a [ 2761.408770] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2761.411708] in-flight: 658:recv_work [ 2761.413511] workqueue ptlrpc_pinger: flags=0x2 [ 2761.415089] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2761.417734] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2761.420466] pool 8: cpus=0-3 flags=0x4 nice=0 hung=830s workers=4 idle: 32984 37 8 [ 2761.423309] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2746s workers=2 idle: 661 [ 2776.056089] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2776.063575] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2776.109775] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2776.117638] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2776.120756] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2776.123297] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2776.124720] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2776.134844] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2776.139641] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2776.142221] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 2776.147803] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 2776.151122] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 2776.155184] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 2776.158696] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 2776.167077] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2776.173169] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2776.181239] Call Trace: [ 2776.182904] [ 2776.186084] ? show_regs.cold.9+0x22/0x2f [ 2776.194502] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2776.203242] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2776.204963] ? watchdog+0x40/0x40 [ 2776.206187] ? hrtimer_interrupt+0x128/0x2c0 [ 2776.212420] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2776.215306] ? apic_timer_interrupt+0xf/0x20 [ 2776.217488] [ 2776.218550] ? native_safe_halt+0x2e/0x50 [ 2776.223937] kvm_wait+0xcd/0xe0 [ 2776.225979] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2776.230779] do_raw_spin_lock+0x177/0x190 [ 2776.233332] _raw_spin_lock+0x21/0x30 [ 2776.237065] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2776.243029] process_one_work+0x2c8/0x700 [ 2776.246394] worker_thread+0x296/0x6e0 [ 2776.252779] ? rescuer_thread+0x570/0x570 [ 2776.256400] kthread+0x1d1/0x200 [ 2776.260517] ? set_kthread_struct+0x70/0x70 [ 2776.263083] ret_from_fork+0x1f/0x30 [ 2788.052169] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 2788.055006] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 2788.057031] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 2788.057045] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2788.057110] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2788.057115] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2788.057117] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2788.057131] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2788.057135] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 2788.057139] RAX: 0000000000001fdb RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2788.057141] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 2788.057143] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2788.057145] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 2788.057147] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 2788.057161] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 2788.057165] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2788.057167] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 2788.057175] Call Trace: [ 2788.057181] [ 2788.057185] ? show_regs.cold.9+0x22/0x2f [ 2788.057195] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2788.057202] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2788.057205] ? watchdog+0x40/0x40 [ 2788.057209] ? hrtimer_interrupt+0x128/0x2c0 [ 2788.057213] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2788.057221] ? apic_timer_interrupt+0xf/0x20 [ 2788.057225] [ 2788.057228] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2788.057233] do_raw_spin_lock+0x177/0x190 [ 2788.057238] _raw_spin_lock+0x21/0x30 [ 2788.059166] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2788.059247] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2788.059259] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2788.059261] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2788.059275] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2788.059279] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2788.059283] RAX: 0000000000002d83 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2788.059286] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 2788.059288] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 2788.059290] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 2788.059293] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 2788.059296] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 2788.059299] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2788.059301] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 2788.059309] Call Trace: [ 2788.059313] [ 2788.059317] ? show_regs.cold.9+0x22/0x2f [ 2788.059327] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2788.059332] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2788.059336] ? watchdog+0x40/0x40 [ 2788.059340] ? hrtimer_interrupt+0x128/0x2c0 [ 2788.059344] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2788.059351] ? apic_timer_interrupt+0xf/0x20 [ 2788.059356] [ 2788.059359] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2788.059363] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2788.059368] do_raw_spin_lock+0x177/0x190 [ 2788.059372] _raw_spin_lock+0x21/0x30 [ 2788.059377] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2788.059672] ? do_raw_spin_unlock+0x75/0x190 [ 2788.059678] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2788.059880] ? do_wait_intr+0xf0/0xf0 [ 2788.059888] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2788.060177] mdc_reint+0x2d/0x110 [mdc] [ 2788.060227] mdc_unlink+0x658/0x870 [mdc] [ 2788.060253] lmv_unlink+0x34d/0xbd0 [lmv] [ 2788.060279] ll_unlink+0x24d/0x7f0 [lustre] [ 2788.060394] vfs_unlink+0x202/0x330 [ 2788.060402] do_unlinkat+0x2fa/0x460 [ 2788.060407] __x64_sys_unlink+0x37/0x50 [ 2788.060410] do_syscall_64+0xc1/0x440 [ 2788.060417] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2788.060424] RIP: 0033:0x7f151bdee10b [ 2788.060430] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2788.060434] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2788.060438] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 2788.060440] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 2788.060442] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2788.060445] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2788.060447] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2788.057243] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2788.060600] ? do_raw_spin_unlock+0x75/0x190 [ 2788.060607] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2788.060798] ? do_wait_intr+0xf0/0xf0 [ 2788.060805] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2788.060997] mdc_close+0x53b/0xd70 [mdc] [ 2788.061032] lmv_close+0x18d/0x490 [lmv] [ 2788.061057] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2788.061226] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2788.061327] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2788.064749] ll_md_close+0x274/0x800 [lustre] [ 2788.064856] ll_file_release+0x128/0xc50 [lustre] [ 2788.064934] __fput+0x10d/0x440 [ 2788.064945] ____fput+0x12/0x20 [ 2788.064949] task_work_run+0xc6/0x110 [ 2788.064959] exit_to_usermode_loop+0x1dd/0x1f0 [ 2788.064965] do_syscall_64+0x3d6/0x440 [ 2788.064970] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2788.064977] RIP: 0033:0x7f151bee3cd8 [ 2788.064986] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2788.064990] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2788.064997] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 2788.064998] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 2788.065000] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2788.065002] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2788.065004] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2788.545606] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2788.580588] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2788.588309] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2788.600799] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2788.604979] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2788.618499] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2788.626797] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 2788.632256] RDX: 0000000000001960 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 2788.636893] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 2788.642198] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 2788.647100] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2788.650744] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 2788.656330] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2788.658418] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2788.661245] Call Trace: [ 2788.662235] [ 2788.662987] ? show_regs.cold.9+0x22/0x2f [ 2788.664557] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2788.666366] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2788.668210] ? watchdog+0x40/0x40 [ 2788.669500] ? hrtimer_interrupt+0x128/0x2c0 [ 2788.671165] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2788.672976] ? apic_timer_interrupt+0xf/0x20 [ 2788.674598] [ 2788.675428] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2788.677465] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2788.679484] do_raw_spin_lock+0x177/0x190 [ 2788.681134] _raw_spin_lock+0x21/0x30 [ 2788.682517] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2788.684623] obd_mod_free+0x46/0x140 [ptlrpc] [ 2788.686717] mdc_commit_open+0x91/0xb0 [mdc] [ 2788.689465] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2788.692347] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2788.699674] after_reply+0xe2c/0x1780 [ptlrpc] [ 2788.701737] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2788.703748] ? __next_timer_interrupt+0x160/0x160 [ 2788.706554] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2788.711651] ? do_wait_intr+0xf0/0xf0 [ 2788.716272] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2788.720524] kthread+0x1d1/0x200 [ 2788.721706] ? set_kthread_struct+0x70/0x70 [ 2788.722771] ret_from_fork+0x1f/0x30 [ 2791.906695] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 861s! [ 2791.911754] Showing busy workqueues and worker pools: [ 2791.913726] workqueue events: flags=0x0 [ 2791.915364] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2791.917954] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2791.921163] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2791.924172] pending: kfree_rcu_monitor [ 2791.926014] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2791.928465] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 2791.932464] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2791.935353] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2791.938508] workqueue events_long: flags=0x0 [ 2791.940423] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2791.943383] pending: lu_objects_flush [obdclass] [ 2791.945894] workqueue events_unbound: flags=0x2 [ 2791.947836] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2791.950757] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2791.955703] workqueue events_power_efficient: flags=0x80 [ 2791.958303] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2791.961101] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 2791.964266] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2791.966613] pending: neigh_periodic_work [ 2791.968678] workqueue mm_percpu_wq: flags=0x8 [ 2791.970579] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2791.973148] pending: vmstat_update, lru_add_drain_per_cpu [ 2791.975444] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2791.977500] pending: vmstat_update [ 2791.978888] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2791.981610] pending: lru_add_drain_per_cpu [ 2791.983388] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2791.986034] pending: lru_add_drain_per_cpu BAR(48) [ 2791.988204] workqueue writeback: flags=0x4a [ 2791.990001] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2791.992343] pending: wb_workfn [ 2791.993813] workqueue xprtiod: flags=0xa [ 2791.995403] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2791.998383] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 2792.001371] workqueue ipv6_addrconf: flags=0x40008 [ 2792.003314] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2792.005777] pending: addrconf_verify_work [ 2792.007716] workqueue knbd0-recv: flags=0x1a [ 2792.009104] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2792.011572] in-flight: 658:recv_work [ 2792.013440] workqueue ptlrpc_pinger: flags=0x2 [ 2792.015194] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2792.017876] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2792.020560] pool 8: cpus=0-3 flags=0x4 nice=0 hung=861s workers=4 idle: 32984 37 8 [ 2792.023575] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2776s workers=2 idle: 661 [ 2804.056028] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2804.061305] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2804.085466] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2804.090866] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2804.095773] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2804.100736] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2804.103431] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2804.112669] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2804.119825] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2804.123240] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 2804.126081] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 2804.129799] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 2804.133027] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 2804.135558] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 2804.138027] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2804.141219] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2804.144554] Call Trace: [ 2804.148498] [ 2804.149645] ? show_regs.cold.9+0x22/0x2f [ 2804.151657] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2804.154458] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2804.156569] ? watchdog+0x40/0x40 [ 2804.158053] ? hrtimer_interrupt+0x128/0x2c0 [ 2804.159828] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2804.161773] ? apic_timer_interrupt+0xf/0x20 [ 2804.163465] [ 2804.164487] ? native_safe_halt+0x2e/0x50 [ 2804.165512] kvm_wait+0xcd/0xe0 [ 2804.166343] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2804.167589] do_raw_spin_lock+0x177/0x190 [ 2804.172583] _raw_spin_lock+0x21/0x30 [ 2804.173629] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2804.182227] process_one_work+0x2c8/0x700 [ 2804.185666] worker_thread+0x296/0x6e0 [ 2804.189450] ? rescuer_thread+0x570/0x570 [ 2804.192856] kthread+0x1d1/0x200 [ 2804.195380] ? set_kthread_struct+0x70/0x70 [ 2804.199372] ret_from_fork+0x1f/0x30 [ 2816.054608] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 2816.058111] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 2816.061011] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl [ 2816.061098] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 2816.061104] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2816.061541] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2816.061547] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2816.061549] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2816.061561] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2816.061565] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2816.061570] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 2816.061572] RDX: 00000000000034b2 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 2816.061574] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 2816.061576] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 2816.061578] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2816.061581] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 2816.061585] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2816.061587] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2816.061595] Call Trace: [ 2816.061600] [ 2816.061606] ? show_regs.cold.9+0x22/0x2f [ 2816.061616] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2816.061623] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2816.061627] ? watchdog+0x40/0x40 [ 2816.061835] ? hrtimer_interrupt+0x128/0x2c0 [ 2816.061839] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2816.061854] ? apic_timer_interrupt+0xf/0x20 [ 2816.061065] i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2816.061988] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2816.061992] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2816.061995] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2816.062001] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2816.062005] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2816.062008] RAX: 0000000000005b21 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2816.062010] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 2816.062012] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 2816.062014] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 2816.062016] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 2816.062018] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 2816.062020] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2816.062022] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 2816.062029] Call Trace: [ 2816.062032] [ 2816.062036] ? show_regs.cold.9+0x22/0x2f [ 2816.062041] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2816.062046] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2816.062049] ? watchdog+0x40/0x40 [ 2816.062053] ? hrtimer_interrupt+0x128/0x2c0 [ 2816.062062] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2816.062067] ? apic_timer_interrupt+0xf/0x20 [ 2816.062072] [ 2816.062075] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2816.062083] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2816.062088] do_raw_spin_lock+0x177/0x190 [ 2816.062093] _raw_spin_lock+0x21/0x30 [ 2816.062128] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2816.063039] ? do_raw_spin_unlock+0x75/0x190 [ 2816.063046] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2816.063308] ? do_wait_intr+0xf0/0xf0 [ 2816.063316] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2816.063659] mdc_reint+0x2d/0x110 [mdc] [ 2816.063706] mdc_unlink+0x658/0x870 [mdc] [ 2816.063738] lmv_unlink+0x34d/0xbd0 [lmv] [ 2816.063768] ll_unlink+0x24d/0x7f0 [lustre] [ 2816.064081] vfs_unlink+0x202/0x330 [ 2816.064091] do_unlinkat+0x2fa/0x460 [ 2816.064097] __x64_sys_unlink+0x37/0x50 [ 2816.064101] do_syscall_64+0xc1/0x440 [ 2816.064106] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2816.064114] RIP: 0033:0x7f151bdee10b [ 2816.064121] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2816.064124] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2816.064128] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 2816.064131] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 2816.064133] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2816.064135] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2816.064137] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2816.068939] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2816.069014] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2816.069021] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2816.069024] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2816.069037] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2816.069041] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2816.069046] RAX: 0000000000007c08 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2816.069048] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 2816.069051] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2816.069053] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 2816.069055] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 2816.069058] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 2816.069061] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2816.069063] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 2816.069071] Call Trace: [ 2816.069076] [ 2816.069082] ? show_regs.cold.9+0x22/0x2f [ 2816.069091] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2816.069097] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2816.069102] ? watchdog+0x40/0x40 [ 2816.069105] ? hrtimer_interrupt+0x128/0x2c0 [ 2816.069109] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2816.069119] ? apic_timer_interrupt+0xf/0x20 [ 2816.069123] [ 2816.069126] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2816.069132] do_raw_spin_lock+0x177/0x190 [ 2816.069136] _raw_spin_lock+0x21/0x30 [ 2816.069141] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2816.069516] ? do_raw_spin_unlock+0x75/0x190 [ 2816.069522] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2816.070107] ? do_wait_intr+0xf0/0xf0 [ 2816.070115] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2816.093145] [ 2816.093158] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2816.093170] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2816.093175] do_raw_spin_lock+0x177/0x190 [ 2816.093181] _raw_spin_lock+0x21/0x30 [ 2816.093189] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2816.093461] obd_mod_free+0x46/0x140 [ptlrpc] [ 2816.097491] mdc_close+0x53b/0xd70 [mdc] [ 2816.119978] mdc_commit_open+0x91/0xb0 [mdc] [ 2816.125944] lmv_close+0x18d/0x490 [lmv] [ 2816.130106] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2816.133398] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2816.147130] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2816.161116] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2816.161319] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2816.161567] ll_md_close+0x274/0x800 [lustre] [ 2816.161648] ll_file_release+0x128/0xc50 [lustre] [ 2816.161866] __fput+0x10d/0x440 [ 2816.161887] ____fput+0x12/0x20 [ 2816.161892] task_work_run+0xc6/0x110 [ 2816.161900] exit_to_usermode_loop+0x1dd/0x1f0 [ 2816.161907] do_syscall_64+0x3d6/0x440 [ 2816.161912] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2816.161921] RIP: 0033:0x7f151bee3cd8 [ 2816.161930] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2816.161934] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2816.161939] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 2816.161941] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 2816.161944] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2816.161946] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2816.161948] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2816.676296] after_reply+0xe2c/0x1780 [ptlrpc] [ 2816.678530] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2816.681129] ? __next_timer_interrupt+0x160/0x160 [ 2816.683448] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2816.685546] ? do_wait_intr+0xf0/0xf0 [ 2816.687250] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2816.692523] kthread+0x1d1/0x200 [ 2816.694812] ? set_kthread_struct+0x70/0x70 [ 2816.697191] ret_from_fork+0x1f/0x30 [ 2822.113032] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 891s! [ 2822.117405] Showing busy workqueues and worker pools: [ 2822.120847] workqueue events: flags=0x0 [ 2822.126468] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2822.131054] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2822.135959] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2822.139853] pending: kfree_rcu_monitor [ 2822.142566] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2822.146527] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 2822.155097] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2822.157923] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2822.163346] workqueue events_long: flags=0x0 [ 2822.166443] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2822.173474] pending: lu_objects_flush [obdclass] [ 2822.181869] workqueue events_unbound: flags=0x2 [ 2822.186089] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2822.192052] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2822.198534] workqueue events_power_efficient: flags=0x80 [ 2822.203139] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2822.211080] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 2822.219085] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2822.221540] pending: neigh_periodic_work [ 2822.234941] workqueue mm_percpu_wq: flags=0x8 [ 2822.237514] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2822.242811] pending: vmstat_update, lru_add_drain_per_cpu [ 2822.245252] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2822.248653] pending: vmstat_update [ 2822.250247] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2822.252796] pending: lru_add_drain_per_cpu [ 2822.257028] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2822.259746] pending: lru_add_drain_per_cpu BAR(48) [ 2822.262484] workqueue writeback: flags=0x4a [ 2822.267300] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2822.274369] pending: wb_workfn [ 2822.277035] workqueue xprtiod: flags=0xa [ 2822.282605] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2822.287859] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 2822.294936] workqueue ipv6_addrconf: flags=0x40008 [ 2822.300493] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2822.306168] pending: addrconf_verify_work [ 2822.309350] workqueue knbd0-recv: flags=0x1a [ 2822.311276] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2822.314445] in-flight: 658:recv_work [ 2822.317539] workqueue ptlrpc_pinger: flags=0x2 [ 2822.321014] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2822.330273] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2822.336090] pool 8: cpus=0-3 flags=0x4 nice=0 hung=891s workers=4 idle: 32984 37 8 [ 2822.341008] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2807s workers=2 idle: 661 [ 2832.056010] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2832.063424] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2832.085576] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2832.089787] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2832.092985] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2832.096081] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2832.098303] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2832.105009] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2832.107545] RAX: 00000000000002af RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2832.110439] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 2832.112916] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 2832.116065] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 2832.119999] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 2832.126934] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 2832.136307] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2832.141580] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2832.151240] Call Trace: [ 2832.152816] [ 2832.153987] ? show_regs.cold.9+0x22/0x2f [ 2832.160591] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2832.164084] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2832.171694] ? watchdog+0x40/0x40 [ 2832.174514] ? hrtimer_interrupt+0x128/0x2c0 [ 2832.179467] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2832.186261] ? apic_timer_interrupt+0xf/0x20 [ 2832.189476] [ 2832.190485] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2832.195171] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2832.201131] do_raw_spin_lock+0x177/0x190 [ 2832.206338] _raw_spin_lock+0x21/0x30 [ 2832.211038] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2832.220381] process_one_work+0x2c8/0x700 [ 2832.224093] worker_thread+0x296/0x6e0 [ 2832.225530] ? rescuer_thread+0x570/0x570 [ 2832.228840] kthread+0x1d1/0x200 [ 2832.230919] ? set_kthread_struct+0x70/0x70 [ 2832.232765] ret_from_fork+0x1f/0x30 [ 2844.055461] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 2844.057241] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 2844.059268] Modules linked in: [ 2844.076958] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2844.077282] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2844.077290] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2844.077293] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2844.077309] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2844.077313] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 2844.077317] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2844.077319] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 2844.077322] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2844.077324] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 2844.077333] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 2844.077337] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 2844.077340] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2844.077342] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 2844.077351] Call Trace: [ 2844.077359] [ 2844.077365] ? show_regs.cold.9+0x22/0x2f [ 2844.077378] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2844.077385] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2844.077390] ? watchdog+0x40/0x40 [ 2844.077394] ? hrtimer_interrupt+0x128/0x2c0 [ 2844.077399] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2844.077406] ? apic_timer_interrupt+0xf/0x20 [ 2844.077410] [ 2844.077414] ? native_safe_halt+0x2e/0x50 [ 2844.077418] kvm_wait+0xcd/0xe0 [ 2844.077425] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2844.077434] do_raw_spin_lock+0x177/0x190 [ 2844.077439] _raw_spin_lock+0x21/0x30 [ 2844.077442] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2844.080211] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2844.080280] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2844.080287] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2844.080289] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2844.080302] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2844.080306] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2844.080311] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 2844.080313] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 2844.080315] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 2844.080317] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 2844.080319] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2844.080322] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 2844.080325] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2844.080327] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2844.080336] Call Trace: [ 2844.080341] [ 2844.080346] ? show_regs.cold.9+0x22/0x2f [ 2844.080356] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2844.080363] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2844.080368] ? watchdog+0x40/0x40 [ 2844.080372] ? hrtimer_interrupt+0x128/0x2c0 [ 2844.080376] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2844.080384] ? apic_timer_interrupt+0xf/0x20 [ 2844.080389] [ 2844.080392] ? native_safe_halt+0x2e/0x50 [ 2844.080395] kvm_wait+0xcd/0xe0 [ 2844.080404] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2844.080410] do_raw_spin_lock+0x177/0x190 [ 2844.080414] _raw_spin_lock+0x21/0x30 [ 2844.080419] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2844.128373] ? do_raw_spin_unlock+0x75/0x190 [ 2844.128394] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2844.137152] obd_mod_free+0x46/0x140 [ptlrpc] [ 2844.150421] ? do_wait_intr+0xf0/0xf0 [ 2844.150447] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2844.150771] mdc_close+0x53b/0xd70 [mdc] [ 2844.150811] lmv_close+0x18d/0x490 [lmv] [ 2844.150970] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2844.151263] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2844.151391] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2844.151535] ll_md_close+0x274/0x800 [lustre] [ 2844.151629] ll_file_release+0x128/0xc50 [lustre] [ 2844.151713] __fput+0x10d/0x440 [ 2844.151723] ____fput+0x12/0x20 [ 2844.151728] task_work_run+0xc6/0x110 [ 2844.151738] exit_to_usermode_loop+0x1dd/0x1f0 [ 2844.151747] do_syscall_64+0x3d6/0x440 [ 2844.151752] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2844.151762] RIP: 0033:0x7f151bee3cd8 [ 2844.151813] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2844.151819] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2844.151824] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 2844.151827] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 2844.151829] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2844.151981] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2844.151983] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2844.567496] mdc_commit_open+0x91/0xb0 [mdc] [ 2844.571815] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2844.577625] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2844.582435] after_reply+0xe2c/0x1780 [ptlrpc] [ 2844.587337] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2844.591204] ? __next_timer_interrupt+0x160/0x160 [ 2844.593694] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2844.596367] ? do_wait_intr+0xf0/0xf0 [ 2844.598636] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2844.601764] kthread+0x1d1/0x200 [ 2844.603620] ? set_kthread_struct+0x70/0x70 [ 2844.605941] ret_from_fork+0x1f/0x30 [ 2848.054814] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 2848.063945] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2848.110933] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2848.125174] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2848.132688] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2848.139944] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2848.150503] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2848.154575] RAX: 0000000000007f86 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2848.162775] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 2848.172345] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 2848.179421] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 2848.192296] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 2848.204955] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 2848.216110] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2848.220905] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 2848.225574] Call Trace: [ 2848.227257] [ 2848.230577] ? show_regs.cold.9+0x22/0x2f [ 2848.233890] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2848.237947] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2848.241067] ? watchdog+0x40/0x40 [ 2848.242829] ? hrtimer_interrupt+0x128/0x2c0 [ 2848.245111] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2848.247075] ? apic_timer_interrupt+0xf/0x20 [ 2848.250448] [ 2848.251797] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2848.256088] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2848.257991] do_raw_spin_lock+0x177/0x190 [ 2848.260167] _raw_spin_lock+0x21/0x30 [ 2848.261642] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2848.264383] ? do_raw_spin_unlock+0x75/0x190 [ 2848.268016] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2848.272737] ? do_wait_intr+0xf0/0xf0 [ 2848.275014] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2848.284574] mdc_reint+0x2d/0x110 [mdc] [ 2848.288325] mdc_unlink+0x658/0x870 [mdc] [ 2848.292533] lmv_unlink+0x34d/0xbd0 [lmv] [ 2848.298248] ll_unlink+0x24d/0x7f0 [lustre] [ 2848.301967] vfs_unlink+0x202/0x330 [ 2848.305678] do_unlinkat+0x2fa/0x460 [ 2848.309168] __x64_sys_unlink+0x37/0x50 [ 2848.313719] do_syscall_64+0xc1/0x440 [ 2848.316867] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2848.319520] RIP: 0033:0x7f151bdee10b [ 2848.321213] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2848.347679] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2848.363146] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 2848.375048] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 2848.380110] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2848.393357] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2848.403413] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2852.835386] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 922s! [ 2852.839161] Showing busy workqueues and worker pools: [ 2852.841120] workqueue events: flags=0x0 [ 2852.843075] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2852.846077] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2852.848846] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2852.851905] pending: kfree_rcu_monitor [ 2852.853589] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2852.856092] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 2852.860333] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2852.862909] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2852.871135] workqueue events_long: flags=0x0 [ 2852.872822] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2852.876375] pending: lu_objects_flush [obdclass] [ 2852.883152] workqueue events_unbound: flags=0x2 [ 2852.888897] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2852.891619] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2852.897798] workqueue events_power_efficient: flags=0x80 [ 2852.903769] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2852.914884] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 2852.917471] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2852.940597] pending: neigh_periodic_work [ 2852.944889] workqueue mm_percpu_wq: flags=0x8 [ 2852.960049] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2852.966246] pending: vmstat_update, lru_add_drain_per_cpu [ 2852.972791] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2852.984458] pending: vmstat_update [ 2852.988185] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2852.994638] pending: lru_add_drain_per_cpu [ 2852.998085] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2853.001974] pending: lru_add_drain_per_cpu BAR(48) [ 2853.005570] workqueue writeback: flags=0x4a [ 2853.010805] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2853.020748] pending: wb_workfn [ 2853.028310] workqueue xprtiod: flags=0xa [ 2853.029862] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2853.032261] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 2853.035271] workqueue ipv6_addrconf: flags=0x40008 [ 2853.037181] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2853.040466] pending: addrconf_verify_work [ 2853.042945] workqueue knbd0-recv: flags=0x1a [ 2853.044829] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2853.047186] in-flight: 658:recv_work [ 2853.048910] workqueue ptlrpc_pinger: flags=0x2 [ 2853.050771] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2853.056247] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2853.063951] pool 8: cpus=0-3 flags=0x4 nice=0 hung=922s workers=4 idle: 32984 37 8 [ 2853.073082] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2837s workers=2 idle: 661 [ 2860.056042] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2860.061555] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2860.122125] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2860.132039] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2860.144075] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2860.154091] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2860.162596] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2860.199488] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2860.204110] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2860.207723] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 2860.212675] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 2860.229691] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 2860.242784] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 2860.251629] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 2860.267771] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2860.272260] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2860.289740] Call Trace: [ 2860.291360] [ 2860.292244] ? show_regs.cold.9+0x22/0x2f [ 2860.295646] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2860.301847] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2860.309429] ? watchdog+0x40/0x40 [ 2860.312570] ? hrtimer_interrupt+0x128/0x2c0 [ 2860.320264] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2860.322522] ? apic_timer_interrupt+0xf/0x20 [ 2860.334250] [ 2860.336635] ? native_safe_halt+0x2e/0x50 [ 2860.342809] kvm_wait+0xcd/0xe0 [ 2860.351055] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2860.355554] do_raw_spin_lock+0x177/0x190 [ 2860.359295] _raw_spin_lock+0x21/0x30 [ 2860.369562] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2860.374727] process_one_work+0x2c8/0x700 [ 2860.380834] worker_thread+0x296/0x6e0 [ 2860.383758] ? rescuer_thread+0x570/0x570 [ 2860.391469] kthread+0x1d1/0x200 [ 2860.393521] ? set_kthread_struct+0x70/0x70 [ 2860.400673] ret_from_fork+0x1f/0x30 [ 2872.057115] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 2872.057682] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 2872.080051] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2872.080125] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2872.080131] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2872.080134] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2872.080148] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2872.080152] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2872.080157] RAX: 000000000000679f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2872.080159] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 2872.080161] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2872.080163] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 2872.080165] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 2872.080168] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 2872.080170] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2872.080172] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 2872.080181] Call Trace: [ 2872.080186] [ 2872.080191] ? show_regs.cold.9+0x22/0x2f [ 2872.080200] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2872.080207] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2872.080212] ? watchdog+0x40/0x40 [ 2872.080216] ? hrtimer_interrupt+0x128/0x2c0 [ 2872.080220] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2872.080228] ? apic_timer_interrupt+0xf/0x20 [ 2872.080233] [ 2872.080235] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2872.080242] do_raw_spin_lock+0x177/0x190 [ 2872.080246] _raw_spin_lock+0x21/0x30 [ 2872.080250] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2872.080540] ? do_raw_spin_unlock+0x75/0x190 [ 2872.080547] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2872.080741] ? do_wait_intr+0xf0/0xf0 [ 2872.080749] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2872.080947] mdc_close+0x53b/0xd70 [mdc] [ 2872.080986] lmv_close+0x18d/0x490 [lmv] [ 2872.081015] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2872.081176] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2872.081283] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2872.081377] ll_md_close+0x274/0x800 [lustre] [ 2872.081462] ll_file_release+0x128/0xc50 [lustre] [ 2872.081545] __fput+0x10d/0x440 [ 2872.081553] ____fput+0x12/0x20 [ 2872.081558] task_work_run+0xc6/0x110 [ 2872.081565] exit_to_usermode_loop+0x1dd/0x1f0 [ 2872.081572] do_syscall_64+0x3d6/0x440 [ 2872.081577] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2872.081583] RIP: 0033:0x7f151bee3cd8 [ 2872.081633] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2872.081637] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2872.081642] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 2872.081644] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 2872.081646] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2872.081649] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2872.081651] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2872.386908] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2872.446852] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2872.463773] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2872.468285] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2872.491817] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2872.515103] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2872.523381] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 2872.532771] RDX: 0000000000006ab3 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 2872.548978] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 2872.554783] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 2872.560581] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2872.566278] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 2872.570306] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2872.574962] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2872.579213] Call Trace: [ 2872.580124] [ 2872.580886] ? show_regs.cold.9+0x22/0x2f [ 2872.594606] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2872.602454] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2872.604221] ? watchdog+0x40/0x40 [ 2872.609454] ? hrtimer_interrupt+0x128/0x2c0 [ 2872.612144] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2872.621540] ? apic_timer_interrupt+0xf/0x20 [ 2872.623569] [ 2872.626143] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2872.628194] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2872.635455] do_raw_spin_lock+0x177/0x190 [ 2872.637632] _raw_spin_lock+0x21/0x30 [ 2872.638869] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2872.643928] obd_mod_free+0x46/0x140 [ptlrpc] [ 2872.654939] mdc_commit_open+0x91/0xb0 [mdc] [ 2872.659161] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2872.663164] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2872.665256] after_reply+0xe2c/0x1780 [ptlrpc] [ 2872.683923] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2872.694566] ? __next_timer_interrupt+0x160/0x160 [ 2872.696248] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2872.719792] ? do_wait_intr+0xf0/0xf0 [ 2872.731633] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2872.736477] kthread+0x1d1/0x200 [ 2872.737648] ? set_kthread_struct+0x70/0x70 [ 2872.745699] ret_from_fork+0x1f/0x30 [ 2880.052033] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 2880.062510] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2880.124532] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2880.132063] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2880.143847] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2880.147442] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2880.172467] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2880.189420] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2880.193532] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 2880.206410] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 2880.214603] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 2880.219755] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 2880.225350] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 2880.234785] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2880.245035] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 2880.252361] Call Trace: [ 2880.257508] [ 2880.262086] ? show_regs.cold.9+0x22/0x2f [ 2880.268820] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2880.272606] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2880.274426] ? watchdog+0x40/0x40 [ 2880.285564] ? hrtimer_interrupt+0x128/0x2c0 [ 2880.293946] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2880.303378] ? apic_timer_interrupt+0xf/0x20 [ 2880.311488] [ 2880.312309] ? native_safe_halt+0x2e/0x50 [ 2880.316542] kvm_wait+0xcd/0xe0 [ 2880.320275] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2880.322990] do_raw_spin_lock+0x177/0x190 [ 2880.330644] _raw_spin_lock+0x21/0x30 [ 2880.335834] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2880.349675] ? do_raw_spin_unlock+0x75/0x190 [ 2880.356573] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2880.376666] ? do_wait_intr+0xf0/0xf0 [ 2880.378831] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2880.392362] mdc_reint+0x2d/0x110 [mdc] [ 2880.395851] mdc_unlink+0x658/0x870 [mdc] [ 2880.403552] lmv_unlink+0x34d/0xbd0 [lmv] [ 2880.415780] ll_unlink+0x24d/0x7f0 [lustre] [ 2880.418908] vfs_unlink+0x202/0x330 [ 2880.429744] do_unlinkat+0x2fa/0x460 [ 2880.432779] __x64_sys_unlink+0x37/0x50 [ 2880.436472] do_syscall_64+0xc1/0x440 [ 2880.440643] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2880.444640] RIP: 0033:0x7f151bdee10b [ 2880.448673] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2880.485802] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2880.499847] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 2880.507041] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 2880.521587] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2880.530826] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2880.533272] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2883.554599] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 952s! [ 2883.568698] Showing busy workqueues and worker pools: [ 2883.577357] workqueue events: flags=0x0 [ 2883.583083] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2883.588354] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2883.597541] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2883.605996] pending: kfree_rcu_monitor [ 2883.609405] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2883.621119] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 2883.636558] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2883.653522] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2883.668030] workqueue events_long: flags=0x0 [ 2883.672695] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2883.680762] pending: lu_objects_flush [obdclass] [ 2883.687750] workqueue events_unbound: flags=0x2 [ 2883.695127] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2883.697563] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2883.713048] workqueue events_power_efficient: flags=0x80 [ 2883.720174] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2883.728540] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 2883.735693] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2883.744338] pending: neigh_periodic_work [ 2883.748981] workqueue mm_percpu_wq: flags=0x8 [ 2883.754773] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2883.763599] pending: vmstat_update, lru_add_drain_per_cpu [ 2883.771213] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2883.782270] pending: vmstat_update [ 2883.786038] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2883.795641] pending: lru_add_drain_per_cpu [ 2883.802434] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2883.810477] pending: lru_add_drain_per_cpu BAR(48) [ 2883.821355] workqueue writeback: flags=0x4a [ 2883.822855] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2883.830795] pending: wb_workfn [ 2883.844305] workqueue xprtiod: flags=0xa [ 2883.848383] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2883.853450] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 2883.866784] workqueue ipv6_addrconf: flags=0x40008 [ 2883.870873] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2883.882355] pending: addrconf_verify_work [ 2883.897287] workqueue knbd0-recv: flags=0x1a [ 2883.899724] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2883.904434] in-flight: 658:recv_work [ 2883.906446] workqueue ptlrpc_pinger: flags=0x2 [ 2883.911193] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2883.915420] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2883.920632] pool 8: cpus=0-3 flags=0x4 nice=0 hung=953s workers=4 idle: 32984 37 8 [ 2883.928452] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2868s workers=2 idle: 661 [ 2884.664166] rcu: INFO: rcu_sched self-detected stall on CPU [ 2884.666327] rcu: 1-...!: (578096 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 2884.669761] (t=960017 jiffies g=249657 q=2744) [ 2884.671331] rcu: rcu_sched kthread starved for 954969 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 2884.675012] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 2884.678288] rcu: RCU grace-period kthread stack dump: [ 2884.680857] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 2884.684156] Call Trace: [ 2884.684826] __schedule+0x351/0xcb0 [ 2884.689946] schedule+0xc0/0x180 [ 2884.691465] schedule_timeout+0xb4/0x190 [ 2884.692828] ? do_raw_spin_unlock+0x75/0x190 [ 2884.694470] ? __next_timer_interrupt+0x160/0x160 [ 2884.696993] ? prepare_to_swait_event+0xf3/0x140 [ 2884.700202] rcu_gp_kthread+0x8dd/0x15b0 [ 2884.701459] ? force_qs_rnp+0x2c0/0x2c0 [ 2884.704978] kthread+0x1d1/0x200 [ 2884.707677] ? set_kthread_struct+0x70/0x70 [ 2884.708974] ret_from_fork+0x1f/0x30 [ 2884.710407] rcu: Stack dump where RCU GP kthread last ran: [ 2884.712736] Sending NMI from CPU 1 to CPUs 0: [ 2884.716742] NMI backtrace for cpu 0 [ 2884.716748] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2884.716750] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2884.716751] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2884.716753] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2884.716755] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 [ 2884.716757] RAX: 0000000000007f83 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2884.716759] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 2884.716760] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 2884.716761] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 2884.716763] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 2884.716764] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 2884.716765] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2884.716766] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 2884.716768] Call Trace: [ 2884.716768] [ 2884.716769] ? show_regs.cold.9+0x22/0x2f [ 2884.716770] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 2884.716771] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 2884.716772] ? nmi_handle+0xbd/0x210 [ 2884.716773] ? default_do_nmi+0x4b/0x1b0 [ 2884.716774] ? do_nmi+0x280/0x380 [ 2884.716775] ? end_repeat_nmi+0x16/0x1f [ 2884.716776] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2884.716777] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2884.716778] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2884.716779] [ 2884.716780] do_raw_spin_lock+0x177/0x190 [ 2884.716781] _raw_spin_lock+0x21/0x30 [ 2884.716782] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2884.716783] ? do_raw_spin_unlock+0x75/0x190 [ 2884.716784] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2884.716785] ? do_wait_intr+0xf0/0xf0 [ 2884.716786] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2884.716787] mdc_reint+0x2d/0x110 [mdc] [ 2884.716788] mdc_unlink+0x658/0x870 [mdc] [ 2884.716789] lmv_unlink+0x34d/0xbd0 [lmv] [ 2884.716790] ll_unlink+0x24d/0x7f0 [lustre] [ 2884.716791] vfs_unlink+0x202/0x330 [ 2884.716791] do_unlinkat+0x2fa/0x460 [ 2884.716792] __x64_sys_unlink+0x37/0x50 [ 2884.716793] do_syscall_64+0xc1/0x440 [ 2884.716794] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2884.716794] RIP: 0033:0x7f151bdee10b [ 2884.716796] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2884.716798] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2884.716801] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 2884.716802] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 2884.716803] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2884.716804] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2884.716805] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2884.726691] NMI backtrace for cpu 1 [ 2884.915967] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2884.924498] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2884.929342] Call Trace: [ 2884.930149] [ 2884.930904] ? dump_stack+0xbb/0x10e [ 2884.932005] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2884.933866] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2884.935729] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2884.937638] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2884.939833] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2884.941556] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 2884.943446] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 2884.944818] ? update_process_times+0x6d/0xc0 [ 2884.945901] ? tick_sched_handle.isra.18+0x27/0x90 [ 2884.948552] ? tick_sched_timer+0x53/0xb0 [ 2884.950311] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2884.951786] ? tick_sched_handle.isra.18+0x90/0x90 [ 2884.954261] ? hrtimer_interrupt+0x128/0x2c0 [ 2884.957149] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2884.959060] ? apic_timer_interrupt+0xf/0x20 [ 2884.961842] [ 2884.964246] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2884.965753] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2884.969418] ? do_raw_spin_lock+0x177/0x190 [ 2884.970888] ? _raw_spin_lock+0x21/0x30 [ 2884.973388] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2884.977204] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2884.979425] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2884.983275] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2884.987407] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2884.989091] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2884.997381] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2885.001795] ? __next_timer_interrupt+0x160/0x160 [ 2885.005007] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2885.007058] ? do_wait_intr+0xf0/0xf0 [ 2885.008773] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2885.016868] ? kthread+0x1d1/0x200 [ 2885.022060] ? set_kthread_struct+0x70/0x70 [ 2885.024574] ? ret_from_fork+0x1f/0x30 [ 2888.058579] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2888.071627] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2888.111032] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2888.114965] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2888.119202] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2888.123576] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2888.126797] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2888.148845] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2888.152820] RAX: 0000000000005883 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2888.162859] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 2888.173962] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 2888.175898] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 2888.184563] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 2888.194874] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 2888.206334] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2888.217727] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2888.226439] Call Trace: [ 2888.227409] [ 2888.228191] ? show_regs.cold.9+0x22/0x2f [ 2888.235666] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2888.236836] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2888.237931] ? watchdog+0x40/0x40 [ 2888.247633] ? hrtimer_interrupt+0x128/0x2c0 [ 2888.260900] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2888.269500] ? apic_timer_interrupt+0xf/0x20 [ 2888.275272] [ 2888.277776] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2888.283342] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2888.285769] do_raw_spin_lock+0x177/0x190 [ 2888.288040] _raw_spin_lock+0x21/0x30 [ 2888.292398] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2888.298662] process_one_work+0x2c8/0x700 [ 2888.302020] worker_thread+0x296/0x6e0 [ 2888.306955] ? rescuer_thread+0x570/0x570 [ 2888.310026] kthread+0x1d1/0x200 [ 2888.314953] ? set_kthread_struct+0x70/0x70 [ 2888.318572] ret_from_fork+0x1f/0x30 [ 2900.058832] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 2900.066862] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2900.099831] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2900.110413] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2900.116030] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2900.122451] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2900.139172] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2900.144229] RAX: 0000000000006896 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2900.159888] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 2900.173639] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2900.179911] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 2900.186442] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 2900.191150] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 2900.197354] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2900.199072] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 2900.204633] Call Trace: [ 2900.205771] [ 2900.206703] ? show_regs.cold.9+0x22/0x2f [ 2900.211808] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2900.215080] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2900.221547] ? watchdog+0x40/0x40 [ 2900.223626] ? hrtimer_interrupt+0x128/0x2c0 [ 2900.226113] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2900.228600] ? apic_timer_interrupt+0xf/0x20 [ 2900.231137] [ 2900.234213] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2900.238162] do_raw_spin_lock+0x177/0x190 [ 2900.239330] _raw_spin_lock+0x21/0x30 [ 2900.245235] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2900.247284] ? do_raw_spin_unlock+0x75/0x190 [ 2900.250980] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2900.254572] ? do_wait_intr+0xf0/0xf0 [ 2900.256857] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2900.263960] mdc_close+0x53b/0xd70 [mdc] [ 2900.266152] lmv_close+0x18d/0x490 [lmv] [ 2900.270518] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2900.277181] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2900.280613] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2900.284153] ll_md_close+0x274/0x800 [lustre] [ 2900.287620] ll_file_release+0x128/0xc50 [lustre] [ 2900.290667] __fput+0x10d/0x440 [ 2900.293594] ____fput+0x12/0x20 [ 2900.296568] task_work_run+0xc6/0x110 [ 2900.299600] exit_to_usermode_loop+0x1dd/0x1f0 [ 2900.303568] do_syscall_64+0x3d6/0x440 [ 2900.307452] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2900.312686] RIP: 0033:0x7f151bee3cd8 [ 2900.315365] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2900.332861] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2900.338481] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 2900.342870] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 2900.347665] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2900.353487] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2900.356930] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2912.052139] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 2912.055408] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2912.058043] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 2912.105274] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2912.105286] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2912.105290] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2912.105304] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2912.105308] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2912.105313] RAX: 000000000000103e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2912.105316] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 2912.105318] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 2912.105320] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 2912.105322] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 2912.105326] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 2912.105328] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2912.105331] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 2912.105340] Call Trace: [ 2912.105345] [ 2912.105351] ? show_regs.cold.9+0x22/0x2f [ 2912.105360] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2912.105366] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2912.105371] ? watchdog+0x40/0x40 [ 2912.105375] ? hrtimer_interrupt+0x128/0x2c0 [ 2912.105378] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2912.105386] ? apic_timer_interrupt+0xf/0x20 [ 2912.105390] [ 2912.105393] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2912.105397] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2912.105402] do_raw_spin_lock+0x177/0x190 [ 2912.105407] _raw_spin_lock+0x21/0x30 [ 2912.105414] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2912.108944] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2912.109033] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2912.109040] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2912.109043] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2912.109145] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2912.109149] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2912.109153] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 2912.109156] RDX: 00000000000037e3 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 2912.109158] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 2912.109160] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 2912.109162] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2912.109165] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 2912.109167] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2912.109170] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2912.109194] Call Trace: [ 2912.109247] [ 2912.109252] ? show_regs.cold.9+0x22/0x2f [ 2912.109262] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2912.109272] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2912.109319] ? watchdog+0x40/0x40 [ 2912.109327] ? hrtimer_interrupt+0x128/0x2c0 [ 2912.109331] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2912.109339] ? apic_timer_interrupt+0xf/0x20 [ 2912.109361] [ 2912.109364] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2912.109387] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2912.109391] do_raw_spin_lock+0x177/0x190 [ 2912.109396] _raw_spin_lock+0x21/0x30 [ 2912.109402] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2912.109776] obd_mod_free+0x46/0x140 [ptlrpc] [ 2912.110021] mdc_commit_open+0x91/0xb0 [mdc] [ 2912.110145] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2912.110457] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2912.110652] after_reply+0xe2c/0x1780 [ptlrpc] [ 2912.110987] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2912.120487] ? do_raw_spin_unlock+0x75/0x190 [ 2912.120511] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2912.121001] ? do_wait_intr+0xf0/0xf0 [ 2912.121009] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2912.121331] mdc_reint+0x2d/0x110 [mdc] [ 2912.121384] mdc_unlink+0x658/0x870 [mdc] [ 2912.121418] lmv_unlink+0x34d/0xbd0 [lmv] [ 2912.121465] ll_unlink+0x24d/0x7f0 [lustre] [ 2912.121586] vfs_unlink+0x202/0x330 [ 2912.121595] do_unlinkat+0x2fa/0x460 [ 2912.121600] __x64_sys_unlink+0x37/0x50 [ 2912.121603] do_syscall_64+0xc1/0x440 [ 2912.121610] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2912.121619] RIP: 0033:0x7f151bdee10b [ 2912.121627] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2912.121630] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2912.121635] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 2912.121638] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 2912.121640] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2912.121642] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2912.121644] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2912.396483] ? __next_timer_interrupt+0x160/0x160 [ 2912.398164] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2912.399761] ? do_wait_intr+0xf0/0xf0 [ 2912.400963] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2912.403008] kthread+0x1d1/0x200 [ 2912.404088] ? set_kthread_struct+0x70/0x70 [ 2912.405646] ret_from_fork+0x1f/0x30 [ 2914.274844] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 983s! [ 2914.281454] Showing busy workqueues and worker pools: [ 2914.283954] workqueue events: flags=0x0 [ 2914.285335] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2914.296090] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2914.303217] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2914.309567] pending: kfree_rcu_monitor [ 2914.314165] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2914.319080] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 2914.323520] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2914.330507] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2914.337171] workqueue events_long: flags=0x0 [ 2914.341640] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2914.344701] pending: lu_objects_flush [obdclass] [ 2914.348191] workqueue events_unbound: flags=0x2 [ 2914.350738] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2914.357155] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2914.361677] workqueue events_power_efficient: flags=0x80 [ 2914.365247] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2914.370437] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 2914.375146] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2914.379093] pending: neigh_periodic_work [ 2914.381159] workqueue mm_percpu_wq: flags=0x8 [ 2914.384177] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2914.388546] pending: vmstat_update, lru_add_drain_per_cpu [ 2914.397056] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2914.400675] pending: vmstat_update [ 2914.402728] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2914.406483] pending: lru_add_drain_per_cpu [ 2914.408277] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2914.411389] pending: lru_add_drain_per_cpu BAR(48) [ 2914.414716] workqueue writeback: flags=0x4a [ 2914.417269] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2914.423475] pending: wb_workfn [ 2914.426705] workqueue xprtiod: flags=0xa [ 2914.429714] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2914.432094] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 2914.435762] workqueue ipv6_addrconf: flags=0x40008 [ 2914.438977] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2914.441917] pending: addrconf_verify_work [ 2914.447126] workqueue knbd0-recv: flags=0x1a [ 2914.452464] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2914.456696] in-flight: 658:recv_work [ 2914.460257] workqueue ptlrpc_pinger: flags=0x2 [ 2914.462192] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2914.465222] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2914.470976] pool 8: cpus=0-3 flags=0x4 nice=0 hung=983s workers=4 idle: 32984 37 8 [ 2914.476059] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2899s workers=2 idle: 661 [ 2916.057374] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2916.066098] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2916.115347] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2916.121435] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2916.125682] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2916.129006] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2916.131123] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2916.140237] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2916.143318] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2916.148584] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 2916.152323] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 2916.155976] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 2916.159389] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 2916.162999] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 2916.167062] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2916.170348] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2916.174364] Call Trace: [ 2916.175555] [ 2916.176573] ? show_regs.cold.9+0x22/0x2f [ 2916.178473] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2916.180325] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2916.182420] ? watchdog+0x40/0x40 [ 2916.184364] ? hrtimer_interrupt+0x128/0x2c0 [ 2916.186363] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2916.190415] ? apic_timer_interrupt+0xf/0x20 [ 2916.195310] [ 2916.200441] ? native_safe_halt+0x2e/0x50 [ 2916.204614] kvm_wait+0xcd/0xe0 [ 2916.207154] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2916.210311] do_raw_spin_lock+0x177/0x190 [ 2916.211903] _raw_spin_lock+0x21/0x30 [ 2916.217366] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2916.224377] process_one_work+0x2c8/0x700 [ 2916.226532] worker_thread+0x296/0x6e0 [ 2916.228557] ? rescuer_thread+0x570/0x570 [ 2916.231230] kthread+0x1d1/0x200 [ 2916.233359] ? set_kthread_struct+0x70/0x70 [ 2916.238537] ret_from_fork+0x1f/0x30 [ 2928.057067] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 2928.062340] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2928.111050] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2928.119289] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2928.126599] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2928.128912] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2928.142860] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2928.151816] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2928.158737] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 2928.167031] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2928.177521] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 2928.186845] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 2928.191563] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 2928.200039] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2928.203754] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 2928.217132] Call Trace: [ 2928.218101] [ 2928.218900] ? show_regs.cold.9+0x22/0x2f [ 2928.224943] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2928.230470] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2928.239156] ? watchdog+0x40/0x40 [ 2928.244174] ? hrtimer_interrupt+0x128/0x2c0 [ 2928.251277] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2928.257067] ? apic_timer_interrupt+0xf/0x20 [ 2928.259158] [ 2928.260467] ? native_safe_halt+0x2e/0x50 [ 2928.264092] kvm_wait+0xcd/0xe0 [ 2928.265771] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2928.268147] do_raw_spin_lock+0x177/0x190 [ 2928.269561] _raw_spin_lock+0x21/0x30 [ 2928.271323] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2928.274139] ? do_raw_spin_unlock+0x75/0x190 [ 2928.275849] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2928.279570] ? do_wait_intr+0xf0/0xf0 [ 2928.281599] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2928.284781] mdc_close+0x53b/0xd70 [mdc] [ 2928.287494] lmv_close+0x18d/0x490 [lmv] [ 2928.289187] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2928.292527] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2928.298498] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2928.301882] ll_md_close+0x274/0x800 [lustre] [ 2928.305895] ll_file_release+0x128/0xc50 [lustre] [ 2928.307599] __fput+0x10d/0x440 [ 2928.309448] ____fput+0x12/0x20 [ 2928.311443] task_work_run+0xc6/0x110 [ 2928.315904] exit_to_usermode_loop+0x1dd/0x1f0 [ 2928.321013] do_syscall_64+0x3d6/0x440 [ 2928.323811] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2928.331992] RIP: 0033:0x7f151bee3cd8 [ 2928.336992] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2928.350891] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2928.355471] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 2928.359499] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 2928.363329] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2928.370311] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2928.383660] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2940.052348] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 2940.055689] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 2940.057974] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2940.058125] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2940.058135] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2940.058138] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2940.058233] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2940.058237] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2940.058242] RAX: 000000000000570c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2940.058244] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 2940.058246] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 2940.058252] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 2940.058257] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 2940.058261] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 2940.058264] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2940.058266] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 2940.058274] Call Trace: [ 2940.058279] [ 2940.058286] ? show_regs.cold.9+0x22/0x2f [ 2940.058297] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2940.058303] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2940.058308] ? watchdog+0x40/0x40 [ 2940.058312] ? hrtimer_interrupt+0x128/0x2c0 [ 2940.058316] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2940.058325] ? apic_timer_interrupt+0xf/0x20 [ 2940.058329] [ 2940.058332] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2940.058337] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2940.058342] do_raw_spin_lock+0x177/0x190 [ 2940.058347] _raw_spin_lock+0x21/0x30 [ 2940.058353] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2940.058895] ? do_raw_spin_unlock+0x75/0x190 [ 2940.058901] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2940.064428] Modules linked in: [ 2940.094101] ? do_wait_intr+0xf0/0xf0 [ 2940.094137] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2940.108752] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2940.108829] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2940.108837] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2940.108840] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2940.108854] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2940.108858] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2940.108863] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 2940.108865] RDX: 0000000000001505 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 2940.108867] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 2940.108870] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 2940.108873] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2940.108876] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 2940.108879] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2940.108882] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2940.108891] Call Trace: [ 2940.108897] [ 2940.108903] ? show_regs.cold.9+0x22/0x2f [ 2940.108914] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2940.108920] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2940.108924] ? watchdog+0x40/0x40 [ 2940.108928] ? hrtimer_interrupt+0x128/0x2c0 [ 2940.108932] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2940.108942] ? apic_timer_interrupt+0xf/0x20 [ 2940.108947] [ 2940.108950] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2940.108956] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2940.108961] do_raw_spin_lock+0x177/0x190 [ 2940.108965] _raw_spin_lock+0x21/0x30 [ 2940.108971] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2940.109334] obd_mod_free+0x46/0x140 [ptlrpc] [ 2940.109547] mdc_commit_open+0x91/0xb0 [mdc] [ 2940.109728] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2940.114949] mdc_reint+0x2d/0x110 [mdc] [ 2940.115003] mdc_unlink+0x658/0x870 [mdc] [ 2940.115031] lmv_unlink+0x34d/0xbd0 [lmv] [ 2940.115062] ll_unlink+0x24d/0x7f0 [lustre] [ 2940.115189] vfs_unlink+0x202/0x330 [ 2940.115198] do_unlinkat+0x2fa/0x460 [ 2940.115204] __x64_sys_unlink+0x37/0x50 [ 2940.115207] do_syscall_64+0xc1/0x440 [ 2940.115214] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2940.115222] RIP: 0033:0x7f151bdee10b [ 2940.115229] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2940.115233] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2940.115238] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 2940.115241] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 2940.115243] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2940.115245] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2940.115248] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2940.411620] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2940.414547] after_reply+0xe2c/0x1780 [ptlrpc] [ 2940.417576] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2940.420224] ? __next_timer_interrupt+0x160/0x160 [ 2940.422397] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2940.424868] ? do_wait_intr+0xf0/0xf0 [ 2940.426625] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2940.428820] kthread+0x1d1/0x200 [ 2940.430139] ? set_kthread_struct+0x70/0x70 [ 2940.433227] ret_from_fork+0x1f/0x30 [ 2944.056281] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2944.060041] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2944.098062] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2944.110699] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2944.120741] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2944.124949] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2944.129399] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2944.147994] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2944.155509] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2944.161482] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 2944.172716] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 2944.183884] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 2944.204610] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 2944.209850] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 2944.217365] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2944.221731] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2944.225975] Call Trace: [ 2944.228887] [ 2944.230993] ? show_regs.cold.9+0x22/0x2f [ 2944.232917] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2944.237755] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2944.241488] ? watchdog+0x40/0x40 [ 2944.245665] ? hrtimer_interrupt+0x128/0x2c0 [ 2944.250170] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2944.253348] ? apic_timer_interrupt+0xf/0x20 [ 2944.258013] [ 2944.258856] ? native_safe_halt+0x2e/0x50 [ 2944.260244] kvm_wait+0xcd/0xe0 [ 2944.262433] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2944.264798] do_raw_spin_lock+0x177/0x190 [ 2944.270808] _raw_spin_lock+0x21/0x30 [ 2944.273585] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2944.279310] process_one_work+0x2c8/0x700 [ 2944.281747] worker_thread+0x296/0x6e0 [ 2944.284238] ? rescuer_thread+0x570/0x570 [ 2944.288429] kthread+0x1d1/0x200 [ 2944.290481] ? set_kthread_struct+0x70/0x70 [ 2944.293264] ret_from_fork+0x1f/0x30 [ 2944.996818] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1014s! [ 2945.005026] Showing busy workqueues and worker pools: [ 2945.013199] workqueue events: flags=0x0 [ 2945.015108] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2945.021789] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2945.026087] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2945.030313] pending: kfree_rcu_monitor [ 2945.037882] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2945.040711] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 2945.046560] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2945.049918] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2945.055456] workqueue events_long: flags=0x0 [ 2945.057411] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2945.061072] pending: lu_objects_flush [obdclass] [ 2945.063912] workqueue events_unbound: flags=0x2 [ 2945.066069] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2945.069470] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2945.074063] workqueue events_power_efficient: flags=0x80 [ 2945.076480] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2945.079579] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 2945.082439] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2945.085445] pending: neigh_periodic_work [ 2945.087302] workqueue mm_percpu_wq: flags=0x8 [ 2945.089129] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2945.092047] pending: vmstat_update, lru_add_drain_per_cpu [ 2945.094898] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2945.097667] pending: vmstat_update [ 2945.099591] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2945.103452] pending: lru_add_drain_per_cpu [ 2945.106334] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2945.109643] pending: lru_add_drain_per_cpu BAR(48) [ 2945.112921] workqueue writeback: flags=0x4a [ 2945.115332] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2945.118983] pending: wb_workfn [ 2945.120811] workqueue xprtiod: flags=0xa [ 2945.122903] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2945.126207] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 2945.136449] workqueue ipv6_addrconf: flags=0x40008 [ 2945.144189] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2945.156180] pending: addrconf_verify_work [ 2945.160955] workqueue knbd0-recv: flags=0x1a [ 2945.164767] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2945.168968] in-flight: 658:recv_work [ 2945.172108] workqueue ptlrpc_pinger: flags=0x2 [ 2945.175824] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2945.180476] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2945.187882] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1014s workers=4 idle: 32984 37 8 [ 2945.195287] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2929s workers=2 idle: 661 [ 2956.057034] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 2956.060765] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2956.097295] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2956.101795] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2956.104362] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2956.106284] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2956.111516] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2956.114205] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2956.116562] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 2956.118810] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2956.120919] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 2956.124025] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 2956.126256] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 2956.129261] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2956.131591] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 2956.134890] Call Trace: [ 2956.136164] [ 2956.137222] ? show_regs.cold.9+0x22/0x2f [ 2956.139032] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2956.145444] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2956.147032] ? watchdog+0x40/0x40 [ 2956.149561] ? hrtimer_interrupt+0x128/0x2c0 [ 2956.152341] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2956.155327] ? apic_timer_interrupt+0xf/0x20 [ 2956.158190] [ 2956.164196] ? native_safe_halt+0x2e/0x50 [ 2956.170248] kvm_wait+0xcd/0xe0 [ 2956.174942] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2956.180944] do_raw_spin_lock+0x177/0x190 [ 2956.183348] _raw_spin_lock+0x21/0x30 [ 2956.184605] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2956.188888] ? do_raw_spin_unlock+0x75/0x190 [ 2956.192813] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2956.197804] ? do_wait_intr+0xf0/0xf0 [ 2956.201791] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2956.206726] mdc_close+0x53b/0xd70 [mdc] [ 2956.208250] lmv_close+0x18d/0x490 [lmv] [ 2956.209958] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2956.213227] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2956.217940] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2956.221161] ll_md_close+0x274/0x800 [lustre] [ 2956.224801] ll_file_release+0x128/0xc50 [lustre] [ 2956.231736] __fput+0x10d/0x440 [ 2956.232873] ____fput+0x12/0x20 [ 2956.235022] task_work_run+0xc6/0x110 [ 2956.236752] exit_to_usermode_loop+0x1dd/0x1f0 [ 2956.239317] do_syscall_64+0x3d6/0x440 [ 2956.241703] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2956.245738] RIP: 0033:0x7f151bee3cd8 [ 2956.247796] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2956.255032] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2956.259020] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 2956.267126] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 2956.275018] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2956.281319] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2956.287868] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2968.056106] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 2968.064341] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2968.105692] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2968.116271] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2968.122628] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2968.125549] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2968.150687] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2968.160021] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 2968.168413] RDX: 0000000000004182 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 2968.173487] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 2968.177724] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 2968.184677] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2968.187992] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 2968.191728] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2968.194191] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2968.196215] Call Trace: [ 2968.198419] [ 2968.200595] ? show_regs.cold.9+0x22/0x2f [ 2968.205647] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2968.208454] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2968.211061] ? watchdog+0x40/0x40 [ 2968.212739] ? hrtimer_interrupt+0x128/0x2c0 [ 2968.214743] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2968.217542] ? apic_timer_interrupt+0xf/0x20 [ 2968.219678] [ 2968.221859] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2968.224858] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2968.227618] do_raw_spin_lock+0x177/0x190 [ 2968.229677] _raw_spin_lock+0x21/0x30 [ 2968.231579] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2968.233905] obd_mod_free+0x46/0x140 [ptlrpc] [ 2968.237760] mdc_commit_open+0x91/0xb0 [mdc] [ 2968.240996] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2968.244232] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2968.251107] after_reply+0xe2c/0x1780 [ptlrpc] [ 2968.254776] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2968.260446] ? __next_timer_interrupt+0x160/0x160 [ 2968.263018] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2968.265592] ? do_wait_intr+0xf0/0xf0 [ 2968.272352] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2968.275708] kthread+0x1d1/0x200 [ 2968.278088] ? set_kthread_struct+0x70/0x70 [ 2968.281014] ret_from_fork+0x1f/0x30 [ 2972.057662] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 2972.059663] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 2972.066794] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2972.066869] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2972.066877] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2972.066880] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2972.066895] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2972.066900] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2972.066904] RAX: 0000000000003de4 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2972.066907] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 2972.066909] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 2972.066911] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 2972.066913] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 2972.066916] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 2972.066919] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2972.066922] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 2972.066931] Call Trace: [ 2972.066936] [ 2972.066942] ? show_regs.cold.9+0x22/0x2f [ 2972.066951] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2972.066958] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2972.066962] ? watchdog+0x40/0x40 [ 2972.066966] ? hrtimer_interrupt+0x128/0x2c0 [ 2972.066970] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2972.066979] ? apic_timer_interrupt+0xf/0x20 [ 2972.066983] [ 2972.066986] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2972.066990] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2972.066995] do_raw_spin_lock+0x177/0x190 [ 2972.066999] _raw_spin_lock+0x21/0x30 [ 2972.067005] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2972.067428] ? do_raw_spin_unlock+0x75/0x190 [ 2972.067434] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2972.067697] ? do_wait_intr+0xf0/0xf0 [ 2972.067704] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2972.072254] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2972.072341] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2972.072348] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2972.075976] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2972.094737] mdc_reint+0x2d/0x110 [mdc] [ 2972.094789] mdc_unlink+0x658/0x870 [mdc] [ 2972.094815] lmv_unlink+0x34d/0xbd0 [lmv] [ 2972.099995] [ 2972.100004] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2972.100022] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2972.100026] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2972.100031] RAX: 000000000000638c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2972.100033] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 2972.100035] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 2972.100037] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 2972.100040] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 2972.100043] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 2972.100046] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2972.100048] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2972.100057] Call Trace: [ 2972.100063] [ 2972.100068] ? show_regs.cold.9+0x22/0x2f [ 2972.100078] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2972.100084] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2972.100089] ? watchdog+0x40/0x40 [ 2972.100094] ? hrtimer_interrupt+0x128/0x2c0 [ 2972.100098] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2972.100105] ? apic_timer_interrupt+0xf/0x20 [ 2972.100110] [ 2972.100113] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2972.100119] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2972.100127] do_raw_spin_lock+0x177/0x190 [ 2972.100132] _raw_spin_lock+0x21/0x30 [ 2972.100137] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2972.100732] process_one_work+0x2c8/0x700 [ 2972.100739] worker_thread+0x296/0x6e0 [ 2972.100744] ? rescuer_thread+0x570/0x570 [ 2972.100749] kthread+0x1d1/0x200 [ 2972.100753] ? set_kthread_struct+0x70/0x70 [ 2972.100757] ret_from_fork+0x1f/0x30 [ 2972.393035] ll_unlink+0x24d/0x7f0 [lustre] [ 2972.394835] vfs_unlink+0x202/0x330 [ 2972.396424] do_unlinkat+0x2fa/0x460 [ 2972.397973] __x64_sys_unlink+0x37/0x50 [ 2972.400590] do_syscall_64+0xc1/0x440 [ 2972.402717] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2972.404846] RIP: 0033:0x7f151bdee10b [ 2972.408489] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2972.418050] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2972.422069] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 2972.425834] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 2972.431633] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2972.435055] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2972.438926] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2975.202536] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1044s! [ 2975.210732] Showing busy workqueues and worker pools: [ 2975.216623] workqueue events: flags=0x0 [ 2975.221711] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2975.225735] pending: cpu_latency_work [ptlrpc], push_to_pool [ 2975.231546] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2975.235307] pending: kfree_rcu_monitor [ 2975.240865] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2975.246815] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 2975.255857] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2975.263095] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 2975.271098] workqueue events_long: flags=0x0 [ 2975.274943] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2975.277798] pending: lu_objects_flush [obdclass] [ 2975.280080] workqueue events_unbound: flags=0x2 [ 2975.281765] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 2975.284588] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 2975.289017] workqueue events_power_efficient: flags=0x80 [ 2975.291367] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2975.294617] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 2975.300108] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2975.305798] pending: neigh_periodic_work [ 2975.308878] workqueue mm_percpu_wq: flags=0x8 [ 2975.313686] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2975.318909] pending: vmstat_update, lru_add_drain_per_cpu [ 2975.324948] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2975.332661] pending: vmstat_update [ 2975.333837] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2975.339568] pending: lru_add_drain_per_cpu [ 2975.342992] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2975.349563] pending: lru_add_drain_per_cpu BAR(48) [ 2975.353807] workqueue writeback: flags=0x4a [ 2975.357891] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2975.364834] pending: wb_workfn [ 2975.367416] workqueue xprtiod: flags=0xa [ 2975.372036] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 2975.377396] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 2975.384882] workqueue ipv6_addrconf: flags=0x40008 [ 2975.388159] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2975.391576] pending: addrconf_verify_work [ 2975.393869] workqueue knbd0-recv: flags=0x1a [ 2975.395869] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2975.398986] in-flight: 658:recv_work [ 2975.400893] workqueue ptlrpc_pinger: flags=0x2 [ 2975.403013] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2975.406060] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 2975.409420] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1044s workers=4 idle: 32984 37 8 [ 2975.412938] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2960s workers=2 idle: 661 [ 2984.057324] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 2984.063595] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2984.101259] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2984.110509] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2984.115437] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2984.123548] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2984.133687] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2984.139090] RAX: 0000000000000783 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 2984.144590] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 2984.151621] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2984.159131] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 2984.163584] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 2984.167494] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 2984.173315] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2984.176638] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 2984.179611] Call Trace: [ 2984.181631] [ 2984.182640] ? show_regs.cold.9+0x22/0x2f [ 2984.184535] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2984.186469] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2984.188054] ? watchdog+0x40/0x40 [ 2984.189461] ? hrtimer_interrupt+0x128/0x2c0 [ 2984.191358] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2984.193072] ? apic_timer_interrupt+0xf/0x20 [ 2984.196058] [ 2984.196791] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2984.200945] do_raw_spin_lock+0x177/0x190 [ 2984.202532] _raw_spin_lock+0x21/0x30 [ 2984.204798] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2984.207209] ? do_raw_spin_unlock+0x75/0x190 [ 2984.210534] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2984.212673] ? do_wait_intr+0xf0/0xf0 [ 2984.214237] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2984.216096] mdc_close+0x53b/0xd70 [mdc] [ 2984.218463] lmv_close+0x18d/0x490 [lmv] [ 2984.219487] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2984.222511] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 2984.226547] ll_md_real_close+0xa5/0x2a0 [lustre] [ 2984.228547] ll_md_close+0x274/0x800 [lustre] [ 2984.230691] ll_file_release+0x128/0xc50 [lustre] [ 2984.233690] __fput+0x10d/0x440 [ 2984.235061] ____fput+0x12/0x20 [ 2984.237225] task_work_run+0xc6/0x110 [ 2984.239694] exit_to_usermode_loop+0x1dd/0x1f0 [ 2984.242396] do_syscall_64+0x3d6/0x440 [ 2984.243858] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2984.245330] RIP: 0033:0x7f151bee3cd8 [ 2984.247328] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 2984.256013] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 2984.261231] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 2984.268069] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 2984.272390] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 2984.276563] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 2984.281498] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2996.055075] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 2996.059198] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2996.096470] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2996.116645] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2996.128108] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2996.129807] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2996.151195] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2996.159311] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 2996.168063] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 2996.171464] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 2996.174343] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 2996.180164] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2996.188027] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 2996.193632] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2996.196282] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 2996.205415] Call Trace: [ 2996.207357] [ 2996.208460] ? show_regs.cold.9+0x22/0x2f [ 2996.210646] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2996.217813] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2996.221083] ? watchdog+0x40/0x40 [ 2996.227435] ? hrtimer_interrupt+0x128/0x2c0 [ 2996.230463] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2996.234646] ? apic_timer_interrupt+0xf/0x20 [ 2996.236596] [ 2996.237542] ? native_safe_halt+0x2e/0x50 [ 2996.239364] kvm_wait+0xcd/0xe0 [ 2996.240990] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2996.243470] do_raw_spin_lock+0x177/0x190 [ 2996.246163] _raw_spin_lock+0x21/0x30 [ 2996.247373] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2996.251809] obd_mod_free+0x46/0x140 [ptlrpc] [ 2996.258652] mdc_commit_open+0x91/0xb0 [mdc] [ 2996.260589] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2996.266682] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2996.270763] after_reply+0xe2c/0x1780 [ptlrpc] [ 2996.273202] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2996.279527] ? __next_timer_interrupt+0x160/0x160 [ 2996.284699] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2996.290396] ? do_wait_intr+0xf0/0xf0 [ 2996.298271] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2996.301303] kthread+0x1d1/0x200 [ 2996.303983] ? set_kthread_struct+0x70/0x70 [ 2996.305686] ret_from_fork+0x1f/0x30 [ 3000.053012] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 3000.056195] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 3000.058333] Modules linked in: lustre(O) [ 3000.061979] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) [ 3000.066174] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3000.066248] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3000.066255] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3000.066258] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3000.066270] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3000.066274] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 [ 3000.078711] fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3000.078775] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3000.078783] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3000.078831] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3000.079145] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3000.079155] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3000.097942] ORIG_RAX: ffffffffffffff13 [ 3000.097952] RAX: 0000000000004c23 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3000.098547] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 3000.098552] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 3000.098555] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 3000.098557] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 3000.098561] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 3000.098564] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3000.098567] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 3000.098576] Call Trace: [ 3000.098582] [ 3000.098588] ? show_regs.cold.9+0x22/0x2f [ 3000.098601] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3000.098609] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3000.098613] ? watchdog+0x40/0x40 [ 3000.098617] ? hrtimer_interrupt+0x128/0x2c0 [ 3000.098621] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3000.098630] ? apic_timer_interrupt+0xf/0x20 [ 3000.098634] [ 3000.098637] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3000.098643] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3000.098648] do_raw_spin_lock+0x177/0x190 [ 3000.098653] _raw_spin_lock+0x21/0x30 [ 3000.098660] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3000.098935] ? do_raw_spin_unlock+0x75/0x190 [ 3000.098940] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3000.099234] ? do_wait_intr+0xf0/0xf0 [ 3000.099242] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3000.099445] mdc_reint+0x2d/0x110 [mdc] [ 3000.099486] mdc_unlink+0x658/0x870 [mdc] [ 3000.099512] lmv_unlink+0x34d/0xbd0 [lmv] [ 3000.112165] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3000.112179] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3000.112181] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 3000.112183] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 3000.112185] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 3000.112187] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 3000.112190] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 3000.112211] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3000.112213] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3000.112221] Call Trace: [ 3000.112228] [ 3000.112233] ? show_regs.cold.9+0x22/0x2f [ 3000.112245] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3000.112251] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3000.112255] ? watchdog+0x40/0x40 [ 3000.112260] ? hrtimer_interrupt+0x128/0x2c0 [ 3000.112264] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3000.112273] ? apic_timer_interrupt+0xf/0x20 [ 3000.112285] [ 3000.112288] ? native_safe_halt+0x2e/0x50 [ 3000.112292] kvm_wait+0xcd/0xe0 [ 3000.112300] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3000.112310] do_raw_spin_lock+0x177/0x190 [ 3000.112314] _raw_spin_lock+0x21/0x30 [ 3000.112318] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3000.112596] process_one_work+0x2c8/0x700 [ 3000.112602] worker_thread+0x296/0x6e0 [ 3000.112606] ? rescuer_thread+0x570/0x570 [ 3000.112609] kthread+0x1d1/0x200 [ 3000.112613] ? set_kthread_struct+0x70/0x70 [ 3000.112617] ret_from_fork+0x1f/0x30 [ 3000.362114] ll_unlink+0x24d/0x7f0 [lustre] [ 3000.365767] vfs_unlink+0x202/0x330 [ 3000.367205] do_unlinkat+0x2fa/0x460 [ 3000.370548] __x64_sys_unlink+0x37/0x50 [ 3000.374111] do_syscall_64+0xc1/0x440 [ 3000.375654] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3000.377892] RIP: 0033:0x7f151bdee10b [ 3000.379279] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3000.390406] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3000.393144] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 3000.400489] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 3000.403605] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3000.406852] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3000.411355] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3005.923092] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1075s! [ 3005.927816] Showing busy workqueues and worker pools: [ 3005.930310] workqueue events: flags=0x0 [ 3005.932819] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3005.940887] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3005.943721] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3005.953460] pending: kfree_rcu_monitor [ 3005.959857] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3005.963877] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 3005.971408] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3005.977416] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3005.982306] workqueue events_long: flags=0x0 [ 3005.985483] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3005.988462] pending: lu_objects_flush [obdclass] [ 3005.992817] workqueue events_unbound: flags=0x2 [ 3005.995843] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3006.000130] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3006.008195] workqueue events_power_efficient: flags=0x80 [ 3006.014360] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3006.016749] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 3006.022775] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3006.028120] pending: neigh_periodic_work [ 3006.029289] workqueue mm_percpu_wq: flags=0x8 [ 3006.032105] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3006.036324] pending: vmstat_update, lru_add_drain_per_cpu [ 3006.041868] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3006.048161] pending: vmstat_update [ 3006.050835] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3006.055789] pending: lru_add_drain_per_cpu [ 3006.059960] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3006.064835] pending: lru_add_drain_per_cpu BAR(48) [ 3006.067317] workqueue writeback: flags=0x4a [ 3006.070356] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3006.073374] pending: wb_workfn [ 3006.074673] workqueue xprtiod: flags=0xa [ 3006.076525] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3006.078870] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3006.084471] workqueue ipv6_addrconf: flags=0x40008 [ 3006.091737] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3006.095785] pending: addrconf_verify_work [ 3006.097201] workqueue knbd0-recv: flags=0x1a [ 3006.101389] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3006.105649] in-flight: 658:recv_work [ 3006.106691] workqueue ptlrpc_pinger: flags=0x2 [ 3006.108501] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3006.110620] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3006.116391] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1075s workers=4 idle: 32984 37 8 [ 3006.122155] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2990s workers=2 idle: 661 [ 3012.057614] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 3012.062854] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3012.100574] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3012.104660] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3012.113390] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3012.116443] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3012.126005] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3012.129722] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3012.133273] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 3012.138743] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3012.145055] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 3012.148321] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 3012.157572] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 3012.166483] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3012.172085] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 3012.178304] Call Trace: [ 3012.179781] [ 3012.181063] ? show_regs.cold.9+0x22/0x2f [ 3012.183466] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3012.186284] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3012.188500] ? watchdog+0x40/0x40 [ 3012.190579] ? hrtimer_interrupt+0x128/0x2c0 [ 3012.193822] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3012.196995] ? apic_timer_interrupt+0xf/0x20 [ 3012.199301] [ 3012.200415] ? native_safe_halt+0x2e/0x50 [ 3012.202991] kvm_wait+0xcd/0xe0 [ 3012.204028] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3012.206120] do_raw_spin_lock+0x177/0x190 [ 3012.208148] _raw_spin_lock+0x21/0x30 [ 3012.209440] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3012.211485] ? do_raw_spin_unlock+0x75/0x190 [ 3012.212966] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3012.215186] ? do_wait_intr+0xf0/0xf0 [ 3012.216546] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3012.218482] mdc_close+0x53b/0xd70 [mdc] [ 3012.219739] lmv_close+0x18d/0x490 [lmv] [ 3012.220690] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3012.222449] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3012.223796] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3012.225089] ll_md_close+0x274/0x800 [lustre] [ 3012.226474] ll_file_release+0x128/0xc50 [lustre] [ 3012.227990] __fput+0x10d/0x440 [ 3012.228959] ____fput+0x12/0x20 [ 3012.230474] task_work_run+0xc6/0x110 [ 3012.232818] exit_to_usermode_loop+0x1dd/0x1f0 [ 3012.234830] do_syscall_64+0x3d6/0x440 [ 3012.236578] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3012.240923] RIP: 0033:0x7f151bee3cd8 [ 3012.242319] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3012.252612] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3012.256609] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 3012.260436] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 3012.264134] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3012.270563] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3012.275066] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3024.055547] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 3024.064468] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3024.101807] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3024.114962] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3024.122673] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3024.129213] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3024.145241] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3024.151711] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 3024.159721] RDX: 00000000000003a3 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 3024.165730] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 3024.172482] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 3024.180076] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3024.183741] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 3024.187545] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3024.190095] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3024.193214] Call Trace: [ 3024.194282] [ 3024.195191] ? show_regs.cold.9+0x22/0x2f [ 3024.196680] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3024.203720] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3024.209518] ? watchdog+0x40/0x40 [ 3024.215149] ? hrtimer_interrupt+0x128/0x2c0 [ 3024.221055] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3024.227392] ? apic_timer_interrupt+0xf/0x20 [ 3024.234059] [ 3024.239313] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3024.247394] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3024.253022] do_raw_spin_lock+0x177/0x190 [ 3024.257519] _raw_spin_lock+0x21/0x30 [ 3024.261613] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3024.264764] obd_mod_free+0x46/0x140 [ptlrpc] [ 3024.269941] mdc_commit_open+0x91/0xb0 [mdc] [ 3024.273527] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3024.281543] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3024.292196] after_reply+0xe2c/0x1780 [ptlrpc] [ 3024.298902] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3024.309490] ? __next_timer_interrupt+0x160/0x160 [ 3024.313785] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3024.318348] ? do_wait_intr+0xf0/0xf0 [ 3024.322421] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3024.325190] kthread+0x1d1/0x200 [ 3024.328704] ? set_kthread_struct+0x70/0x70 [ 3024.331385] ret_from_fork+0x1f/0x30 [ 3028.056277] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 3028.061286] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3028.100898] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3028.115492] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3028.126918] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3028.138373] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3028.144489] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3028.164251] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3028.170879] RAX: 0000000000007f91 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3028.178134] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 3028.185258] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 3028.191259] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 3028.195213] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 3028.199451] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 3028.208240] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3028.212237] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3028.216194] Call Trace: [ 3028.216935] [ 3028.217650] ? show_regs.cold.9+0x22/0x2f [ 3028.220194] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3028.222315] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3028.224572] ? watchdog+0x40/0x40 [ 3028.225555] ? hrtimer_interrupt+0x128/0x2c0 [ 3028.232333] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3028.234553] ? apic_timer_interrupt+0xf/0x20 [ 3028.236492] [ 3028.237791] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3028.241940] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3028.244568] do_raw_spin_lock+0x177/0x190 [ 3028.246279] _raw_spin_lock+0x21/0x30 [ 3028.249021] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3028.252613] process_one_work+0x2c8/0x700 [ 3028.254903] worker_thread+0x296/0x6e0 [ 3028.256636] ? rescuer_thread+0x570/0x570 [ 3028.258575] kthread+0x1d1/0x200 [ 3028.260038] ? set_kthread_struct+0x70/0x70 [ 3028.262423] ret_from_fork+0x1f/0x30 [ 3032.054134] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 3032.057480] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3032.093621] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3032.098631] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3032.104116] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3032.111891] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3032.123560] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3032.133250] RAX: 0000000000006d9a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3032.135767] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 3032.143435] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 3032.146519] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 3032.149285] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 3032.152349] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 3032.156629] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3032.159106] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 3032.170267] Call Trace: [ 3032.172612] [ 3032.173703] ? show_regs.cold.9+0x22/0x2f [ 3032.176066] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3032.179667] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3032.181030] ? watchdog+0x40/0x40 [ 3032.181957] ? hrtimer_interrupt+0x128/0x2c0 [ 3032.186759] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3032.188796] ? apic_timer_interrupt+0xf/0x20 [ 3032.190468] [ 3032.193123] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3032.195246] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3032.199307] do_raw_spin_lock+0x177/0x190 [ 3032.201386] _raw_spin_lock+0x21/0x30 [ 3032.207391] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3032.212122] ? do_raw_spin_unlock+0x75/0x190 [ 3032.214573] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3032.216754] ? do_wait_intr+0xf0/0xf0 [ 3032.218293] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3032.220124] mdc_reint+0x2d/0x110 [mdc] [ 3032.221278] mdc_unlink+0x658/0x870 [mdc] [ 3032.222423] lmv_unlink+0x34d/0xbd0 [lmv] [ 3032.224437] ll_unlink+0x24d/0x7f0 [lustre] [ 3032.228950] vfs_unlink+0x202/0x330 [ 3032.230107] do_unlinkat+0x2fa/0x460 [ 3032.231517] __x64_sys_unlink+0x37/0x50 [ 3032.232550] do_syscall_64+0xc1/0x440 [ 3032.233582] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3032.238090] RIP: 0033:0x7f151bdee10b [ 3032.243292] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3032.251746] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3032.254538] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 3032.264860] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 3032.268341] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3032.271556] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3032.274624] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3036.128045] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1105s! [ 3036.130946] Showing busy workqueues and worker pools: [ 3036.136973] workqueue events: flags=0x0 [ 3036.138284] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3036.151124] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3036.157280] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3036.160630] pending: kfree_rcu_monitor [ 3036.163887] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3036.166495] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 3036.173367] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3036.179500] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3036.184011] workqueue events_long: flags=0x0 [ 3036.189487] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3036.196140] pending: lu_objects_flush [obdclass] [ 3036.200061] workqueue events_unbound: flags=0x2 [ 3036.202255] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3036.204878] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3036.211780] workqueue events_power_efficient: flags=0x80 [ 3036.213395] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3036.217384] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 3036.220632] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3036.224570] pending: neigh_periodic_work [ 3036.226488] workqueue mm_percpu_wq: flags=0x8 [ 3036.228162] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3036.236046] pending: vmstat_update, lru_add_drain_per_cpu [ 3036.239333] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3036.246294] pending: vmstat_update [ 3036.252039] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3036.254251] pending: lru_add_drain_per_cpu [ 3036.255369] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3036.259094] pending: lru_add_drain_per_cpu BAR(48) [ 3036.261474] workqueue writeback: flags=0x4a [ 3036.267373] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3036.272469] pending: wb_workfn [ 3036.274956] workqueue xprtiod: flags=0xa [ 3036.276452] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3036.281819] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3036.286396] workqueue ipv6_addrconf: flags=0x40008 [ 3036.289792] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3036.296586] pending: addrconf_verify_work [ 3036.300019] workqueue knbd0-recv: flags=0x1a [ 3036.303200] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3036.310940] in-flight: 658:recv_work [ 3036.314459] workqueue ptlrpc_pinger: flags=0x2 [ 3036.317571] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3036.325339] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3036.330833] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1105s workers=4 idle: 32984 37 8 [ 3036.338396] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3021s workers=2 idle: 661 [ 3040.057013] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 3040.062064] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3040.080924] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3040.085554] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3040.088983] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 3040.091476] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 3040.097665] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3040.101114] RAX: 00000000000060bd RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3040.104494] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 3040.106853] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3040.109105] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 3040.111972] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 3040.115848] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 3040.119371] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3040.121672] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 3040.124756] Call Trace: [ 3040.126452] [ 3040.127316] ? show_regs.cold.9+0x22/0x2f [ 3040.129153] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3040.131213] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3040.133069] ? watchdog+0x40/0x40 [ 3040.135213] ? hrtimer_interrupt+0x128/0x2c0 [ 3040.137507] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3040.139799] ? apic_timer_interrupt+0xf/0x20 [ 3040.141855] [ 3040.142643] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 3040.144980] do_raw_spin_lock+0x177/0x190 [ 3040.146549] _raw_spin_lock+0x21/0x30 [ 3040.148210] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3040.150597] ? do_raw_spin_unlock+0x75/0x190 [ 3040.152635] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3040.154989] ? do_wait_intr+0xf0/0xf0 [ 3040.156800] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3040.159177] mdc_close+0x53b/0xd70 [mdc] [ 3040.160829] lmv_close+0x18d/0x490 [lmv] [ 3040.162970] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3040.165336] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3040.167767] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3040.169979] ll_md_close+0x274/0x800 [lustre] [ 3040.171995] ll_file_release+0x128/0xc50 [lustre] [ 3040.174380] __fput+0x10d/0x440 [ 3040.176497] ____fput+0x12/0x20 [ 3040.177836] task_work_run+0xc6/0x110 [ 3040.179385] exit_to_usermode_loop+0x1dd/0x1f0 [ 3040.182644] do_syscall_64+0x3d6/0x440 [ 3040.183969] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3040.187015] RIP: 0033:0x7f151bee3cd8 [ 3040.188537] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3040.198204] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3040.202257] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 3040.205982] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 3040.210677] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3040.214501] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3040.217843] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3052.060352] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 3052.070314] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3052.119149] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3052.131968] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3052.140296] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3052.144163] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3052.159338] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3052.166449] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 3052.171700] RDX: 0000000000000d85 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 3052.178938] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 3052.181559] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 3052.189415] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3052.195383] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 3052.200269] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3052.209622] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3052.214755] Call Trace: [ 3052.216653] [ 3052.218676] ? show_regs.cold.9+0x22/0x2f [ 3052.226759] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3052.229190] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3052.230877] ? watchdog+0x40/0x40 [ 3052.233494] ? hrtimer_interrupt+0x128/0x2c0 [ 3052.235036] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3052.242375] ? apic_timer_interrupt+0xf/0x20 [ 3052.243851] [ 3052.244581] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3052.251703] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3052.255959] do_raw_spin_lock+0x177/0x190 [ 3052.259152] _raw_spin_lock+0x21/0x30 [ 3052.262875] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3052.265314] obd_mod_free+0x46/0x140 [ptlrpc] [ 3052.268425] mdc_commit_open+0x91/0xb0 [mdc] [ 3052.270560] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3052.274830] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3052.284088] after_reply+0xe2c/0x1780 [ptlrpc] [ 3052.290183] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3052.298649] ? __next_timer_interrupt+0x160/0x160 [ 3052.302895] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3052.306206] ? do_wait_intr+0xf0/0xf0 [ 3052.311028] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3052.317162] kthread+0x1d1/0x200 [ 3052.323918] ? set_kthread_struct+0x70/0x70 [ 3052.330859] ret_from_fork+0x1f/0x30 [ 3056.056058] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 3056.061081] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3056.086133] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3056.092135] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3056.098935] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3056.102328] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3056.104497] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3056.119100] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3056.121827] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3056.125979] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 3056.129323] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 3056.132606] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 3056.135900] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 3056.143085] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 3056.152948] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3056.157789] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3056.160379] Call Trace: [ 3056.161306] [ 3056.161931] ? show_regs.cold.9+0x22/0x2f [ 3056.163306] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3056.164923] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3056.166318] ? watchdog+0x40/0x40 [ 3056.168273] ? hrtimer_interrupt+0x128/0x2c0 [ 3056.169717] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3056.171159] ? apic_timer_interrupt+0xf/0x20 [ 3056.172636] [ 3056.173325] ? native_safe_halt+0x2e/0x50 [ 3056.175149] kvm_wait+0xcd/0xe0 [ 3056.177139] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3056.179877] do_raw_spin_lock+0x177/0x190 [ 3056.182151] _raw_spin_lock+0x21/0x30 [ 3056.184244] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3056.186911] process_one_work+0x2c8/0x700 [ 3056.188896] worker_thread+0x296/0x6e0 [ 3056.190826] ? rescuer_thread+0x570/0x570 [ 3056.192941] kthread+0x1d1/0x200 [ 3056.194601] ? set_kthread_struct+0x70/0x70 [ 3056.197390] ret_from_fork+0x1f/0x30 [ 3064.052612] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 3064.066641] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3064.109915] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3064.121812] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3064.131397] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3064.138531] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3064.154414] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3064.163105] RAX: 0000000000007c87 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3064.168964] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 3064.171635] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 3064.182483] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 3064.189628] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 3064.198875] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 3064.218849] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3064.226753] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 3064.234840] Call Trace: [ 3064.237543] [ 3064.238826] ? show_regs.cold.9+0x22/0x2f [ 3064.245834] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3064.252009] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3064.259984] ? watchdog+0x40/0x40 [ 3064.261839] ? hrtimer_interrupt+0x128/0x2c0 [ 3064.267871] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3064.278504] ? apic_timer_interrupt+0xf/0x20 [ 3064.280931] [ 3064.285601] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3064.295596] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3064.305742] do_raw_spin_lock+0x177/0x190 [ 3064.308210] _raw_spin_lock+0x21/0x30 [ 3064.309294] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3064.321093] ? do_raw_spin_unlock+0x75/0x190 [ 3064.331974] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3064.339202] ? do_wait_intr+0xf0/0xf0 [ 3064.344018] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3064.350761] mdc_reint+0x2d/0x110 [mdc] [ 3064.353055] mdc_unlink+0x658/0x870 [mdc] [ 3064.357004] lmv_unlink+0x34d/0xbd0 [lmv] [ 3064.360994] ll_unlink+0x24d/0x7f0 [lustre] [ 3064.363652] vfs_unlink+0x202/0x330 [ 3064.367003] do_unlinkat+0x2fa/0x460 [ 3064.370523] __x64_sys_unlink+0x37/0x50 [ 3064.372743] do_syscall_64+0xc1/0x440 [ 3064.374019] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3064.377771] RIP: 0033:0x7f151bdee10b [ 3064.379915] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3064.404819] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3064.417693] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 3064.430063] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 3064.435264] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3064.442729] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3064.450279] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3064.667201] rcu: INFO: rcu_sched self-detected stall on CPU [ 3064.669746] rcu: 1-...!: (692153 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 3064.676929] (t=1140020 jiffies g=249657 q=2751) [ 3064.678670] rcu: rcu_sched kthread starved for 1134972 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 3064.684946] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3064.691869] rcu: RCU grace-period kthread stack dump: [ 3064.694643] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3064.699643] Call Trace: [ 3064.700929] __schedule+0x351/0xcb0 [ 3064.712656] schedule+0xc0/0x180 [ 3064.717600] schedule_timeout+0xb4/0x190 [ 3064.727586] ? do_raw_spin_unlock+0x75/0x190 [ 3064.730626] ? __next_timer_interrupt+0x160/0x160 [ 3064.734060] ? prepare_to_swait_event+0xf3/0x140 [ 3064.736057] rcu_gp_kthread+0x8dd/0x15b0 [ 3064.741588] ? force_qs_rnp+0x2c0/0x2c0 [ 3064.743505] kthread+0x1d1/0x200 [ 3064.749539] ? set_kthread_struct+0x70/0x70 [ 3064.751115] ret_from_fork+0x1f/0x30 [ 3064.751988] rcu: Stack dump where RCU GP kthread last ran: [ 3064.757505] NMI backtrace for cpu 1 [ 3064.763260] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3064.771730] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3064.776824] Call Trace: [ 3064.777744] [ 3064.778356] ? dump_stack+0xbb/0x10e [ 3064.780696] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3064.788896] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3064.791960] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3064.796835] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3064.800613] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3064.803576] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 3064.807715] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 3064.810996] ? update_process_times+0x6d/0xc0 [ 3064.820257] ? tick_sched_handle.isra.18+0x27/0x90 [ 3064.824580] ? tick_sched_timer+0x53/0xb0 [ 3064.830217] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3064.834696] ? tick_sched_handle.isra.18+0x90/0x90 [ 3064.838522] ? hrtimer_interrupt+0x128/0x2c0 [ 3064.843039] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3064.847064] ? apic_timer_interrupt+0xf/0x20 [ 3064.849367] [ 3064.850002] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3064.861932] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3064.863652] ? do_raw_spin_lock+0x177/0x190 [ 3064.873147] ? _raw_spin_lock+0x21/0x30 [ 3064.874289] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3064.886000] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3064.895735] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3064.897437] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3064.919252] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3064.925093] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3064.936373] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3064.942830] ? __next_timer_interrupt+0x160/0x160 [ 3064.944849] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3064.948150] ? do_wait_intr+0xf0/0xf0 [ 3064.952459] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3064.956966] ? kthread+0x1d1/0x200 [ 3064.958846] ? set_kthread_struct+0x70/0x70 [ 3064.960640] ? ret_from_fork+0x1f/0x30 [ 3064.965140] NMI backtrace for cpu 1 [ 3064.967348] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3064.974316] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3064.982964] Call Trace: [ 3064.985983] [ 3064.988873] ? dump_stack+0xbb/0x10e [ 3064.990304] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3064.996583] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3065.000847] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3065.002450] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3065.005316] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3065.008415] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 3065.010105] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 3065.012957] ? update_process_times+0x6d/0xc0 [ 3065.015031] ? tick_sched_handle.isra.18+0x27/0x90 [ 3065.018209] ? tick_sched_timer+0x53/0xb0 [ 3065.020160] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3065.021820] ? tick_sched_handle.isra.18+0x90/0x90 [ 3065.024032] ? hrtimer_interrupt+0x128/0x2c0 [ 3065.025965] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3065.027929] ? apic_timer_interrupt+0xf/0x20 [ 3065.030003] [ 3065.030992] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3065.033267] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3065.035604] ? do_raw_spin_lock+0x177/0x190 [ 3065.037444] ? _raw_spin_lock+0x21/0x30 [ 3065.038836] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3065.041260] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3065.043681] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3065.045677] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3065.048119] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3065.050614] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3065.052550] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3065.054432] ? __next_timer_interrupt+0x160/0x160 [ 3065.056192] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3065.058173] ? do_wait_intr+0xf0/0xf0 [ 3065.059522] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3065.061634] ? kthread+0x1d1/0x200 [ 3065.062879] ? set_kthread_struct+0x70/0x70 [ 3065.064668] ? ret_from_fork+0x1f/0x30 [ 3066.852806] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1136s! [ 3066.865166] Showing busy workqueues and worker pools: [ 3066.870930] workqueue events: flags=0x0 [ 3066.874864] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3066.880410] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3066.886441] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3066.891251] pending: kfree_rcu_monitor [ 3066.892927] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3066.898948] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 3066.907137] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3066.913939] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3066.921887] workqueue events_long: flags=0x0 [ 3066.924177] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3066.926289] pending: lu_objects_flush [obdclass] [ 3066.931636] workqueue events_unbound: flags=0x2 [ 3066.932830] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3066.940211] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3066.945317] workqueue events_power_efficient: flags=0x80 [ 3066.950526] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3066.955957] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 3066.962147] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3066.968897] pending: neigh_periodic_work [ 3066.972956] workqueue mm_percpu_wq: flags=0x8 [ 3066.976918] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3066.979732] pending: vmstat_update, lru_add_drain_per_cpu [ 3066.982514] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3066.985705] pending: vmstat_update [ 3066.988454] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3066.992229] pending: lru_add_drain_per_cpu [ 3066.994802] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3066.998037] pending: lru_add_drain_per_cpu BAR(48) [ 3067.001013] workqueue writeback: flags=0x4a [ 3067.003080] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3067.006086] pending: wb_workfn [ 3067.007879] workqueue xprtiod: flags=0xa [ 3067.009824] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3067.012816] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3067.016765] workqueue ipv6_addrconf: flags=0x40008 [ 3067.019187] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3067.022527] pending: addrconf_verify_work [ 3067.024916] workqueue knbd0-recv: flags=0x1a [ 3067.026960] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3067.030012] in-flight: 658:recv_work [ 3067.031962] workqueue ptlrpc_pinger: flags=0x2 [ 3067.035224] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3067.040766] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3067.046555] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1136s workers=4 idle: 32984 37 8 [ 3067.051579] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3051s workers=2 idle: 661 [ 3068.057106] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 3068.060189] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3068.083741] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3068.088778] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3068.092413] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3068.095348] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3068.104076] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3068.107499] RAX: 0000000000000a3a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3068.112042] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 3068.116411] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3068.120267] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 3068.123665] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 3068.126711] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 3068.130960] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3068.133674] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 3068.136939] Call Trace: [ 3068.137971] [ 3068.139020] ? show_regs.cold.9+0x22/0x2f [ 3068.141017] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3068.142976] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3068.144955] ? watchdog+0x40/0x40 [ 3068.146941] ? hrtimer_interrupt+0x128/0x2c0 [ 3068.149536] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3068.152399] ? apic_timer_interrupt+0xf/0x20 [ 3068.154373] [ 3068.155458] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3068.158177] do_raw_spin_lock+0x177/0x190 [ 3068.160372] _raw_spin_lock+0x21/0x30 [ 3068.162142] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3068.164921] ? do_raw_spin_unlock+0x75/0x190 [ 3068.166979] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3068.170040] ? do_wait_intr+0xf0/0xf0 [ 3068.172398] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3068.174926] mdc_close+0x53b/0xd70 [mdc] [ 3068.176814] lmv_close+0x18d/0x490 [lmv] [ 3068.178549] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3068.181720] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3068.184399] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3068.188206] ll_md_close+0x274/0x800 [lustre] [ 3068.190717] ll_file_release+0x128/0xc50 [lustre] [ 3068.193030] __fput+0x10d/0x440 [ 3068.195214] ____fput+0x12/0x20 [ 3068.199023] task_work_run+0xc6/0x110 [ 3068.202126] exit_to_usermode_loop+0x1dd/0x1f0 [ 3068.207034] do_syscall_64+0x3d6/0x440 [ 3068.210525] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3068.217051] RIP: 0033:0x7f151bee3cd8 [ 3068.218826] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3068.239927] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3068.248940] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 3068.256796] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 3068.262162] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3068.267377] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3068.270652] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3084.056021] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 3084.058629] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3084.080770] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3084.085638] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3084.089091] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3084.091694] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3084.094362] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3084.102103] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3084.105362] RAX: 0000000000004d8c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3084.108606] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 3084.111370] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 3084.114585] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 3084.118264] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 3084.121343] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 3084.125287] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3084.127632] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3084.131857] Call Trace: [ 3084.135289] [ 3084.136517] ? show_regs.cold.9+0x22/0x2f [ 3084.140773] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3084.143752] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3084.146249] ? watchdog+0x40/0x40 [ 3084.148026] ? hrtimer_interrupt+0x128/0x2c0 [ 3084.150450] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3084.153371] ? apic_timer_interrupt+0xf/0x20 [ 3084.155630] [ 3084.156674] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3084.159671] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3084.162954] do_raw_spin_lock+0x177/0x190 [ 3084.165969] _raw_spin_lock+0x21/0x30 [ 3084.167973] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3084.171602] process_one_work+0x2c8/0x700 [ 3084.174356] worker_thread+0x296/0x6e0 [ 3084.179340] ? rescuer_thread+0x570/0x570 [ 3084.184519] kthread+0x1d1/0x200 [ 3084.186465] ? set_kthread_struct+0x70/0x70 [ 3084.191637] ret_from_fork+0x1f/0x30 [ 3092.054282] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 3092.055025] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 3092.061738] Modules linked in: [ 3092.073678] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) [ 3092.074834] lustre(O) osp(O) ofd(O) lod(O) mdt(O) [ 3092.086705] ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) [ 3092.094318] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3092.094403] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3092.094410] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3092.094413] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3092.094428] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3092.094433] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3092.094437] RAX: 000000000000290f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3092.094439] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 3092.094442] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 3092.094444] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 3092.094446] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 3092.094449] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 3092.094452] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3092.094455] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 3092.094463] Call Trace: [ 3092.094468] [ 3092.094474] ? show_regs.cold.9+0x22/0x2f [ 3092.094483] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3092.094489] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3092.094493] ? watchdog+0x40/0x40 [ 3092.094497] ? hrtimer_interrupt+0x128/0x2c0 [ 3092.094501] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3092.094510] ? apic_timer_interrupt+0xf/0x20 [ 3092.094514] [ 3092.094517] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3092.094521] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3092.094526] do_raw_spin_lock+0x177/0x190 [ 3092.094531] _raw_spin_lock+0x21/0x30 [ 3092.094536] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3092.095017] ? do_raw_spin_unlock+0x75/0x190 [ 3092.095023] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3092.095464] ? do_wait_intr+0xf0/0xf0 [ 3092.095473] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3092.096263] mdc_reint+0x2d/0x110 [mdc] [ 3092.096304] mdc_unlink+0x658/0x870 [mdc] [ 3092.096330] lmv_unlink+0x34d/0xbd0 [lmv] [ 3092.096362] ll_unlink+0x24d/0x7f0 [lustre] [ 3092.096501] vfs_unlink+0x202/0x330 [ 3092.096510] do_unlinkat+0x2fa/0x460 [ 3092.096515] __x64_sys_unlink+0x37/0x50 [ 3092.096520] do_syscall_64+0xc1/0x440 [ 3092.096527] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3092.096534] RIP: 0033:0x7f151bdee10b [ 3092.096543] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3092.096546] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3092.096551] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 3092.096554] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 3092.096555] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3092.096557] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3092.099585] obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3092.099634] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3092.099642] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3092.099644] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3092.099657] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3092.099661] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 3092.099666] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 3092.099668] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 3092.099670] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 3092.099673] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 3092.099675] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3092.099678] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 3092.099681] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3092.099683] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3092.099694] Call Trace: [ 3092.099699] [ 3092.099704] ? show_regs.cold.9+0x22/0x2f [ 3092.099715] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3092.099720] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3092.099724] ? watchdog+0x40/0x40 [ 3092.099728] ? hrtimer_interrupt+0x128/0x2c0 [ 3092.099732] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3092.099738] ? apic_timer_interrupt+0xf/0x20 [ 3092.099742] [ 3092.099745] ? native_safe_halt+0x2e/0x50 [ 3092.099749] kvm_wait+0xcd/0xe0 [ 3092.099757] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3092.099763] do_raw_spin_lock+0x177/0x190 [ 3092.099767] _raw_spin_lock+0x21/0x30 [ 3092.099773] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3092.100049] obd_mod_free+0x46/0x140 [ptlrpc] [ 3092.100263] mdc_commit_open+0x91/0xb0 [mdc] [ 3092.100315] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3092.117741] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3092.121193] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3092.121453] after_reply+0xe2c/0x1780 [ptlrpc] [ 3092.121648] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3092.121841] ? __next_timer_interrupt+0x160/0x160 [ 3092.121850] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3092.122025] ? do_wait_intr+0xf0/0xf0 [ 3092.122109] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3092.122292] kthread+0x1d1/0x200 [ 3092.122296] ? set_kthread_struct+0x70/0x70 [ 3092.122299] ret_from_fork+0x1f/0x30 [ 3096.058235] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 3096.073007] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3096.117934] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3096.126409] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3096.131900] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3096.136216] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3096.158289] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3096.168554] RAX: 0000000000002fb1 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3096.173333] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 3096.182035] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3096.189865] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 3096.206573] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 3096.221844] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 3096.233034] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3096.254731] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 3096.267101] Call Trace: [ 3096.270763] [ 3096.271550] ? show_regs.cold.9+0x22/0x2f [ 3096.278869] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3096.288998] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3096.293881] ? watchdog+0x40/0x40 [ 3096.302705] ? hrtimer_interrupt+0x128/0x2c0 [ 3096.304240] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3096.319885] ? apic_timer_interrupt+0xf/0x20 [ 3096.325988] [ 3096.329904] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3096.333151] do_raw_spin_lock+0x177/0x190 [ 3096.339935] _raw_spin_lock+0x21/0x30 [ 3096.342198] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3096.344903] ? do_raw_spin_unlock+0x75/0x190 [ 3096.347969] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3096.356411] ? do_wait_intr+0xf0/0xf0 [ 3096.367118] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3096.377670] mdc_close+0x53b/0xd70 [mdc] [ 3096.381432] lmv_close+0x18d/0x490 [lmv] [ 3096.387641] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3096.393852] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3096.403388] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3096.407178] ll_md_close+0x274/0x800 [lustre] [ 3096.423091] ll_file_release+0x128/0xc50 [lustre] [ 3096.424976] __fput+0x10d/0x440 [ 3096.435839] ____fput+0x12/0x20 [ 3096.442434] task_work_run+0xc6/0x110 [ 3096.451056] exit_to_usermode_loop+0x1dd/0x1f0 [ 3096.456346] do_syscall_64+0x3d6/0x440 [ 3096.464575] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3096.467700] RIP: 0033:0x7f151bee3cd8 [ 3096.470071] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3096.488220] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3096.494708] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 3096.505407] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 3096.510269] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3096.521150] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3096.527064] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3097.059589] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1166s! [ 3097.072792] Showing busy workqueues and worker pools: [ 3097.078792] workqueue events: flags=0x0 [ 3097.084122] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3097.092606] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3097.101230] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3097.107139] pending: kfree_rcu_monitor [ 3097.108495] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3097.113818] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 3097.122918] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3097.132785] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3097.136605] workqueue events_long: flags=0x0 [ 3097.143030] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3097.150829] pending: lu_objects_flush [obdclass] [ 3097.156980] workqueue events_unbound: flags=0x2 [ 3097.165908] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3097.174713] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3097.191721] workqueue events_power_efficient: flags=0x80 [ 3097.197635] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3097.208834] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 3097.212864] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3097.217241] pending: neigh_periodic_work [ 3097.219097] workqueue mm_percpu_wq: flags=0x8 [ 3097.224237] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3097.230792] pending: vmstat_update, lru_add_drain_per_cpu [ 3097.233896] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3097.237138] pending: vmstat_update [ 3097.239742] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3097.242968] pending: lru_add_drain_per_cpu [ 3097.245235] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3097.249054] pending: lru_add_drain_per_cpu BAR(48) [ 3097.253109] workqueue writeback: flags=0x4a [ 3097.257563] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3097.263745] pending: wb_workfn [ 3097.265445] workqueue xprtiod: flags=0xa [ 3097.269537] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3097.271863] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3097.276959] workqueue ipv6_addrconf: flags=0x40008 [ 3097.283624] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3097.290220] pending: addrconf_verify_work [ 3097.291994] workqueue knbd0-recv: flags=0x1a [ 3097.296409] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3097.301965] in-flight: 658:recv_work [ 3097.304067] workqueue ptlrpc_pinger: flags=0x2 [ 3097.305868] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3097.314136] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3097.318519] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1166s workers=4 idle: 32984 37 8 [ 3097.325259] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3082s workers=2 idle: 661 [ 3112.065648] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 3112.083042] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3112.131396] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3112.142814] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3112.156549] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3112.169143] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3112.174643] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3112.196773] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3112.209195] RAX: 0000000000006283 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3112.219085] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 3112.222171] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 3112.223812] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 3112.227637] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 3112.237438] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 3112.244749] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3112.255678] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3112.261806] Call Trace: [ 3112.267279] [ 3112.268013] ? show_regs.cold.9+0x22/0x2f [ 3112.270779] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3112.273798] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3112.277503] ? watchdog+0x40/0x40 [ 3112.278723] ? hrtimer_interrupt+0x128/0x2c0 [ 3112.282280] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3112.283990] ? apic_timer_interrupt+0xf/0x20 [ 3112.286608] [ 3112.287602] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3112.290544] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3112.293366] do_raw_spin_lock+0x177/0x190 [ 3112.295247] _raw_spin_lock+0x21/0x30 [ 3112.297026] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3112.299686] process_one_work+0x2c8/0x700 [ 3112.301532] worker_thread+0x296/0x6e0 [ 3112.303702] ? rescuer_thread+0x570/0x570 [ 3112.305748] kthread+0x1d1/0x200 [ 3112.307671] ? set_kthread_struct+0x70/0x70 [ 3112.309821] ret_from_fork+0x1f/0x30 [ 3120.055664] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 3120.065294] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3120.117472] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3120.136133] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3120.148374] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3120.155029] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3120.167533] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3120.174058] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 3120.182837] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 3120.193499] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 3120.195923] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 3120.209544] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3120.217733] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 3120.230130] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3120.235027] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3120.241513] Call Trace: [ 3120.246031] [ 3120.251564] ? show_regs.cold.9+0x22/0x2f [ 3120.253381] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3120.257528] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3120.260034] ? watchdog+0x40/0x40 [ 3120.262139] ? hrtimer_interrupt+0x128/0x2c0 [ 3120.264579] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3120.270602] ? apic_timer_interrupt+0xf/0x20 [ 3120.282238] [ 3120.282765] ? native_safe_halt+0x2e/0x50 [ 3120.284025] kvm_wait+0xcd/0xe0 [ 3120.293306] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3120.300915] do_raw_spin_lock+0x177/0x190 [ 3120.305419] _raw_spin_lock+0x21/0x30 [ 3120.307855] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3120.314917] obd_mod_free+0x46/0x140 [ptlrpc] [ 3120.323008] mdc_commit_open+0x91/0xb0 [mdc] [ 3120.331979] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3120.340156] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3120.359291] after_reply+0xe2c/0x1780 [ptlrpc] [ 3120.369474] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3120.381379] ? __next_timer_interrupt+0x160/0x160 [ 3120.382974] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3120.402426] ? do_wait_intr+0xf0/0xf0 [ 3120.411363] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3120.422876] kthread+0x1d1/0x200 [ 3120.426205] ? set_kthread_struct+0x70/0x70 [ 3120.428439] ret_from_fork+0x1f/0x30 [ 3124.056342] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 3124.058603] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 3124.058880] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3124.058964] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3124.058970] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3124.058973] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3124.058986] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3124.058989] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3124.058993] RAX: 0000000000004c85 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3124.058996] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 3124.058998] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 3124.059000] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 3124.059002] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 3124.059005] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 3124.059007] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3124.059009] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 3124.059017] Call Trace: [ 3124.059022] [ 3124.059027] ? show_regs.cold.9+0x22/0x2f [ 3124.059036] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3124.059041] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3124.059045] ? watchdog+0x40/0x40 [ 3124.059049] ? hrtimer_interrupt+0x128/0x2c0 [ 3124.059052] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3124.059060] ? apic_timer_interrupt+0xf/0x20 [ 3124.059066] [ 3124.059068] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3124.059073] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3124.059078] do_raw_spin_lock+0x177/0x190 [ 3124.059082] _raw_spin_lock+0x21/0x30 [ 3124.059088] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3124.059393] ? do_raw_spin_unlock+0x75/0x190 [ 3124.059398] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3124.059619] ? do_wait_intr+0xf0/0xf0 [ 3124.059627] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3124.059836] mdc_reint+0x2d/0x110 [mdc] [ 3124.059876] mdc_unlink+0x658/0x870 [mdc] [ 3124.062042] lmv_unlink+0x34d/0xbd0 [lmv] [ 3124.062077] ll_unlink+0x24d/0x7f0 [lustre] [ 3124.062193] vfs_unlink+0x202/0x330 [ 3124.062202] do_unlinkat+0x2fa/0x460 [ 3124.062207] __x64_sys_unlink+0x37/0x50 [ 3124.062211] do_syscall_64+0xc1/0x440 [ 3124.062217] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3124.062226] RIP: 0033:0x7f151bdee10b [ 3124.062233] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3124.062237] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3124.062242] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 3124.062245] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 3124.062246] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3124.062249] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3124.062251] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3124.232680] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3124.254149] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3124.258354] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3124.261485] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3124.263700] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3124.270538] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3124.272917] RAX: 000000000000185b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3124.275906] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 3124.278566] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3124.281347] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 3124.283970] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 3124.286630] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 3124.293584] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3124.295378] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 3124.297356] Call Trace: [ 3124.298358] [ 3124.299294] ? show_regs.cold.9+0x22/0x2f [ 3124.300676] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3124.302331] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3124.304402] ? watchdog+0x40/0x40 [ 3124.306238] ? hrtimer_interrupt+0x128/0x2c0 [ 3124.308109] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3124.311226] ? apic_timer_interrupt+0xf/0x20 [ 3124.312755] [ 3124.313577] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3124.319737] do_raw_spin_lock+0x177/0x190 [ 3124.322056] _raw_spin_lock+0x21/0x30 [ 3124.323973] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3124.331393] ? do_raw_spin_unlock+0x75/0x190 [ 3124.336154] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3124.341755] ? do_wait_intr+0xf0/0xf0 [ 3124.343950] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3124.350075] mdc_close+0x53b/0xd70 [mdc] [ 3124.351623] lmv_close+0x18d/0x490 [lmv] [ 3124.355763] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3124.359363] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3124.362506] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3124.365265] ll_md_close+0x274/0x800 [lustre] [ 3124.366958] ll_file_release+0x128/0xc50 [lustre] [ 3124.370487] __fput+0x10d/0x440 [ 3124.372543] ____fput+0x12/0x20 [ 3124.378718] task_work_run+0xc6/0x110 [ 3124.382412] exit_to_usermode_loop+0x1dd/0x1f0 [ 3124.393462] do_syscall_64+0x3d6/0x440 [ 3124.397271] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3124.404186] RIP: 0033:0x7f151bee3cd8 [ 3124.410320] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3124.427245] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3124.435449] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 3124.451703] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 3124.464843] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3124.482381] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3124.492264] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3127.776628] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1197s! [ 3127.793026] Showing busy workqueues and worker pools: [ 3127.798118] workqueue events: flags=0x0 [ 3127.804076] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3127.814173] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3127.819709] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3127.826074] pending: kfree_rcu_monitor [ 3127.830513] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3127.837763] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 3127.843490] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3127.848616] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3127.853178] workqueue events_long: flags=0x0 [ 3127.856257] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3127.859477] pending: lu_objects_flush [obdclass] [ 3127.864014] workqueue events_unbound: flags=0x2 [ 3127.868607] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3127.871514] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3127.877038] workqueue events_power_efficient: flags=0x80 [ 3127.879110] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3127.881242] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 3127.885147] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3127.888362] pending: neigh_periodic_work [ 3127.890375] workqueue mm_percpu_wq: flags=0x8 [ 3127.892393] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3127.895497] pending: vmstat_update, lru_add_drain_per_cpu [ 3127.898103] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3127.900957] pending: vmstat_update [ 3127.903131] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3127.906430] pending: lru_add_drain_per_cpu [ 3127.908931] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3127.912140] pending: lru_add_drain_per_cpu BAR(48) [ 3127.914673] workqueue writeback: flags=0x4a [ 3127.916687] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3127.919659] pending: wb_workfn [ 3127.922515] workqueue xprtiod: flags=0xa [ 3127.924041] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3127.926425] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3127.930514] workqueue ipv6_addrconf: flags=0x40008 [ 3127.932627] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3127.936121] pending: addrconf_verify_work [ 3127.938192] workqueue knbd0-recv: flags=0x1a [ 3127.940091] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3127.943192] in-flight: 658:recv_work [ 3127.945021] workqueue ptlrpc_pinger: flags=0x2 [ 3127.946992] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3127.949846] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3127.952866] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1197s workers=4 idle: 32984 37 8 [ 3127.956644] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3112s workers=2 idle: 661 [ 3140.056030] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3140.059583] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3140.093398] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3140.101224] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3140.107839] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3140.113474] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3140.117949] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3140.131587] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3140.138090] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3140.143256] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 3140.149286] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 3140.154819] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 3140.159310] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 3140.162941] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 3140.171327] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3140.176347] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3140.181848] Call Trace: [ 3140.183139] [ 3140.184329] ? show_regs.cold.9+0x22/0x2f [ 3140.187647] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3140.191335] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3140.193648] ? watchdog+0x40/0x40 [ 3140.194867] ? hrtimer_interrupt+0x128/0x2c0 [ 3140.203004] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3140.212241] ? apic_timer_interrupt+0xf/0x20 [ 3140.215377] [ 3140.217354] ? native_safe_halt+0x2e/0x50 [ 3140.220269] kvm_wait+0xcd/0xe0 [ 3140.222398] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3140.226214] do_raw_spin_lock+0x177/0x190 [ 3140.229224] _raw_spin_lock+0x21/0x30 [ 3140.232207] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3140.236226] process_one_work+0x2c8/0x700 [ 3140.239287] worker_thread+0x296/0x6e0 [ 3140.241637] ? rescuer_thread+0x570/0x570 [ 3140.244568] kthread+0x1d1/0x200 [ 3140.247177] ? set_kthread_struct+0x70/0x70 [ 3140.249256] ret_from_fork+0x1f/0x30 [ 3148.055010] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 3148.059986] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3148.099010] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3148.105123] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3148.109650] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3148.114268] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3148.125148] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3148.127897] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 3148.134006] RDX: 000000000000431e RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 3148.137216] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 3148.144880] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 3148.149999] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3148.157999] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 3148.165037] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3148.172109] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3148.181157] Call Trace: [ 3148.183700] [ 3148.185859] ? show_regs.cold.9+0x22/0x2f [ 3148.191888] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3148.195008] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3148.201870] ? watchdog+0x40/0x40 [ 3148.203805] ? hrtimer_interrupt+0x128/0x2c0 [ 3148.207562] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3148.212292] ? apic_timer_interrupt+0xf/0x20 [ 3148.219206] [ 3148.221373] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3148.224164] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3148.232930] do_raw_spin_lock+0x177/0x190 [ 3148.235221] _raw_spin_lock+0x21/0x30 [ 3148.237929] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3148.241013] obd_mod_free+0x46/0x140 [ptlrpc] [ 3148.245001] mdc_commit_open+0x91/0xb0 [mdc] [ 3148.248125] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3148.252501] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3148.259292] after_reply+0xe2c/0x1780 [ptlrpc] [ 3148.262566] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3148.266036] ? __next_timer_interrupt+0x160/0x160 [ 3148.269146] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3148.273255] ? do_wait_intr+0xf0/0xf0 [ 3148.279254] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3148.285257] kthread+0x1d1/0x200 [ 3148.286451] ? set_kthread_struct+0x70/0x70 [ 3148.292042] ret_from_fork+0x1f/0x30 [ 3152.052022] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 3152.055721] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3152.057098] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 3152.077520] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3152.077534] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3152.077537] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3152.077550] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3152.077554] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3152.077560] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3152.077637] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 3152.077640] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 3152.077642] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 3152.077644] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 3152.077648] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 3152.077650] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3152.077652] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 3152.077663] Call Trace: [ 3152.077668] [ 3152.077673] ? show_regs.cold.9+0x22/0x2f [ 3152.077683] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3152.077690] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3152.077694] ? watchdog+0x40/0x40 [ 3152.077698] ? hrtimer_interrupt+0x128/0x2c0 [ 3152.077707] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3152.077714] ? apic_timer_interrupt+0xf/0x20 [ 3152.077719] [ 3152.077722] ? native_safe_halt+0x2e/0x50 [ 3152.077726] kvm_wait+0xcd/0xe0 [ 3152.077737] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3152.077745] do_raw_spin_lock+0x177/0x190 [ 3152.077749] _raw_spin_lock+0x21/0x30 [ 3152.077759] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3152.078083] ? do_raw_spin_unlock+0x75/0x190 [ 3152.078089] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3152.085422] Modules linked in: [ 3152.091154] ? do_wait_intr+0xf0/0xf0 [ 3152.091175] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3152.091573] mdc_reint+0x2d/0x110 [mdc] [ 3152.091611] mdc_unlink+0x658/0x870 [mdc] [ 3152.091660] lmv_unlink+0x34d/0xbd0 [lmv] [ 3152.091694] ll_unlink+0x24d/0x7f0 [lustre] [ 3152.091923] vfs_unlink+0x202/0x330 [ 3152.091934] do_unlinkat+0x2fa/0x460 [ 3152.091939] __x64_sys_unlink+0x37/0x50 [ 3152.091943] do_syscall_64+0xc1/0x440 [ 3152.091950] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3152.091960] RIP: 0033:0x7f151bdee10b [ 3152.091968] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3152.091972] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3152.091978] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 3152.091980] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 3152.091982] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3152.091985] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3152.091987] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3152.222835] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3152.237286] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3152.240845] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3152.243596] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3152.245057] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3152.250572] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3152.253767] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3152.256743] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 3152.261585] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3152.265370] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 3152.268663] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 3152.271966] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 3152.275360] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3152.277891] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 3152.281178] Call Trace: [ 3152.282005] [ 3152.282715] ? show_regs.cold.9+0x22/0x2f [ 3152.284350] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3152.289399] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3152.291572] ? watchdog+0x40/0x40 [ 3152.293330] ? hrtimer_interrupt+0x128/0x2c0 [ 3152.294814] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3152.296427] ? apic_timer_interrupt+0xf/0x20 [ 3152.297985] [ 3152.298778] ? native_safe_halt+0x2e/0x50 [ 3152.300070] kvm_wait+0xcd/0xe0 [ 3152.301126] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3152.304267] do_raw_spin_lock+0x177/0x190 [ 3152.308916] _raw_spin_lock+0x21/0x30 [ 3152.311161] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3152.316079] ? do_raw_spin_unlock+0x75/0x190 [ 3152.319403] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3152.320990] ? do_wait_intr+0xf0/0xf0 [ 3152.322678] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3152.324649] mdc_close+0x53b/0xd70 [mdc] [ 3152.326243] lmv_close+0x18d/0x490 [lmv] [ 3152.330284] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3152.332781] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3152.340436] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3152.343224] ll_md_close+0x274/0x800 [lustre] [ 3152.347343] ll_file_release+0x128/0xc50 [lustre] [ 3152.349405] __fput+0x10d/0x440 [ 3152.350884] ____fput+0x12/0x20 [ 3152.351902] task_work_run+0xc6/0x110 [ 3152.353087] exit_to_usermode_loop+0x1dd/0x1f0 [ 3152.355891] do_syscall_64+0x3d6/0x440 [ 3152.357240] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3152.361059] RIP: 0033:0x7f151bee3cd8 [ 3152.363061] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3152.372463] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3152.374580] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 3152.380462] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 3152.383107] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3152.388672] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3152.390510] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3157.985052] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1227s! [ 3157.988044] Showing busy workqueues and worker pools: [ 3157.992415] workqueue events: flags=0x0 [ 3157.996328] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3158.001156] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3158.008414] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3158.017286] pending: kfree_rcu_monitor [ 3158.019643] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3158.030977] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 3158.040239] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3158.044149] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3158.048226] workqueue events_long: flags=0x0 [ 3158.049587] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3158.051855] pending: lu_objects_flush [obdclass] [ 3158.054201] workqueue events_unbound: flags=0x2 [ 3158.056141] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3158.058596] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3158.061288] workqueue events_power_efficient: flags=0x80 [ 3158.063356] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3158.066391] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 3158.070762] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3158.074117] pending: neigh_periodic_work [ 3158.076668] workqueue mm_percpu_wq: flags=0x8 [ 3158.078983] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3158.081464] pending: vmstat_update, lru_add_drain_per_cpu [ 3158.087339] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3158.097012] pending: vmstat_update [ 3158.101516] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3158.108610] pending: lru_add_drain_per_cpu [ 3158.112882] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3158.116304] pending: lru_add_drain_per_cpu BAR(48) [ 3158.119194] workqueue writeback: flags=0x4a [ 3158.120658] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3158.122706] pending: wb_workfn [ 3158.124311] workqueue xprtiod: flags=0xa [ 3158.126544] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3158.129377] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3158.133016] workqueue ipv6_addrconf: flags=0x40008 [ 3158.134461] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3158.136401] pending: addrconf_verify_work [ 3158.138045] workqueue knbd0-recv: flags=0x1a [ 3158.139743] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3158.143133] in-flight: 658:recv_work [ 3158.144992] workqueue ptlrpc_pinger: flags=0x2 [ 3158.147181] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3158.150129] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3158.153187] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1227s workers=4 idle: 32984 37 8 [ 3158.156504] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3142s workers=2 idle: 661 [ 3168.056139] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3168.059698] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3168.100133] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3168.105249] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3168.112709] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3168.119928] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3168.128575] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3168.145174] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3168.151323] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3168.154270] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 3168.157684] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 3168.161273] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 3168.164446] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 3168.167247] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 3168.170885] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3168.175633] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3168.179738] Call Trace: [ 3168.181154] [ 3168.182469] ? show_regs.cold.9+0x22/0x2f [ 3168.184480] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3168.188270] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3168.190633] ? watchdog+0x40/0x40 [ 3168.192492] ? hrtimer_interrupt+0x128/0x2c0 [ 3168.195147] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3168.197448] ? apic_timer_interrupt+0xf/0x20 [ 3168.199571] [ 3168.200449] ? native_safe_halt+0x2e/0x50 [ 3168.202274] kvm_wait+0xcd/0xe0 [ 3168.203532] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3168.205736] do_raw_spin_lock+0x177/0x190 [ 3168.207564] _raw_spin_lock+0x21/0x30 [ 3168.209393] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3168.211909] process_one_work+0x2c8/0x700 [ 3168.213665] worker_thread+0x296/0x6e0 [ 3168.214995] ? rescuer_thread+0x570/0x570 [ 3168.216826] kthread+0x1d1/0x200 [ 3168.218359] ? set_kthread_struct+0x70/0x70 [ 3168.220193] ret_from_fork+0x1f/0x30 [ 3176.055046] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 3176.064413] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3176.137131] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3176.161122] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3176.174326] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3176.181122] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3176.209451] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3176.216679] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 3176.221187] RDX: 00000000000049b0 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 3176.226187] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 3176.237483] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 3176.242129] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3176.246814] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 3176.250358] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3176.251946] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3176.255157] Call Trace: [ 3176.256238] [ 3176.256960] ? show_regs.cold.9+0x22/0x2f [ 3176.258684] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3176.260765] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3176.262824] ? watchdog+0x40/0x40 [ 3176.264236] ? hrtimer_interrupt+0x128/0x2c0 [ 3176.265954] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3176.267194] ? apic_timer_interrupt+0xf/0x20 [ 3176.269486] [ 3176.271666] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3176.276547] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3176.283231] do_raw_spin_lock+0x177/0x190 [ 3176.284664] _raw_spin_lock+0x21/0x30 [ 3176.286316] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3176.288712] obd_mod_free+0x46/0x140 [ptlrpc] [ 3176.290986] mdc_commit_open+0x91/0xb0 [mdc] [ 3176.293748] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3176.298360] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3176.308111] after_reply+0xe2c/0x1780 [ptlrpc] [ 3176.311761] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3176.314947] ? __next_timer_interrupt+0x160/0x160 [ 3176.317079] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3176.319222] ? do_wait_intr+0xf0/0xf0 [ 3176.320194] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3176.325925] kthread+0x1d1/0x200 [ 3176.327665] ? set_kthread_struct+0x70/0x70 [ 3176.333431] ret_from_fork+0x1f/0x30 [ 3180.060212] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 3180.066762] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3180.114843] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3180.120015] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3180.124496] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3180.133446] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3180.152040] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3180.157404] RAX: 0000000000007c9e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3180.161568] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 3180.167088] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3180.171985] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 3180.177248] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 3180.181325] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 3180.186178] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3180.194498] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 3180.199122] Call Trace: [ 3180.205019] [ 3180.206676] ? show_regs.cold.9+0x22/0x2f [ 3180.214942] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3180.220530] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3180.228335] ? watchdog+0x40/0x40 [ 3180.232133] ? hrtimer_interrupt+0x128/0x2c0 [ 3180.234968] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3180.236282] ? apic_timer_interrupt+0xf/0x20 [ 3180.237462] [ 3180.238081] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3180.243259] do_raw_spin_lock+0x177/0x190 [ 3180.252435] _raw_spin_lock+0x21/0x30 [ 3180.255064] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3180.258089] ? do_raw_spin_unlock+0x75/0x190 [ 3180.262208] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3180.265824] ? do_wait_intr+0xf0/0xf0 [ 3180.267429] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3180.271136] mdc_close+0x53b/0xd70 [mdc] [ 3180.274135] lmv_close+0x18d/0x490 [lmv] [ 3180.276440] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3180.278564] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3180.284099] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3180.290815] ll_md_close+0x274/0x800 [lustre] [ 3180.294413] ll_file_release+0x128/0xc50 [lustre] [ 3180.298382] __fput+0x10d/0x440 [ 3180.300840] ____fput+0x12/0x20 [ 3180.303291] task_work_run+0xc6/0x110 [ 3180.309655] exit_to_usermode_loop+0x1dd/0x1f0 [ 3180.312303] do_syscall_64+0x3d6/0x440 [ 3180.314175] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3180.320393] RIP: 0033:0x7f151bee3cd8 [ 3180.323043] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3180.336694] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3180.341451] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 3180.344747] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 3180.349308] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3180.353472] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3180.363959] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3184.054699] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 3184.064696] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3184.096808] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3184.113182] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3184.118053] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3184.124419] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3184.140822] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3184.144670] RAX: 0000000000007908 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3184.150019] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 3184.158578] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 3184.172609] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 3184.180710] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 3184.191121] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 3184.200544] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3184.205700] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 3184.210359] Call Trace: [ 3184.213937] [ 3184.215633] ? show_regs.cold.9+0x22/0x2f [ 3184.217678] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3184.220464] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3184.227342] ? watchdog+0x40/0x40 [ 3184.229509] ? hrtimer_interrupt+0x128/0x2c0 [ 3184.231659] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3184.235432] ? apic_timer_interrupt+0xf/0x20 [ 3184.237622] [ 3184.238635] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3184.241340] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3184.243752] do_raw_spin_lock+0x177/0x190 [ 3184.245838] _raw_spin_lock+0x21/0x30 [ 3184.247514] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3184.250108] ? do_raw_spin_unlock+0x75/0x190 [ 3184.252028] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3184.254520] ? do_wait_intr+0xf0/0xf0 [ 3184.255905] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3184.258149] mdc_reint+0x2d/0x110 [mdc] [ 3184.259774] mdc_unlink+0x658/0x870 [mdc] [ 3184.261468] lmv_unlink+0x34d/0xbd0 [lmv] [ 3184.263298] ll_unlink+0x24d/0x7f0 [lustre] [ 3184.265064] vfs_unlink+0x202/0x330 [ 3184.266669] do_unlinkat+0x2fa/0x460 [ 3184.268431] __x64_sys_unlink+0x37/0x50 [ 3184.269753] do_syscall_64+0xc1/0x440 [ 3184.271416] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3184.273438] RIP: 0033:0x7f151bdee10b [ 3184.274883] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3184.282937] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3184.286585] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 3184.289648] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 3184.292520] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3184.295539] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3184.298960] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3188.193460] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1257s! [ 3188.208263] Showing busy workqueues and worker pools: [ 3188.210076] workqueue events: flags=0x0 [ 3188.219552] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3188.225032] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3188.235757] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3188.243661] pending: kfree_rcu_monitor [ 3188.245584] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3188.250545] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 3188.255070] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3188.259331] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3188.273782] workqueue events_long: flags=0x0 [ 3188.278465] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3188.288857] pending: lu_objects_flush [obdclass] [ 3188.300396] workqueue events_unbound: flags=0x2 [ 3188.306217] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3188.315291] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3188.322602] workqueue events_power_efficient: flags=0x80 [ 3188.329758] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3188.337427] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 3188.343416] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3188.355242] pending: neigh_periodic_work [ 3188.362084] workqueue mm_percpu_wq: flags=0x8 [ 3188.366628] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3188.371480] pending: vmstat_update, lru_add_drain_per_cpu [ 3188.376437] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3188.380588] pending: vmstat_update [ 3188.382619] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3188.394008] pending: lru_add_drain_per_cpu [ 3188.395715] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3188.411337] pending: lru_add_drain_per_cpu BAR(48) [ 3188.415466] workqueue writeback: flags=0x4a [ 3188.422825] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3188.430783] pending: wb_workfn [ 3188.435059] workqueue xprtiod: flags=0xa [ 3188.439978] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3188.445427] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3188.464695] workqueue ipv6_addrconf: flags=0x40008 [ 3188.468449] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3188.478462] pending: addrconf_verify_work [ 3188.483310] workqueue knbd0-recv: flags=0x1a [ 3188.489696] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3188.500619] in-flight: 658:recv_work [ 3188.505536] workqueue ptlrpc_pinger: flags=0x2 [ 3188.513533] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3188.521504] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3188.534515] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1257s workers=4 idle: 32984 37 8 [ 3188.546685] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3173s workers=2 idle: 661 [ 3196.056511] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3196.070282] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3196.185131] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3196.192786] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3196.203447] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3196.220992] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3196.232847] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3196.247393] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3196.252543] RAX: 0000000000007c85 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3196.271426] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 3196.290402] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 3196.304065] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 3196.317205] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 3196.319680] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 3196.348003] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3196.361982] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3196.388947] Call Trace: [ 3196.397022] [ 3196.398093] ? show_regs.cold.9+0x22/0x2f [ 3196.413242] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3196.421141] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3196.429795] ? watchdog+0x40/0x40 [ 3196.434597] ? hrtimer_interrupt+0x128/0x2c0 [ 3196.441375] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3196.443151] ? apic_timer_interrupt+0xf/0x20 [ 3196.453065] [ 3196.453860] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3196.469770] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3196.484148] do_raw_spin_lock+0x177/0x190 [ 3196.486056] _raw_spin_lock+0x21/0x30 [ 3196.488366] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3196.501345] process_one_work+0x2c8/0x700 [ 3196.508273] worker_thread+0x296/0x6e0 [ 3196.514231] ? rescuer_thread+0x570/0x570 [ 3196.517659] kthread+0x1d1/0x200 [ 3196.519961] ? set_kthread_struct+0x70/0x70 [ 3196.521577] ret_from_fork+0x1f/0x30 [ 3204.056088] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 3204.063785] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3204.110080] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3204.122570] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3204.131181] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3204.139884] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3204.156448] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3204.162972] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 3204.169998] RDX: 0000000000006605 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 3204.178961] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 3204.187798] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 3204.196764] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3204.207076] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 3204.215368] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3204.219809] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3204.230450] Call Trace: [ 3204.233524] [ 3204.236083] ? show_regs.cold.9+0x22/0x2f [ 3204.240000] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3204.241769] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3204.244129] ? watchdog+0x40/0x40 [ 3204.246362] ? hrtimer_interrupt+0x128/0x2c0 [ 3204.250300] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3204.254765] ? apic_timer_interrupt+0xf/0x20 [ 3204.263045] [ 3204.265438] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3204.272955] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3204.277857] do_raw_spin_lock+0x177/0x190 [ 3204.280890] _raw_spin_lock+0x21/0x30 [ 3204.284049] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3204.290873] obd_mod_free+0x46/0x140 [ptlrpc] [ 3204.295765] mdc_commit_open+0x91/0xb0 [mdc] [ 3204.298445] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3204.302816] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3204.307105] after_reply+0xe2c/0x1780 [ptlrpc] [ 3204.310535] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3204.313448] ? __next_timer_interrupt+0x160/0x160 [ 3204.315714] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3204.318225] ? do_wait_intr+0xf0/0xf0 [ 3204.320320] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3204.322701] kthread+0x1d1/0x200 [ 3204.324339] ? set_kthread_struct+0x70/0x70 [ 3204.326847] ret_from_fork+0x1f/0x30 [ 3208.057067] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 3208.060562] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3208.096680] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3208.107690] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3208.113813] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3208.116372] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3208.127189] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3208.132697] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3208.136228] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 3208.138861] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3208.140827] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 3208.143485] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 3208.146086] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 3208.149209] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3208.151285] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 3208.154052] Call Trace: [ 3208.154832] [ 3208.155410] ? show_regs.cold.9+0x22/0x2f [ 3208.156877] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3208.158802] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3208.160550] ? watchdog+0x40/0x40 [ 3208.161540] ? hrtimer_interrupt+0x128/0x2c0 [ 3208.163194] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3208.164664] ? apic_timer_interrupt+0xf/0x20 [ 3208.166190] [ 3208.166912] ? native_safe_halt+0x2e/0x50 [ 3208.168533] kvm_wait+0xcd/0xe0 [ 3208.169563] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3208.174119] do_raw_spin_lock+0x177/0x190 [ 3208.176589] _raw_spin_lock+0x21/0x30 [ 3208.178486] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3208.180701] ? do_raw_spin_unlock+0x75/0x190 [ 3208.182329] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3208.184376] ? do_wait_intr+0xf0/0xf0 [ 3208.186059] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3208.188295] mdc_close+0x53b/0xd70 [mdc] [ 3208.189440] lmv_close+0x18d/0x490 [lmv] [ 3208.190395] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3208.192537] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3208.194708] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3208.198388] ll_md_close+0x274/0x800 [lustre] [ 3208.200703] ll_file_release+0x128/0xc50 [lustre] [ 3208.204724] __fput+0x10d/0x440 [ 3208.206900] ____fput+0x12/0x20 [ 3208.212695] task_work_run+0xc6/0x110 [ 3208.214719] exit_to_usermode_loop+0x1dd/0x1f0 [ 3208.219247] do_syscall_64+0x3d6/0x440 [ 3208.221957] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3208.225006] RIP: 0033:0x7f151bee3cd8 [ 3208.229279] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3208.241657] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3208.247497] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 3208.254499] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 3208.262527] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3208.273281] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3208.278725] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3216.054551] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 3216.061938] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3216.111555] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3216.117502] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3216.125781] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3216.131429] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3216.149197] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3216.153291] RAX: 0000000000006e85 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3216.157627] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 3216.162288] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 3216.167265] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 3216.174223] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 3216.184486] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 3216.188673] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3216.191036] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 3216.196216] Call Trace: [ 3216.196993] [ 3216.197703] ? show_regs.cold.9+0x22/0x2f [ 3216.206336] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3216.210309] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3216.218010] ? watchdog+0x40/0x40 [ 3216.220288] ? hrtimer_interrupt+0x128/0x2c0 [ 3216.222373] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3216.227059] ? apic_timer_interrupt+0xf/0x20 [ 3216.229596] [ 3216.231318] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3216.238326] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3216.241198] do_raw_spin_lock+0x177/0x190 [ 3216.243246] _raw_spin_lock+0x21/0x30 [ 3216.245003] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3216.248451] ? do_raw_spin_unlock+0x75/0x190 [ 3216.253882] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3216.255714] ? do_wait_intr+0xf0/0xf0 [ 3216.257818] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3216.262132] mdc_reint+0x2d/0x110 [mdc] [ 3216.264160] mdc_unlink+0x658/0x870 [mdc] [ 3216.267129] lmv_unlink+0x34d/0xbd0 [lmv] [ 3216.269352] ll_unlink+0x24d/0x7f0 [lustre] [ 3216.271451] vfs_unlink+0x202/0x330 [ 3216.273554] do_unlinkat+0x2fa/0x460 [ 3216.276932] __x64_sys_unlink+0x37/0x50 [ 3216.279312] do_syscall_64+0xc1/0x440 [ 3216.283540] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3216.286673] RIP: 0033:0x7f151bdee10b [ 3216.289635] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3216.304292] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3216.309411] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 3216.314316] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 3216.318209] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3216.322836] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3216.326356] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3218.917830] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1288s! [ 3218.921425] Showing busy workqueues and worker pools: [ 3218.929168] workqueue events: flags=0x0 [ 3218.931395] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3218.935341] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3218.938484] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3218.941637] pending: kfree_rcu_monitor [ 3218.946435] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3218.950928] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 3218.958475] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3218.961430] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3218.966233] workqueue events_long: flags=0x0 [ 3218.970442] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3218.974616] pending: lu_objects_flush [obdclass] [ 3218.977359] workqueue events_unbound: flags=0x2 [ 3218.979401] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3218.983023] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3218.990251] workqueue events_power_efficient: flags=0x80 [ 3218.994979] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3219.000465] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 3219.005494] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3219.009385] pending: neigh_periodic_work [ 3219.011479] workqueue mm_percpu_wq: flags=0x8 [ 3219.014165] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3219.020043] pending: vmstat_update, lru_add_drain_per_cpu [ 3219.022887] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3219.033055] pending: vmstat_update [ 3219.036355] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3219.042326] pending: lru_add_drain_per_cpu [ 3219.044976] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3219.050849] pending: lru_add_drain_per_cpu BAR(48) [ 3219.054357] workqueue writeback: flags=0x4a [ 3219.057301] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3219.060630] pending: wb_workfn [ 3219.062627] workqueue xprtiod: flags=0xa [ 3219.070014] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3219.075548] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3219.080946] workqueue ipv6_addrconf: flags=0x40008 [ 3219.083978] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3219.087700] pending: addrconf_verify_work [ 3219.089717] workqueue knbd0-recv: flags=0x1a [ 3219.095616] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3219.101184] in-flight: 658:recv_work [ 3219.103937] workqueue ptlrpc_pinger: flags=0x2 [ 3219.106707] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3219.110557] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3219.117056] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1288s workers=4 idle: 32984 37 8 [ 3219.121923] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3203s workers=2 idle: 661 [ 3224.057229] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3224.061424] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3224.087759] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3224.093398] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3224.096292] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3224.098716] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3224.100609] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3224.106990] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3224.109240] RAX: 0000000000007f3a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3224.111662] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 3224.113793] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 3224.116183] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 3224.118901] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 3224.121551] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 3224.124418] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3224.126239] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3224.129647] Call Trace: [ 3224.130772] [ 3224.131726] ? show_regs.cold.9+0x22/0x2f [ 3224.133323] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3224.136008] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3224.138623] ? watchdog+0x40/0x40 [ 3224.140558] ? hrtimer_interrupt+0x128/0x2c0 [ 3224.142827] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3224.145432] ? apic_timer_interrupt+0xf/0x20 [ 3224.146865] [ 3224.147824] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3224.150066] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3224.152283] do_raw_spin_lock+0x177/0x190 [ 3224.153884] _raw_spin_lock+0x21/0x30 [ 3224.155194] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3224.157358] process_one_work+0x2c8/0x700 [ 3224.159130] worker_thread+0x296/0x6e0 [ 3224.160654] ? rescuer_thread+0x570/0x570 [ 3224.162102] kthread+0x1d1/0x200 [ 3224.163093] ? set_kthread_struct+0x70/0x70 [ 3224.164326] ret_from_fork+0x1f/0x30 [ 3232.058864] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 3232.069391] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3232.126627] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3232.149107] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3232.165054] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3232.171981] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3232.184061] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3232.190232] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 3232.197830] RDX: 00000000000039c8 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 3232.202056] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 3232.205053] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 3232.207880] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3232.215181] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 3232.224038] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3232.228687] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3232.238737] Call Trace: [ 3232.247077] [ 3232.251189] ? show_regs.cold.9+0x22/0x2f [ 3232.256555] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3232.269207] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3232.280166] ? watchdog+0x40/0x40 [ 3232.285328] ? hrtimer_interrupt+0x128/0x2c0 [ 3232.290663] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3232.293270] ? apic_timer_interrupt+0xf/0x20 [ 3232.301305] [ 3232.302623] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3232.308623] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3232.310586] do_raw_spin_lock+0x177/0x190 [ 3232.314521] _raw_spin_lock+0x21/0x30 [ 3232.318408] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3232.327857] obd_mod_free+0x46/0x140 [ptlrpc] [ 3232.336764] mdc_commit_open+0x91/0xb0 [mdc] [ 3232.342007] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3232.346256] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3232.348317] after_reply+0xe2c/0x1780 [ptlrpc] [ 3232.352233] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3232.358009] ? __next_timer_interrupt+0x160/0x160 [ 3232.360571] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3232.362643] ? do_wait_intr+0xf0/0xf0 [ 3232.365069] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3232.366655] kthread+0x1d1/0x200 [ 3232.368157] ? set_kthread_struct+0x70/0x70 [ 3232.370056] ret_from_fork+0x1f/0x30 [ 3236.057068] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 3236.060908] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3236.085041] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3236.091088] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3236.095630] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3236.098856] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3236.108990] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3236.112622] RAX: 0000000000001e81 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3236.116515] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 3236.119921] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3236.123602] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 3236.127506] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 3236.130870] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 3236.134916] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3236.137907] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 3236.141599] Call Trace: [ 3236.142662] [ 3236.143456] ? show_regs.cold.9+0x22/0x2f [ 3236.145885] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3236.148429] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3236.150778] ? watchdog+0x40/0x40 [ 3236.152720] ? hrtimer_interrupt+0x128/0x2c0 [ 3236.155617] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3236.158903] ? apic_timer_interrupt+0xf/0x20 [ 3236.161697] [ 3236.163123] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3236.166645] do_raw_spin_lock+0x177/0x190 [ 3236.168839] _raw_spin_lock+0x21/0x30 [ 3236.171636] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3236.174866] ? do_raw_spin_unlock+0x75/0x190 [ 3236.177312] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3236.185432] ? do_wait_intr+0xf0/0xf0 [ 3236.187443] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3236.190716] mdc_close+0x53b/0xd70 [mdc] [ 3236.192623] lmv_close+0x18d/0x490 [lmv] [ 3236.194888] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3236.198394] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3236.201594] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3236.203812] ll_md_close+0x274/0x800 [lustre] [ 3236.207022] ll_file_release+0x128/0xc50 [lustre] [ 3236.209369] __fput+0x10d/0x440 [ 3236.210851] ____fput+0x12/0x20 [ 3236.212622] task_work_run+0xc6/0x110 [ 3236.214599] exit_to_usermode_loop+0x1dd/0x1f0 [ 3236.217025] do_syscall_64+0x3d6/0x440 [ 3236.219023] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3236.221331] RIP: 0033:0x7f151bee3cd8 [ 3236.223183] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3236.231320] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3236.234113] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 3236.237011] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 3236.239813] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3236.243039] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3236.246642] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3244.052186] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 3244.056013] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3244.076369] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3244.081574] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3244.088852] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3244.093825] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3244.114674] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3244.117322] RAX: 0000000000007e23 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3244.128412] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 3244.137368] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 3244.143112] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 3244.148498] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 3244.156450] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 3244.170278] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3244.176348] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 3244.187118] Call Trace: [ 3244.191373] [ 3244.193103] ? show_regs.cold.9+0x22/0x2f [ 3244.195192] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3244.204293] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3244.213948] ? watchdog+0x40/0x40 [ 3244.215111] ? hrtimer_interrupt+0x128/0x2c0 [ 3244.220757] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3244.224573] ? apic_timer_interrupt+0xf/0x20 [ 3244.227535] [ 3244.229052] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3244.240117] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3244.247717] do_raw_spin_lock+0x177/0x190 [ 3244.252406] _raw_spin_lock+0x21/0x30 [ 3244.255755] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3244.257512] ? do_raw_spin_unlock+0x75/0x190 [ 3244.259252] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3244.261523] ? do_wait_intr+0xf0/0xf0 [ 3244.263132] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3244.271549] mdc_reint+0x2d/0x110 [mdc] [ 3244.276483] mdc_unlink+0x658/0x870 [mdc] [ 3244.282514] lmv_unlink+0x34d/0xbd0 [lmv] [ 3244.284974] ll_unlink+0x24d/0x7f0 [lustre] [ 3244.287181] vfs_unlink+0x202/0x330 [ 3244.289911] do_unlinkat+0x2fa/0x460 [ 3244.294258] __x64_sys_unlink+0x37/0x50 [ 3244.296450] do_syscall_64+0xc1/0x440 [ 3244.299430] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3244.303330] RIP: 0033:0x7f151bdee10b [ 3244.306339] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3244.327705] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3244.337845] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 3244.345941] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 3244.354104] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3244.366479] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3244.378618] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3244.670042] rcu: INFO: rcu_sched self-detected stall on CPU [ 3244.674324] rcu: 1-...!: (796502 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 3244.684774] (t=1320023 jiffies g=249657 q=2826) [ 3244.688078] rcu: rcu_sched kthread starved for 1314975 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 3244.706747] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3244.714339] rcu: RCU grace-period kthread stack dump: [ 3244.719876] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3244.725276] Call Trace: [ 3244.726278] __schedule+0x351/0xcb0 [ 3244.728907] schedule+0xc0/0x180 [ 3244.730797] schedule_timeout+0xb4/0x190 [ 3244.733482] ? do_raw_spin_unlock+0x75/0x190 [ 3244.735398] ? __next_timer_interrupt+0x160/0x160 [ 3244.737944] ? prepare_to_swait_event+0xf3/0x140 [ 3244.740079] rcu_gp_kthread+0x8dd/0x15b0 [ 3244.741996] ? force_qs_rnp+0x2c0/0x2c0 [ 3244.744134] kthread+0x1d1/0x200 [ 3244.745927] ? set_kthread_struct+0x70/0x70 [ 3244.755151] ret_from_fork+0x1f/0x30 [ 3244.759405] rcu: Stack dump where RCU GP kthread last ran: [ 3244.763809] Sending NMI from CPU 1 to CPUs 2: [ 3244.765961] NMI backtrace for cpu 2 [ 3244.765966] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3244.765968] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3244.765969] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3244.765972] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3244.765973] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3244.765975] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 [ 3244.765976] RAX: 0000000000004b85 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3244.765978] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 3244.765979] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 3244.765981] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 3244.765982] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 3244.765983] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 3244.765984] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3244.765985] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3244.765986] Call Trace: [ 3244.765987] [ 3244.765988] ? show_regs.cold.9+0x22/0x2f [ 3244.765989] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 3244.765990] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 3244.765991] ? nmi_handle+0xbd/0x210 [ 3244.765992] ? default_do_nmi+0x4b/0x1b0 [ 3244.765993] ? do_nmi+0x280/0x380 [ 3244.765994] ? end_repeat_nmi+0x16/0x1f [ 3244.765995] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3244.765996] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3244.765997] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3244.765998] [ 3244.765999] do_raw_spin_lock+0x177/0x190 [ 3244.766000] _raw_spin_lock+0x21/0x30 [ 3244.766001] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3244.766002] process_one_work+0x2c8/0x700 [ 3244.766003] worker_thread+0x296/0x6e0 [ 3244.766004] ? rescuer_thread+0x570/0x570 [ 3244.766005] kthread+0x1d1/0x200 [ 3244.766006] ? set_kthread_struct+0x70/0x70 [ 3244.766007] ret_from_fork+0x1f/0x30 [ 3244.770013] NMI backtrace for cpu 1 [ 3244.910751] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3244.919003] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3244.925952] Call Trace: [ 3244.927405] [ 3244.930254] ? dump_stack+0xbb/0x10e [ 3244.934851] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3244.937977] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3244.941466] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3244.945572] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3244.949813] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3244.954112] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 3244.956236] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 3244.961235] ? update_process_times+0x6d/0xc0 [ 3244.965862] ? tick_sched_handle.isra.18+0x27/0x90 [ 3244.969139] ? tick_sched_timer+0x53/0xb0 [ 3244.972508] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3244.975544] ? tick_sched_handle.isra.18+0x90/0x90 [ 3244.980479] ? hrtimer_interrupt+0x128/0x2c0 [ 3244.983039] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3244.988669] ? apic_timer_interrupt+0xf/0x20 [ 3244.993509] [ 3244.994342] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3244.998885] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3245.003901] ? do_raw_spin_lock+0x177/0x190 [ 3245.008179] ? _raw_spin_lock+0x21/0x30 [ 3245.010573] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3245.014934] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3245.020185] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3245.023870] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3245.027854] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3245.033420] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3245.036382] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3245.040039] ? __next_timer_interrupt+0x160/0x160 [ 3245.042567] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3245.049644] ? do_wait_intr+0xf0/0xf0 [ 3245.052413] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3245.055416] ? kthread+0x1d1/0x200 [ 3245.056654] ? set_kthread_struct+0x70/0x70 [ 3245.061632] ? ret_from_fork+0x1f/0x30 [ 3249.122263] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1318s! [ 3249.131737] Showing busy workqueues and worker pools: [ 3249.138530] workqueue events: flags=0x0 [ 3249.143409] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3249.150649] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3249.154409] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3249.158022] pending: kfree_rcu_monitor [ 3249.161141] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3249.165160] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 3249.174184] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3249.180775] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3249.184837] workqueue events_long: flags=0x0 [ 3249.194406] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3249.197159] pending: lu_objects_flush [obdclass] [ 3249.204493] workqueue events_unbound: flags=0x2 [ 3249.206831] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3249.210523] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3249.223193] workqueue events_power_efficient: flags=0x80 [ 3249.230429] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3249.239602] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 3249.245148] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3249.248906] pending: neigh_periodic_work [ 3249.251226] workqueue mm_percpu_wq: flags=0x8 [ 3249.257651] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3249.261180] pending: vmstat_update, lru_add_drain_per_cpu [ 3249.264508] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3249.271118] pending: vmstat_update [ 3249.273492] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3249.277923] pending: lru_add_drain_per_cpu [ 3249.281526] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3249.288368] pending: lru_add_drain_per_cpu BAR(48) [ 3249.294137] workqueue writeback: flags=0x4a [ 3249.297985] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3249.301447] pending: wb_workfn [ 3249.302801] workqueue xprtiod: flags=0xa [ 3249.306859] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3249.313237] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3249.322306] workqueue ipv6_addrconf: flags=0x40008 [ 3249.324911] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3249.328109] pending: addrconf_verify_work [ 3249.335204] workqueue knbd0-recv: flags=0x1a [ 3249.336787] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3249.340290] in-flight: 658:recv_work [ 3249.342469] workqueue ptlrpc_pinger: flags=0x2 [ 3249.344753] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3249.348424] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3249.360601] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1318s workers=4 idle: 32984 37 8 [ 3249.368199] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3234s workers=2 idle: 661 [ 3264.058378] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 3264.064550] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3264.096592] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3264.104848] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3264.111561] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3264.115660] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3264.138018] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3264.145380] RAX: 0000000000007815 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3264.159391] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 3264.170241] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3264.181298] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 3264.186250] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 3264.187998] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 3264.204608] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3264.208888] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 3264.210914] Call Trace: [ 3264.218169] [ 3264.218944] ? show_regs.cold.9+0x22/0x2f [ 3264.224190] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3264.226316] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3264.228567] ? watchdog+0x40/0x40 [ 3264.230041] ? hrtimer_interrupt+0x128/0x2c0 [ 3264.232339] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3264.237139] ? apic_timer_interrupt+0xf/0x20 [ 3264.241154] [ 3264.243143] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3264.247167] do_raw_spin_lock+0x177/0x190 [ 3264.252262] _raw_spin_lock+0x21/0x30 [ 3264.254362] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3264.258185] ? do_raw_spin_unlock+0x75/0x190 [ 3264.260722] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3264.278477] ? do_wait_intr+0xf0/0xf0 [ 3264.283044] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3264.285407] mdc_close+0x53b/0xd70 [mdc] [ 3264.291899] lmv_close+0x18d/0x490 [lmv] [ 3264.293313] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3264.300817] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3264.307224] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3264.314203] ll_md_close+0x274/0x800 [lustre] [ 3264.315797] ll_file_release+0x128/0xc50 [lustre] [ 3264.324074] __fput+0x10d/0x440 [ 3264.325752] ____fput+0x12/0x20 [ 3264.331759] task_work_run+0xc6/0x110 [ 3264.333111] exit_to_usermode_loop+0x1dd/0x1f0 [ 3264.340617] do_syscall_64+0x3d6/0x440 [ 3264.344696] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3264.349450] RIP: 0033:0x7f151bee3cd8 [ 3264.354443] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3264.369609] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3264.381805] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 3264.391740] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 3264.399947] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3264.406573] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3264.416044] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3272.055006] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 3272.056038] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 3272.059841] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3272.059918] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3272.059926] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3272.059996] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3272.060010] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3272.060014] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3272.060019] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 3272.060021] RDX: 000000000000429e RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 3272.060023] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 3272.060025] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 3272.060027] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3272.060030] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 3272.060032] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3272.060035] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3272.060043] Call Trace: [ 3272.060049] [ 3272.060054] ? show_regs.cold.9+0x22/0x2f [ 3272.060063] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3272.060068] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3272.060072] ? watchdog+0x40/0x40 [ 3272.060076] ? hrtimer_interrupt+0x128/0x2c0 [ 3272.060080] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3272.060088] ? apic_timer_interrupt+0xf/0x20 [ 3272.060093] [ 3272.060096] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3272.060101] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3272.060107] do_raw_spin_lock+0x177/0x190 [ 3272.060112] _raw_spin_lock+0x21/0x30 [ 3272.060118] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3272.061358] obd_mod_free+0x46/0x140 [ptlrpc] [ 3272.062032] mdc_commit_open+0x91/0xb0 [mdc] [ 3272.062075] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3272.068811] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) [ 3272.121307] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3272.121979] after_reply+0xe2c/0x1780 [ptlrpc] [ 3272.122193] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3272.122510] ? __next_timer_interrupt+0x160/0x160 [ 3272.122522] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3272.122777] ? do_wait_intr+0xf0/0xf0 [ 3272.122788] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3272.122995] kthread+0x1d1/0x200 [ 3272.123002] ? set_kthread_struct+0x70/0x70 [ 3272.123005] ret_from_fork+0x1f/0x30 [ 3272.234529] mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3272.258441] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3272.264445] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3272.268015] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3272.271386] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3272.273274] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3272.282624] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3272.286390] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3272.289449] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 3272.292819] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 3272.296894] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 3272.300403] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 3272.304704] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 3272.307636] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3272.309911] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3272.313986] Call Trace: [ 3272.315477] [ 3272.316684] ? show_regs.cold.9+0x22/0x2f [ 3272.318617] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3272.321866] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3272.324188] ? watchdog+0x40/0x40 [ 3272.325717] ? hrtimer_interrupt+0x128/0x2c0 [ 3272.327847] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3272.330646] ? apic_timer_interrupt+0xf/0x20 [ 3272.332951] [ 3272.334153] ? native_safe_halt+0x2e/0x50 [ 3272.336656] kvm_wait+0xcd/0xe0 [ 3272.338251] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3272.341266] do_raw_spin_lock+0x177/0x190 [ 3272.343531] _raw_spin_lock+0x21/0x30 [ 3272.345466] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3272.348850] process_one_work+0x2c8/0x700 [ 3272.351216] worker_thread+0x296/0x6e0 [ 3272.353313] ? rescuer_thread+0x570/0x570 [ 3272.355797] kthread+0x1d1/0x200 [ 3272.357627] ? set_kthread_struct+0x70/0x70 [ 3272.359995] ret_from_fork+0x1f/0x30 [ 3276.052021] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 3276.061587] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3276.131666] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3276.151710] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3276.160657] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3276.161924] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3276.188273] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3276.200660] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3276.210112] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 3276.219963] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 3276.225566] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 3276.234252] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 3276.244388] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 3276.251552] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3276.254020] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 3276.263522] Call Trace: [ 3276.269059] [ 3276.270726] ? show_regs.cold.9+0x22/0x2f [ 3276.276737] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3276.280109] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3276.285762] ? watchdog+0x40/0x40 [ 3276.288756] ? hrtimer_interrupt+0x128/0x2c0 [ 3276.294720] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3276.297391] ? apic_timer_interrupt+0xf/0x20 [ 3276.301175] [ 3276.302383] ? native_safe_halt+0x2e/0x50 [ 3276.310684] kvm_wait+0xcd/0xe0 [ 3276.313041] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3276.317960] do_raw_spin_lock+0x177/0x190 [ 3276.321000] _raw_spin_lock+0x21/0x30 [ 3276.328854] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3276.334116] ? do_raw_spin_unlock+0x75/0x190 [ 3276.344790] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3276.351861] ? do_wait_intr+0xf0/0xf0 [ 3276.356519] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3276.359808] mdc_reint+0x2d/0x110 [mdc] [ 3276.367429] mdc_unlink+0x658/0x870 [mdc] [ 3276.372182] lmv_unlink+0x34d/0xbd0 [lmv] [ 3276.377226] ll_unlink+0x24d/0x7f0 [lustre] [ 3276.381027] vfs_unlink+0x202/0x330 [ 3276.386169] do_unlinkat+0x2fa/0x460 [ 3276.389157] __x64_sys_unlink+0x37/0x50 [ 3276.395717] do_syscall_64+0xc1/0x440 [ 3276.400001] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3276.401789] RIP: 0033:0x7f151bdee10b [ 3276.406030] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3276.430527] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3276.435921] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 3276.442909] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 3276.449118] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3276.469254] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3276.480398] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3279.848182] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1349s! [ 3279.852919] Showing busy workqueues and worker pools: [ 3279.855451] workqueue events: flags=0x0 [ 3279.856694] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3279.863858] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3279.870090] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3279.873113] pending: kfree_rcu_monitor [ 3279.876168] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3279.880874] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 3279.888431] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3279.893124] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3279.897010] workqueue events_long: flags=0x0 [ 3279.902106] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3279.906739] pending: lu_objects_flush [obdclass] [ 3279.913743] workqueue events_unbound: flags=0x2 [ 3279.915245] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3279.921617] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3279.931460] workqueue events_power_efficient: flags=0x80 [ 3279.937047] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3279.938995] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 3279.955859] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3279.960681] pending: neigh_periodic_work [ 3279.962898] workqueue mm_percpu_wq: flags=0x8 [ 3279.964702] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3279.967106] pending: vmstat_update, lru_add_drain_per_cpu [ 3279.969683] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3279.972224] pending: vmstat_update [ 3279.973589] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3279.976365] pending: lru_add_drain_per_cpu [ 3279.978130] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3279.980799] pending: lru_add_drain_per_cpu BAR(48) [ 3279.983350] workqueue writeback: flags=0x4a [ 3279.984979] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3279.987526] pending: wb_workfn [ 3279.988910] workqueue xprtiod: flags=0xa [ 3279.990674] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3279.993432] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3279.996742] workqueue ipv6_addrconf: flags=0x40008 [ 3279.998601] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3280.000748] pending: addrconf_verify_work [ 3280.002408] workqueue knbd0-recv: flags=0x1a [ 3280.003973] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3280.006181] in-flight: 658:recv_work [ 3280.007674] workqueue ptlrpc_pinger: flags=0x2 [ 3280.009437] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3280.011698] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3280.014638] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1349s workers=4 idle: 32984 37 8 [ 3280.017814] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3264s workers=2 idle: 661 [ 3292.061277] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 3292.066344] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3292.091007] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3292.097032] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3292.102041] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3292.106676] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3292.118832] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3292.121689] RAX: 0000000000006343 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3292.126206] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 3292.129329] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3292.134353] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 3292.140083] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 3292.144610] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 3292.149867] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3292.154365] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 3292.157690] Call Trace: [ 3292.158760] [ 3292.160604] ? show_regs.cold.9+0x22/0x2f [ 3292.167285] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3292.170982] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3292.173646] ? watchdog+0x40/0x40 [ 3292.174561] ? hrtimer_interrupt+0x128/0x2c0 [ 3292.175851] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3292.177648] ? apic_timer_interrupt+0xf/0x20 [ 3292.184480] [ 3292.185681] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3292.188759] do_raw_spin_lock+0x177/0x190 [ 3292.190627] _raw_spin_lock+0x21/0x30 [ 3292.192317] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3292.197004] ? do_raw_spin_unlock+0x75/0x190 [ 3292.198937] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3292.202566] ? do_wait_intr+0xf0/0xf0 [ 3292.204198] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3292.206559] mdc_close+0x53b/0xd70 [mdc] [ 3292.208710] lmv_close+0x18d/0x490 [lmv] [ 3292.212581] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3292.215877] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3292.218405] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3292.220501] ll_md_close+0x274/0x800 [lustre] [ 3292.222761] ll_file_release+0x128/0xc50 [lustre] [ 3292.224913] __fput+0x10d/0x440 [ 3292.226456] ____fput+0x12/0x20 [ 3292.229414] task_work_run+0xc6/0x110 [ 3292.230984] exit_to_usermode_loop+0x1dd/0x1f0 [ 3292.232947] do_syscall_64+0x3d6/0x440 [ 3292.234643] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3292.237202] RIP: 0033:0x7f151bee3cd8 [ 3292.238742] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3292.249934] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3292.254638] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 3292.258638] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 3292.263078] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3292.266994] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3292.272718] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3300.055010] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 3300.058211] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3300.059924] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3300.060011] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3300.060018] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3300.060020] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3300.060034] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3300.060038] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3300.060043] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 3300.060045] RDX: 0000000000004304 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 3300.060056] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 3300.060059] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 3300.060061] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3300.060064] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 3300.060067] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3300.060069] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3300.060078] Call Trace: [ 3300.060083] [ 3300.060089] ? show_regs.cold.9+0x22/0x2f [ 3300.060098] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3300.060105] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3300.060110] ? watchdog+0x40/0x40 [ 3300.060114] ? hrtimer_interrupt+0x128/0x2c0 [ 3300.060118] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3300.060126] ? apic_timer_interrupt+0xf/0x20 [ 3300.060130] [ 3300.060138] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3300.060143] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3300.060151] do_raw_spin_lock+0x177/0x190 [ 3300.060156] _raw_spin_lock+0x21/0x30 [ 3300.060166] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3300.060738] obd_mod_free+0x46/0x140 [ptlrpc] [ 3300.060953] mdc_commit_open+0x91/0xb0 [mdc] [ 3300.060996] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3300.061225] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3300.065490] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3300.065568] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3300.065576] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3300.065583] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3300.065844] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3300.065853] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3300.065856] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3300.065860] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3300.065863] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 3300.065865] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 3300.065866] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 3300.065869] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 3300.065872] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 3300.065875] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3300.065877] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3300.065886] Call Trace: [ 3300.065890] [ 3300.065895] ? show_regs.cold.9+0x22/0x2f [ 3300.065903] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3300.065909] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3300.065913] ? watchdog+0x40/0x40 [ 3300.065917] ? hrtimer_interrupt+0x128/0x2c0 [ 3300.065921] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3300.065928] ? apic_timer_interrupt+0xf/0x20 [ 3300.065932] [ 3300.065935] ? native_safe_halt+0x2e/0x50 [ 3300.065939] kvm_wait+0xcd/0xe0 [ 3300.065946] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3300.065953] do_raw_spin_lock+0x177/0x190 [ 3300.065957] _raw_spin_lock+0x21/0x30 [ 3300.065960] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3300.099390] after_reply+0xe2c/0x1780 [ptlrpc] [ 3300.099851] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3300.100399] ? __next_timer_interrupt+0x160/0x160 [ 3300.100411] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3300.100817] ? do_wait_intr+0xf0/0xf0 [ 3300.100833] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3300.101616] kthread+0x1d1/0x200 [ 3300.101625] ? set_kthread_struct+0x70/0x70 [ 3300.101630] ret_from_fork+0x1f/0x30 [ 3300.398493] process_one_work+0x2c8/0x700 [ 3300.401304] worker_thread+0x296/0x6e0 [ 3300.403936] ? rescuer_thread+0x570/0x570 [ 3300.407746] kthread+0x1d1/0x200 [ 3300.410122] ? set_kthread_struct+0x70/0x70 [ 3300.412831] ret_from_fork+0x1f/0x30 [ 3304.053513] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 3304.061850] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3304.108523] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3304.121358] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3304.133074] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3304.140761] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3304.163457] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3304.173950] RAX: 0000000000002595 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3304.182925] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 3304.191178] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 3304.195614] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 3304.202479] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 3304.208726] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 3304.220299] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3304.226506] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 3304.232589] Call Trace: [ 3304.233796] [ 3304.235525] ? show_regs.cold.9+0x22/0x2f [ 3304.238759] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3304.241903] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3304.246333] ? watchdog+0x40/0x40 [ 3304.248526] ? hrtimer_interrupt+0x128/0x2c0 [ 3304.254429] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3304.263718] ? apic_timer_interrupt+0xf/0x20 [ 3304.268462] [ 3304.272119] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3304.279434] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3304.284715] do_raw_spin_lock+0x177/0x190 [ 3304.287253] _raw_spin_lock+0x21/0x30 [ 3304.289363] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3304.293265] ? do_raw_spin_unlock+0x75/0x190 [ 3304.296291] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3304.301431] ? do_wait_intr+0xf0/0xf0 [ 3304.305563] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3304.311139] mdc_reint+0x2d/0x110 [mdc] [ 3304.314329] mdc_unlink+0x658/0x870 [mdc] [ 3304.320790] lmv_unlink+0x34d/0xbd0 [lmv] [ 3304.323540] ll_unlink+0x24d/0x7f0 [lustre] [ 3304.327378] vfs_unlink+0x202/0x330 [ 3304.329671] do_unlinkat+0x2fa/0x460 [ 3304.332778] __x64_sys_unlink+0x37/0x50 [ 3304.335379] do_syscall_64+0xc1/0x440 [ 3304.339565] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3304.343905] RIP: 0033:0x7f151bdee10b [ 3304.345293] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3304.360151] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3304.372616] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 3304.382444] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 3304.393539] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3304.399369] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3304.404459] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3310.050158] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1379s! [ 3310.053968] Showing busy workqueues and worker pools: [ 3310.060790] workqueue events: flags=0x0 [ 3310.063185] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3310.067731] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3310.070892] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3310.075644] pending: kfree_rcu_monitor [ 3310.078838] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3310.088730] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 3310.102400] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3310.108954] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3310.116609] workqueue events_long: flags=0x0 [ 3310.124480] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3310.128210] pending: lu_objects_flush [obdclass] [ 3310.130605] workqueue events_unbound: flags=0x2 [ 3310.134380] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3310.139054] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3310.143908] workqueue events_power_efficient: flags=0x80 [ 3310.147076] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3310.150433] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 3310.155769] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3310.158711] pending: neigh_periodic_work [ 3310.161199] workqueue mm_percpu_wq: flags=0x8 [ 3310.163754] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3310.167774] pending: vmstat_update, lru_add_drain_per_cpu [ 3310.173593] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3310.180357] pending: vmstat_update [ 3310.187655] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3310.191628] pending: lru_add_drain_per_cpu [ 3310.197237] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3310.201704] pending: lru_add_drain_per_cpu BAR(48) [ 3310.206398] workqueue writeback: flags=0x4a [ 3310.209268] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3310.213629] pending: wb_workfn [ 3310.218798] workqueue xprtiod: flags=0xa [ 3310.226426] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3310.231150] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3310.237598] workqueue ipv6_addrconf: flags=0x40008 [ 3310.239926] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3310.243467] pending: addrconf_verify_work [ 3310.246217] workqueue knbd0-recv: flags=0x1a [ 3310.248582] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3310.252008] in-flight: 658:recv_work [ 3310.256395] workqueue ptlrpc_pinger: flags=0x2 [ 3310.262242] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3310.270296] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3310.274265] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1379s workers=4 idle: 32984 37 8 [ 3310.281495] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3295s workers=2 idle: 661 [ 3320.057440] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 3320.062288] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3320.105793] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3320.116913] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3320.121296] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3320.124739] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3320.134836] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3320.141578] RAX: 0000000000005c89 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3320.146118] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 3320.154262] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3320.159457] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 3320.164270] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 3320.169285] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 3320.177347] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3320.189407] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 3320.195741] Call Trace: [ 3320.197790] [ 3320.200733] ? show_regs.cold.9+0x22/0x2f [ 3320.202721] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3320.207078] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3320.209974] ? watchdog+0x40/0x40 [ 3320.217387] ? hrtimer_interrupt+0x128/0x2c0 [ 3320.222673] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3320.232222] ? apic_timer_interrupt+0xf/0x20 [ 3320.234583] [ 3320.237560] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3320.242070] do_raw_spin_lock+0x177/0x190 [ 3320.246825] _raw_spin_lock+0x21/0x30 [ 3320.248817] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3320.255064] ? do_raw_spin_unlock+0x75/0x190 [ 3320.267446] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3320.270621] ? do_wait_intr+0xf0/0xf0 [ 3320.272844] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3320.284802] mdc_close+0x53b/0xd70 [mdc] [ 3320.293043] lmv_close+0x18d/0x490 [lmv] [ 3320.298435] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3320.300849] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3320.307148] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3320.310998] ll_md_close+0x274/0x800 [lustre] [ 3320.316377] ll_file_release+0x128/0xc50 [lustre] [ 3320.319376] __fput+0x10d/0x440 [ 3320.321776] ____fput+0x12/0x20 [ 3320.323624] task_work_run+0xc6/0x110 [ 3320.326393] exit_to_usermode_loop+0x1dd/0x1f0 [ 3320.329968] do_syscall_64+0x3d6/0x440 [ 3320.333402] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3320.337807] RIP: 0033:0x7f151bee3cd8 [ 3320.342080] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3320.369740] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3320.374618] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 3320.382313] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 3320.388275] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3320.396730] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3320.400621] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3328.055013] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 3328.057791] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3328.058689] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3328.096383] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3328.096397] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3328.096401] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3328.096416] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3328.096421] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3328.096427] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 3328.096429] RDX: 0000000000005b10 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 3328.096432] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 3328.096433] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 3328.096435] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3328.096439] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 3328.096442] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3328.096444] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3328.096452] Call Trace: [ 3328.096458] [ 3328.096463] ? show_regs.cold.9+0x22/0x2f [ 3328.096473] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3328.096479] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3328.096483] ? watchdog+0x40/0x40 [ 3328.096487] ? hrtimer_interrupt+0x128/0x2c0 [ 3328.096491] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3328.096501] ? apic_timer_interrupt+0xf/0x20 [ 3328.096505] [ 3328.096507] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3328.096512] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3328.096518] do_raw_spin_lock+0x177/0x190 [ 3328.096523] _raw_spin_lock+0x21/0x30 [ 3328.096553] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3328.097027] obd_mod_free+0x46/0x140 [ptlrpc] [ 3328.097226] mdc_commit_open+0x91/0xb0 [mdc] [ 3328.097259] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3328.097460] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3328.097854] after_reply+0xe2c/0x1780 [ptlrpc] [ 3328.098059] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3328.098251] ? __next_timer_interrupt+0x160/0x160 [ 3328.098260] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3328.098460] ? do_wait_intr+0xf0/0xf0 [ 3328.098469] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3328.101073] kthread+0x1d1/0x200 [ 3328.101082] ? set_kthread_struct+0x70/0x70 [ 3328.101086] ret_from_fork+0x1f/0x30 [ 3328.311439] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3328.347000] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3328.357348] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3328.364741] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3328.370113] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3328.376828] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3328.390340] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3328.398665] RAX: 0000000000006f0a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3328.407720] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 3328.418660] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 3328.427474] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 3328.436989] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 3328.445278] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 3328.448043] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3328.450281] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3328.452572] Call Trace: [ 3328.455149] [ 3328.455778] ? show_regs.cold.9+0x22/0x2f [ 3328.459015] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3328.460985] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3328.462655] ? watchdog+0x40/0x40 [ 3328.464627] ? hrtimer_interrupt+0x128/0x2c0 [ 3328.468940] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3328.470482] ? apic_timer_interrupt+0xf/0x20 [ 3328.482294] [ 3328.482941] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3328.484428] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3328.487927] do_raw_spin_lock+0x177/0x190 [ 3328.489219] _raw_spin_lock+0x21/0x30 [ 3328.498678] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3328.507936] process_one_work+0x2c8/0x700 [ 3328.510703] worker_thread+0x296/0x6e0 [ 3328.511995] ? rescuer_thread+0x570/0x570 [ 3328.514018] kthread+0x1d1/0x200 [ 3328.516184] ? set_kthread_struct+0x70/0x70 [ 3328.517365] ret_from_fork+0x1f/0x30 [ 3336.053465] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 3336.058665] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3336.079207] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3336.085300] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3336.088610] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3336.090600] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3336.097377] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3336.101039] RAX: 0000000000000298 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3336.105541] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 3336.108127] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 3336.110642] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 3336.114967] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 3336.119165] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 3336.121772] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3336.123548] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 3336.126424] Call Trace: [ 3336.127498] [ 3336.128387] ? show_regs.cold.9+0x22/0x2f [ 3336.129947] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3336.131608] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3336.133422] ? watchdog+0x40/0x40 [ 3336.134587] ? hrtimer_interrupt+0x128/0x2c0 [ 3336.137314] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3336.139322] ? apic_timer_interrupt+0xf/0x20 [ 3336.148822] [ 3336.151457] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3336.159365] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3336.166889] do_raw_spin_lock+0x177/0x190 [ 3336.168050] _raw_spin_lock+0x21/0x30 [ 3336.169753] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3336.172474] ? do_raw_spin_unlock+0x75/0x190 [ 3336.174436] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3336.176142] ? do_wait_intr+0xf0/0xf0 [ 3336.177558] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3336.182366] mdc_reint+0x2d/0x110 [mdc] [ 3336.184217] mdc_unlink+0x658/0x870 [mdc] [ 3336.186097] lmv_unlink+0x34d/0xbd0 [lmv] [ 3336.189705] ll_unlink+0x24d/0x7f0 [lustre] [ 3336.191161] vfs_unlink+0x202/0x330 [ 3336.196071] do_unlinkat+0x2fa/0x460 [ 3336.198011] __x64_sys_unlink+0x37/0x50 [ 3336.199234] do_syscall_64+0xc1/0x440 [ 3336.201172] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3336.203078] RIP: 0033:0x7f151bdee10b [ 3336.204470] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3336.215473] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3336.217867] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 3336.219998] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 3336.222104] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3336.226197] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3336.229320] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3340.769026] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1410s! [ 3340.776052] Showing busy workqueues and worker pools: [ 3340.778376] workqueue events: flags=0x0 [ 3340.779385] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3340.782621] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3340.788177] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3340.794590] pending: kfree_rcu_monitor [ 3340.797335] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3340.809250] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 3340.813896] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3340.820225] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3340.824549] workqueue events_long: flags=0x0 [ 3340.828254] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3340.830745] pending: lu_objects_flush [obdclass] [ 3340.840967] workqueue events_unbound: flags=0x2 [ 3340.843399] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3340.846902] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3340.859339] workqueue events_power_efficient: flags=0x80 [ 3340.867453] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3340.874021] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 3340.879535] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3340.887635] pending: neigh_periodic_work [ 3340.890351] workqueue mm_percpu_wq: flags=0x8 [ 3340.897130] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3340.903075] pending: vmstat_update, lru_add_drain_per_cpu [ 3340.905408] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3340.913246] pending: vmstat_update [ 3340.915919] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3340.920837] pending: lru_add_drain_per_cpu [ 3340.928433] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3340.938166] pending: lru_add_drain_per_cpu BAR(48) [ 3340.942649] workqueue writeback: flags=0x4a [ 3340.944081] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3340.950328] pending: wb_workfn [ 3340.953582] workqueue xprtiod: flags=0xa [ 3340.956227] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3340.959835] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3340.964885] workqueue ipv6_addrconf: flags=0x40008 [ 3340.968640] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3340.973757] pending: addrconf_verify_work [ 3340.976847] workqueue knbd0-recv: flags=0x1a [ 3340.981264] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3340.992507] in-flight: 658:recv_work [ 3340.994626] workqueue ptlrpc_pinger: flags=0x2 [ 3341.001771] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3341.016637] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3341.024509] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1410s workers=4 idle: 32984 37 8 [ 3341.030107] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3325s workers=2 idle: 661 [ 3348.059653] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 3348.075253] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3348.137711] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3348.146112] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3348.151846] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3348.155780] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3348.164428] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3348.169376] RAX: 0000000000007b87 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3348.173915] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 3348.177789] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3348.180200] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 3348.184082] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 3348.186680] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 3348.191745] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3348.195094] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 3348.198673] Call Trace: [ 3348.199988] [ 3348.200968] ? show_regs.cold.9+0x22/0x2f [ 3348.202925] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3348.204753] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3348.207069] ? watchdog+0x40/0x40 [ 3348.210371] ? hrtimer_interrupt+0x128/0x2c0 [ 3348.213652] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3348.217907] ? apic_timer_interrupt+0xf/0x20 [ 3348.220421] [ 3348.225360] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3348.230431] do_raw_spin_lock+0x177/0x190 [ 3348.234226] _raw_spin_lock+0x21/0x30 [ 3348.236423] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3348.240198] ? do_raw_spin_unlock+0x75/0x190 [ 3348.242570] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3348.248756] ? do_wait_intr+0xf0/0xf0 [ 3348.252188] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3348.255888] mdc_close+0x53b/0xd70 [mdc] [ 3348.257370] lmv_close+0x18d/0x490 [lmv] [ 3348.262225] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3348.265241] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3348.270810] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3348.274669] ll_md_close+0x274/0x800 [lustre] [ 3348.277244] ll_file_release+0x128/0xc50 [lustre] [ 3348.280070] __fput+0x10d/0x440 [ 3348.282289] ____fput+0x12/0x20 [ 3348.284145] task_work_run+0xc6/0x110 [ 3348.286212] exit_to_usermode_loop+0x1dd/0x1f0 [ 3348.289972] do_syscall_64+0x3d6/0x440 [ 3348.292185] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3348.297002] RIP: 0033:0x7f151bee3cd8 [ 3348.299002] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3348.309577] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3348.313073] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 3348.316171] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 3348.320128] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3348.323322] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3348.333284] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3356.055434] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 3356.056463] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3356.061250] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3356.061327] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3356.061335] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3356.061337] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3356.061352] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3356.061356] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3356.061361] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 3356.061363] RDX: 0000000000007f80 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 3356.061365] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 3356.061368] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 3356.061370] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3356.061373] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 3356.061376] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3356.061378] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3356.061387] Call Trace: [ 3356.061392] [ 3356.061397] ? show_regs.cold.9+0x22/0x2f [ 3356.061406] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3356.061413] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3356.061417] ? watchdog+0x40/0x40 [ 3356.061421] ? hrtimer_interrupt+0x128/0x2c0 [ 3356.061425] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3356.061434] ? apic_timer_interrupt+0xf/0x20 [ 3356.061438] [ 3356.061441] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3356.061446] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3356.066285] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 [ 3356.104062] do_raw_spin_lock+0x177/0x190 [ 3356.112874] auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3356.112917] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3356.112924] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3356.112930] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3356.113207] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3356.113216] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3356.113220] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3356.113224] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3356.113226] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 3356.113228] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 3356.113230] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 3356.113233] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 3356.113236] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 3356.113239] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3356.113241] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3356.113250] Call Trace: [ 3356.113255] [ 3356.113260] ? show_regs.cold.9+0x22/0x2f [ 3356.113270] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3356.113276] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3356.113281] ? watchdog+0x40/0x40 [ 3356.113285] ? hrtimer_interrupt+0x128/0x2c0 [ 3356.113289] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3356.113295] ? apic_timer_interrupt+0xf/0x20 [ 3356.113300] [ 3356.113303] ? native_safe_halt+0x2e/0x50 [ 3356.113307] kvm_wait+0xcd/0xe0 [ 3356.113313] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3356.113320] do_raw_spin_lock+0x177/0x190 [ 3356.113324] _raw_spin_lock+0x21/0x30 [ 3356.113328] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3356.113537] process_one_work+0x2c8/0x700 [ 3356.113544] worker_thread+0x296/0x6e0 [ 3356.113547] ? rescuer_thread+0x570/0x570 [ 3356.113551] kthread+0x1d1/0x200 [ 3356.113555] ? set_kthread_struct+0x70/0x70 [ 3356.113558] ret_from_fork+0x1f/0x30 [ 3356.343285] _raw_spin_lock+0x21/0x30 [ 3356.345022] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3356.347474] obd_mod_free+0x46/0x140 [ptlrpc] [ 3356.349670] mdc_commit_open+0x91/0xb0 [mdc] [ 3356.351630] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3356.354116] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3356.356864] after_reply+0xe2c/0x1780 [ptlrpc] [ 3356.359072] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3356.361888] ? __next_timer_interrupt+0x160/0x160 [ 3356.364160] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3356.365933] ? do_wait_intr+0xf0/0xf0 [ 3356.367355] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3356.372147] kthread+0x1d1/0x200 [ 3356.375101] ? set_kthread_struct+0x70/0x70 [ 3356.377420] ret_from_fork+0x1f/0x30 [ 3368.054797] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 3368.062025] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3368.124593] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3368.136271] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3368.142004] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3368.150061] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3368.164797] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3368.169003] RAX: 0000000000004532 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3368.171986] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 3368.175008] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 3368.179574] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 3368.184017] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 3368.186610] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 3368.190989] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3368.193733] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 3368.197318] Call Trace: [ 3368.199651] [ 3368.200488] ? show_regs.cold.9+0x22/0x2f [ 3368.202736] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3368.204680] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3368.207483] ? watchdog+0x40/0x40 [ 3368.209257] ? hrtimer_interrupt+0x128/0x2c0 [ 3368.211514] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3368.213364] ? apic_timer_interrupt+0xf/0x20 [ 3368.215822] [ 3368.216637] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3368.219389] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3368.221353] do_raw_spin_lock+0x177/0x190 [ 3368.224435] _raw_spin_lock+0x21/0x30 [ 3368.226225] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3368.229979] ? do_raw_spin_unlock+0x75/0x190 [ 3368.232461] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3368.235646] ? do_wait_intr+0xf0/0xf0 [ 3368.237857] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3368.241303] mdc_reint+0x2d/0x110 [mdc] [ 3368.245841] mdc_unlink+0x658/0x870 [mdc] [ 3368.248439] lmv_unlink+0x34d/0xbd0 [lmv] [ 3368.252151] ll_unlink+0x24d/0x7f0 [lustre] [ 3368.254800] vfs_unlink+0x202/0x330 [ 3368.257736] do_unlinkat+0x2fa/0x460 [ 3368.259755] __x64_sys_unlink+0x37/0x50 [ 3368.262685] do_syscall_64+0xc1/0x440 [ 3368.265659] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3368.268320] RIP: 0033:0x7f151bdee10b [ 3368.275691] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3368.285668] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3368.289397] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 3368.291797] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 3368.298974] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3368.302350] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3368.306814] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3371.497638] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1440s! [ 3371.504883] Showing busy workqueues and worker pools: [ 3371.511605] workqueue events: flags=0x0 [ 3371.518327] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3371.524321] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3371.527778] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3371.534697] pending: kfree_rcu_monitor [ 3371.538860] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3371.543181] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 3371.550505] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3371.554446] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3371.560155] workqueue events_long: flags=0x0 [ 3371.564537] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3371.575821] pending: lu_objects_flush [obdclass] [ 3371.577934] workqueue events_unbound: flags=0x2 [ 3371.582592] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3371.593042] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3371.608099] workqueue events_power_efficient: flags=0x80 [ 3371.615576] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3371.618092] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 3371.623278] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3371.627723] pending: neigh_periodic_work [ 3371.640106] workqueue mm_percpu_wq: flags=0x8 [ 3371.641732] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3371.655833] pending: vmstat_update, lru_add_drain_per_cpu [ 3371.662869] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3371.666708] pending: vmstat_update [ 3371.672042] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3371.680207] pending: lru_add_drain_per_cpu [ 3371.683012] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3371.691741] pending: lru_add_drain_per_cpu BAR(48) [ 3371.694652] workqueue writeback: flags=0x4a [ 3371.701599] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3371.710553] pending: wb_workfn [ 3371.714130] workqueue xprtiod: flags=0xa [ 3371.716730] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3371.726872] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3371.741763] workqueue ipv6_addrconf: flags=0x40008 [ 3371.748597] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3371.758842] pending: addrconf_verify_work [ 3371.762630] workqueue knbd0-recv: flags=0x1a [ 3371.776344] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3371.788612] in-flight: 658:recv_work [ 3371.801187] workqueue ptlrpc_pinger: flags=0x2 [ 3371.809933] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3371.818815] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3371.828646] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1441s workers=4 idle: 32984 37 8 [ 3371.834063] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3356s workers=2 idle: 661 [ 3376.057028] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 3376.060730] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3376.095220] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3376.104351] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3376.110066] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3376.112780] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3376.125303] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3376.138808] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3376.144676] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 3376.160177] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3376.172121] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 3376.174079] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 3376.186272] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 3376.193428] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3376.197518] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 3376.203147] Call Trace: [ 3376.204804] [ 3376.207848] ? show_regs.cold.9+0x22/0x2f [ 3376.210235] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3376.213515] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3376.218101] ? watchdog+0x40/0x40 [ 3376.221760] ? hrtimer_interrupt+0x128/0x2c0 [ 3376.224228] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3376.227473] ? apic_timer_interrupt+0xf/0x20 [ 3376.231472] [ 3376.232514] ? native_safe_halt+0x2e/0x50 [ 3376.235495] kvm_wait+0xcd/0xe0 [ 3376.238197] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3376.240512] do_raw_spin_lock+0x177/0x190 [ 3376.242362] _raw_spin_lock+0x21/0x30 [ 3376.244040] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3376.247454] ? do_raw_spin_unlock+0x75/0x190 [ 3376.249436] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3376.252176] ? do_wait_intr+0xf0/0xf0 [ 3376.254243] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3376.256753] mdc_close+0x53b/0xd70 [mdc] [ 3376.258407] lmv_close+0x18d/0x490 [lmv] [ 3376.261235] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3376.264210] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3376.268147] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3376.270860] ll_md_close+0x274/0x800 [lustre] [ 3376.273408] ll_file_release+0x128/0xc50 [lustre] [ 3376.276267] __fput+0x10d/0x440 [ 3376.277584] ____fput+0x12/0x20 [ 3376.279048] task_work_run+0xc6/0x110 [ 3376.281197] exit_to_usermode_loop+0x1dd/0x1f0 [ 3376.283141] do_syscall_64+0x3d6/0x440 [ 3376.284608] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3376.286769] RIP: 0033:0x7f151bee3cd8 [ 3376.289175] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3376.298012] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3376.301040] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 3376.304594] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 3376.308295] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3376.312294] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3376.317029] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3384.055022] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 3384.057819] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul [ 3384.057887] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3384.057896] Modules linked in: [ 3384.072011] crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3384.072042] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3384.072050] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3384.072053] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3384.072065] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3384.072069] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3384.072073] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 3384.072076] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 3384.072078] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 3384.072080] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 3384.072082] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3384.072084] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 3384.072087] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3384.072089] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3384.072098] Call Trace: [ 3384.072103] [ 3384.072109] ? show_regs.cold.9+0x22/0x2f [ 3384.072118] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3384.072125] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3384.072129] ? watchdog+0x40/0x40 [ 3384.072133] ? hrtimer_interrupt+0x128/0x2c0 [ 3384.072137] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3384.072144] ? apic_timer_interrupt+0xf/0x20 [ 3384.072148] [ 3384.072151] ? native_safe_halt+0x2e/0x50 [ 3384.072155] kvm_wait+0xcd/0xe0 [ 3384.072162] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3384.072169] do_raw_spin_lock+0x177/0x190 [ 3384.072174] _raw_spin_lock+0x21/0x30 [ 3384.072180] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3384.072599] obd_mod_free+0x46/0x140 [ptlrpc] [ 3384.072812] mdc_commit_open+0x91/0xb0 [mdc] [ 3384.072849] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3384.076660] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3384.076730] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3384.076737] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3384.076743] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3384.077017] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3384.077027] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3384.077032] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3384.077036] RAX: 0000000000007e82 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3384.077038] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 3384.077041] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 3384.077043] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 3384.077045] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 3384.077048] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 3384.077051] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3384.077053] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3384.077061] Call Trace: [ 3384.077066] [ 3384.077071] ? show_regs.cold.9+0x22/0x2f [ 3384.077079] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3384.077086] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3384.077091] ? watchdog+0x40/0x40 [ 3384.077095] ? hrtimer_interrupt+0x128/0x2c0 [ 3384.077100] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3384.077109] ? apic_timer_interrupt+0xf/0x20 [ 3384.077113] [ 3384.077116] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3384.077122] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3384.077127] do_raw_spin_lock+0x177/0x190 [ 3384.077132] _raw_spin_lock+0x21/0x30 [ 3384.077137] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3384.077356] process_one_work+0x2c8/0x700 [ 3384.077362] worker_thread+0x296/0x6e0 [ 3384.077380] ? rescuer_thread+0x570/0x570 [ 3384.077383] kthread+0x1d1/0x200 [ 3384.077387] ? set_kthread_struct+0x70/0x70 [ 3384.077390] ret_from_fork+0x1f/0x30 [ 3384.266449] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3384.273806] after_reply+0xe2c/0x1780 [ptlrpc] [ 3384.277460] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3384.280377] ? __next_timer_interrupt+0x160/0x160 [ 3384.283130] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3384.291562] ? do_wait_intr+0xf0/0xf0 [ 3384.292912] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3384.294576] kthread+0x1d1/0x200 [ 3384.295491] ? set_kthread_struct+0x70/0x70 [ 3384.296600] ret_from_fork+0x1f/0x30 [ 3396.052032] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 3396.057837] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3396.076039] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3396.080319] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3396.083982] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3396.086301] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3396.095663] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 3396.099090] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3396.102456] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 3396.105455] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 3396.108363] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 3396.112299] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 3396.115890] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 3396.120635] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3396.129464] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 3396.132311] Call Trace: [ 3396.133527] [ 3396.134162] ? show_regs.cold.9+0x22/0x2f [ 3396.135612] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3396.137580] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3396.139110] ? watchdog+0x40/0x40 [ 3396.140525] ? hrtimer_interrupt+0x128/0x2c0 [ 3396.142220] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3396.145782] ? apic_timer_interrupt+0xf/0x20 [ 3396.147699] [ 3396.148454] ? native_safe_halt+0x2e/0x50 [ 3396.149824] kvm_wait+0xcd/0xe0 [ 3396.151291] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3396.153086] do_raw_spin_lock+0x177/0x190 [ 3396.154776] _raw_spin_lock+0x21/0x30 [ 3396.156063] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3396.158776] ? do_raw_spin_unlock+0x75/0x190 [ 3396.161191] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3396.163760] ? do_wait_intr+0xf0/0xf0 [ 3396.166857] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3396.170359] mdc_reint+0x2d/0x110 [mdc] [ 3396.171616] mdc_unlink+0x658/0x870 [mdc] [ 3396.174020] lmv_unlink+0x34d/0xbd0 [lmv] [ 3396.176836] ll_unlink+0x24d/0x7f0 [lustre] [ 3396.182128] vfs_unlink+0x202/0x330 [ 3396.187609] do_unlinkat+0x2fa/0x460 [ 3396.191951] __x64_sys_unlink+0x37/0x50 [ 3396.193589] do_syscall_64+0xc1/0x440 [ 3396.195817] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3396.198726] RIP: 0033:0x7f151bdee10b [ 3396.201297] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3396.214537] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3396.218106] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 3396.221285] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 3396.223921] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3396.226683] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3396.229362] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3402.208141] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1471s! [ 3402.212279] Showing busy workqueues and worker pools: [ 3402.214606] workqueue events: flags=0x0 [ 3402.216204] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3402.219595] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3402.222533] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3402.225799] pending: kfree_rcu_monitor [ 3402.227814] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3402.230891] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 3402.235363] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3402.238857] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3402.242408] workqueue events_long: flags=0x0 [ 3402.243786] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3402.246858] pending: lu_objects_flush [obdclass] [ 3402.249328] workqueue events_unbound: flags=0x2 [ 3402.250801] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3402.253908] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3402.258406] workqueue events_power_efficient: flags=0x80 [ 3402.261046] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3402.263799] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 3402.267327] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3402.270370] pending: neigh_periodic_work [ 3402.272351] workqueue mm_percpu_wq: flags=0x8 [ 3402.274511] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3402.277578] pending: vmstat_update, lru_add_drain_per_cpu [ 3402.279808] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3402.282296] pending: vmstat_update [ 3402.283776] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3402.287180] pending: lru_add_drain_per_cpu [ 3402.288751] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3402.291626] pending: lru_add_drain_per_cpu BAR(48) [ 3402.293952] workqueue writeback: flags=0x4a [ 3402.296290] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3402.298796] pending: wb_workfn [ 3402.300442] workqueue xprtiod: flags=0xa [ 3402.302309] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3402.305260] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3402.311663] workqueue ipv6_addrconf: flags=0x40008 [ 3402.315876] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3402.320625] pending: addrconf_verify_work [ 3402.322480] workqueue knbd0-recv: flags=0x1a [ 3402.324957] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3402.328208] in-flight: 658:recv_work [ 3402.329914] workqueue ptlrpc_pinger: flags=0x2 [ 3402.331959] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3402.335949] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3402.339333] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1471s workers=4 idle: 32984 37 8 [ 3402.343488] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3387s workers=2 idle: 661 [ 3404.062183] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 3404.081252] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3404.167338] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3404.187935] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3404.203066] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3404.204862] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3404.238451] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3404.246118] RAX: 0000000000007f09 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3404.253521] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 3404.258771] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3404.263294] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 3404.266536] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 3404.270587] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 3404.273899] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3404.278583] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 3404.284640] Call Trace: [ 3404.289940] [ 3404.290827] ? show_regs.cold.9+0x22/0x2f [ 3404.294258] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3404.297303] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3404.299514] ? watchdog+0x40/0x40 [ 3404.301194] ? hrtimer_interrupt+0x128/0x2c0 [ 3404.304031] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3404.309776] ? apic_timer_interrupt+0xf/0x20 [ 3404.313593] [ 3404.318266] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3404.324262] do_raw_spin_lock+0x177/0x190 [ 3404.326599] _raw_spin_lock+0x21/0x30 [ 3404.327879] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3404.331889] ? do_raw_spin_unlock+0x75/0x190 [ 3404.333903] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3404.336797] ? do_wait_intr+0xf0/0xf0 [ 3404.338358] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3404.340673] mdc_close+0x53b/0xd70 [mdc] [ 3404.342684] lmv_close+0x18d/0x490 [lmv] [ 3404.346567] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3404.350266] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3404.353033] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3404.355167] ll_md_close+0x274/0x800 [lustre] [ 3404.357297] ll_file_release+0x128/0xc50 [lustre] [ 3404.366832] __fput+0x10d/0x440 [ 3404.367915] ____fput+0x12/0x20 [ 3404.369928] task_work_run+0xc6/0x110 [ 3404.372924] exit_to_usermode_loop+0x1dd/0x1f0 [ 3404.377032] do_syscall_64+0x3d6/0x440 [ 3404.378808] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3404.387557] RIP: 0033:0x7f151bee3cd8 [ 3404.396007] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3404.411546] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3404.415941] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 3404.420538] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 3404.426320] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3404.435024] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3404.438623] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3412.055060] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 3412.056030] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3412.059133] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3412.059216] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3412.059224] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3412.059228] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3412.059242] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3412.059246] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3412.059250] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 3412.059253] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 3412.059255] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 3412.059257] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 3412.059260] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3412.059263] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 3412.059265] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3412.059268] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3412.059275] Call Trace: [ 3412.059280] [ 3412.059284] ? show_regs.cold.9+0x22/0x2f [ 3412.059293] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3412.059299] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3412.059303] ? watchdog+0x40/0x40 [ 3412.059307] ? hrtimer_interrupt+0x128/0x2c0 [ 3412.059311] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3412.059318] ? apic_timer_interrupt+0xf/0x20 [ 3412.059322] [ 3412.059325] ? native_safe_halt+0x2e/0x50 [ 3412.059329] kvm_wait+0xcd/0xe0 [ 3412.059336] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3412.059343] do_raw_spin_lock+0x177/0x190 [ 3412.059347] _raw_spin_lock+0x21/0x30 [ 3412.059351] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3412.059638] obd_mod_free+0x46/0x140 [ptlrpc] [ 3412.060060] mdc_commit_open+0x91/0xb0 [mdc] [ 3412.064444] Modules linked in: lustre(O) [ 3412.089837] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3412.093193] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3412.097808] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3412.097882] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3412.097890] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3412.097897] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3412.111943] after_reply+0xe2c/0x1780 [ptlrpc] [ 3412.112226] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3412.112435] ? __next_timer_interrupt+0x160/0x160 [ 3412.112445] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3412.112646] ? do_wait_intr+0xf0/0xf0 [ 3412.112655] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3412.112859] kthread+0x1d1/0x200 [ 3412.112865] ? set_kthread_struct+0x70/0x70 [ 3412.112869] ret_from_fork+0x1f/0x30 [ 3412.358234] [ 3412.358584] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3412.361500] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3412.368120] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3412.371690] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3412.375473] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 3412.378792] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 3412.386310] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 3412.390408] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 3412.395876] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 3412.408857] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3412.414329] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3412.424852] Call Trace: [ 3412.429804] [ 3412.432469] ? show_regs.cold.9+0x22/0x2f [ 3412.433854] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3412.442879] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3412.449521] ? watchdog+0x40/0x40 [ 3412.451423] ? hrtimer_interrupt+0x128/0x2c0 [ 3412.454366] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3412.456783] ? apic_timer_interrupt+0xf/0x20 [ 3412.458365] [ 3412.460809] ? native_safe_halt+0x2e/0x50 [ 3412.464190] kvm_wait+0xcd/0xe0 [ 3412.467786] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3412.471444] do_raw_spin_lock+0x177/0x190 [ 3412.482362] _raw_spin_lock+0x21/0x30 [ 3412.484298] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3412.492725] process_one_work+0x2c8/0x700 [ 3412.496294] worker_thread+0x296/0x6e0 [ 3412.501883] ? rescuer_thread+0x570/0x570 [ 3412.503632] kthread+0x1d1/0x200 [ 3412.504789] ? set_kthread_struct+0x70/0x70 [ 3412.508926] ret_from_fork+0x1f/0x30 [ 3424.673563] rcu: INFO: rcu_sched self-detected stall on CPU [ 3424.681137] rcu: 1-...!: (903047 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 3424.689660] (t=1500026 jiffies g=249657 q=2883) [ 3424.692610] rcu: rcu_sched kthread starved for 1494978 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 3424.703852] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3424.717261] rcu: RCU grace-period kthread stack dump: [ 3424.719052] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3424.726408] Call Trace: [ 3424.728699] __schedule+0x351/0xcb0 [ 3424.729988] schedule+0xc0/0x180 [ 3424.733234] schedule_timeout+0xb4/0x190 [ 3424.734615] ? do_raw_spin_unlock+0x75/0x190 [ 3424.737402] ? __next_timer_interrupt+0x160/0x160 [ 3424.742044] ? prepare_to_swait_event+0xf3/0x140 [ 3424.757437] rcu_gp_kthread+0x8dd/0x15b0 [ 3424.762733] ? force_qs_rnp+0x2c0/0x2c0 [ 3424.771094] kthread+0x1d1/0x200 [ 3424.773821] ? set_kthread_struct+0x70/0x70 [ 3424.779658] ret_from_fork+0x1f/0x30 [ 3424.786420] rcu: Stack dump where RCU GP kthread last ran: [ 3424.788308] Sending NMI from CPU 1 to CPUs 0: [ 3424.792763] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [ 3424.796443] NMI backtrace for cpu 1 [ 3424.806090] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3424.810786] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3424.816167] Call Trace: [ 3424.817649] [ 3424.818527] ? dump_stack+0xbb/0x10e [ 3424.820260] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3424.823261] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3424.825678] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3424.828827] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3424.832576] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3424.835596] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 3424.838353] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 3424.841111] ? update_process_times+0x6d/0xc0 [ 3424.843384] ? tick_sched_handle.isra.18+0x27/0x90 [ 3424.845532] ? tick_sched_timer+0x53/0xb0 [ 3424.847432] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3424.849367] ? tick_sched_handle.isra.18+0x90/0x90 [ 3424.852258] ? hrtimer_interrupt+0x128/0x2c0 [ 3424.854508] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3424.859595] ? apic_timer_interrupt+0xf/0x20 [ 3424.866243] [ 3424.874168] ? native_safe_halt+0x2e/0x50 [ 3424.878531] ? kvm_wait+0xcd/0xe0 [ 3424.882682] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3424.892717] ? do_raw_spin_lock+0x177/0x190 [ 3424.898789] ? _raw_spin_lock+0x21/0x30 [ 3424.902129] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3424.914689] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3424.933038] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3424.934547] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3424.950699] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3424.958909] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3424.973441] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3424.975611] ? __next_timer_interrupt+0x160/0x160 [ 3424.987713] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3424.993445] ? do_wait_intr+0xf0/0xf0 [ 3424.999412] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3425.006386] ? kthread+0x1d1/0x200 [ 3425.008492] ? set_kthread_struct+0x70/0x70 [ 3425.011433] ? ret_from_fork+0x1f/0x30 [ 3432.058465] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 3432.060985] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3432.122721] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3432.131892] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3432.137756] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3432.141732] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3432.155350] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3432.164684] RAX: 0000000000005683 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3432.168978] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 3432.172486] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3432.176502] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 3432.180663] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 3432.183002] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 3432.185326] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3432.187757] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 3432.190702] Call Trace: [ 3432.191522] [ 3432.192545] ? show_regs.cold.9+0x22/0x2f [ 3432.195414] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3432.197566] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3432.199623] ? watchdog+0x40/0x40 [ 3432.201294] ? hrtimer_interrupt+0x128/0x2c0 [ 3432.203405] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3432.205508] ? apic_timer_interrupt+0xf/0x20 [ 3432.208274] [ 3432.216165] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3432.222746] do_raw_spin_lock+0x177/0x190 [ 3432.228022] _raw_spin_lock+0x21/0x30 [ 3432.229362] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3432.237018] ? do_raw_spin_unlock+0x75/0x190 [ 3432.239741] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3432.247471] ? do_wait_intr+0xf0/0xf0 [ 3432.250439] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3432.252487] mdc_close+0x53b/0xd70 [mdc] [ 3432.257954] lmv_close+0x18d/0x490 [lmv] [ 3432.261481] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3432.264723] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3432.273864] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3432.277555] ll_md_close+0x274/0x800 [lustre] [ 3432.281336] ll_file_release+0x128/0xc50 [lustre] [ 3432.284409] __fput+0x10d/0x440 [ 3432.291730] ____fput+0x12/0x20 [ 3432.295613] task_work_run+0xc6/0x110 [ 3432.299305] exit_to_usermode_loop+0x1dd/0x1f0 [ 3432.302660] do_syscall_64+0x3d6/0x440 [ 3432.305829] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3432.310127] RIP: 0033:0x7f151bee3cd8 [ 3432.316343] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3432.342726] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3432.353731] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 3432.357638] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 3432.362023] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3432.367351] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3432.371961] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3432.421030] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1501s! [ 3432.437207] Showing busy workqueues and worker pools: [ 3432.439200] workqueue events: flags=0x0 [ 3432.445269] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3432.455674] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3432.465280] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3432.469482] pending: kfree_rcu_monitor [ 3432.472389] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3432.475068] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 3432.480694] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3432.484598] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3432.488013] workqueue events_long: flags=0x0 [ 3432.489663] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3432.498520] pending: lu_objects_flush [obdclass] [ 3432.502290] workqueue events_unbound: flags=0x2 [ 3432.504704] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3432.509537] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3432.524869] workqueue events_power_efficient: flags=0x80 [ 3432.528948] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3432.533157] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 3432.535864] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3432.547247] pending: neigh_periodic_work [ 3432.554872] workqueue mm_percpu_wq: flags=0x8 [ 3432.556462] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3432.571878] pending: vmstat_update, lru_add_drain_per_cpu [ 3432.575522] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3432.586681] pending: vmstat_update [ 3432.588954] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3432.593304] pending: lru_add_drain_per_cpu [ 3432.597166] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3432.603966] pending: lru_add_drain_per_cpu BAR(48) [ 3432.608851] workqueue writeback: flags=0x4a [ 3432.612477] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3432.615671] pending: wb_workfn [ 3432.617696] workqueue xprtiod: flags=0xa [ 3432.620694] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3432.625616] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3432.630318] workqueue ipv6_addrconf: flags=0x40008 [ 3432.638786] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3432.642433] pending: addrconf_verify_work [ 3432.644803] workqueue knbd0-recv: flags=0x1a [ 3432.649571] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3432.656930] in-flight: 658:recv_work [ 3432.661785] workqueue ptlrpc_pinger: flags=0x2 [ 3432.664333] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3432.668993] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3432.685504] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1502s workers=4 idle: 32984 37 8 [ 3432.695711] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3417s workers=2 idle: 661 [ 3440.056174] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3440.060214] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3440.087667] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3440.097860] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3440.102110] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3440.110165] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3440.116245] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3440.124362] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3440.130323] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3440.135844] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 3440.143049] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 3440.147292] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 3440.150197] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 3440.152403] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 3440.155563] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3440.158349] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3440.163355] Call Trace: [ 3440.164177] [ 3440.164627] ? show_regs.cold.9+0x22/0x2f [ 3440.166079] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3440.174453] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3440.180731] ? watchdog+0x40/0x40 [ 3440.181634] ? hrtimer_interrupt+0x128/0x2c0 [ 3440.183609] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3440.185061] ? apic_timer_interrupt+0xf/0x20 [ 3440.190534] [ 3440.195339] ? native_safe_halt+0x2e/0x50 [ 3440.198289] kvm_wait+0xcd/0xe0 [ 3440.202129] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3440.205249] do_raw_spin_lock+0x177/0x190 [ 3440.208859] _raw_spin_lock+0x21/0x30 [ 3440.210147] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3440.215432] process_one_work+0x2c8/0x700 [ 3440.217797] worker_thread+0x296/0x6e0 [ 3440.219286] ? rescuer_thread+0x570/0x570 [ 3440.220993] kthread+0x1d1/0x200 [ 3440.222331] ? set_kthread_struct+0x70/0x70 [ 3440.224382] ret_from_fork+0x1f/0x30 [ 3452.055934] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 3452.059971] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3452.092267] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3452.097411] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3452.103055] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3452.111193] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3452.132429] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3452.136256] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 3452.142380] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 3452.154555] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 3452.166346] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 3452.172240] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3452.180520] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 3452.193016] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3452.196542] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3452.201217] Call Trace: [ 3452.202360] [ 3452.203852] ? show_regs.cold.9+0x22/0x2f [ 3452.206432] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3452.208991] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3452.212930] ? watchdog+0x40/0x40 [ 3452.214460] ? hrtimer_interrupt+0x128/0x2c0 [ 3452.218363] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3452.223151] ? apic_timer_interrupt+0xf/0x20 [ 3452.224415] [ 3452.225086] ? native_safe_halt+0x2e/0x50 [ 3452.231216] kvm_wait+0xcd/0xe0 [ 3452.233195] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3452.240940] do_raw_spin_lock+0x177/0x190 [ 3452.243650] _raw_spin_lock+0x21/0x30 [ 3452.245838] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3452.248974] obd_mod_free+0x46/0x140 [ptlrpc] [ 3452.254030] mdc_commit_open+0x91/0xb0 [mdc] [ 3452.259359] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3452.265532] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3452.270477] after_reply+0xe2c/0x1780 [ptlrpc] [ 3452.272021] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3452.275498] ? __next_timer_interrupt+0x160/0x160 [ 3452.279521] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3452.287939] ? do_wait_intr+0xf0/0xf0 [ 3452.291517] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3452.294256] kthread+0x1d1/0x200 [ 3452.295818] ? set_kthread_struct+0x70/0x70 [ 3452.297790] ret_from_fork+0x1f/0x30 [ 3460.052297] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 3460.057501] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3460.060995] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 3460.075376] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3460.075388] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3460.075391] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3460.075405] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3460.075409] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3460.075414] RAX: 0000000000007a15 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3460.075417] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 3460.075419] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 3460.075421] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 3460.075423] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 3460.075427] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 3460.075430] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3460.075432] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 3460.075440] Call Trace: [ 3460.075445] [ 3460.075449] ? show_regs.cold.9+0x22/0x2f [ 3460.075461] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3460.075467] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3460.075472] ? watchdog+0x40/0x40 [ 3460.075476] ? hrtimer_interrupt+0x128/0x2c0 [ 3460.075480] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3460.075488] ? apic_timer_interrupt+0xf/0x20 [ 3460.075493] [ 3460.075496] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3460.075501] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3460.075505] do_raw_spin_lock+0x177/0x190 [ 3460.075510] _raw_spin_lock+0x21/0x30 [ 3460.075517] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3460.075832] ? do_raw_spin_unlock+0x75/0x190 [ 3460.075837] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3460.076035] ? do_wait_intr+0xf0/0xf0 [ 3460.076042] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3460.076236] mdc_reint+0x2d/0x110 [mdc] [ 3460.076279] mdc_unlink+0x658/0x870 [mdc] [ 3460.076303] lmv_unlink+0x34d/0xbd0 [lmv] [ 3460.076333] ll_unlink+0x24d/0x7f0 [lustre] [ 3460.076481] vfs_unlink+0x202/0x330 [ 3460.076488] do_unlinkat+0x2fa/0x460 [ 3460.076492] __x64_sys_unlink+0x37/0x50 [ 3460.076495] do_syscall_64+0xc1/0x440 [ 3460.076503] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3460.076508] RIP: 0033:0x7f151bdee10b [ 3460.076514] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3460.076517] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3460.076521] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 3460.076524] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 3460.076526] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3460.076528] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3460.076530] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3460.205526] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3460.222710] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3460.225927] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3460.229913] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3460.231942] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3460.237768] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3460.240788] RAX: 0000000000007490 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3460.243316] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 3460.245537] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3460.248275] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 3460.251864] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 3460.254403] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 3460.257186] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3460.259030] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 3460.260911] Call Trace: [ 3460.261625] [ 3460.262368] ? show_regs.cold.9+0x22/0x2f [ 3460.263522] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3460.265406] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3460.266807] ? watchdog+0x40/0x40 [ 3460.267834] ? hrtimer_interrupt+0x128/0x2c0 [ 3460.269518] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3460.271559] ? apic_timer_interrupt+0xf/0x20 [ 3460.272806] [ 3460.273427] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3460.275172] do_raw_spin_lock+0x177/0x190 [ 3460.276459] _raw_spin_lock+0x21/0x30 [ 3460.277726] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3460.279461] ? do_raw_spin_unlock+0x75/0x190 [ 3460.281131] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3460.282923] ? do_wait_intr+0xf0/0xf0 [ 3460.284196] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3460.286202] mdc_close+0x53b/0xd70 [mdc] [ 3460.287625] lmv_close+0x18d/0x490 [lmv] [ 3460.288973] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3460.290864] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3460.292835] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3460.294415] ll_md_close+0x274/0x800 [lustre] [ 3460.296245] ll_file_release+0x128/0xc50 [lustre] [ 3460.297734] __fput+0x10d/0x440 [ 3460.298667] ____fput+0x12/0x20 [ 3460.299944] task_work_run+0xc6/0x110 [ 3460.301279] exit_to_usermode_loop+0x1dd/0x1f0 [ 3460.303087] do_syscall_64+0x3d6/0x440 [ 3460.304361] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3460.306330] RIP: 0033:0x7f151bee3cd8 [ 3460.307532] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3460.314294] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3460.317090] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 3460.319528] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 3460.322183] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3460.324815] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3460.327354] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3463.142122] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1532s! [ 3463.150516] Showing busy workqueues and worker pools: [ 3463.156110] workqueue events: flags=0x0 [ 3463.157452] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3463.160905] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3463.166239] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3463.174410] pending: kfree_rcu_monitor [ 3463.176926] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3463.181096] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 3463.194932] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3463.201087] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3463.209856] workqueue events_long: flags=0x0 [ 3463.216702] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3463.222413] pending: lu_objects_flush [obdclass] [ 3463.228020] workqueue events_unbound: flags=0x2 [ 3463.231381] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3463.237336] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3463.249330] workqueue events_power_efficient: flags=0x80 [ 3463.256669] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3463.261417] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 3463.272576] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3463.281849] pending: neigh_periodic_work [ 3463.289358] workqueue mm_percpu_wq: flags=0x8 [ 3463.296789] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3463.305303] pending: vmstat_update, lru_add_drain_per_cpu [ 3463.313517] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3463.319482] pending: vmstat_update [ 3463.321880] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3463.326140] pending: lru_add_drain_per_cpu [ 3463.332863] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3463.339799] pending: lru_add_drain_per_cpu BAR(48) [ 3463.344778] workqueue writeback: flags=0x4a [ 3463.347671] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3463.355694] pending: wb_workfn [ 3463.359431] workqueue xprtiod: flags=0xa [ 3463.367340] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3463.376837] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3463.381541] workqueue ipv6_addrconf: flags=0x40008 [ 3463.384093] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3463.391448] pending: addrconf_verify_work [ 3463.395624] workqueue knbd0-recv: flags=0x1a [ 3463.397762] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3463.404839] in-flight: 658:recv_work [ 3463.413581] workqueue ptlrpc_pinger: flags=0x2 [ 3463.420114] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3463.430697] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3463.438750] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1532s workers=4 idle: 32984 37 8 [ 3463.449221] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3448s workers=2 idle: 661 [ 3468.056204] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3468.063651] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3468.120615] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3468.132535] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3468.135896] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3468.140323] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3468.142907] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3468.158354] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3468.165423] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3468.169877] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 3468.175773] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 3468.185640] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 3468.188165] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 3468.195453] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 3468.206368] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3468.211536] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3468.218862] Call Trace: [ 3468.220174] [ 3468.221387] ? show_regs.cold.9+0x22/0x2f [ 3468.223314] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3468.226822] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3468.230761] ? watchdog+0x40/0x40 [ 3468.233128] ? hrtimer_interrupt+0x128/0x2c0 [ 3468.236821] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3468.242847] ? apic_timer_interrupt+0xf/0x20 [ 3468.248816] [ 3468.249748] ? native_safe_halt+0x2e/0x50 [ 3468.250883] kvm_wait+0xcd/0xe0 [ 3468.251953] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3468.255106] do_raw_spin_lock+0x177/0x190 [ 3468.257669] _raw_spin_lock+0x21/0x30 [ 3468.259229] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3468.262543] process_one_work+0x2c8/0x700 [ 3468.264361] worker_thread+0x296/0x6e0 [ 3468.266330] ? rescuer_thread+0x570/0x570 [ 3468.268574] kthread+0x1d1/0x200 [ 3468.270015] ? set_kthread_struct+0x70/0x70 [ 3468.272057] ret_from_fork+0x1f/0x30 [ 3480.056133] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 3480.069689] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3480.108530] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3480.114894] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3480.121515] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3480.124256] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3480.141614] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3480.146384] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 3480.152231] RDX: 0000000000006ff5 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 3480.162777] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 3480.167298] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 3480.169253] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3480.177691] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 3480.186262] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3480.189739] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3480.194208] Call Trace: [ 3480.198874] [ 3480.200304] ? show_regs.cold.9+0x22/0x2f [ 3480.201789] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3480.206121] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3480.211636] ? watchdog+0x40/0x40 [ 3480.214276] ? hrtimer_interrupt+0x128/0x2c0 [ 3480.219515] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3480.223366] ? apic_timer_interrupt+0xf/0x20 [ 3480.225174] [ 3480.230846] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3480.236859] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3480.239032] do_raw_spin_lock+0x177/0x190 [ 3480.243705] _raw_spin_lock+0x21/0x30 [ 3480.252903] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3480.262111] obd_mod_free+0x46/0x140 [ptlrpc] [ 3480.269885] mdc_commit_open+0x91/0xb0 [mdc] [ 3480.273166] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3480.276143] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3480.278718] after_reply+0xe2c/0x1780 [ptlrpc] [ 3480.282169] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3480.290707] ? __next_timer_interrupt+0x160/0x160 [ 3480.298650] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3480.304632] ? do_wait_intr+0xf0/0xf0 [ 3480.305987] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3480.315767] kthread+0x1d1/0x200 [ 3480.320623] ? set_kthread_struct+0x70/0x70 [ 3480.325236] ret_from_fork+0x1f/0x30 [ 3488.055640] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 3488.060262] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 3488.066435] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3488.066520] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3488.066528] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3488.066531] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3488.066545] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3488.066550] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3488.066554] RAX: 000000000000744f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3488.066557] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 3488.066559] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 3488.066561] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 3488.066563] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 3488.066567] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 3488.066569] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3488.066572] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 3488.066580] Call Trace: [ 3488.066586] [ 3488.066592] ? show_regs.cold.9+0x22/0x2f [ 3488.066602] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3488.066608] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3488.066613] ? watchdog+0x40/0x40 [ 3488.066616] ? hrtimer_interrupt+0x128/0x2c0 [ 3488.066621] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3488.066630] ? apic_timer_interrupt+0xf/0x20 [ 3488.066635] [ 3488.066638] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3488.066643] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3488.066648] do_raw_spin_lock+0x177/0x190 [ 3488.066653] _raw_spin_lock+0x21/0x30 [ 3488.066658] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3488.080839] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3488.080928] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3488.080935] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3488.080937] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3488.080952] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3488.080956] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3488.080961] RAX: 0000000000004946 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3488.080963] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 3488.080965] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3488.080978] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 3488.080981] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 3488.080984] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 3488.080987] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3488.080989] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 3488.080997] Call Trace: [ 3488.081002] [ 3488.081007] ? show_regs.cold.9+0x22/0x2f [ 3488.081016] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3488.081022] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3488.081026] ? watchdog+0x40/0x40 [ 3488.081031] ? hrtimer_interrupt+0x128/0x2c0 [ 3488.081035] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3488.081044] ? apic_timer_interrupt+0xf/0x20 [ 3488.081048] [ 3488.081051] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3488.081058] do_raw_spin_lock+0x177/0x190 [ 3488.081063] _raw_spin_lock+0x21/0x30 [ 3488.081067] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3488.081339] ? do_raw_spin_unlock+0x75/0x190 [ 3488.081345] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3488.083297] ? do_wait_intr+0xf0/0xf0 [ 3488.083310] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3488.084070] mdc_close+0x53b/0xd70 [mdc] [ 3488.084107] lmv_close+0x18d/0x490 [lmv] [ 3488.084134] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3488.084581] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3488.084698] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3488.084937] ll_md_close+0x274/0x800 [lustre] [ 3488.085019] ll_file_release+0x128/0xc50 [lustre] [ 3488.085195] __fput+0x10d/0x440 [ 3488.085205] ____fput+0x12/0x20 [ 3488.085209] task_work_run+0xc6/0x110 [ 3488.085216] exit_to_usermode_loop+0x1dd/0x1f0 [ 3488.085293] do_syscall_64+0x3d6/0x440 [ 3488.085299] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3488.085308] RIP: 0033:0x7f151bee3cd8 [ 3488.085315] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3488.085319] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3488.085324] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 3488.085327] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 3488.085329] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3488.085331] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3488.085333] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3488.816441] ? do_raw_spin_unlock+0x75/0x190 [ 3488.820599] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3488.824813] ? do_wait_intr+0xf0/0xf0 [ 3488.827123] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3488.832752] mdc_reint+0x2d/0x110 [mdc] [ 3488.834843] mdc_unlink+0x658/0x870 [mdc] [ 3488.838496] lmv_unlink+0x34d/0xbd0 [lmv] [ 3488.840005] ll_unlink+0x24d/0x7f0 [lustre] [ 3488.848126] vfs_unlink+0x202/0x330 [ 3488.850822] do_unlinkat+0x2fa/0x460 [ 3488.855867] __x64_sys_unlink+0x37/0x50 [ 3488.860457] do_syscall_64+0xc1/0x440 [ 3488.862150] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3488.868377] RIP: 0033:0x7f151bdee10b [ 3488.869531] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3488.883032] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3488.886612] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 3488.891619] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 3488.899429] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3488.904666] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3488.910202] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3493.860231] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1563s! [ 3493.870117] Showing busy workqueues and worker pools: [ 3493.873695] workqueue events: flags=0x0 [ 3493.879883] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3493.883676] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3493.888147] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3493.892145] pending: kfree_rcu_monitor [ 3493.894488] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3493.899125] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 3493.905064] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3493.909484] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3493.914486] workqueue events_long: flags=0x0 [ 3493.917033] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3493.920526] pending: lu_objects_flush [obdclass] [ 3493.922751] workqueue events_unbound: flags=0x2 [ 3493.924250] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3493.927479] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3493.934211] workqueue events_power_efficient: flags=0x80 [ 3493.937952] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3493.942506] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 3493.951737] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3493.954187] pending: neigh_periodic_work [ 3493.962655] workqueue mm_percpu_wq: flags=0x8 [ 3493.970560] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3493.979735] pending: vmstat_update, lru_add_drain_per_cpu [ 3493.985438] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3493.987656] pending: vmstat_update [ 3493.998150] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3494.005478] pending: lru_add_drain_per_cpu [ 3494.011919] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3494.017786] pending: lru_add_drain_per_cpu BAR(48) [ 3494.023992] workqueue writeback: flags=0x4a [ 3494.041414] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3494.055506] pending: wb_workfn [ 3494.061965] workqueue xprtiod: flags=0xa [ 3494.082887] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3494.085360] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3494.104286] workqueue ipv6_addrconf: flags=0x40008 [ 3494.112379] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3494.120794] pending: addrconf_verify_work [ 3494.121943] workqueue knbd0-recv: flags=0x1a [ 3494.123269] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3494.137125] in-flight: 658:recv_work [ 3494.139121] workqueue ptlrpc_pinger: flags=0x2 [ 3494.140266] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3494.150763] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3494.155719] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1563s workers=4 idle: 32984 37 8 [ 3494.159249] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3478s workers=2 idle: 661 [ 3496.056018] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3496.060171] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3496.088039] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3496.094862] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3496.099413] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3496.102648] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3496.105023] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3496.113440] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3496.116474] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3496.119989] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 3496.123366] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 3496.126496] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 3496.129940] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 3496.133451] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 3496.137622] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3496.140844] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3496.144371] Call Trace: [ 3496.145908] [ 3496.146869] ? show_regs.cold.9+0x22/0x2f [ 3496.148354] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3496.150440] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3496.154370] ? watchdog+0x40/0x40 [ 3496.157109] ? hrtimer_interrupt+0x128/0x2c0 [ 3496.160302] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3496.163967] ? apic_timer_interrupt+0xf/0x20 [ 3496.167133] [ 3496.168669] ? native_safe_halt+0x2e/0x50 [ 3496.170829] kvm_wait+0xcd/0xe0 [ 3496.173380] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3496.176355] do_raw_spin_lock+0x177/0x190 [ 3496.178953] _raw_spin_lock+0x21/0x30 [ 3496.181375] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3496.185925] process_one_work+0x2c8/0x700 [ 3496.188351] worker_thread+0x296/0x6e0 [ 3496.191083] ? rescuer_thread+0x570/0x570 [ 3496.193674] kthread+0x1d1/0x200 [ 3496.195951] ? set_kthread_struct+0x70/0x70 [ 3496.198699] ret_from_fork+0x1f/0x30 [ 3508.055035] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 3508.059713] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3508.103411] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3508.120242] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3508.128560] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3508.131678] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3508.148437] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3508.156107] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 3508.160959] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 3508.170602] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 3508.178624] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 3508.184972] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3508.190848] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 3508.196066] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3508.203499] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3508.216160] Call Trace: [ 3508.218940] [ 3508.222310] ? show_regs.cold.9+0x22/0x2f [ 3508.223712] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3508.226790] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3508.228733] ? watchdog+0x40/0x40 [ 3508.230605] ? hrtimer_interrupt+0x128/0x2c0 [ 3508.232503] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3508.235316] ? apic_timer_interrupt+0xf/0x20 [ 3508.236817] [ 3508.237518] ? native_safe_halt+0x2e/0x50 [ 3508.241215] kvm_wait+0xcd/0xe0 [ 3508.243011] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3508.245687] do_raw_spin_lock+0x177/0x190 [ 3508.248122] _raw_spin_lock+0x21/0x30 [ 3508.250531] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3508.252621] obd_mod_free+0x46/0x140 [ptlrpc] [ 3508.259156] mdc_commit_open+0x91/0xb0 [mdc] [ 3508.260512] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3508.264878] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3508.266751] after_reply+0xe2c/0x1780 [ptlrpc] [ 3508.271477] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3508.275742] ? __next_timer_interrupt+0x160/0x160 [ 3508.278217] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3508.280738] ? do_wait_intr+0xf0/0xf0 [ 3508.282724] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3508.285104] kthread+0x1d1/0x200 [ 3508.286348] ? set_kthread_struct+0x70/0x70 [ 3508.288338] ret_from_fork+0x1f/0x30 [ 3516.057022] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 3516.060266] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3516.084223] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3516.087835] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3516.090343] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3516.091516] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3516.098873] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3516.101495] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3516.104120] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 3516.106833] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3516.109582] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 3516.112435] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 3516.114363] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 3516.116739] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3516.119029] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 3516.121800] Call Trace: [ 3516.122367] [ 3516.122777] ? show_regs.cold.9+0x22/0x2f [ 3516.123657] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3516.126607] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3516.128931] ? watchdog+0x40/0x40 [ 3516.130578] ? hrtimer_interrupt+0x128/0x2c0 [ 3516.132899] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3516.135220] ? apic_timer_interrupt+0xf/0x20 [ 3516.137875] [ 3516.138818] ? native_safe_halt+0x2e/0x50 [ 3516.140287] kvm_wait+0xcd/0xe0 [ 3516.141513] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3516.143461] do_raw_spin_lock+0x177/0x190 [ 3516.145273] _raw_spin_lock+0x21/0x30 [ 3516.147246] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3516.150777] ? do_raw_spin_unlock+0x75/0x190 [ 3516.152944] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3516.155794] ? do_wait_intr+0xf0/0xf0 [ 3516.157138] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3516.161309] mdc_close+0x53b/0xd70 [mdc] [ 3516.163074] lmv_close+0x18d/0x490 [lmv] [ 3516.165431] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3516.171140] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3516.173071] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3516.175056] ll_md_close+0x274/0x800 [lustre] [ 3516.177899] ll_file_release+0x128/0xc50 [lustre] [ 3516.180448] __fput+0x10d/0x440 [ 3516.183324] ____fput+0x12/0x20 [ 3516.185244] task_work_run+0xc6/0x110 [ 3516.188196] exit_to_usermode_loop+0x1dd/0x1f0 [ 3516.191569] do_syscall_64+0x3d6/0x440 [ 3516.194449] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3516.197927] RIP: 0033:0x7f151bee3cd8 [ 3516.199143] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3516.213194] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3516.220056] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 3516.232188] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 3516.239923] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3516.245051] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3516.250545] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3520.052030] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 3520.055586] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3520.087310] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3520.098766] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3520.111444] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3520.119083] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3520.146270] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3520.155032] RAX: 0000000000005828 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3520.159865] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 3520.166020] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 3520.174333] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 3520.177423] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 3520.182068] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 3520.189330] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3520.196938] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 3520.200797] Call Trace: [ 3520.204765] [ 3520.205612] ? show_regs.cold.9+0x22/0x2f [ 3520.210448] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3520.212725] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3520.216589] ? watchdog+0x40/0x40 [ 3520.218970] ? hrtimer_interrupt+0x128/0x2c0 [ 3520.221835] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3520.230764] ? apic_timer_interrupt+0xf/0x20 [ 3520.231996] [ 3520.232517] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3520.235825] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3520.242598] do_raw_spin_lock+0x177/0x190 [ 3520.245346] _raw_spin_lock+0x21/0x30 [ 3520.248258] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3520.250138] ? do_raw_spin_unlock+0x75/0x190 [ 3520.252063] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3520.254366] ? do_wait_intr+0xf0/0xf0 [ 3520.255768] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3520.257855] mdc_reint+0x2d/0x110 [mdc] [ 3520.259128] mdc_unlink+0x658/0x870 [mdc] [ 3520.260661] lmv_unlink+0x34d/0xbd0 [lmv] [ 3520.262061] ll_unlink+0x24d/0x7f0 [lustre] [ 3520.264110] vfs_unlink+0x202/0x330 [ 3520.265111] do_unlinkat+0x2fa/0x460 [ 3520.265875] __x64_sys_unlink+0x37/0x50 [ 3520.267339] do_syscall_64+0xc1/0x440 [ 3520.268462] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3520.270302] RIP: 0033:0x7f151bdee10b [ 3520.271639] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3520.277645] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3520.280021] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 3520.282569] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 3520.285371] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3520.287606] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3520.291241] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3524.057579] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 3524.071559] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3524.101155] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3524.106359] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3524.109560] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3524.113316] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3524.116139] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3524.126253] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3524.140117] RAX: 0000000000007e20 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3524.144407] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 3524.151458] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 3524.155505] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 3524.159101] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 3524.164238] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 3524.174352] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3524.182538] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3524.196757] Call Trace: [ 3524.197635] [ 3524.198379] ? show_regs.cold.9+0x22/0x2f [ 3524.201531] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3524.204372] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3524.207312] ? watchdog+0x40/0x40 [ 3524.209314] ? hrtimer_interrupt+0x128/0x2c0 [ 3524.214332] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3524.217054] ? apic_timer_interrupt+0xf/0x20 [ 3524.219238] [ 3524.221255] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3524.226539] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3524.234473] do_raw_spin_lock+0x177/0x190 [ 3524.245438] _raw_spin_lock+0x21/0x30 [ 3524.250530] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3524.263630] process_one_work+0x2c8/0x700 [ 3524.265959] worker_thread+0x296/0x6e0 [ 3524.268350] ? rescuer_thread+0x570/0x570 [ 3524.272761] kthread+0x1d1/0x200 [ 3524.280499] ? set_kthread_struct+0x70/0x70 [ 3524.290049] ret_from_fork+0x1f/0x30 [ 3524.584087] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1593s! [ 3524.586840] Showing busy workqueues and worker pools: [ 3524.589329] workqueue events: flags=0x0 [ 3524.591058] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3524.594167] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3524.596845] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3524.599922] pending: kfree_rcu_monitor [ 3524.601596] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3524.605067] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 3524.610020] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3524.613151] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3524.617224] workqueue events_long: flags=0x0 [ 3524.619453] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3524.623703] pending: lu_objects_flush [obdclass] [ 3524.627650] workqueue events_unbound: flags=0x2 [ 3524.631425] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3524.634558] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3524.644387] workqueue events_power_efficient: flags=0x80 [ 3524.646929] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3524.650437] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 3524.657420] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3524.666880] pending: neigh_periodic_work [ 3524.668706] workqueue mm_percpu_wq: flags=0x8 [ 3524.673477] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3524.677422] pending: vmstat_update, lru_add_drain_per_cpu [ 3524.679300] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3524.684551] pending: vmstat_update [ 3524.687018] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3524.691707] pending: lru_add_drain_per_cpu [ 3524.694414] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3524.699410] pending: lru_add_drain_per_cpu BAR(48) [ 3524.702820] workqueue writeback: flags=0x4a [ 3524.705816] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3524.709691] pending: wb_workfn [ 3524.712418] workqueue xprtiod: flags=0xa [ 3524.719914] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3524.724230] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3524.728024] workqueue ipv6_addrconf: flags=0x40008 [ 3524.730087] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3524.733649] pending: addrconf_verify_work [ 3524.736481] workqueue knbd0-recv: flags=0x1a [ 3524.738678] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3524.742039] in-flight: 658:recv_work [ 3524.744215] workqueue ptlrpc_pinger: flags=0x2 [ 3524.747687] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3524.749738] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3524.756477] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1594s workers=4 idle: 32984 37 8 [ 3524.763428] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3509s workers=2 idle: 661 [ 3536.055007] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 3536.059404] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3536.090104] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3536.097359] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3536.102007] RIP: 0010:__pv_queued_spin_lock_slowpath+0x185/0x6d0 [ 3536.105191] Code: 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 48 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 <0f> b6 03 84 c0 75 e6 48 83 05 2c d6 dc 02 01 44 89 f8 f0 66 44 0f [ 3536.116150] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3536.120693] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 3536.124743] RDX: 0000000000007bc1 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 3536.128454] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 3536.131831] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 3536.135684] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3536.139669] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 3536.143946] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3536.146949] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3536.151424] Call Trace: [ 3536.152753] [ 3536.154209] ? show_regs.cold.9+0x22/0x2f [ 3536.156436] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3536.160703] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3536.164410] ? watchdog+0x40/0x40 [ 3536.166524] ? hrtimer_interrupt+0x128/0x2c0 [ 3536.169781] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3536.172879] ? apic_timer_interrupt+0xf/0x20 [ 3536.176025] [ 3536.177927] ? __pv_queued_spin_lock_slowpath+0x185/0x6d0 [ 3536.180640] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3536.184721] do_raw_spin_lock+0x177/0x190 [ 3536.187039] _raw_spin_lock+0x21/0x30 [ 3536.189105] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3536.192380] obd_mod_free+0x46/0x140 [ptlrpc] [ 3536.195104] mdc_commit_open+0x91/0xb0 [mdc] [ 3536.197293] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3536.200373] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3536.203641] after_reply+0xe2c/0x1780 [ptlrpc] [ 3536.207535] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3536.210870] ? __next_timer_interrupt+0x160/0x160 [ 3536.214069] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3536.217218] ? do_wait_intr+0xf0/0xf0 [ 3536.219841] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3536.223070] kthread+0x1d1/0x200 [ 3536.225887] ? set_kthread_struct+0x70/0x70 [ 3536.228538] ret_from_fork+0x1f/0x30 [ 3544.058150] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 3544.061823] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3544.102288] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3544.106542] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3544.110145] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3544.112625] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3544.120809] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3544.126316] RAX: 0000000000002b28 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3544.128855] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 3544.131491] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3544.135323] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 3544.146035] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 3544.148672] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 3544.155690] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3544.166421] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 3544.180232] Call Trace: [ 3544.183873] [ 3544.185985] ? show_regs.cold.9+0x22/0x2f [ 3544.189993] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3544.193756] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3544.195805] ? watchdog+0x40/0x40 [ 3544.197662] ? hrtimer_interrupt+0x128/0x2c0 [ 3544.199580] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3544.205896] ? apic_timer_interrupt+0xf/0x20 [ 3544.208128] [ 3544.209268] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3544.212458] do_raw_spin_lock+0x177/0x190 [ 3544.214211] _raw_spin_lock+0x21/0x30 [ 3544.215865] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3544.218234] ? do_raw_spin_unlock+0x75/0x190 [ 3544.219975] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3544.222416] ? do_wait_intr+0xf0/0xf0 [ 3544.225197] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3544.228178] mdc_close+0x53b/0xd70 [mdc] [ 3544.229955] lmv_close+0x18d/0x490 [lmv] [ 3544.231206] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3544.232973] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3544.234666] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3544.236271] ll_md_close+0x274/0x800 [lustre] [ 3544.237770] ll_file_release+0x128/0xc50 [lustre] [ 3544.239116] __fput+0x10d/0x440 [ 3544.239979] ____fput+0x12/0x20 [ 3544.240936] task_work_run+0xc6/0x110 [ 3544.241949] exit_to_usermode_loop+0x1dd/0x1f0 [ 3544.243073] do_syscall_64+0x3d6/0x440 [ 3544.244576] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3544.247009] RIP: 0033:0x7f151bee3cd8 [ 3544.248265] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3544.256222] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3544.259585] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 3544.261641] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 3544.263998] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3544.266893] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3544.269969] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3552.052646] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 3552.056261] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 3552.058086] Modules linked in: lustre(O) [ 3552.071129] Modules linked in: [ 3552.071132] osp(O) ofd(O) [ 3552.074546] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3552.074621] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3552.074629] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3552.074635] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3552.075256] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3552.075264] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3552.075267] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3552.075369] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3552.075370] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 3552.075372] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 3552.075373] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 3552.075375] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 3552.075377] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 3552.075379] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3552.075381] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3552.075387] Call Trace: [ 3552.075391] [ 3552.075395] ? show_regs.cold.9+0x22/0x2f [ 3552.075403] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3552.075408] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3552.075410] ? watchdog+0x40/0x40 [ 3552.075413] ? hrtimer_interrupt+0x128/0x2c0 [ 3552.075416] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3552.075420] ? apic_timer_interrupt+0xf/0x20 [ 3552.075424] [ 3552.075426] ? native_safe_halt+0x2e/0x50 [ 3552.075428] kvm_wait+0xcd/0xe0 [ 3552.075433] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3552.075439] do_raw_spin_lock+0x177/0x190 [ 3552.075497] _raw_spin_lock+0x21/0x30 [ 3552.075503] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3552.075689] process_one_work+0x2c8/0x700 [ 3552.075696] worker_thread+0x296/0x6e0 [ 3552.075700] ? rescuer_thread+0x570/0x570 [ 3552.075702] kthread+0x1d1/0x200 [ 3552.075705] ? set_kthread_struct+0x70/0x70 [ 3552.075707] ret_from_fork+0x1f/0x30 [ 3552.248885] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3552.274271] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3552.280676] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3552.290281] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3552.297614] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3552.316768] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3552.321103] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3552.324308] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 3552.329693] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 3552.333676] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 3552.337292] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 3552.339917] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 3552.344522] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3552.348052] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 3552.352235] Call Trace: [ 3552.353523] [ 3552.354843] ? show_regs.cold.9+0x22/0x2f [ 3552.358556] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3552.360595] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3552.364191] ? watchdog+0x40/0x40 [ 3552.366225] ? hrtimer_interrupt+0x128/0x2c0 [ 3552.373884] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3552.380707] ? apic_timer_interrupt+0xf/0x20 [ 3552.383689] [ 3552.385021] ? native_safe_halt+0x2e/0x50 [ 3552.386999] kvm_wait+0xcd/0xe0 [ 3552.388329] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3552.391054] do_raw_spin_lock+0x177/0x190 [ 3552.392662] _raw_spin_lock+0x21/0x30 [ 3552.396022] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3552.400840] ? do_raw_spin_unlock+0x75/0x190 [ 3552.403186] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3552.405399] ? do_wait_intr+0xf0/0xf0 [ 3552.407515] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3552.410102] mdc_reint+0x2d/0x110 [mdc] [ 3552.412019] mdc_unlink+0x658/0x870 [mdc] [ 3552.414216] lmv_unlink+0x34d/0xbd0 [lmv] [ 3552.415235] ll_unlink+0x24d/0x7f0 [lustre] [ 3552.419180] vfs_unlink+0x202/0x330 [ 3552.423951] do_unlinkat+0x2fa/0x460 [ 3552.430701] __x64_sys_unlink+0x37/0x50 [ 3552.433105] do_syscall_64+0xc1/0x440 [ 3552.435147] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3552.438392] RIP: 0033:0x7f151bdee10b [ 3552.441118] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3552.451091] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3552.454703] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 3552.458100] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 3552.461312] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3552.465084] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3552.468067] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3554.784236] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1624s! [ 3554.788595] Showing busy workqueues and worker pools: [ 3554.798075] workqueue events: flags=0x0 [ 3554.800929] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3554.806806] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3554.810448] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3554.813137] pending: kfree_rcu_monitor [ 3554.814529] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3554.818859] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 3554.827282] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3554.833241] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3554.836943] workqueue events_long: flags=0x0 [ 3554.842893] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3554.848440] pending: lu_objects_flush [obdclass] [ 3554.852321] workqueue events_unbound: flags=0x2 [ 3554.856444] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3554.858426] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3554.864659] workqueue events_power_efficient: flags=0x80 [ 3554.870787] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3554.874471] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 3554.877263] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3554.879852] pending: neigh_periodic_work [ 3554.881453] workqueue mm_percpu_wq: flags=0x8 [ 3554.883207] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3554.885767] pending: vmstat_update, lru_add_drain_per_cpu [ 3554.888200] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3554.890856] pending: vmstat_update [ 3554.892397] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3554.894879] pending: lru_add_drain_per_cpu [ 3554.896812] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3554.899473] pending: lru_add_drain_per_cpu BAR(48) [ 3554.901356] workqueue writeback: flags=0x4a [ 3554.902966] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3554.905448] pending: wb_workfn [ 3554.906863] workqueue xprtiod: flags=0xa [ 3554.908548] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3554.911166] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3554.914278] workqueue ipv6_addrconf: flags=0x40008 [ 3554.916294] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3554.918847] pending: addrconf_verify_work [ 3554.920015] workqueue knbd0-recv: flags=0x1a [ 3554.921070] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3554.922632] in-flight: 658:recv_work [ 3554.924055] workqueue ptlrpc_pinger: flags=0x2 [ 3554.925738] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3554.927918] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3554.930290] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1624s workers=4 idle: 32984 37 8 [ 3554.933066] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3539s workers=2 idle: 661 [ 3564.056774] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 3564.060953] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3564.086930] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3564.093999] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3564.104132] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3564.108560] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3564.125937] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3564.129434] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 3564.140127] RDX: 00000000000012e5 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 3564.146146] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 3564.150439] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 3564.154941] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3564.159410] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 3564.163457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3564.166000] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3564.176968] Call Trace: [ 3564.180094] [ 3564.187809] ? show_regs.cold.9+0x22/0x2f [ 3564.189168] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3564.199282] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3564.204901] ? watchdog+0x40/0x40 [ 3564.208969] ? hrtimer_interrupt+0x128/0x2c0 [ 3564.214015] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3564.222300] ? apic_timer_interrupt+0xf/0x20 [ 3564.229248] [ 3564.234877] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3564.240454] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3564.244100] do_raw_spin_lock+0x177/0x190 [ 3564.247257] _raw_spin_lock+0x21/0x30 [ 3564.250035] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3564.253258] obd_mod_free+0x46/0x140 [ptlrpc] [ 3564.255955] mdc_commit_open+0x91/0xb0 [mdc] [ 3564.258177] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3564.260358] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3564.263099] after_reply+0xe2c/0x1780 [ptlrpc] [ 3564.265303] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3564.267866] ? __next_timer_interrupt+0x160/0x160 [ 3564.269848] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3564.271855] ? do_wait_intr+0xf0/0xf0 [ 3564.273402] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3564.275959] kthread+0x1d1/0x200 [ 3564.277365] ? set_kthread_struct+0x70/0x70 [ 3564.278997] ret_from_fork+0x1f/0x30 [ 3572.058471] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 3572.069925] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3572.120934] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3572.135364] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3572.142537] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3572.152320] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3572.175019] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3572.177791] RAX: 00000000000027d4 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3572.188624] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 3572.195145] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3572.203786] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 3572.210188] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 3572.219314] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 3572.227165] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3572.236303] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 3572.241021] Call Trace: [ 3572.245240] [ 3572.248794] ? show_regs.cold.9+0x22/0x2f [ 3572.251653] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3572.258649] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3572.263639] ? watchdog+0x40/0x40 [ 3572.267999] ? hrtimer_interrupt+0x128/0x2c0 [ 3572.274124] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3572.279822] ? apic_timer_interrupt+0xf/0x20 [ 3572.281392] [ 3572.282158] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3572.285927] do_raw_spin_lock+0x177/0x190 [ 3572.292240] _raw_spin_lock+0x21/0x30 [ 3572.296646] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3572.305385] ? do_raw_spin_unlock+0x75/0x190 [ 3572.309195] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3572.312734] ? do_wait_intr+0xf0/0xf0 [ 3572.314620] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3572.317739] mdc_close+0x53b/0xd70 [mdc] [ 3572.319701] lmv_close+0x18d/0x490 [lmv] [ 3572.322472] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3572.327573] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3572.332380] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3572.335254] ll_md_close+0x274/0x800 [lustre] [ 3572.348256] ll_file_release+0x128/0xc50 [lustre] [ 3572.354478] __fput+0x10d/0x440 [ 3572.358136] ____fput+0x12/0x20 [ 3572.363153] task_work_run+0xc6/0x110 [ 3572.370223] exit_to_usermode_loop+0x1dd/0x1f0 [ 3572.372929] do_syscall_64+0x3d6/0x440 [ 3572.374364] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3572.379080] RIP: 0033:0x7f151bee3cd8 [ 3572.381672] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3572.407923] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3572.415443] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 3572.419135] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 3572.427629] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3572.435660] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3572.444634] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3580.053828] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 3580.056058] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 3580.056821] Modules linked in: [ 3580.073357] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3580.073438] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3580.073446] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3580.073452] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3580.073742] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3580.073752] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3580.073756] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3580.073761] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3580.073763] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 3580.073765] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 3580.073767] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 3580.073770] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 3580.073774] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 3580.073777] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3580.073779] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3580.073788] Call Trace: [ 3580.073793] [ 3580.073799] ? show_regs.cold.9+0x22/0x2f [ 3580.073810] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3580.073816] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3580.073820] ? watchdog+0x40/0x40 [ 3580.073824] ? hrtimer_interrupt+0x128/0x2c0 [ 3580.073829] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3580.073836] ? apic_timer_interrupt+0xf/0x20 [ 3580.073851] [ 3580.073854] ? native_safe_halt+0x2e/0x50 [ 3580.073859] kvm_wait+0xcd/0xe0 [ 3580.073875] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3580.073883] do_raw_spin_lock+0x177/0x190 [ 3580.073887] _raw_spin_lock+0x21/0x30 [ 3580.073891] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3580.074104] process_one_work+0x2c8/0x700 [ 3580.074111] worker_thread+0x296/0x6e0 [ 3580.074116] ? rescuer_thread+0x570/0x570 [ 3580.074120] kthread+0x1d1/0x200 [ 3580.074124] ? set_kthread_struct+0x70/0x70 [ 3580.074129] ret_from_fork+0x1f/0x30 [ 3580.265985] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3580.299212] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3580.307073] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3580.312086] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3580.315172] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3580.334314] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3580.337090] RAX: 00000000000009f8 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3580.342971] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 3580.355097] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 3580.361014] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 3580.364105] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 3580.367151] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 3580.377056] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3580.380939] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 3580.385084] Call Trace: [ 3580.386476] [ 3580.387012] ? show_regs.cold.9+0x22/0x2f [ 3580.392987] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3580.398156] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3580.399689] ? watchdog+0x40/0x40 [ 3580.400997] ? hrtimer_interrupt+0x128/0x2c0 [ 3580.405046] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3580.407232] ? apic_timer_interrupt+0xf/0x20 [ 3580.408391] [ 3580.408972] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3580.414311] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3580.416159] do_raw_spin_lock+0x177/0x190 [ 3580.420414] _raw_spin_lock+0x21/0x30 [ 3580.422187] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3580.425492] ? do_raw_spin_unlock+0x75/0x190 [ 3580.431853] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3580.436344] ? do_wait_intr+0xf0/0xf0 [ 3580.437377] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3580.439861] mdc_reint+0x2d/0x110 [mdc] [ 3580.444654] mdc_unlink+0x658/0x870 [mdc] [ 3580.445899] lmv_unlink+0x34d/0xbd0 [lmv] [ 3580.447098] ll_unlink+0x24d/0x7f0 [lustre] [ 3580.455064] vfs_unlink+0x202/0x330 [ 3580.456013] do_unlinkat+0x2fa/0x460 [ 3580.456948] __x64_sys_unlink+0x37/0x50 [ 3580.459300] do_syscall_64+0xc1/0x440 [ 3580.460636] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3580.469444] RIP: 0033:0x7f151bdee10b [ 3580.471138] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3580.489230] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3580.495049] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 3580.504089] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 3580.507009] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3580.509331] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3580.514621] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3584.993092] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1654s! [ 3584.998259] Showing busy workqueues and worker pools: [ 3585.003074] workqueue events: flags=0x0 [ 3585.021554] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3585.034879] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3585.048887] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3585.072048] pending: kfree_rcu_monitor [ 3585.083603] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3585.095784] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 3585.110411] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3585.121440] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3585.133646] workqueue events_long: flags=0x0 [ 3585.135289] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3585.150599] pending: lu_objects_flush [obdclass] [ 3585.161700] workqueue events_unbound: flags=0x2 [ 3585.177565] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3585.181644] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3585.196760] workqueue events_power_efficient: flags=0x80 [ 3585.201025] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3585.204978] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 3585.209840] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3585.221096] pending: neigh_periodic_work [ 3585.227465] workqueue mm_percpu_wq: flags=0x8 [ 3585.231037] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3585.234375] pending: vmstat_update, lru_add_drain_per_cpu [ 3585.238534] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3585.242537] pending: vmstat_update [ 3585.244557] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3585.248169] pending: lru_add_drain_per_cpu [ 3585.252270] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3585.257433] pending: lru_add_drain_per_cpu BAR(48) [ 3585.261524] workqueue writeback: flags=0x4a [ 3585.265462] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3585.268996] pending: wb_workfn [ 3585.274666] workqueue xprtiod: flags=0xa [ 3585.278104] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3585.284776] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3585.292629] workqueue ipv6_addrconf: flags=0x40008 [ 3585.300902] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3585.304703] pending: addrconf_verify_work [ 3585.309574] workqueue knbd0-recv: flags=0x1a [ 3585.318117] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3585.322768] in-flight: 658:recv_work [ 3585.327867] workqueue ptlrpc_pinger: flags=0x2 [ 3585.333261] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3585.344219] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3585.353438] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1654s workers=4 idle: 32984 37 8 [ 3585.359143] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3570s workers=2 idle: 661 [ 3592.057161] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 3592.071617] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3592.121996] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3592.140572] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3592.146559] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3592.150824] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3592.172299] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3592.184648] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 3592.192660] RDX: 0000000000003afe RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 3592.200618] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 3592.205558] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 3592.214306] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3592.218767] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 3592.228651] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3592.232319] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3592.242385] Call Trace: [ 3592.245459] [ 3592.247186] ? show_regs.cold.9+0x22/0x2f [ 3592.251146] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3592.257037] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3592.258698] ? watchdog+0x40/0x40 [ 3592.264991] ? hrtimer_interrupt+0x128/0x2c0 [ 3592.271054] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3592.282051] ? apic_timer_interrupt+0xf/0x20 [ 3592.292570] [ 3592.294051] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3592.299501] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3592.306451] do_raw_spin_lock+0x177/0x190 [ 3592.315780] _raw_spin_lock+0x21/0x30 [ 3592.319609] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3592.326458] obd_mod_free+0x46/0x140 [ptlrpc] [ 3592.331662] mdc_commit_open+0x91/0xb0 [mdc] [ 3592.339665] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3592.348062] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3592.354086] after_reply+0xe2c/0x1780 [ptlrpc] [ 3592.360682] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3592.362770] ? __next_timer_interrupt+0x160/0x160 [ 3592.378924] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3592.383336] ? do_wait_intr+0xf0/0xf0 [ 3592.387499] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3592.394440] kthread+0x1d1/0x200 [ 3592.398119] ? set_kthread_struct+0x70/0x70 [ 3592.406053] ret_from_fork+0x1f/0x30 [ 3600.057061] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 3600.060311] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3600.114594] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3600.128187] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3600.141567] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3600.146694] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3600.167331] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3600.174734] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3600.187850] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 3600.195480] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3600.199842] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 3600.206183] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 3600.212770] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 3600.224483] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3600.232710] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 3600.240124] Call Trace: [ 3600.245911] [ 3600.249593] ? show_regs.cold.9+0x22/0x2f [ 3600.255053] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3600.260988] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3600.264040] ? watchdog+0x40/0x40 [ 3600.265196] ? hrtimer_interrupt+0x128/0x2c0 [ 3600.269314] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3600.279992] ? apic_timer_interrupt+0xf/0x20 [ 3600.288340] [ 3600.289582] ? native_safe_halt+0x2e/0x50 [ 3600.301842] kvm_wait+0xcd/0xe0 [ 3600.304015] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3600.313253] do_raw_spin_lock+0x177/0x190 [ 3600.323824] _raw_spin_lock+0x21/0x30 [ 3600.326628] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3600.333872] ? do_raw_spin_unlock+0x75/0x190 [ 3600.341089] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3600.346304] ? do_wait_intr+0xf0/0xf0 [ 3600.352626] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3600.363417] mdc_close+0x53b/0xd70 [mdc] [ 3600.370670] lmv_close+0x18d/0x490 [lmv] [ 3600.376738] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3600.396070] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3600.405682] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3600.410491] ll_md_close+0x274/0x800 [lustre] [ 3600.414871] ll_file_release+0x128/0xc50 [lustre] [ 3600.418185] __fput+0x10d/0x440 [ 3600.422078] ____fput+0x12/0x20 [ 3600.424615] task_work_run+0xc6/0x110 [ 3600.434486] exit_to_usermode_loop+0x1dd/0x1f0 [ 3600.437429] do_syscall_64+0x3d6/0x440 [ 3600.440860] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3600.449838] RIP: 0033:0x7f151bee3cd8 [ 3600.452179] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3600.461782] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3600.465366] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 3600.468330] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 3600.472418] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3600.476976] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3600.481670] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3604.677941] rcu: INFO: rcu_sched self-detected stall on CPU [ 3604.680329] rcu: 1-...!: (1014537 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 3604.687400] (t=1680030 jiffies g=249657 q=2909) [ 3604.689286] rcu: rcu_sched kthread starved for 1674982 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 3604.694451] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3604.698332] rcu: RCU grace-period kthread stack dump: [ 3604.700613] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3604.705031] Call Trace: [ 3604.706434] __schedule+0x351/0xcb0 [ 3604.708435] schedule+0xc0/0x180 [ 3604.709637] schedule_timeout+0xb4/0x190 [ 3604.712011] ? do_raw_spin_unlock+0x75/0x190 [ 3604.713912] ? __next_timer_interrupt+0x160/0x160 [ 3604.715601] ? prepare_to_swait_event+0xf3/0x140 [ 3604.719335] rcu_gp_kthread+0x8dd/0x15b0 [ 3604.720715] ? force_qs_rnp+0x2c0/0x2c0 [ 3604.723041] kthread+0x1d1/0x200 [ 3604.724170] ? set_kthread_struct+0x70/0x70 [ 3604.725623] ret_from_fork+0x1f/0x30 [ 3604.726754] rcu: Stack dump where RCU GP kthread last ran: [ 3604.729438] Sending NMI from CPU 1 to CPUs 0: [ 3604.732111] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [ 3604.733082] NMI backtrace for cpu 1 [ 3604.739259] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3604.747965] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3604.755653] Call Trace: [ 3604.756922] [ 3604.766234] ? dump_stack+0xbb/0x10e [ 3604.768195] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3604.770717] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3604.772708] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3604.776291] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3604.781285] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3604.784573] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 3604.787293] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 3604.788692] ? update_process_times+0x6d/0xc0 [ 3604.790847] ? tick_sched_handle.isra.18+0x27/0x90 [ 3604.794284] ? tick_sched_timer+0x53/0xb0 [ 3604.795373] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3604.797356] ? tick_sched_handle.isra.18+0x90/0x90 [ 3604.799515] ? hrtimer_interrupt+0x128/0x2c0 [ 3604.802646] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3604.804602] ? apic_timer_interrupt+0xf/0x20 [ 3604.806440] [ 3604.807467] ? native_safe_halt+0x2e/0x50 [ 3604.810299] ? kvm_wait+0xcd/0xe0 [ 3604.811511] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3604.813624] ? do_raw_spin_lock+0x177/0x190 [ 3604.816281] ? _raw_spin_lock+0x21/0x30 [ 3604.819640] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3604.823054] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3604.825806] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3604.827869] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3604.830449] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3604.833124] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3604.835050] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3604.836902] ? __next_timer_interrupt+0x160/0x160 [ 3604.838685] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3604.841266] ? do_wait_intr+0xf0/0xf0 [ 3604.843174] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3604.846610] ? kthread+0x1d1/0x200 [ 3604.847914] ? set_kthread_struct+0x70/0x70 [ 3604.849669] ? ret_from_fork+0x1f/0x30 [ 3608.060727] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 3608.070468] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3608.149391] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3608.167912] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3608.181397] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3608.191856] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3608.203849] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3608.229291] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3608.242720] RAX: 0000000000007f84 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3608.252441] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 3608.274345] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 3608.293400] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 3608.304874] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 3608.317405] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 3608.334810] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3608.341933] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3608.355529] Call Trace: [ 3608.358349] [ 3608.361355] ? show_regs.cold.9+0x22/0x2f [ 3608.367580] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3608.377587] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3608.386436] ? watchdog+0x40/0x40 [ 3608.391849] ? hrtimer_interrupt+0x128/0x2c0 [ 3608.398256] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3608.399914] ? apic_timer_interrupt+0xf/0x20 [ 3608.415139] [ 3608.417388] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3608.424791] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3608.433840] do_raw_spin_lock+0x177/0x190 [ 3608.436201] _raw_spin_lock+0x21/0x30 [ 3608.440833] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3608.449396] process_one_work+0x2c8/0x700 [ 3608.460876] worker_thread+0x296/0x6e0 [ 3608.462243] ? rescuer_thread+0x570/0x570 [ 3608.476514] kthread+0x1d1/0x200 [ 3608.491998] ? set_kthread_struct+0x70/0x70 [ 3608.494573] ret_from_fork+0x1f/0x30 [ 3615.715916] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1685s! [ 3615.724462] Showing busy workqueues and worker pools: [ 3615.728372] workqueue events: flags=0x0 [ 3615.733290] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3615.740149] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3615.742365] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3615.754670] pending: kfree_rcu_monitor [ 3615.760384] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3615.771025] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 3615.780284] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3615.785899] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3615.794620] workqueue events_long: flags=0x0 [ 3615.799929] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3615.806789] pending: lu_objects_flush [obdclass] [ 3615.814206] workqueue events_unbound: flags=0x2 [ 3615.823207] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3615.828243] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3615.835251] workqueue events_power_efficient: flags=0x80 [ 3615.838926] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3615.844596] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 3615.849180] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3615.854495] pending: neigh_periodic_work [ 3615.857201] workqueue mm_percpu_wq: flags=0x8 [ 3615.859766] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3615.863597] pending: vmstat_update, lru_add_drain_per_cpu [ 3615.868375] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3615.871184] pending: vmstat_update [ 3615.872570] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3615.876977] pending: lru_add_drain_per_cpu [ 3615.879905] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3615.883167] pending: lru_add_drain_per_cpu BAR(48) [ 3615.886175] workqueue writeback: flags=0x4a [ 3615.888611] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3615.892264] pending: wb_workfn [ 3615.894323] workqueue xprtiod: flags=0xa [ 3615.900616] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3615.913256] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3615.923078] workqueue ipv6_addrconf: flags=0x40008 [ 3615.930358] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3615.940154] pending: addrconf_verify_work [ 3615.942714] workqueue knbd0-recv: flags=0x1a [ 3615.946569] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3615.950751] in-flight: 658:recv_work [ 3615.953165] workqueue ptlrpc_pinger: flags=0x2 [ 3615.959757] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3615.967742] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3615.972969] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1685s workers=4 idle: 32984 37 8 [ 3615.978488] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3600s workers=2 idle: 661 [ 3628.057066] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 3628.060861] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3628.084901] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3628.090752] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3628.094441] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3628.097240] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3628.106541] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3628.110430] RAX: 0000000000000d0d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3628.114332] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 3628.117813] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3628.121796] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 3628.125240] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 3628.128564] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 3628.132425] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3628.138395] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 3628.141316] Call Trace: [ 3628.142476] [ 3628.143434] ? show_regs.cold.9+0x22/0x2f [ 3628.145394] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3628.147584] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3628.149790] ? watchdog+0x40/0x40 [ 3628.151403] ? hrtimer_interrupt+0x128/0x2c0 [ 3628.153433] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3628.155688] ? apic_timer_interrupt+0xf/0x20 [ 3628.157835] [ 3628.158807] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3628.161166] do_raw_spin_lock+0x177/0x190 [ 3628.162394] _raw_spin_lock+0x21/0x30 [ 3628.163457] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3628.165699] ? do_raw_spin_unlock+0x75/0x190 [ 3628.167511] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3628.170549] ? do_wait_intr+0xf0/0xf0 [ 3628.172092] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3628.173797] mdc_close+0x53b/0xd70 [mdc] [ 3628.175083] lmv_close+0x18d/0x490 [lmv] [ 3628.176595] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3628.179338] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3628.181756] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3628.184427] ll_md_close+0x274/0x800 [lustre] [ 3628.187146] ll_file_release+0x128/0xc50 [lustre] [ 3628.189435] __fput+0x10d/0x440 [ 3628.190862] ____fput+0x12/0x20 [ 3628.192489] task_work_run+0xc6/0x110 [ 3628.194212] exit_to_usermode_loop+0x1dd/0x1f0 [ 3628.196231] do_syscall_64+0x3d6/0x440 [ 3628.197972] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3628.200196] RIP: 0033:0x7f151bee3cd8 [ 3628.201827] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3628.210133] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3628.213456] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 3628.216523] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 3628.219685] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3628.222589] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3628.225990] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3632.055526] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 3632.063404] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3632.093208] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3632.099412] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3632.103205] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3632.104896] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3632.117138] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3632.122024] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 3632.130769] RDX: 0000000000000fb6 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 3632.133504] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 3632.141345] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 3632.143904] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3632.151241] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 3632.158529] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3632.164666] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3632.171801] Call Trace: [ 3632.174950] [ 3632.176340] ? show_regs.cold.9+0x22/0x2f [ 3632.177867] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3632.180477] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3632.182818] ? watchdog+0x40/0x40 [ 3632.184940] ? hrtimer_interrupt+0x128/0x2c0 [ 3632.191223] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3632.194039] ? apic_timer_interrupt+0xf/0x20 [ 3632.196292] [ 3632.197745] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3632.201531] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3632.203550] do_raw_spin_lock+0x177/0x190 [ 3632.212208] _raw_spin_lock+0x21/0x30 [ 3632.217066] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3632.221174] obd_mod_free+0x46/0x140 [ptlrpc] [ 3632.228184] mdc_commit_open+0x91/0xb0 [mdc] [ 3632.232787] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3632.236610] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3632.244112] after_reply+0xe2c/0x1780 [ptlrpc] [ 3632.247123] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3632.254457] ? __next_timer_interrupt+0x160/0x160 [ 3632.261952] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3632.265310] ? do_wait_intr+0xf0/0xf0 [ 3632.267233] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3632.270805] kthread+0x1d1/0x200 [ 3632.273382] ? set_kthread_struct+0x70/0x70 [ 3632.282813] ret_from_fork+0x1f/0x30 [ 3636.059058] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 3636.066485] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3636.110243] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3636.122476] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3636.131805] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3636.141222] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3636.147085] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3636.164668] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 3636.172689] RAX: 0000000000004b97 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3636.180404] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 3636.183850] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 3636.186415] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 3636.191249] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 3636.203666] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 3636.209800] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3636.212370] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3636.216208] Call Trace: [ 3636.218601] [ 3636.219424] ? show_regs.cold.9+0x22/0x2f [ 3636.221721] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3636.228318] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3636.232407] ? watchdog+0x40/0x40 [ 3636.234293] ? hrtimer_interrupt+0x128/0x2c0 [ 3636.237297] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3636.239405] ? apic_timer_interrupt+0xf/0x20 [ 3636.241741] [ 3636.243289] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3636.245967] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3636.248070] do_raw_spin_lock+0x177/0x190 [ 3636.252128] _raw_spin_lock+0x21/0x30 [ 3636.254262] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3636.258918] process_one_work+0x2c8/0x700 [ 3636.261385] worker_thread+0x296/0x6e0 [ 3636.266155] ? rescuer_thread+0x570/0x570 [ 3636.269365] kthread+0x1d1/0x200 [ 3636.273354] ? set_kthread_struct+0x70/0x70 [ 3636.276030] ret_from_fork+0x1f/0x30 [ 3640.055417] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 3640.067236] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3640.096139] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3640.100652] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3640.105059] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3640.107468] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3640.116335] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3640.120011] RAX: 0000000000006a08 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3640.123072] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 3640.127059] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 3640.131191] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 3640.134664] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 3640.137865] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 3640.142452] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3640.146519] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 3640.151072] Call Trace: [ 3640.151833] [ 3640.152662] ? show_regs.cold.9+0x22/0x2f [ 3640.154957] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3640.158243] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3640.162042] ? watchdog+0x40/0x40 [ 3640.163891] ? hrtimer_interrupt+0x128/0x2c0 [ 3640.165207] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3640.166603] ? apic_timer_interrupt+0xf/0x20 [ 3640.168503] [ 3640.169115] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3640.172305] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3640.175111] do_raw_spin_lock+0x177/0x190 [ 3640.177219] _raw_spin_lock+0x21/0x30 [ 3640.178260] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3640.182020] ? do_raw_spin_unlock+0x75/0x190 [ 3640.184247] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3640.186621] ? do_wait_intr+0xf0/0xf0 [ 3640.188910] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3640.190835] mdc_reint+0x2d/0x110 [mdc] [ 3640.194064] mdc_unlink+0x658/0x870 [mdc] [ 3640.195920] lmv_unlink+0x34d/0xbd0 [lmv] [ 3640.197519] ll_unlink+0x24d/0x7f0 [lustre] [ 3640.201396] vfs_unlink+0x202/0x330 [ 3640.202731] do_unlinkat+0x2fa/0x460 [ 3640.204145] __x64_sys_unlink+0x37/0x50 [ 3640.205932] do_syscall_64+0xc1/0x440 [ 3640.207356] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3640.209543] RIP: 0033:0x7f151bdee10b [ 3640.210586] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3640.219353] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3640.221469] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 3640.224290] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 3640.228365] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3640.231091] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3640.234174] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3646.433030] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1715s! [ 3646.437780] Showing busy workqueues and worker pools: [ 3646.441162] workqueue events: flags=0x0 [ 3646.444055] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3646.446128] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3646.450615] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3646.454398] pending: kfree_rcu_monitor [ 3646.456851] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3646.460594] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 3646.467547] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3646.471967] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3646.477407] workqueue events_long: flags=0x0 [ 3646.479671] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3646.483430] pending: lu_objects_flush [obdclass] [ 3646.486210] workqueue events_unbound: flags=0x2 [ 3646.489955] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3646.492016] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3646.502071] workqueue events_power_efficient: flags=0x80 [ 3646.505094] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3646.508435] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 3646.512858] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3646.517230] pending: neigh_periodic_work [ 3646.520207] workqueue mm_percpu_wq: flags=0x8 [ 3646.524423] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3646.529656] pending: vmstat_update, lru_add_drain_per_cpu [ 3646.535253] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3646.539386] pending: vmstat_update [ 3646.541630] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3646.545193] pending: lru_add_drain_per_cpu [ 3646.552681] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3646.563465] pending: lru_add_drain_per_cpu BAR(48) [ 3646.568117] workqueue writeback: flags=0x4a [ 3646.573150] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3646.577856] pending: wb_workfn [ 3646.579605] workqueue xprtiod: flags=0xa [ 3646.581473] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3646.584947] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3646.588180] workqueue ipv6_addrconf: flags=0x40008 [ 3646.589993] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3646.592623] pending: addrconf_verify_work [ 3646.594144] workqueue knbd0-recv: flags=0x1a [ 3646.595435] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3646.597538] in-flight: 658:recv_work [ 3646.599209] workqueue ptlrpc_pinger: flags=0x2 [ 3646.600470] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3646.606240] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3646.613211] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1716s workers=4 idle: 32984 37 8 [ 3646.621154] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3631s workers=2 idle: 661 [ 3656.061089] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 3656.074633] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3656.119865] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3656.135206] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3656.145814] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3656.149494] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3656.155956] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3656.158017] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3656.162787] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 3656.166492] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3656.170586] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 3656.175163] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 3656.181094] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 3656.188687] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3656.192931] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 3656.195568] Call Trace: [ 3656.197188] [ 3656.198071] ? show_regs.cold.9+0x22/0x2f [ 3656.199895] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3656.204114] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3656.206662] ? watchdog+0x40/0x40 [ 3656.208097] ? hrtimer_interrupt+0x128/0x2c0 [ 3656.213739] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3656.217365] ? apic_timer_interrupt+0xf/0x20 [ 3656.222019] [ 3656.222771] ? native_safe_halt+0x2e/0x50 [ 3656.226790] kvm_wait+0xcd/0xe0 [ 3656.229118] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3656.233387] do_raw_spin_lock+0x177/0x190 [ 3656.234826] _raw_spin_lock+0x21/0x30 [ 3656.236198] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3656.238073] ? do_raw_spin_unlock+0x75/0x190 [ 3656.245629] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3656.250357] ? do_wait_intr+0xf0/0xf0 [ 3656.257291] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3656.261225] mdc_close+0x53b/0xd70 [mdc] [ 3656.265318] lmv_close+0x18d/0x490 [lmv] [ 3656.267421] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3656.271878] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3656.277483] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3656.281179] ll_md_close+0x274/0x800 [lustre] [ 3656.283696] ll_file_release+0x128/0xc50 [lustre] [ 3656.285983] __fput+0x10d/0x440 [ 3656.288019] ____fput+0x12/0x20 [ 3656.289491] task_work_run+0xc6/0x110 [ 3656.291237] exit_to_usermode_loop+0x1dd/0x1f0 [ 3656.297838] do_syscall_64+0x3d6/0x440 [ 3656.300929] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3656.304159] RIP: 0033:0x7f151bee3cd8 [ 3656.305429] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3656.328554] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3656.335097] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 3656.340379] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 3656.343468] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3656.351651] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3656.361453] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3660.055063] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 3660.060219] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3660.120412] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3660.132534] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3660.139265] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3660.142361] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3660.165804] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3660.173274] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 3660.177706] RDX: 0000000000005138 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 3660.187217] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 3660.194201] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 3660.198581] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3660.202707] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 3660.206516] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3660.208946] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3660.212612] Call Trace: [ 3660.213547] [ 3660.214412] ? show_regs.cold.9+0x22/0x2f [ 3660.216202] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3660.218338] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3660.220406] ? watchdog+0x40/0x40 [ 3660.221972] ? hrtimer_interrupt+0x128/0x2c0 [ 3660.224254] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3660.226537] ? apic_timer_interrupt+0xf/0x20 [ 3660.228377] [ 3660.229145] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3660.231239] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3660.233965] do_raw_spin_lock+0x177/0x190 [ 3660.235469] _raw_spin_lock+0x21/0x30 [ 3660.236648] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3660.239001] obd_mod_free+0x46/0x140 [ptlrpc] [ 3660.242295] mdc_commit_open+0x91/0xb0 [mdc] [ 3660.244163] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3660.246852] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3660.250021] after_reply+0xe2c/0x1780 [ptlrpc] [ 3660.252906] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3660.255966] ? __next_timer_interrupt+0x160/0x160 [ 3660.258767] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3660.263193] ? do_wait_intr+0xf0/0xf0 [ 3660.269241] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3660.275126] kthread+0x1d1/0x200 [ 3660.278106] ? set_kthread_struct+0x70/0x70 [ 3660.284154] ret_from_fork+0x1f/0x30 [ 3664.056451] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3664.068143] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3664.122114] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3664.129140] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3664.147055] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3664.157993] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3664.166271] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3664.188222] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3664.195704] RAX: 0000000000004d84 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3664.199845] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 3664.204124] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 3664.212913] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 3664.217363] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 3664.224210] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 3664.241297] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3664.247009] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3664.259957] Call Trace: [ 3664.264914] [ 3664.266663] ? show_regs.cold.9+0x22/0x2f [ 3664.271759] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3664.277595] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3664.281403] ? watchdog+0x40/0x40 [ 3664.282444] ? hrtimer_interrupt+0x128/0x2c0 [ 3664.286495] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3664.292727] ? apic_timer_interrupt+0xf/0x20 [ 3664.295685] [ 3664.296612] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3664.299499] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3664.305481] do_raw_spin_lock+0x177/0x190 [ 3664.309732] _raw_spin_lock+0x21/0x30 [ 3664.312185] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3664.314474] process_one_work+0x2c8/0x700 [ 3664.322560] worker_thread+0x296/0x6e0 [ 3664.323527] ? rescuer_thread+0x570/0x570 [ 3664.330536] kthread+0x1d1/0x200 [ 3664.331749] ? set_kthread_struct+0x70/0x70 [ 3664.336046] ret_from_fork+0x1f/0x30 [ 3672.053190] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 3672.057702] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3672.084884] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3672.094541] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3672.103816] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3672.110964] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3672.126320] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3672.136414] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3672.146063] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 3672.150111] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 3672.163195] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 3672.170020] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 3672.173997] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 3672.185688] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3672.196596] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 3672.201616] Call Trace: [ 3672.203890] [ 3672.205152] ? show_regs.cold.9+0x22/0x2f [ 3672.207033] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3672.209244] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3672.211766] ? watchdog+0x40/0x40 [ 3672.213723] ? hrtimer_interrupt+0x128/0x2c0 [ 3672.220409] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3672.227372] ? apic_timer_interrupt+0xf/0x20 [ 3672.229017] [ 3672.229937] ? native_safe_halt+0x2e/0x50 [ 3672.231597] kvm_wait+0xcd/0xe0 [ 3672.235614] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3672.238318] do_raw_spin_lock+0x177/0x190 [ 3672.240899] _raw_spin_lock+0x21/0x30 [ 3672.242694] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3672.245413] ? do_raw_spin_unlock+0x75/0x190 [ 3672.247673] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3672.250300] ? do_wait_intr+0xf0/0xf0 [ 3672.252114] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3672.254328] mdc_reint+0x2d/0x110 [mdc] [ 3672.256008] mdc_unlink+0x658/0x870 [mdc] [ 3672.257740] lmv_unlink+0x34d/0xbd0 [lmv] [ 3672.259093] ll_unlink+0x24d/0x7f0 [lustre] [ 3672.260628] vfs_unlink+0x202/0x330 [ 3672.261691] do_unlinkat+0x2fa/0x460 [ 3672.262738] __x64_sys_unlink+0x37/0x50 [ 3672.263912] do_syscall_64+0xc1/0x440 [ 3672.264999] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3672.266586] RIP: 0033:0x7f151bdee10b [ 3672.268028] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3672.278741] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3672.290679] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 3672.296098] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 3672.301025] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3672.305591] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3672.315771] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3676.641854] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1746s! [ 3676.654013] Showing busy workqueues and worker pools: [ 3676.657060] workqueue events: flags=0x0 [ 3676.665275] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3676.672652] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3676.680229] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3676.688169] pending: kfree_rcu_monitor [ 3676.693309] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3676.698633] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 3676.720622] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3676.727316] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3676.735966] workqueue events_long: flags=0x0 [ 3676.740691] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3676.748893] pending: lu_objects_flush [obdclass] [ 3676.755620] workqueue events_unbound: flags=0x2 [ 3676.757959] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3676.762092] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3676.770885] workqueue events_power_efficient: flags=0x80 [ 3676.780860] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3676.784185] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 3676.787208] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3676.791169] pending: neigh_periodic_work [ 3676.799192] workqueue mm_percpu_wq: flags=0x8 [ 3676.801739] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3676.808003] pending: vmstat_update, lru_add_drain_per_cpu [ 3676.811626] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3676.818079] pending: vmstat_update [ 3676.822486] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3676.828655] pending: lru_add_drain_per_cpu [ 3676.838737] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3676.847520] pending: lru_add_drain_per_cpu BAR(48) [ 3676.850189] workqueue writeback: flags=0x4a [ 3676.851975] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3676.854311] pending: wb_workfn [ 3676.856212] workqueue xprtiod: flags=0xa [ 3676.857658] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3676.869459] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3676.874060] workqueue ipv6_addrconf: flags=0x40008 [ 3676.881385] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3676.887035] pending: addrconf_verify_work [ 3676.892048] workqueue knbd0-recv: flags=0x1a [ 3676.898654] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3676.906192] in-flight: 658:recv_work [ 3676.910597] workqueue ptlrpc_pinger: flags=0x2 [ 3676.918175] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3676.921183] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3676.924944] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1746s workers=4 idle: 32984 37 8 [ 3676.929034] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3661s workers=2 idle: 661 [ 3684.057028] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 3684.061112] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3684.080341] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3684.084064] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3684.089781] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3684.092418] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3684.101947] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3684.104431] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3684.107392] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 3684.110510] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3684.114299] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 3684.117842] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 3684.121012] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 3684.124026] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3684.125940] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 3684.128489] Call Trace: [ 3684.129443] [ 3684.130037] ? show_regs.cold.9+0x22/0x2f [ 3684.131507] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3684.133052] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3684.134718] ? watchdog+0x40/0x40 [ 3684.136028] ? hrtimer_interrupt+0x128/0x2c0 [ 3684.137633] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3684.139553] ? apic_timer_interrupt+0xf/0x20 [ 3684.142543] [ 3684.143616] ? native_safe_halt+0x2e/0x50 [ 3684.146010] kvm_wait+0xcd/0xe0 [ 3684.147792] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3684.151100] do_raw_spin_lock+0x177/0x190 [ 3684.153439] _raw_spin_lock+0x21/0x30 [ 3684.155491] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3684.158119] ? do_raw_spin_unlock+0x75/0x190 [ 3684.161031] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3684.167517] ? do_wait_intr+0xf0/0xf0 [ 3684.170082] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3684.172641] mdc_close+0x53b/0xd70 [mdc] [ 3684.174694] lmv_close+0x18d/0x490 [lmv] [ 3684.175797] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3684.179433] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3684.185580] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3684.187853] ll_md_close+0x274/0x800 [lustre] [ 3684.189956] ll_file_release+0x128/0xc50 [lustre] [ 3684.192797] __fput+0x10d/0x440 [ 3684.194873] ____fput+0x12/0x20 [ 3684.196591] task_work_run+0xc6/0x110 [ 3684.198493] exit_to_usermode_loop+0x1dd/0x1f0 [ 3684.201464] do_syscall_64+0x3d6/0x440 [ 3684.205636] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3684.211030] RIP: 0033:0x7f151bee3cd8 [ 3684.212167] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3684.231255] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3684.237324] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 3684.243474] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 3684.247069] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3684.255534] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3684.260507] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3688.056866] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 3688.063828] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3688.098013] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3688.105510] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3688.110875] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3688.113573] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3688.122469] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3688.126052] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 3688.129407] RDX: 0000000000005819 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 3688.132582] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 3688.136159] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 3688.139643] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3688.142134] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 3688.148829] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3688.150986] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3688.157491] Call Trace: [ 3688.159666] [ 3688.163674] ? show_regs.cold.9+0x22/0x2f [ 3688.170091] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3688.177183] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3688.185064] ? watchdog+0x40/0x40 [ 3688.192255] ? hrtimer_interrupt+0x128/0x2c0 [ 3688.199309] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3688.208056] ? apic_timer_interrupt+0xf/0x20 [ 3688.212982] [ 3688.213659] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3688.215354] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3688.224545] do_raw_spin_lock+0x177/0x190 [ 3688.230216] _raw_spin_lock+0x21/0x30 [ 3688.236125] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3688.243017] obd_mod_free+0x46/0x140 [ptlrpc] [ 3688.245681] mdc_commit_open+0x91/0xb0 [mdc] [ 3688.247297] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3688.253216] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3688.261691] after_reply+0xe2c/0x1780 [ptlrpc] [ 3688.271561] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3688.278440] ? __next_timer_interrupt+0x160/0x160 [ 3688.285060] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3688.290429] ? do_wait_intr+0xf0/0xf0 [ 3688.292745] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3688.299710] kthread+0x1d1/0x200 [ 3688.306317] ? set_kthread_struct+0x70/0x70 [ 3688.309002] ret_from_fork+0x1f/0x30 [ 3692.056010] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3692.067960] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3692.097992] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3692.107663] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3692.112703] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3692.117221] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3692.119122] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3692.127025] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3692.130336] RAX: 00000000000008f6 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3692.134372] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 3692.138598] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 3692.142408] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 3692.145991] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 3692.149743] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 3692.153682] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3692.156467] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3692.161795] Call Trace: [ 3692.165275] [ 3692.166117] ? show_regs.cold.9+0x22/0x2f [ 3692.168198] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3692.170446] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3692.172966] ? watchdog+0x40/0x40 [ 3692.173973] ? hrtimer_interrupt+0x128/0x2c0 [ 3692.176531] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3692.179037] ? apic_timer_interrupt+0xf/0x20 [ 3692.180810] [ 3692.181839] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3692.184061] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3692.186573] do_raw_spin_lock+0x177/0x190 [ 3692.188733] _raw_spin_lock+0x21/0x30 [ 3692.190490] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3692.192890] process_one_work+0x2c8/0x700 [ 3692.194733] worker_thread+0x296/0x6e0 [ 3692.196484] ? rescuer_thread+0x570/0x570 [ 3692.198526] kthread+0x1d1/0x200 [ 3692.199927] ? set_kthread_struct+0x70/0x70 [ 3692.202227] ret_from_fork+0x1f/0x30 [ 3704.059384] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 3704.067002] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3704.104525] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3704.115293] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3704.124195] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3704.128745] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3704.141214] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3704.146925] RAX: 0000000000007508 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3704.149604] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 3704.155239] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 3704.159007] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 3704.163956] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 3704.169004] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 3704.171868] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3704.175949] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 3704.178667] Call Trace: [ 3704.179730] [ 3704.180496] ? show_regs.cold.9+0x22/0x2f [ 3704.183784] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3704.187858] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3704.190227] ? watchdog+0x40/0x40 [ 3704.191785] ? hrtimer_interrupt+0x128/0x2c0 [ 3704.194766] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3704.196926] ? apic_timer_interrupt+0xf/0x20 [ 3704.199615] [ 3704.201416] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3704.204745] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3704.208701] do_raw_spin_lock+0x177/0x190 [ 3704.212045] _raw_spin_lock+0x21/0x30 [ 3704.213672] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3704.216909] ? do_raw_spin_unlock+0x75/0x190 [ 3704.218047] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3704.220660] ? do_wait_intr+0xf0/0xf0 [ 3704.221990] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3704.226242] mdc_reint+0x2d/0x110 [mdc] [ 3704.227661] mdc_unlink+0x658/0x870 [mdc] [ 3704.229869] lmv_unlink+0x34d/0xbd0 [lmv] [ 3704.231932] ll_unlink+0x24d/0x7f0 [lustre] [ 3704.234626] vfs_unlink+0x202/0x330 [ 3704.236116] do_unlinkat+0x2fa/0x460 [ 3704.238048] __x64_sys_unlink+0x37/0x50 [ 3704.239304] do_syscall_64+0xc1/0x440 [ 3704.240279] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3704.243049] RIP: 0033:0x7f151bdee10b [ 3704.244571] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3704.251293] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3704.256632] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 3704.259108] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 3704.261257] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3704.265886] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3704.269829] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3707.361745] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1776s! [ 3707.371637] Showing busy workqueues and worker pools: [ 3707.374979] workqueue events: flags=0x0 [ 3707.377278] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3707.385241] pending: cpu_latency_work [ptlrpc], push_to_pool [ 3707.387758] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3707.393828] pending: kfree_rcu_monitor [ 3707.407620] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3707.415352] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 3707.431582] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3707.438420] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3707.450672] workqueue events_long: flags=0x0 [ 3707.454572] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3707.457377] pending: lu_objects_flush [obdclass] [ 3707.464616] workqueue events_unbound: flags=0x2 [ 3707.467573] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3707.476472] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3707.486289] workqueue events_power_efficient: flags=0x80 [ 3707.491447] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3707.497582] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 3707.505330] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3707.509657] pending: neigh_periodic_work [ 3707.514535] workqueue mm_percpu_wq: flags=0x8 [ 3707.519808] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3707.526929] pending: vmstat_update, lru_add_drain_per_cpu [ 3707.532359] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3707.539720] pending: vmstat_update [ 3707.541641] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3707.545696] pending: lru_add_drain_per_cpu [ 3707.554303] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3707.559075] pending: lru_add_drain_per_cpu BAR(48) [ 3707.562689] workqueue writeback: flags=0x4a [ 3707.566595] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3707.568966] pending: wb_workfn [ 3707.576460] workqueue xprtiod: flags=0xa [ 3707.579905] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3707.590127] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3707.601384] workqueue ipv6_addrconf: flags=0x40008 [ 3707.604381] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3707.611216] pending: addrconf_verify_work [ 3707.614598] workqueue knbd0-recv: flags=0x1a [ 3707.620781] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3707.624928] in-flight: 658:recv_work [ 3707.627372] workqueue ptlrpc_pinger: flags=0x2 [ 3707.634217] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3707.636766] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3707.644382] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1777s workers=4 idle: 32984 37 8 [ 3707.654128] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3692s workers=2 idle: 661 [ 3712.058144] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 3712.061345] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3712.081384] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3712.086723] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3712.090857] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3712.093646] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3712.101786] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3712.109888] RAX: 000000000000200a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3712.112830] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 3712.116838] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3712.121626] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 3712.126840] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 3712.132742] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 3712.138974] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3712.142618] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 3712.147191] Call Trace: [ 3712.148385] [ 3712.149624] ? show_regs.cold.9+0x22/0x2f [ 3712.152839] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3712.155624] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3712.157800] ? watchdog+0x40/0x40 [ 3712.159599] ? hrtimer_interrupt+0x128/0x2c0 [ 3712.161856] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3712.164723] ? apic_timer_interrupt+0xf/0x20 [ 3712.167755] [ 3712.169146] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3712.171974] do_raw_spin_lock+0x177/0x190 [ 3712.174693] _raw_spin_lock+0x21/0x30 [ 3712.176856] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3712.180836] ? do_raw_spin_unlock+0x75/0x190 [ 3712.184702] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3712.187749] ? do_wait_intr+0xf0/0xf0 [ 3712.190114] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3712.193499] mdc_close+0x53b/0xd70 [mdc] [ 3712.195758] lmv_close+0x18d/0x490 [lmv] [ 3712.203739] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3712.209396] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3712.214639] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3712.217779] ll_md_close+0x274/0x800 [lustre] [ 3712.220643] ll_file_release+0x128/0xc50 [lustre] [ 3712.223936] __fput+0x10d/0x440 [ 3712.227485] ____fput+0x12/0x20 [ 3712.230506] task_work_run+0xc6/0x110 [ 3712.232383] exit_to_usermode_loop+0x1dd/0x1f0 [ 3712.234662] do_syscall_64+0x3d6/0x440 [ 3712.237154] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3712.239112] RIP: 0033:0x7f151bee3cd8 [ 3712.240367] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3712.252251] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3712.256979] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 3712.260864] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 3712.264824] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3712.268907] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3712.272204] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3716.055491] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 3716.073822] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3716.132584] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3716.145124] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3716.157941] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3716.172294] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3716.188780] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3716.193835] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 3716.199231] RDX: 0000000000007716 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 3716.203405] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 3716.207004] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 3716.210558] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3716.214317] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 3716.218123] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3716.220416] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3716.224418] Call Trace: [ 3716.225726] [ 3716.226611] ? show_regs.cold.9+0x22/0x2f [ 3716.228464] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3716.230244] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3716.232220] ? watchdog+0x40/0x40 [ 3716.233499] ? hrtimer_interrupt+0x128/0x2c0 [ 3716.236581] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3716.238827] ? apic_timer_interrupt+0xf/0x20 [ 3716.240957] [ 3716.241719] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3716.244706] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3716.246732] do_raw_spin_lock+0x177/0x190 [ 3716.248448] _raw_spin_lock+0x21/0x30 [ 3716.249739] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3716.251979] obd_mod_free+0x46/0x140 [ptlrpc] [ 3716.254139] mdc_commit_open+0x91/0xb0 [mdc] [ 3716.255776] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3716.257558] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3716.260468] after_reply+0xe2c/0x1780 [ptlrpc] [ 3716.262097] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3716.264674] ? __next_timer_interrupt+0x160/0x160 [ 3716.266226] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3716.268103] ? do_wait_intr+0xf0/0xf0 [ 3716.269490] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3716.271658] kthread+0x1d1/0x200 [ 3716.272591] ? set_kthread_struct+0x70/0x70 [ 3716.274207] ret_from_fork+0x1f/0x30 [ 3720.059906] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3720.069828] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3720.121541] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3720.136576] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3720.150119] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3720.159781] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3720.165438] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3720.189851] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3720.194512] RAX: 0000000000006a8f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3720.203626] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 3720.216547] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 3720.221386] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 3720.230872] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 3720.243867] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 3720.253595] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3720.262428] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3720.269687] Call Trace: [ 3720.278636] [ 3720.281550] ? show_regs.cold.9+0x22/0x2f [ 3720.296844] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3720.310082] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3720.318476] ? watchdog+0x40/0x40 [ 3720.321077] ? hrtimer_interrupt+0x128/0x2c0 [ 3720.324552] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3720.329159] ? apic_timer_interrupt+0xf/0x20 [ 3720.332512] [ 3720.338418] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3720.344224] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3720.350161] do_raw_spin_lock+0x177/0x190 [ 3720.353594] _raw_spin_lock+0x21/0x30 [ 3720.359420] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3720.365606] process_one_work+0x2c8/0x700 [ 3720.367841] worker_thread+0x296/0x6e0 [ 3720.369428] ? rescuer_thread+0x570/0x570 [ 3720.376343] kthread+0x1d1/0x200 [ 3720.379520] ? set_kthread_struct+0x70/0x70 [ 3720.383522] ret_from_fork+0x1f/0x30 [ 3732.052025] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 3732.058445] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3732.086438] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3732.093371] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3732.099099] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3732.101364] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3732.112826] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3732.117198] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3732.122680] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 3732.126859] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 3732.131601] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 3732.136226] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 3732.139568] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 3732.144749] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3732.148354] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 3732.151867] Call Trace: [ 3732.153685] [ 3732.154364] ? show_regs.cold.9+0x22/0x2f [ 3732.156729] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3732.159014] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3732.161200] ? watchdog+0x40/0x40 [ 3732.163253] ? hrtimer_interrupt+0x128/0x2c0 [ 3732.167049] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3732.168879] ? apic_timer_interrupt+0xf/0x20 [ 3732.171300] [ 3732.172354] ? native_safe_halt+0x2e/0x50 [ 3732.175641] kvm_wait+0xcd/0xe0 [ 3732.178079] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3732.180623] do_raw_spin_lock+0x177/0x190 [ 3732.182591] _raw_spin_lock+0x21/0x30 [ 3732.184939] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3732.188738] ? do_raw_spin_unlock+0x75/0x190 [ 3732.191465] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3732.195044] ? do_wait_intr+0xf0/0xf0 [ 3732.197015] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3732.201103] mdc_reint+0x2d/0x110 [mdc] [ 3732.203333] mdc_unlink+0x658/0x870 [mdc] [ 3732.205156] lmv_unlink+0x34d/0xbd0 [lmv] [ 3732.208434] ll_unlink+0x24d/0x7f0 [lustre] [ 3732.211522] vfs_unlink+0x202/0x330 [ 3732.213054] do_unlinkat+0x2fa/0x460 [ 3732.214836] __x64_sys_unlink+0x37/0x50 [ 3732.216707] do_syscall_64+0xc1/0x440 [ 3732.218705] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3732.222600] RIP: 0033:0x7f151bdee10b [ 3732.224582] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3732.235967] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3732.242890] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 3732.247464] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 3732.252066] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3732.255682] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3732.259977] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3738.088394] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1807s! [ 3738.097453] Showing busy workqueues and worker pools: [ 3738.107247] workqueue events: flags=0x0 [ 3738.112189] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3738.121613] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 3738.133150] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3738.138570] pending: kfree_rcu_monitor [ 3738.149326] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3738.158421] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 3738.168188] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3738.171722] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3738.177479] workqueue events_long: flags=0x0 [ 3738.179145] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3738.182161] pending: lu_objects_flush [obdclass] [ 3738.188345] workqueue events_unbound: flags=0x2 [ 3738.190866] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3738.196905] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3738.206270] workqueue events_power_efficient: flags=0x80 [ 3738.209179] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3738.212086] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 3738.215039] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3738.217791] pending: neigh_periodic_work [ 3738.219604] workqueue mm_percpu_wq: flags=0x8 [ 3738.221388] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3738.227257] pending: vmstat_update, lru_add_drain_per_cpu [ 3738.231335] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3738.236301] pending: vmstat_update [ 3738.238756] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3738.243079] pending: lru_add_drain_per_cpu [ 3738.244309] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3738.251090] pending: lru_add_drain_per_cpu BAR(48) [ 3738.253636] workqueue writeback: flags=0x4a [ 3738.260095] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3738.262760] pending: wb_workfn [ 3738.264310] workqueue xprtiod: flags=0xa [ 3738.266218] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3738.268694] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3738.278301] workqueue ipv6_addrconf: flags=0x40008 [ 3738.281753] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3738.288630] pending: addrconf_verify_work [ 3738.293802] workqueue knbd0-recv: flags=0x1a [ 3738.296762] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3738.305628] in-flight: 658:recv_work [ 3738.311776] workqueue ptlrpc_pinger: flags=0x2 [ 3738.314646] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3738.318468] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3738.323368] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1807s workers=4 idle: 32984 37 8 [ 3738.330348] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3723s workers=2 idle: 661 [ 3740.060934] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 3740.070118] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3740.110185] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3740.119285] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3740.126872] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3740.134496] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3740.153110] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3740.165316] RAX: 000000000000458d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3740.176749] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 3740.191033] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3740.205751] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 3740.215147] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 3740.222875] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 3740.233479] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3740.244794] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 3740.254457] Call Trace: [ 3740.255315] [ 3740.256035] ? show_regs.cold.9+0x22/0x2f [ 3740.268762] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3740.270538] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3740.283484] ? watchdog+0x40/0x40 [ 3740.292331] ? hrtimer_interrupt+0x128/0x2c0 [ 3740.301334] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3740.303180] ? apic_timer_interrupt+0xf/0x20 [ 3740.313511] [ 3740.315089] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3740.319408] do_raw_spin_lock+0x177/0x190 [ 3740.320776] _raw_spin_lock+0x21/0x30 [ 3740.326169] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3740.334284] ? do_raw_spin_unlock+0x75/0x190 [ 3740.337150] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3740.340483] ? do_wait_intr+0xf0/0xf0 [ 3740.343467] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3740.346531] mdc_close+0x53b/0xd70 [mdc] [ 3740.349245] lmv_close+0x18d/0x490 [lmv] [ 3740.351531] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3740.355057] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3740.358316] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3740.361019] ll_md_close+0x274/0x800 [lustre] [ 3740.363317] ll_file_release+0x128/0xc50 [lustre] [ 3740.366071] __fput+0x10d/0x440 [ 3740.367740] ____fput+0x12/0x20 [ 3740.370586] task_work_run+0xc6/0x110 [ 3740.372551] exit_to_usermode_loop+0x1dd/0x1f0 [ 3740.374469] do_syscall_64+0x3d6/0x440 [ 3740.376250] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3740.379599] RIP: 0033:0x7f151bee3cd8 [ 3740.381655] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3740.392740] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3740.403096] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 3740.407140] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 3740.413286] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3740.418294] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3740.422620] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3744.057192] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 3744.066691] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3744.153577] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3744.171602] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3744.183242] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3744.193543] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3744.216574] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3744.224312] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 3744.230701] RDX: 00000000000067b5 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 3744.238623] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 3744.248614] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 3744.254346] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3744.259531] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 3744.266629] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3744.272960] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3744.288478] Call Trace: [ 3744.289499] [ 3744.291558] ? show_regs.cold.9+0x22/0x2f [ 3744.296058] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3744.303693] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3744.305468] ? watchdog+0x40/0x40 [ 3744.315183] ? hrtimer_interrupt+0x128/0x2c0 [ 3744.320470] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3744.325604] ? apic_timer_interrupt+0xf/0x20 [ 3744.327627] [ 3744.331672] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3744.335296] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3744.341155] do_raw_spin_lock+0x177/0x190 [ 3744.346225] _raw_spin_lock+0x21/0x30 [ 3744.354333] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3744.358963] obd_mod_free+0x46/0x140 [ptlrpc] [ 3744.360350] mdc_commit_open+0x91/0xb0 [mdc] [ 3744.367506] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3744.370304] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3744.376395] after_reply+0xe2c/0x1780 [ptlrpc] [ 3744.383502] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3744.387312] ? __next_timer_interrupt+0x160/0x160 [ 3744.389504] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3744.395452] ? do_wait_intr+0xf0/0xf0 [ 3744.398453] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3744.403198] kthread+0x1d1/0x200 [ 3744.405580] ? set_kthread_struct+0x70/0x70 [ 3744.408564] ret_from_fork+0x1f/0x30 [ 3748.058495] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3748.066174] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3748.115327] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3748.121553] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3748.128918] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3748.132241] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3748.135265] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3748.148678] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 3748.154471] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3748.159187] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 3748.162809] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 3748.166376] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 3748.170487] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 3748.176350] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 3748.187258] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3748.191232] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3748.195831] Call Trace: [ 3748.197425] [ 3748.198437] ? show_regs.cold.9+0x22/0x2f [ 3748.203034] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3748.208358] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3748.210670] ? watchdog+0x40/0x40 [ 3748.213394] ? hrtimer_interrupt+0x128/0x2c0 [ 3748.219596] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3748.223014] ? apic_timer_interrupt+0xf/0x20 [ 3748.225805] [ 3748.227177] ? native_safe_halt+0x2e/0x50 [ 3748.230026] kvm_wait+0xcd/0xe0 [ 3748.232070] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3748.234399] do_raw_spin_lock+0x177/0x190 [ 3748.237076] _raw_spin_lock+0x21/0x30 [ 3748.239344] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3748.241652] process_one_work+0x2c8/0x700 [ 3748.244563] worker_thread+0x296/0x6e0 [ 3748.246514] ? rescuer_thread+0x570/0x570 [ 3748.248328] kthread+0x1d1/0x200 [ 3748.250442] ? set_kthread_struct+0x70/0x70 [ 3748.252579] ret_from_fork+0x1f/0x30 [ 3764.052717] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 3764.059765] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3764.108443] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3764.122801] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3764.131077] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3764.134719] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3764.148285] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3764.152260] RAX: 0000000000007f84 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3764.156364] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 3764.161584] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 3764.165995] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 3764.178443] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 3764.191886] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 3764.195408] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3764.200690] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 3764.203208] Call Trace: [ 3764.204478] [ 3764.205532] ? show_regs.cold.9+0x22/0x2f [ 3764.208018] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3764.213506] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3764.215078] ? watchdog+0x40/0x40 [ 3764.217694] ? hrtimer_interrupt+0x128/0x2c0 [ 3764.221885] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3764.226583] ? apic_timer_interrupt+0xf/0x20 [ 3764.228230] [ 3764.229065] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3764.233817] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3764.236049] do_raw_spin_lock+0x177/0x190 [ 3764.241517] _raw_spin_lock+0x21/0x30 [ 3764.243075] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3764.245496] ? do_raw_spin_unlock+0x75/0x190 [ 3764.248429] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3764.253428] ? do_wait_intr+0xf0/0xf0 [ 3764.254667] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3764.265003] mdc_reint+0x2d/0x110 [mdc] [ 3764.269755] mdc_unlink+0x658/0x870 [mdc] [ 3764.271384] lmv_unlink+0x34d/0xbd0 [lmv] [ 3764.273279] ll_unlink+0x24d/0x7f0 [lustre] [ 3764.275425] vfs_unlink+0x202/0x330 [ 3764.277063] do_unlinkat+0x2fa/0x460 [ 3764.278219] __x64_sys_unlink+0x37/0x50 [ 3764.279389] do_syscall_64+0xc1/0x440 [ 3764.280697] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3764.282134] RIP: 0033:0x7f151bdee10b [ 3764.283942] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3764.291341] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3764.296916] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 3764.299482] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 3764.309449] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3764.317934] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3764.321235] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3768.059720] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 3768.064207] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3768.085942] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3768.091410] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3768.095243] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3768.097426] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3768.104254] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3768.107037] RAX: 0000000000005419 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3768.109302] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 3768.111587] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3768.114072] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 3768.116619] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 3768.128705] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 3768.135307] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3768.138044] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 3768.140687] Call Trace: [ 3768.141588] [ 3768.142272] ? show_regs.cold.9+0x22/0x2f [ 3768.144178] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3768.146316] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3768.148204] ? watchdog+0x40/0x40 [ 3768.149198] ? hrtimer_interrupt+0x128/0x2c0 [ 3768.150836] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3768.152633] ? apic_timer_interrupt+0xf/0x20 [ 3768.154478] [ 3768.156409] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3768.160421] do_raw_spin_lock+0x177/0x190 [ 3768.163298] _raw_spin_lock+0x21/0x30 [ 3768.170921] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3768.172737] ? do_raw_spin_unlock+0x75/0x190 [ 3768.174293] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3768.176401] ? do_wait_intr+0xf0/0xf0 [ 3768.178024] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3768.180270] mdc_close+0x53b/0xd70 [mdc] [ 3768.182178] lmv_close+0x18d/0x490 [lmv] [ 3768.183706] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3768.185693] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3768.188103] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3768.190132] ll_md_close+0x274/0x800 [lustre] [ 3768.192218] ll_file_release+0x128/0xc50 [lustre] [ 3768.195559] __fput+0x10d/0x440 [ 3768.196746] ____fput+0x12/0x20 [ 3768.197740] task_work_run+0xc6/0x110 [ 3768.200157] exit_to_usermode_loop+0x1dd/0x1f0 [ 3768.204075] do_syscall_64+0x3d6/0x440 [ 3768.208126] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3768.212956] RIP: 0033:0x7f151bee3cd8 [ 3768.216405] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3768.226892] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3768.232181] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 3768.238375] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 3768.241923] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3768.246487] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3768.250024] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3768.803019] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1838s! [ 3768.807055] Showing busy workqueues and worker pools: [ 3768.810212] workqueue events: flags=0x0 [ 3768.812553] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3768.816118] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 3768.823898] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3768.829780] pending: kfree_rcu_monitor [ 3768.834376] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3768.841367] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 3768.847251] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3768.852231] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3768.857023] workqueue events_long: flags=0x0 [ 3768.860917] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3768.864480] pending: lu_objects_flush [obdclass] [ 3768.870438] workqueue events_unbound: flags=0x2 [ 3768.872488] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3768.879305] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3768.885689] workqueue events_power_efficient: flags=0x80 [ 3768.888410] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3768.891582] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 3768.894257] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3768.897238] pending: neigh_periodic_work [ 3768.899597] workqueue mm_percpu_wq: flags=0x8 [ 3768.901133] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3768.904276] pending: vmstat_update, lru_add_drain_per_cpu [ 3768.907072] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3768.909999] pending: vmstat_update [ 3768.911561] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3768.914569] pending: lru_add_drain_per_cpu [ 3768.916465] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3768.919721] pending: lru_add_drain_per_cpu BAR(48) [ 3768.921406] workqueue writeback: flags=0x4a [ 3768.923258] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3768.926449] pending: wb_workfn [ 3768.928589] workqueue xprtiod: flags=0xa [ 3768.930718] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3768.933488] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3768.937180] workqueue ipv6_addrconf: flags=0x40008 [ 3768.938975] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3768.942677] pending: addrconf_verify_work [ 3768.944182] workqueue knbd0-recv: flags=0x1a [ 3768.945838] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3768.948587] in-flight: 658:recv_work [ 3768.950285] workqueue ptlrpc_pinger: flags=0x2 [ 3768.952623] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3768.955448] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3768.958473] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1838s workers=4 idle: 32984 37 8 [ 3768.962243] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3753s workers=2 idle: 661 [ 3772.055143] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 3772.058845] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3772.100506] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3772.115036] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3772.124201] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3772.126454] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3772.151810] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3772.159316] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 3772.166990] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 3772.170747] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 3772.176487] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 3772.184241] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3772.191059] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 3772.205736] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3772.216042] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3772.229454] Call Trace: [ 3772.230389] [ 3772.231861] ? show_regs.cold.9+0x22/0x2f [ 3772.242266] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3772.251967] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3772.260111] ? watchdog+0x40/0x40 [ 3772.264488] ? hrtimer_interrupt+0x128/0x2c0 [ 3772.266775] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3772.269084] ? apic_timer_interrupt+0xf/0x20 [ 3772.271036] [ 3772.272507] ? native_safe_halt+0x2e/0x50 [ 3772.273454] kvm_wait+0xcd/0xe0 [ 3772.277074] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3772.283196] do_raw_spin_lock+0x177/0x190 [ 3772.285420] _raw_spin_lock+0x21/0x30 [ 3772.287225] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3772.289984] obd_mod_free+0x46/0x140 [ptlrpc] [ 3772.293019] mdc_commit_open+0x91/0xb0 [mdc] [ 3772.294997] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3772.297681] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3772.300387] after_reply+0xe2c/0x1780 [ptlrpc] [ 3772.303910] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3772.306689] ? __next_timer_interrupt+0x160/0x160 [ 3772.309008] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3772.312044] ? do_wait_intr+0xf0/0xf0 [ 3772.313915] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3772.316383] kthread+0x1d1/0x200 [ 3772.317853] ? set_kthread_struct+0x70/0x70 [ 3772.326617] ret_from_fork+0x1f/0x30 [ 3776.056310] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3776.068881] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3776.113329] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3776.125820] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3776.133874] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3776.147808] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3776.151329] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3776.171332] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3776.182038] RAX: 0000000000007a84 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3776.186007] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 3776.194222] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 3776.197793] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 3776.201152] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 3776.203375] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 3776.205789] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3776.207362] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3776.211301] Call Trace: [ 3776.212328] [ 3776.213251] ? show_regs.cold.9+0x22/0x2f [ 3776.214787] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3776.216324] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3776.217841] ? watchdog+0x40/0x40 [ 3776.218863] ? hrtimer_interrupt+0x128/0x2c0 [ 3776.220183] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3776.221858] ? apic_timer_interrupt+0xf/0x20 [ 3776.224023] [ 3776.224889] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3776.227057] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3776.229234] do_raw_spin_lock+0x177/0x190 [ 3776.232394] _raw_spin_lock+0x21/0x30 [ 3776.236328] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3776.238398] process_one_work+0x2c8/0x700 [ 3776.241991] worker_thread+0x296/0x6e0 [ 3776.243069] ? rescuer_thread+0x570/0x570 [ 3776.245021] kthread+0x1d1/0x200 [ 3776.246279] ? set_kthread_struct+0x70/0x70 [ 3776.248789] ret_from_fork+0x1f/0x30 [ 3784.680299] rcu: INFO: rcu_sched self-detected stall on CPU [ 3784.683045] rcu: 1-...!: (1123770 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 3784.690708] (t=1860033 jiffies g=249657 q=2963) [ 3784.693072] rcu: rcu_sched kthread starved for 1854985 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 3784.699131] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3784.703842] rcu: RCU grace-period kthread stack dump: [ 3784.706932] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3784.716752] Call Trace: [ 3784.718284] __schedule+0x351/0xcb0 [ 3784.719973] schedule+0xc0/0x180 [ 3784.721838] schedule_timeout+0xb4/0x190 [ 3784.723874] ? do_raw_spin_unlock+0x75/0x190 [ 3784.726086] ? __next_timer_interrupt+0x160/0x160 [ 3784.728927] ? prepare_to_swait_event+0xf3/0x140 [ 3784.732556] rcu_gp_kthread+0x8dd/0x15b0 [ 3784.734467] ? force_qs_rnp+0x2c0/0x2c0 [ 3784.736458] kthread+0x1d1/0x200 [ 3784.738836] ? set_kthread_struct+0x70/0x70 [ 3784.740834] ret_from_fork+0x1f/0x30 [ 3784.742677] rcu: Stack dump where RCU GP kthread last ran: [ 3784.745145] NMI backtrace for cpu 1 [ 3784.746830] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3784.752259] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3784.758860] Call Trace: [ 3784.760196] [ 3784.761075] ? dump_stack+0xbb/0x10e [ 3784.763621] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3784.766613] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3784.769629] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3784.772508] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3784.775334] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3784.778898] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 3784.781632] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 3784.784233] ? update_process_times+0x6d/0xc0 [ 3784.786572] ? tick_sched_handle.isra.18+0x27/0x90 [ 3784.789931] ? tick_sched_timer+0x53/0xb0 [ 3784.792308] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3784.794423] ? tick_sched_handle.isra.18+0x90/0x90 [ 3784.797300] ? hrtimer_interrupt+0x128/0x2c0 [ 3784.799823] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3784.802255] ? apic_timer_interrupt+0xf/0x20 [ 3784.804983] [ 3784.806111] ? native_safe_halt+0x2e/0x50 [ 3784.808606] ? kvm_wait+0xcd/0xe0 [ 3784.810197] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3784.812721] ? do_raw_spin_lock+0x177/0x190 [ 3784.815091] ? _raw_spin_lock+0x21/0x30 [ 3784.817159] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3784.820490] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3784.823494] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3784.826625] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3784.830322] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3784.833647] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3784.837310] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3784.841292] ? __next_timer_interrupt+0x160/0x160 [ 3784.843883] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3784.846468] ? do_wait_intr+0xf0/0xf0 [ 3784.848518] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3784.851501] ? kthread+0x1d1/0x200 [ 3784.853500] ? set_kthread_struct+0x70/0x70 [ 3784.855393] ? ret_from_fork+0x1f/0x30 [ 3784.857719] NMI backtrace for cpu 1 [ 3784.860050] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3784.867762] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3784.873059] Call Trace: [ 3784.874425] [ 3784.875609] ? dump_stack+0xbb/0x10e [ 3784.877901] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3784.881289] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3784.884118] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3784.886893] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3784.889891] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3784.893116] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 3784.895992] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 3784.897836] ? update_process_times+0x6d/0xc0 [ 3784.900518] ? tick_sched_handle.isra.18+0x27/0x90 [ 3784.903420] ? tick_sched_timer+0x53/0xb0 [ 3784.905642] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3784.908188] ? tick_sched_handle.isra.18+0x90/0x90 [ 3784.910883] ? hrtimer_interrupt+0x128/0x2c0 [ 3784.913124] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3784.915376] ? apic_timer_interrupt+0xf/0x20 [ 3784.917664] [ 3784.918900] ? native_safe_halt+0x2e/0x50 [ 3784.921055] ? kvm_wait+0xcd/0xe0 [ 3784.922696] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3784.925434] ? do_raw_spin_lock+0x177/0x190 [ 3784.927535] ? _raw_spin_lock+0x21/0x30 [ 3784.930102] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3784.933935] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3784.937042] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3784.939640] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3784.943555] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3784.947260] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3784.950918] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3784.954556] ? __next_timer_interrupt+0x160/0x160 [ 3784.956851] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3784.959687] ? do_wait_intr+0xf0/0xf0 [ 3784.962503] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3784.965747] ? kthread+0x1d1/0x200 [ 3784.967668] ? set_kthread_struct+0x70/0x70 [ 3784.970644] ? ret_from_fork+0x1f/0x30 [ 3796.052365] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 3796.055040] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3796.057062] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 3796.071854] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3796.074719] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3796.074800] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3796.074808] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3796.074811] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3796.074824] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3796.074828] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3796.074832] RAX: 0000000000001e99 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3796.074835] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 3796.074837] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3796.074839] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 3796.074841] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 3796.074844] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 3796.074847] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3796.074849] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 3796.074856] Call Trace: [ 3796.074860] [ 3796.074866] ? show_regs.cold.9+0x22/0x2f [ 3796.074875] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3796.074881] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3796.074885] ? watchdog+0x40/0x40 [ 3796.074890] ? hrtimer_interrupt+0x128/0x2c0 [ 3796.074893] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3796.074902] ? apic_timer_interrupt+0xf/0x20 [ 3796.074907] [ 3796.074910] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3796.074917] do_raw_spin_lock+0x177/0x190 [ 3796.074922] _raw_spin_lock+0x21/0x30 [ 3796.074927] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3796.075263] ? do_raw_spin_unlock+0x75/0x190 [ 3796.075268] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3796.078929] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3796.078939] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3796.078955] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3796.078960] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3796.078966] RAX: 0000000000007dae RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3796.078968] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 3796.078971] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 3796.078973] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 3796.078975] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 3796.078979] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 3796.094607] ? do_wait_intr+0xf0/0xf0 [ 3796.098035] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3796.098048] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 3796.098056] Call Trace: [ 3796.101170] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3796.103094] [ 3796.109154] mdc_close+0x53b/0xd70 [mdc] [ 3796.111628] ? show_regs.cold.9+0x22/0x2f [ 3796.111650] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3796.114149] lmv_close+0x18d/0x490 [lmv] [ 3796.116571] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3796.119012] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3796.121363] ? watchdog+0x40/0x40 [ 3796.123327] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3796.125556] ? hrtimer_interrupt+0x128/0x2c0 [ 3796.125569] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3796.125577] ? apic_timer_interrupt+0xf/0x20 [ 3796.125581] [ 3796.125586] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3796.125592] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3796.125597] do_raw_spin_lock+0x177/0x190 [ 3796.125602] _raw_spin_lock+0x21/0x30 [ 3796.125608] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3796.127460] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3796.129877] ? do_raw_spin_unlock+0x75/0x190 [ 3796.129894] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3796.130146] ? do_wait_intr+0xf0/0xf0 [ 3796.130153] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3796.131160] ll_md_close+0x274/0x800 [lustre] [ 3796.131891] mdc_reint+0x2d/0x110 [mdc] [ 3796.131933] mdc_unlink+0x658/0x870 [mdc] [ 3796.133629] ll_file_release+0x128/0xc50 [lustre] [ 3796.133738] __fput+0x10d/0x440 [ 3796.133747] ____fput+0x12/0x20 [ 3796.133751] task_work_run+0xc6/0x110 [ 3796.133759] exit_to_usermode_loop+0x1dd/0x1f0 [ 3796.133765] do_syscall_64+0x3d6/0x440 [ 3796.133770] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3796.133778] RIP: 0033:0x7f151bee3cd8 [ 3796.133786] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3796.133790] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3796.133795] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 3796.133797] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 3796.133799] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3796.133802] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3796.133804] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3796.260769] lmv_unlink+0x34d/0xbd0 [lmv] [ 3796.262141] ll_unlink+0x24d/0x7f0 [lustre] [ 3796.263714] vfs_unlink+0x202/0x330 [ 3796.264845] do_unlinkat+0x2fa/0x460 [ 3796.266050] __x64_sys_unlink+0x37/0x50 [ 3796.267498] do_syscall_64+0xc1/0x440 [ 3796.268778] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3796.270651] RIP: 0033:0x7f151bdee10b [ 3796.271884] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3796.278648] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3796.281402] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 3796.283798] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 3796.286059] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3796.288187] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3796.290653] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3799.017091] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1868s! [ 3799.020491] Showing busy workqueues and worker pools: [ 3799.022478] workqueue events: flags=0x0 [ 3799.024528] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3799.032355] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 3799.036310] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3799.039200] pending: kfree_rcu_monitor [ 3799.041060] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3799.044048] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 3799.047855] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3799.050806] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3799.054037] workqueue events_long: flags=0x0 [ 3799.055816] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3799.058422] pending: lu_objects_flush [obdclass] [ 3799.060818] workqueue events_unbound: flags=0x2 [ 3799.062844] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3799.065339] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3799.069792] workqueue events_power_efficient: flags=0x80 [ 3799.071908] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3799.074618] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 3799.077669] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3799.080245] pending: neigh_periodic_work [ 3799.082070] workqueue mm_percpu_wq: flags=0x8 [ 3799.083904] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3799.086837] pending: vmstat_update, lru_add_drain_per_cpu [ 3799.089240] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3799.092082] pending: vmstat_update [ 3799.093707] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3799.096381] pending: lru_add_drain_per_cpu [ 3799.098187] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3799.101150] pending: lru_add_drain_per_cpu BAR(48) [ 3799.103525] workqueue writeback: flags=0x4a [ 3799.105384] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3799.108152] pending: wb_workfn [ 3799.109884] workqueue xprtiod: flags=0xa [ 3799.111734] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3799.114462] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3799.118275] workqueue ipv6_addrconf: flags=0x40008 [ 3799.120360] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3799.123589] pending: addrconf_verify_work [ 3799.125605] workqueue knbd0-recv: flags=0x1a [ 3799.127538] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3799.130422] in-flight: 658:recv_work [ 3799.132187] workqueue ptlrpc_pinger: flags=0x2 [ 3799.134073] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3799.136796] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3799.139722] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1868s workers=4 idle: 32984 37 8 [ 3799.143464] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3783s workers=2 idle: 661 [ 3804.057204] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3804.062443] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3804.096467] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3804.101113] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3804.112582] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3804.115071] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3804.121124] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3804.134599] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3804.140875] RAX: 0000000000006219 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3804.147522] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 3804.155715] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 3804.160981] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 3804.167395] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 3804.174475] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 3804.181847] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3804.187994] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3804.197153] Call Trace: [ 3804.201218] [ 3804.202064] ? show_regs.cold.9+0x22/0x2f [ 3804.205861] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3804.216005] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3804.221547] ? watchdog+0x40/0x40 [ 3804.222856] ? hrtimer_interrupt+0x128/0x2c0 [ 3804.225784] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3804.228523] ? apic_timer_interrupt+0xf/0x20 [ 3804.230620] [ 3804.232164] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3804.241922] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3804.244735] do_raw_spin_lock+0x177/0x190 [ 3804.246157] _raw_spin_lock+0x21/0x30 [ 3804.249450] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3804.251882] process_one_work+0x2c8/0x700 [ 3804.253588] worker_thread+0x296/0x6e0 [ 3804.255675] ? rescuer_thread+0x570/0x570 [ 3804.257094] kthread+0x1d1/0x200 [ 3804.258616] ? set_kthread_struct+0x70/0x70 [ 3804.260141] ret_from_fork+0x1f/0x30 [ 3812.060409] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 3812.068048] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3812.093899] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3812.099830] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3812.104774] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3812.106968] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3812.117380] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3812.121181] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 3812.125001] RDX: 0000000000007428 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 3812.130039] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 3812.134332] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 3812.141834] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3812.147313] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 3812.152712] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3812.154304] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3812.159450] Call Trace: [ 3812.160722] [ 3812.168099] ? show_regs.cold.9+0x22/0x2f [ 3812.170414] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3812.172843] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3812.175182] ? watchdog+0x40/0x40 [ 3812.176972] ? hrtimer_interrupt+0x128/0x2c0 [ 3812.179178] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3812.181402] ? apic_timer_interrupt+0xf/0x20 [ 3812.185894] [ 3812.189045] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3812.192319] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3812.195885] do_raw_spin_lock+0x177/0x190 [ 3812.198277] _raw_spin_lock+0x21/0x30 [ 3812.200620] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3812.205426] obd_mod_free+0x46/0x140 [ptlrpc] [ 3812.208656] mdc_commit_open+0x91/0xb0 [mdc] [ 3812.211033] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3812.214409] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3812.218779] after_reply+0xe2c/0x1780 [ptlrpc] [ 3812.222335] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3812.225919] ? __next_timer_interrupt+0x160/0x160 [ 3812.229345] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3812.232032] ? do_wait_intr+0xf0/0xf0 [ 3812.234795] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3812.239021] kthread+0x1d1/0x200 [ 3812.243646] ? set_kthread_struct+0x70/0x70 [ 3812.249103] ret_from_fork+0x1f/0x30 [ 3824.052522] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 3824.057371] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3824.057571] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 3824.098592] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3824.106716] Modules linked in: [ 3824.113312] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3824.113322] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3824.113409] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3824.113414] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3824.113419] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3824.113422] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 3824.113424] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 3824.113426] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 3824.113429] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 3824.113432] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 3824.113435] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3824.113438] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 3824.113446] Call Trace: [ 3824.113462] [ 3824.113467] ? show_regs.cold.9+0x22/0x2f [ 3824.113476] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3824.113482] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3824.113487] ? watchdog+0x40/0x40 [ 3824.113491] ? hrtimer_interrupt+0x128/0x2c0 [ 3824.113495] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3824.113502] ? apic_timer_interrupt+0xf/0x20 [ 3824.113507] [ 3824.113509] ? native_safe_halt+0x2e/0x50 [ 3824.113513] kvm_wait+0xcd/0xe0 [ 3824.113520] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3824.113526] do_raw_spin_lock+0x177/0x190 [ 3824.113531] _raw_spin_lock+0x21/0x30 [ 3824.113537] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3824.113825] ? do_raw_spin_unlock+0x75/0x190 [ 3824.113831] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3824.114606] ? do_wait_intr+0xf0/0xf0 [ 3824.114614] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3824.114816] mdc_reint+0x2d/0x110 [mdc] [ 3824.114865] mdc_unlink+0x658/0x870 [mdc] [ 3824.114902] lmv_unlink+0x34d/0xbd0 [lmv] [ 3824.114940] ll_unlink+0x24d/0x7f0 [lustre] [ 3824.115158] vfs_unlink+0x202/0x330 [ 3824.115166] do_unlinkat+0x2fa/0x460 [ 3824.115171] __x64_sys_unlink+0x37/0x50 [ 3824.115175] do_syscall_64+0xc1/0x440 [ 3824.115181] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3824.115188] RIP: 0033:0x7f151bdee10b [ 3824.115219] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3824.115224] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3824.115228] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 3824.115230] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 3824.115232] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3824.115234] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3824.115236] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3824.297646] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3824.318140] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3824.323555] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3824.328298] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3824.330545] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3824.338366] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3824.341877] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3824.345523] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 3824.348920] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3824.353866] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 3824.357399] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 3824.362121] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 3824.365604] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3824.368519] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 3824.371706] Call Trace: [ 3824.373888] [ 3824.376048] ? show_regs.cold.9+0x22/0x2f [ 3824.379789] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3824.382267] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3824.384643] ? watchdog+0x40/0x40 [ 3824.386199] ? hrtimer_interrupt+0x128/0x2c0 [ 3824.388857] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3824.391527] ? apic_timer_interrupt+0xf/0x20 [ 3824.393587] [ 3824.394625] ? native_safe_halt+0x2e/0x50 [ 3824.396529] kvm_wait+0xcd/0xe0 [ 3824.398447] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3824.401197] do_raw_spin_lock+0x177/0x190 [ 3824.403344] _raw_spin_lock+0x21/0x30 [ 3824.405062] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3824.407357] ? do_raw_spin_unlock+0x75/0x190 [ 3824.408957] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3824.411175] ? do_wait_intr+0xf0/0xf0 [ 3824.412371] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3824.414632] mdc_close+0x53b/0xd70 [mdc] [ 3824.416400] lmv_close+0x18d/0x490 [lmv] [ 3824.418133] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3824.420678] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3824.422891] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3824.425084] ll_md_close+0x274/0x800 [lustre] [ 3824.427075] ll_file_release+0x128/0xc50 [lustre] [ 3824.428898] __fput+0x10d/0x440 [ 3824.429899] ____fput+0x12/0x20 [ 3824.431342] task_work_run+0xc6/0x110 [ 3824.432758] exit_to_usermode_loop+0x1dd/0x1f0 [ 3824.434217] do_syscall_64+0x3d6/0x440 [ 3824.435639] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3824.437551] RIP: 0033:0x7f151bee3cd8 [ 3824.439274] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3824.445610] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3824.448608] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 3824.451496] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 3824.453965] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3824.456943] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3824.459601] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3829.220107] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1898s! [ 3829.232926] Showing busy workqueues and worker pools: [ 3829.242158] workqueue events: flags=0x0 [ 3829.244610] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3829.250632] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 3829.265721] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3829.272186] pending: kfree_rcu_monitor [ 3829.275404] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3829.278698] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 3829.285544] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3829.291865] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3829.297635] workqueue events_long: flags=0x0 [ 3829.300410] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3829.302940] pending: lu_objects_flush [obdclass] [ 3829.307318] workqueue events_unbound: flags=0x2 [ 3829.312193] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3829.315260] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3829.322115] workqueue events_power_efficient: flags=0x80 [ 3829.325179] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3829.331085] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 3829.334366] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3829.338350] pending: neigh_periodic_work [ 3829.342363] workqueue mm_percpu_wq: flags=0x8 [ 3829.348734] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3829.357285] pending: vmstat_update, lru_add_drain_per_cpu [ 3829.362011] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3829.366182] pending: vmstat_update [ 3829.368541] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3829.373359] pending: lru_add_drain_per_cpu [ 3829.375130] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3829.378384] pending: lru_add_drain_per_cpu BAR(48) [ 3829.387043] workqueue writeback: flags=0x4a [ 3829.390060] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3829.394107] pending: wb_workfn [ 3829.395899] workqueue xprtiod: flags=0xa [ 3829.399701] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3829.409145] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3829.418650] workqueue ipv6_addrconf: flags=0x40008 [ 3829.425586] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3829.429077] pending: addrconf_verify_work [ 3829.432081] workqueue knbd0-recv: flags=0x1a [ 3829.434860] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3829.439640] in-flight: 658:recv_work [ 3829.442103] workqueue ptlrpc_pinger: flags=0x2 [ 3829.444214] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3829.452639] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3829.463606] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1898s workers=4 idle: 32984 37 8 [ 3829.470093] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3814s workers=2 idle: 661 [ 3832.062670] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3832.077748] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3832.122067] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3832.136707] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3832.143008] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3832.153625] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3832.158068] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3832.173230] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3832.181267] RAX: 0000000000007f0c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3832.187512] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 3832.195628] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 3832.199273] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 3832.205440] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 3832.208670] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 3832.213297] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3832.216393] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3832.220117] Call Trace: [ 3832.223117] [ 3832.224064] ? show_regs.cold.9+0x22/0x2f [ 3832.228482] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3832.233704] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3832.237962] ? watchdog+0x40/0x40 [ 3832.239784] ? hrtimer_interrupt+0x128/0x2c0 [ 3832.243205] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3832.246204] ? apic_timer_interrupt+0xf/0x20 [ 3832.248265] [ 3832.249423] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3832.253073] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3832.256273] do_raw_spin_lock+0x177/0x190 [ 3832.259793] _raw_spin_lock+0x21/0x30 [ 3832.261816] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3832.265864] process_one_work+0x2c8/0x700 [ 3832.275474] worker_thread+0x296/0x6e0 [ 3832.280182] ? rescuer_thread+0x570/0x570 [ 3832.284341] kthread+0x1d1/0x200 [ 3832.287529] ? set_kthread_struct+0x70/0x70 [ 3832.296834] ret_from_fork+0x1f/0x30 [ 3840.055277] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 3840.060739] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3840.094252] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3840.104185] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3840.111545] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3840.116959] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3840.130228] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3840.136002] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 3840.140495] RDX: 0000000000005262 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 3840.148053] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 3840.154260] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 3840.161256] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3840.169102] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 3840.175081] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3840.177952] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3840.183075] Call Trace: [ 3840.184136] [ 3840.186083] ? show_regs.cold.9+0x22/0x2f [ 3840.188863] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3840.191761] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3840.194261] ? watchdog+0x40/0x40 [ 3840.197150] ? hrtimer_interrupt+0x128/0x2c0 [ 3840.200131] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3840.204408] ? apic_timer_interrupt+0xf/0x20 [ 3840.207714] [ 3840.209125] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3840.212318] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3840.215221] do_raw_spin_lock+0x177/0x190 [ 3840.217201] _raw_spin_lock+0x21/0x30 [ 3840.220023] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3840.223150] obd_mod_free+0x46/0x140 [ptlrpc] [ 3840.226260] mdc_commit_open+0x91/0xb0 [mdc] [ 3840.229313] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3840.233181] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3840.238162] after_reply+0xe2c/0x1780 [ptlrpc] [ 3840.241197] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3840.249800] ? __next_timer_interrupt+0x160/0x160 [ 3840.255015] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3840.258146] ? do_wait_intr+0xf0/0xf0 [ 3840.260286] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3840.264146] kthread+0x1d1/0x200 [ 3840.266535] ? set_kthread_struct+0x70/0x70 [ 3840.268935] ret_from_fork+0x1f/0x30 [ 3852.060441] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 3852.070100] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3852.111786] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3852.118920] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3852.123898] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3852.127563] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3852.144062] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 3852.148287] RAX: 000000000000788b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3852.155138] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 3852.166715] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3852.174429] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 3852.179204] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 3852.189233] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 3852.194543] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3852.204081] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 3852.206367] Call Trace: [ 3852.213349] [ 3852.215105] ? show_regs.cold.9+0x22/0x2f [ 3852.218906] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3852.222610] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3852.225671] ? watchdog+0x40/0x40 [ 3852.228225] ? hrtimer_interrupt+0x128/0x2c0 [ 3852.233065] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3852.236268] ? apic_timer_interrupt+0xf/0x20 [ 3852.240612] [ 3852.241549] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3852.243763] do_raw_spin_lock+0x177/0x190 [ 3852.245922] _raw_spin_lock+0x21/0x30 [ 3852.248964] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3852.252259] ? do_raw_spin_unlock+0x75/0x190 [ 3852.257343] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3852.262773] ? do_wait_intr+0xf0/0xf0 [ 3852.267941] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3852.271313] mdc_close+0x53b/0xd70 [mdc] [ 3852.275014] lmv_close+0x18d/0x490 [lmv] [ 3852.276755] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3852.280021] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3852.282201] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3852.284282] ll_md_close+0x274/0x800 [lustre] [ 3852.285560] ll_file_release+0x128/0xc50 [lustre] [ 3852.287950] __fput+0x10d/0x440 [ 3852.288621] ____fput+0x12/0x20 [ 3852.289660] task_work_run+0xc6/0x110 [ 3852.290703] exit_to_usermode_loop+0x1dd/0x1f0 [ 3852.292817] do_syscall_64+0x3d6/0x440 [ 3852.294781] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3852.298106] RIP: 0033:0x7f151bee3cd8 [ 3852.299887] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3852.308047] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3852.311301] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 3852.314812] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 3852.317844] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3852.320807] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3852.324098] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3856.052010] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 3856.057557] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3856.101244] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3856.113226] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3856.122370] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3856.129970] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3856.153737] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 [ 3856.160385] RAX: 0000000000007300 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3856.165258] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 3856.168115] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 3856.171211] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 3856.174278] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 3856.177498] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 3856.180990] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3856.183533] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 3856.186480] Call Trace: [ 3856.187419] [ 3856.188553] ? show_regs.cold.9+0x22/0x2f [ 3856.190406] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3856.192671] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3856.195147] ? watchdog+0x40/0x40 [ 3856.196597] ? hrtimer_interrupt+0x128/0x2c0 [ 3856.199255] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3856.201645] ? apic_timer_interrupt+0xf/0x20 [ 3856.204284] [ 3856.205319] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3856.208372] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3856.210557] do_raw_spin_lock+0x177/0x190 [ 3856.212376] _raw_spin_lock+0x21/0x30 [ 3856.217074] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3856.220388] ? do_raw_spin_unlock+0x75/0x190 [ 3856.221769] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3856.224154] ? do_wait_intr+0xf0/0xf0 [ 3856.226072] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3856.229194] mdc_reint+0x2d/0x110 [mdc] [ 3856.231339] mdc_unlink+0x658/0x870 [mdc] [ 3856.233505] lmv_unlink+0x34d/0xbd0 [lmv] [ 3856.236766] ll_unlink+0x24d/0x7f0 [lustre] [ 3856.238860] vfs_unlink+0x202/0x330 [ 3856.240622] do_unlinkat+0x2fa/0x460 [ 3856.242680] __x64_sys_unlink+0x37/0x50 [ 3856.249491] do_syscall_64+0xc1/0x440 [ 3856.251782] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3856.257872] RIP: 0033:0x7f151bdee10b [ 3856.260173] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3856.268624] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3856.272835] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 3856.276563] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 3856.280066] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3856.283563] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3856.286588] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3859.936526] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1929s! [ 3859.968827] Showing busy workqueues and worker pools: [ 3859.978091] workqueue events: flags=0x0 [ 3859.987528] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3860.003935] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 3860.023762] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3860.032657] pending: kfree_rcu_monitor [ 3860.044835] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3860.055339] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 3860.058419] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3860.063072] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3860.063095] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3860.063452] workqueue events_long: flags=0x0 [ 3860.063455] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3860.063464] pending: lu_objects_flush [obdclass] [ 3860.063624] workqueue events_unbound: flags=0x2 [ 3860.063627] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3860.063633] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3860.063654] workqueue events_power_efficient: flags=0x80 [ 3860.063656] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3860.063663] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 3860.063902] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3860.063911] pending: neigh_periodic_work [ 3860.063919] workqueue mm_percpu_wq: flags=0x8 [ 3860.063921] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3860.063928] pending: vmstat_update, lru_add_drain_per_cpu [ 3860.063939] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3860.063944] pending: vmstat_update [ 3860.063948] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3860.063955] pending: lru_add_drain_per_cpu [ 3860.063961] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3860.063967] pending: lru_add_drain_per_cpu BAR(48) [ 3860.063977] workqueue writeback: flags=0x4a [ 3860.063979] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3860.063984] pending: wb_workfn [ 3860.063997] workqueue xprtiod: flags=0xa [ 3860.064085] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3860.064092] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3860.064111] workqueue ipv6_addrconf: flags=0x40008 [ 3860.064114] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3860.064120] pending: addrconf_verify_work [ 3860.064131] workqueue knbd0-recv: flags=0x1a [ 3860.064134] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3860.064140] in-flight: 658:recv_work [ 3860.064150] workqueue ptlrpc_pinger: flags=0x2 [ 3860.064153] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3860.064158] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3860.064551] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1929s workers=4 idle: 32984 37 8 [ 3860.064564] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3844s workers=2 idle: 661 [ 3860.284174] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3860.323584] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3860.339549] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3860.343903] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3860.353382] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3860.358067] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3860.368359] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3860.372433] RAX: 0000000000007f94 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3860.378624] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 3860.386389] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 3860.390205] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 3860.394455] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 3860.397567] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 3860.410967] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3860.419241] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3860.427301] Call Trace: [ 3860.436927] [ 3860.439715] ? show_regs.cold.9+0x22/0x2f [ 3860.445901] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3860.451663] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3860.457463] ? watchdog+0x40/0x40 [ 3860.458696] ? hrtimer_interrupt+0x128/0x2c0 [ 3860.460194] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3860.465409] ? apic_timer_interrupt+0xf/0x20 [ 3860.476458] [ 3860.480198] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3860.489106] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3860.496022] do_raw_spin_lock+0x177/0x190 [ 3860.499868] _raw_spin_lock+0x21/0x30 [ 3860.504486] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3860.513733] process_one_work+0x2c8/0x700 [ 3860.524071] worker_thread+0x296/0x6e0 [ 3860.527111] ? rescuer_thread+0x570/0x570 [ 3860.540312] kthread+0x1d1/0x200 [ 3860.541374] ? set_kthread_struct+0x70/0x70 [ 3860.547600] ret_from_fork+0x1f/0x30 [ 3868.056304] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 3868.064645] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3868.106515] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3868.116181] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3868.122924] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3868.133408] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3868.152695] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3868.158078] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 3868.165750] RDX: 0000000000005230 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 3868.171943] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 3868.178085] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 3868.186117] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3868.194351] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 3868.203437] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3868.210098] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3868.220387] Call Trace: [ 3868.221396] [ 3868.223398] ? show_regs.cold.9+0x22/0x2f [ 3868.226006] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3868.233814] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3868.238078] ? watchdog+0x40/0x40 [ 3868.241710] ? hrtimer_interrupt+0x128/0x2c0 [ 3868.246275] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3868.252285] ? apic_timer_interrupt+0xf/0x20 [ 3868.256308] [ 3868.257836] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3868.261420] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3868.268099] do_raw_spin_lock+0x177/0x190 [ 3868.274255] _raw_spin_lock+0x21/0x30 [ 3868.277404] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3868.283871] obd_mod_free+0x46/0x140 [ptlrpc] [ 3868.289924] mdc_commit_open+0x91/0xb0 [mdc] [ 3868.293443] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3868.296428] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3868.301245] after_reply+0xe2c/0x1780 [ptlrpc] [ 3868.304300] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3868.310876] ? __next_timer_interrupt+0x160/0x160 [ 3868.313192] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3868.315880] ? do_wait_intr+0xf0/0xf0 [ 3868.317714] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3868.322802] kthread+0x1d1/0x200 [ 3868.324679] ? set_kthread_struct+0x70/0x70 [ 3868.326656] ret_from_fork+0x1f/0x30 [ 3880.060140] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 3880.069430] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3880.113136] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3880.128019] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3880.136529] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3880.141885] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3880.163763] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3880.168761] RAX: 0000000000007e8b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3880.172369] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 3880.176397] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3880.186186] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 3880.198232] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 3880.200138] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 3880.216113] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3880.228341] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 3880.246231] Call Trace: [ 3880.248960] [ 3880.253802] ? show_regs.cold.9+0x22/0x2f [ 3880.261497] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3880.267172] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3880.268661] ? watchdog+0x40/0x40 [ 3880.276222] ? hrtimer_interrupt+0x128/0x2c0 [ 3880.284737] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3880.291180] ? apic_timer_interrupt+0xf/0x20 [ 3880.295159] [ 3880.297441] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3880.300179] do_raw_spin_lock+0x177/0x190 [ 3880.302709] _raw_spin_lock+0x21/0x30 [ 3880.306677] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3880.309421] ? do_raw_spin_unlock+0x75/0x190 [ 3880.312800] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3880.316548] ? do_wait_intr+0xf0/0xf0 [ 3880.318405] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3880.321153] mdc_close+0x53b/0xd70 [mdc] [ 3880.323250] lmv_close+0x18d/0x490 [lmv] [ 3880.326708] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3880.335285] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3880.341209] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3880.345230] ll_md_close+0x274/0x800 [lustre] [ 3880.350126] ll_file_release+0x128/0xc50 [lustre] [ 3880.359168] __fput+0x10d/0x440 [ 3880.364562] ____fput+0x12/0x20 [ 3880.369063] task_work_run+0xc6/0x110 [ 3880.377627] exit_to_usermode_loop+0x1dd/0x1f0 [ 3880.383238] do_syscall_64+0x3d6/0x440 [ 3880.391717] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3880.395094] RIP: 0033:0x7f151bee3cd8 [ 3880.396059] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3880.428703] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3880.446201] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 3880.466254] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 3880.471303] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3880.478332] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3880.490874] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3888.052064] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 3888.055636] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3888.056202] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 3888.100715] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3888.100731] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3888.100733] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3888.100746] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3888.100750] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3888.100756] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3888.100758] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 3888.100761] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 3888.100763] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 3888.100765] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 3888.100768] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 3888.100771] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3888.100774] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 3888.100958] Call Trace: [ 3888.100965] [ 3888.100971] ? show_regs.cold.9+0x22/0x2f [ 3888.100981] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3888.100986] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3888.100990] ? watchdog+0x40/0x40 [ 3888.100994] ? hrtimer_interrupt+0x128/0x2c0 [ 3888.100998] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3888.101005] ? apic_timer_interrupt+0xf/0x20 [ 3888.101009] [ 3888.101012] ? native_safe_halt+0x2e/0x50 [ 3888.101016] kvm_wait+0xcd/0xe0 [ 3888.101024] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3888.101032] do_raw_spin_lock+0x177/0x190 [ 3888.101037] _raw_spin_lock+0x21/0x30 [ 3888.101046] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3888.101438] ? do_raw_spin_unlock+0x75/0x190 [ 3888.101444] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3888.101762] ? do_wait_intr+0xf0/0xf0 [ 3888.101771] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3888.105266] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3888.105355] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3888.105374] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3888.105381] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3888.105777] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3888.105788] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3888.105793] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3888.105797] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3888.105799] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 3888.105801] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 3888.105804] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 3888.105806] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 3888.105809] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 3888.105817] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3888.105819] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3888.105826] Call Trace: [ 3888.105832] [ 3888.105838] ? show_regs.cold.9+0x22/0x2f [ 3888.105847] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3888.105853] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3888.105861] ? watchdog+0x40/0x40 [ 3888.105867] ? hrtimer_interrupt+0x128/0x2c0 [ 3888.105870] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3888.105876] ? apic_timer_interrupt+0xf/0x20 [ 3888.105881] [ 3888.105884] ? native_safe_halt+0x2e/0x50 [ 3888.105964] kvm_wait+0xcd/0xe0 [ 3888.105998] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3888.106081] do_raw_spin_lock+0x177/0x190 [ 3888.106086] _raw_spin_lock+0x21/0x30 [ 3888.106090] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3888.106497] process_one_work+0x2c8/0x700 [ 3888.106505] worker_thread+0x296/0x6e0 [ 3888.106510] ? rescuer_thread+0x570/0x570 [ 3888.106514] kthread+0x1d1/0x200 [ 3888.106518] ? set_kthread_struct+0x70/0x70 [ 3888.106522] ret_from_fork+0x1f/0x30 [ 3888.548873] mdc_reint+0x2d/0x110 [mdc] [ 3888.551908] mdc_unlink+0x658/0x870 [mdc] [ 3888.555971] lmv_unlink+0x34d/0xbd0 [lmv] [ 3888.560336] ll_unlink+0x24d/0x7f0 [lustre] [ 3888.563675] vfs_unlink+0x202/0x330 [ 3888.566324] do_unlinkat+0x2fa/0x460 [ 3888.569309] __x64_sys_unlink+0x37/0x50 [ 3888.572923] do_syscall_64+0xc1/0x440 [ 3888.575371] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3888.578959] RIP: 0033:0x7f151bdee10b [ 3888.581117] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3888.596907] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3888.603211] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 3888.607497] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 3888.612146] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3888.621950] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3888.633864] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3890.150675] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1959s! [ 3890.156431] Showing busy workqueues and worker pools: [ 3890.159142] workqueue events: flags=0x0 [ 3890.166430] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3890.170755] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 3890.178076] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3890.181529] pending: kfree_rcu_monitor [ 3890.190452] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3890.196142] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 3890.199700] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3890.202230] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3890.208212] workqueue events_long: flags=0x0 [ 3890.209784] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3890.219855] pending: lu_objects_flush [obdclass] [ 3890.224853] workqueue events_unbound: flags=0x2 [ 3890.226844] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3890.231848] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3890.242056] workqueue events_power_efficient: flags=0x80 [ 3890.246231] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3890.249801] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 3890.254836] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3890.259869] pending: neigh_periodic_work [ 3890.261645] workqueue mm_percpu_wq: flags=0x8 [ 3890.265908] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3890.268760] pending: vmstat_update, lru_add_drain_per_cpu [ 3890.271662] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3890.277624] pending: vmstat_update [ 3890.280787] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3890.282515] pending: lru_add_drain_per_cpu [ 3890.288422] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3890.292246] pending: lru_add_drain_per_cpu BAR(48) [ 3890.294500] workqueue writeback: flags=0x4a [ 3890.296611] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3890.299175] pending: wb_workfn [ 3890.300204] workqueue xprtiod: flags=0xa [ 3890.304228] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3890.307741] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3890.320765] workqueue ipv6_addrconf: flags=0x40008 [ 3890.322882] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3890.332198] pending: addrconf_verify_work [ 3890.339622] workqueue knbd0-recv: flags=0x1a [ 3890.343831] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3890.350847] in-flight: 658:recv_work [ 3890.354723] workqueue ptlrpc_pinger: flags=0x2 [ 3890.358681] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3890.363796] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3890.371576] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1959s workers=4 idle: 32984 37 8 [ 3890.376695] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3875s workers=2 idle: 661 [ 3896.059501] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 3896.070185] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3896.122522] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3896.125948] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3896.128374] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3896.130170] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3896.134876] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3896.137517] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 3896.139451] RDX: 0000000000002614 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 3896.141528] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 3896.143734] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 3896.146132] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3896.148502] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 3896.152120] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3896.154078] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3896.156253] Call Trace: [ 3896.157050] [ 3896.157608] ? show_regs.cold.9+0x22/0x2f [ 3896.158697] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3896.160030] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3896.161810] ? watchdog+0x40/0x40 [ 3896.163211] ? hrtimer_interrupt+0x128/0x2c0 [ 3896.164747] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3896.166452] ? apic_timer_interrupt+0xf/0x20 [ 3896.167665] [ 3896.168260] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3896.169881] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3896.171711] do_raw_spin_lock+0x177/0x190 [ 3896.173093] _raw_spin_lock+0x21/0x30 [ 3896.174528] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3896.176230] obd_mod_free+0x46/0x140 [ptlrpc] [ 3896.178049] mdc_commit_open+0x91/0xb0 [mdc] [ 3896.179754] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3896.182204] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3896.184067] after_reply+0xe2c/0x1780 [ptlrpc] [ 3896.185767] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3896.187811] ? __next_timer_interrupt+0x160/0x160 [ 3896.189134] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3896.191367] ? do_wait_intr+0xf0/0xf0 [ 3896.192515] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3896.194415] kthread+0x1d1/0x200 [ 3896.195438] ? set_kthread_struct+0x70/0x70 [ 3896.196752] ret_from_fork+0x1f/0x30 [ 3908.057008] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 3908.076270] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3908.140057] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3908.155352] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3908.164853] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3908.170478] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3908.192440] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3908.200471] RAX: 0000000000006049 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3908.209377] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 3908.217027] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3908.225053] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 3908.237440] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 3908.247775] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 3908.257064] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3908.260103] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 3908.270327] Call Trace: [ 3908.271272] [ 3908.272027] ? show_regs.cold.9+0x22/0x2f [ 3908.274523] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3908.280907] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3908.283983] ? watchdog+0x40/0x40 [ 3908.286312] ? hrtimer_interrupt+0x128/0x2c0 [ 3908.288820] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3908.291408] ? apic_timer_interrupt+0xf/0x20 [ 3908.305949] [ 3908.310392] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3908.315376] do_raw_spin_lock+0x177/0x190 [ 3908.323030] _raw_spin_lock+0x21/0x30 [ 3908.325796] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3908.330663] ? do_raw_spin_unlock+0x75/0x190 [ 3908.333294] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3908.338320] ? do_wait_intr+0xf0/0xf0 [ 3908.343510] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3908.352265] mdc_close+0x53b/0xd70 [mdc] [ 3908.360099] lmv_close+0x18d/0x490 [lmv] [ 3908.361489] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3908.370480] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3908.373734] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3908.375511] ll_md_close+0x274/0x800 [lustre] [ 3908.386141] ll_file_release+0x128/0xc50 [lustre] [ 3908.396269] __fput+0x10d/0x440 [ 3908.398137] ____fput+0x12/0x20 [ 3908.407546] task_work_run+0xc6/0x110 [ 3908.412402] exit_to_usermode_loop+0x1dd/0x1f0 [ 3908.423962] do_syscall_64+0x3d6/0x440 [ 3908.427424] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3908.442350] RIP: 0033:0x7f151bee3cd8 [ 3908.449982] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3908.481135] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3908.495033] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 3908.506489] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 3908.522969] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3908.538116] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3908.549408] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3916.054292] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 3916.056069] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 3916.065641] Modules linked in: lustre(O) [ 3916.081465] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3916.081551] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3916.081557] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3916.081564] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3916.085038] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3916.085064] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3916.085702] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3916.085708] RAX: 000000000000044e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3916.085710] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 3916.085712] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 3916.085714] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 3916.085716] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 3916.085719] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 3916.085722] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3916.085724] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3916.085732] Call Trace: [ 3916.085737] [ 3916.085743] ? show_regs.cold.9+0x22/0x2f [ 3916.085754] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3916.085760] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3916.085764] ? watchdog+0x40/0x40 [ 3916.085768] ? hrtimer_interrupt+0x128/0x2c0 [ 3916.085772] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3916.085781] ? apic_timer_interrupt+0xf/0x20 [ 3916.085785] [ 3916.085787] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3916.085792] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3916.085800] do_raw_spin_lock+0x177/0x190 [ 3916.085805] _raw_spin_lock+0x21/0x30 [ 3916.085810] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3916.088861] process_one_work+0x2c8/0x700 [ 3916.088874] worker_thread+0x296/0x6e0 [ 3916.088879] ? rescuer_thread+0x570/0x570 [ 3916.088883] kthread+0x1d1/0x200 [ 3916.088888] ? set_kthread_struct+0x70/0x70 [ 3916.088891] ret_from_fork+0x1f/0x30 [ 3916.420915] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3916.452814] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3916.461043] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3916.467045] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3916.469212] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3916.497262] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 3916.504797] RAX: 0000000000007f8a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3916.511062] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 3916.516999] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 3916.524758] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 3916.527671] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 3916.542985] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 3916.553669] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3916.559804] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 3916.563044] Call Trace: [ 3916.564887] [ 3916.565834] ? show_regs.cold.9+0x22/0x2f [ 3916.570883] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3916.573313] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3916.574753] ? watchdog+0x40/0x40 [ 3916.579037] ? hrtimer_interrupt+0x128/0x2c0 [ 3916.587068] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3916.598425] ? apic_timer_interrupt+0xf/0x20 [ 3916.602265] [ 3916.603082] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3916.609215] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3916.612328] do_raw_spin_lock+0x177/0x190 [ 3916.614900] _raw_spin_lock+0x21/0x30 [ 3916.618391] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3916.623496] ? do_raw_spin_unlock+0x75/0x190 [ 3916.627276] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3916.637030] ? do_wait_intr+0xf0/0xf0 [ 3916.638441] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3916.649551] mdc_reint+0x2d/0x110 [mdc] [ 3916.651097] mdc_unlink+0x658/0x870 [mdc] [ 3916.656313] lmv_unlink+0x34d/0xbd0 [lmv] [ 3916.661845] ll_unlink+0x24d/0x7f0 [lustre] [ 3916.667995] vfs_unlink+0x202/0x330 [ 3916.673791] do_unlinkat+0x2fa/0x460 [ 3916.679996] __x64_sys_unlink+0x37/0x50 [ 3916.684699] do_syscall_64+0xc1/0x440 [ 3916.693033] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3916.698433] RIP: 0033:0x7f151bdee10b [ 3916.706385] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3916.730888] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3916.734381] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 3916.748231] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 3916.755270] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3916.766397] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3916.779804] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3920.864094] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1990s! [ 3920.869272] Showing busy workqueues and worker pools: [ 3920.872118] workqueue events: flags=0x0 [ 3920.883726] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3920.892367] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 3920.900936] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3920.906704] pending: kfree_rcu_monitor [ 3920.913113] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3920.922306] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 3920.932233] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3920.939923] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3920.946673] workqueue events_long: flags=0x0 [ 3920.955695] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3920.965196] pending: lu_objects_flush [obdclass] [ 3920.969046] workqueue events_unbound: flags=0x2 [ 3920.980687] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3920.986110] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3920.997417] workqueue events_power_efficient: flags=0x80 [ 3920.999429] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3921.011477] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 3921.023135] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3921.034769] pending: neigh_periodic_work [ 3921.038663] workqueue mm_percpu_wq: flags=0x8 [ 3921.040313] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3921.051931] pending: vmstat_update, lru_add_drain_per_cpu [ 3921.057513] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3921.060038] pending: vmstat_update [ 3921.065353] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3921.070769] pending: lru_add_drain_per_cpu [ 3921.075020] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3921.081911] pending: lru_add_drain_per_cpu BAR(48) [ 3921.085018] workqueue writeback: flags=0x4a [ 3921.088179] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3921.091922] pending: wb_workfn [ 3921.097941] workqueue xprtiod: flags=0xa [ 3921.103361] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3921.107863] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3921.123231] workqueue ipv6_addrconf: flags=0x40008 [ 3921.129313] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3921.137796] pending: addrconf_verify_work [ 3921.143867] workqueue knbd0-recv: flags=0x1a [ 3921.145405] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3921.157584] in-flight: 658:recv_work [ 3921.162137] workqueue ptlrpc_pinger: flags=0x2 [ 3921.166212] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3921.170297] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3921.176875] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1990s workers=4 idle: 32984 37 8 [ 3921.182199] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3905s workers=2 idle: 661 [ 3924.056369] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 3924.061752] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3924.091987] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3924.099873] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3924.111240] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3924.113175] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3924.130943] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3924.136335] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 3924.145801] RDX: 0000000000007b21 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 3924.156349] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 3924.164983] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 3924.175039] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3924.190520] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 3924.200373] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3924.209499] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3924.211895] Call Trace: [ 3924.220608] [ 3924.221350] ? show_regs.cold.9+0x22/0x2f [ 3924.222789] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3924.236796] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3924.243258] ? watchdog+0x40/0x40 [ 3924.247392] ? hrtimer_interrupt+0x128/0x2c0 [ 3924.252692] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3924.258293] ? apic_timer_interrupt+0xf/0x20 [ 3924.262311] [ 3924.267742] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3924.271787] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3924.277315] do_raw_spin_lock+0x177/0x190 [ 3924.281137] _raw_spin_lock+0x21/0x30 [ 3924.282970] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3924.286207] obd_mod_free+0x46/0x140 [ptlrpc] [ 3924.290864] mdc_commit_open+0x91/0xb0 [mdc] [ 3924.293436] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3924.301110] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3924.305239] after_reply+0xe2c/0x1780 [ptlrpc] [ 3924.313797] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3924.321196] ? __next_timer_interrupt+0x160/0x160 [ 3924.326823] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3924.329934] ? do_wait_intr+0xf0/0xf0 [ 3924.332209] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3924.336798] kthread+0x1d1/0x200 [ 3924.340382] ? set_kthread_struct+0x70/0x70 [ 3924.344003] ret_from_fork+0x1f/0x30 [ 3936.057019] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 3936.061713] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3936.093961] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3936.102128] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3936.109637] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3936.113143] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3936.151274] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3936.165071] RAX: 00000000000047b7 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3936.174458] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 3936.188222] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3936.191670] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 3936.201644] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 3936.208145] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 3936.216321] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3936.227195] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 3936.232221] Call Trace: [ 3936.233353] [ 3936.234387] ? show_regs.cold.9+0x22/0x2f [ 3936.239424] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3936.244614] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3936.248337] ? watchdog+0x40/0x40 [ 3936.254132] ? hrtimer_interrupt+0x128/0x2c0 [ 3936.255862] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3936.263658] ? apic_timer_interrupt+0xf/0x20 [ 3936.269707] [ 3936.273405] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3936.276416] do_raw_spin_lock+0x177/0x190 [ 3936.279746] _raw_spin_lock+0x21/0x30 [ 3936.283441] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3936.290194] ? do_raw_spin_unlock+0x75/0x190 [ 3936.295643] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3936.302630] ? do_wait_intr+0xf0/0xf0 [ 3936.305780] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3936.314088] mdc_close+0x53b/0xd70 [mdc] [ 3936.315414] lmv_close+0x18d/0x490 [lmv] [ 3936.320444] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3936.326844] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3936.329842] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3936.337181] ll_md_close+0x274/0x800 [lustre] [ 3936.340359] ll_file_release+0x128/0xc50 [lustre] [ 3936.343092] __fput+0x10d/0x440 [ 3936.344695] ____fput+0x12/0x20 [ 3936.347100] task_work_run+0xc6/0x110 [ 3936.350365] exit_to_usermode_loop+0x1dd/0x1f0 [ 3936.355060] do_syscall_64+0x3d6/0x440 [ 3936.356927] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3936.359062] RIP: 0033:0x7f151bee3cd8 [ 3936.360349] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3936.369095] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3936.371415] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 3936.374251] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 3936.377606] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3936.380621] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3936.383690] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3944.058132] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 3944.062415] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3944.104001] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3944.115688] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3944.126762] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3944.137864] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3944.147343] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3944.162974] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 3944.171093] RAX: 0000000000003a19 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3944.178500] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 3944.185673] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 3944.193488] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 3944.197429] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 3944.200655] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 3944.208365] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3944.214400] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3944.221006] Call Trace: [ 3944.222136] [ 3944.225795] ? show_regs.cold.9+0x22/0x2f [ 3944.228303] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3944.232576] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3944.236386] ? watchdog+0x40/0x40 [ 3944.240413] ? hrtimer_interrupt+0x128/0x2c0 [ 3944.242699] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3944.246767] ? apic_timer_interrupt+0xf/0x20 [ 3944.248784] [ 3944.254029] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3944.255636] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3944.261311] do_raw_spin_lock+0x177/0x190 [ 3944.263114] _raw_spin_lock+0x21/0x30 [ 3944.268808] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3944.271158] process_one_work+0x2c8/0x700 [ 3944.272909] worker_thread+0x296/0x6e0 [ 3944.278468] ? rescuer_thread+0x570/0x570 [ 3944.280299] kthread+0x1d1/0x200 [ 3944.283299] ? set_kthread_struct+0x70/0x70 [ 3944.285055] ret_from_fork+0x1f/0x30 [ 3948.059939] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 3948.071569] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3948.140005] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3948.153224] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3948.167463] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3948.178614] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3948.196243] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3948.202110] RAX: 00000000000071a0 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3948.211809] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 3948.224835] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 3948.234081] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 3948.240943] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 3948.250526] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 3948.256498] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3948.262276] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 3948.269881] Call Trace: [ 3948.271491] [ 3948.275490] ? show_regs.cold.9+0x22/0x2f [ 3948.281800] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3948.287217] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3948.295519] ? watchdog+0x40/0x40 [ 3948.300497] ? hrtimer_interrupt+0x128/0x2c0 [ 3948.305298] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3948.315644] ? apic_timer_interrupt+0xf/0x20 [ 3948.327460] [ 3948.329724] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3948.336475] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3948.341648] do_raw_spin_lock+0x177/0x190 [ 3948.351227] _raw_spin_lock+0x21/0x30 [ 3948.358449] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3948.363916] ? do_raw_spin_unlock+0x75/0x190 [ 3948.374412] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3948.384244] ? do_wait_intr+0xf0/0xf0 [ 3948.389650] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3948.396440] mdc_reint+0x2d/0x110 [mdc] [ 3948.402632] mdc_unlink+0x658/0x870 [mdc] [ 3948.404238] lmv_unlink+0x34d/0xbd0 [lmv] [ 3948.405869] ll_unlink+0x24d/0x7f0 [lustre] [ 3948.413138] vfs_unlink+0x202/0x330 [ 3948.416752] do_unlinkat+0x2fa/0x460 [ 3948.423857] __x64_sys_unlink+0x37/0x50 [ 3948.428971] do_syscall_64+0xc1/0x440 [ 3948.430336] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3948.446393] RIP: 0033:0x7f151bdee10b [ 3948.461067] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3948.474567] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3948.480547] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 3948.487137] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 3948.497521] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3948.512688] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3948.519967] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3951.588026] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2020s! [ 3951.595105] Showing busy workqueues and worker pools: [ 3951.596710] workqueue events: flags=0x0 [ 3951.598146] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3951.600140] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 3951.604350] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3951.608350] pending: kfree_rcu_monitor [ 3951.613390] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3951.617047] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 3951.622638] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3951.626741] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3951.630122] workqueue events_long: flags=0x0 [ 3951.631772] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3951.634462] pending: lu_objects_flush [obdclass] [ 3951.636581] workqueue events_unbound: flags=0x2 [ 3951.638409] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3951.641265] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3951.647612] workqueue events_power_efficient: flags=0x80 [ 3951.650687] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3951.654592] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 3951.659412] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3951.664382] pending: neigh_periodic_work [ 3951.667337] workqueue mm_percpu_wq: flags=0x8 [ 3951.670280] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3951.672629] pending: vmstat_update, lru_add_drain_per_cpu [ 3951.678241] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3951.682106] pending: vmstat_update [ 3951.683474] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3951.689399] pending: lru_add_drain_per_cpu [ 3951.693492] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3951.697432] pending: lru_add_drain_per_cpu BAR(48) [ 3951.700907] workqueue writeback: flags=0x4a [ 3951.702248] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3951.705309] pending: wb_workfn [ 3951.709373] workqueue xprtiod: flags=0xa [ 3951.712316] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3951.717220] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3951.726553] workqueue ipv6_addrconf: flags=0x40008 [ 3951.728709] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3951.732277] pending: addrconf_verify_work [ 3951.733662] workqueue knbd0-recv: flags=0x1a [ 3951.738758] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3951.741629] in-flight: 658:recv_work [ 3951.744618] workqueue ptlrpc_pinger: flags=0x2 [ 3951.746513] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3951.751692] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3951.754123] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2021s workers=4 idle: 32984 37 8 [ 3951.757655] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3936s workers=2 idle: 661 [ 3952.056651] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 3952.067166] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3952.119026] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3952.135641] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3952.150124] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3952.161060] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3952.177993] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3952.186254] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 3952.193078] RDX: 0000000000003591 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 3952.201651] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 3952.206335] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 3952.212755] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3952.219571] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 3952.230072] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3952.232305] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3952.241175] Call Trace: [ 3952.243380] [ 3952.244320] ? show_regs.cold.9+0x22/0x2f [ 3952.247052] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3952.256557] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3952.261009] ? watchdog+0x40/0x40 [ 3952.266560] ? hrtimer_interrupt+0x128/0x2c0 [ 3952.272420] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3952.281745] ? apic_timer_interrupt+0xf/0x20 [ 3952.288180] [ 3952.291897] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3952.298067] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3952.305987] do_raw_spin_lock+0x177/0x190 [ 3952.309474] _raw_spin_lock+0x21/0x30 [ 3952.314556] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3952.321416] obd_mod_free+0x46/0x140 [ptlrpc] [ 3952.328904] mdc_commit_open+0x91/0xb0 [mdc] [ 3952.334404] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3952.343598] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3952.348172] after_reply+0xe2c/0x1780 [ptlrpc] [ 3952.357124] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3952.360386] ? __next_timer_interrupt+0x160/0x160 [ 3952.362006] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3952.367525] ? do_wait_intr+0xf0/0xf0 [ 3952.368823] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3952.376423] kthread+0x1d1/0x200 [ 3952.379139] ? set_kthread_struct+0x70/0x70 [ 3952.386333] ret_from_fork+0x1f/0x30 [ 3964.059240] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 3964.067144] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3964.121563] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3964.136698] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3964.143751] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3964.145798] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3964.168334] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 3964.175210] RAX: 0000000000007e68 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3964.185933] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 3964.189764] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3964.195101] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 3964.207007] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 3964.211764] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 3964.221600] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3964.230136] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 3964.239056] Call Trace: [ 3964.244366] [ 3964.250328] ? show_regs.cold.9+0x22/0x2f [ 3964.257464] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3964.262265] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3964.264508] ? watchdog+0x40/0x40 [ 3964.266540] ? hrtimer_interrupt+0x128/0x2c0 [ 3964.268656] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3964.273483] ? apic_timer_interrupt+0xf/0x20 [ 3964.274784] [ 3964.275392] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3964.278089] do_raw_spin_lock+0x177/0x190 [ 3964.286500] _raw_spin_lock+0x21/0x30 [ 3964.291217] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3964.301286] ? do_raw_spin_unlock+0x75/0x190 [ 3964.308765] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3964.314325] ? do_wait_intr+0xf0/0xf0 [ 3964.318096] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3964.320132] mdc_close+0x53b/0xd70 [mdc] [ 3964.324252] lmv_close+0x18d/0x490 [lmv] [ 3964.325884] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3964.328745] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3964.330972] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3964.333095] ll_md_close+0x274/0x800 [lustre] [ 3964.335236] ll_file_release+0x128/0xc50 [lustre] [ 3964.337310] __fput+0x10d/0x440 [ 3964.338754] ____fput+0x12/0x20 [ 3964.346285] task_work_run+0xc6/0x110 [ 3964.355469] exit_to_usermode_loop+0x1dd/0x1f0 [ 3964.357056] do_syscall_64+0x3d6/0x440 [ 3964.361903] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3964.365710] RIP: 0033:0x7f151bee3cd8 [ 3964.367828] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3964.384360] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3964.389906] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 3964.399418] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 3964.415714] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3964.430455] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3964.446760] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3964.685915] rcu: INFO: rcu_sched self-detected stall on CPU [ 3964.691254] rcu: 1-...!: (1221655 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 3964.707132] (t=2040038 jiffies g=249657 q=3011) [ 3964.708898] rcu: rcu_sched kthread starved for 2034990 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 3964.721856] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3964.743151] rcu: RCU grace-period kthread stack dump: [ 3964.749324] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3964.760977] Call Trace: [ 3964.762115] __schedule+0x351/0xcb0 [ 3964.763918] schedule+0xc0/0x180 [ 3964.766348] schedule_timeout+0xb4/0x190 [ 3964.768203] ? do_raw_spin_unlock+0x75/0x190 [ 3964.770284] ? __next_timer_interrupt+0x160/0x160 [ 3964.772361] ? prepare_to_swait_event+0xf3/0x140 [ 3964.774713] rcu_gp_kthread+0x8dd/0x15b0 [ 3964.776658] ? force_qs_rnp+0x2c0/0x2c0 [ 3964.778250] kthread+0x1d1/0x200 [ 3964.780136] ? set_kthread_struct+0x70/0x70 [ 3964.781964] ret_from_fork+0x1f/0x30 [ 3964.783361] rcu: Stack dump where RCU GP kthread last ran: [ 3964.785361] Sending NMI from CPU 1 to CPUs 2: [ 3964.787526] NMI backtrace for cpu 2 [ 3964.787532] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3964.787534] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3964.787536] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3964.787538] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3964.787540] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3964.787541] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 [ 3964.787543] RAX: 000000000000751d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3964.787544] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 3964.787545] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 3964.787547] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 3964.787548] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 3964.787550] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 3964.787551] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3964.787552] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3964.787554] Call Trace: [ 3964.787555] [ 3964.787555] ? show_regs.cold.9+0x22/0x2f [ 3964.787556] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 3964.787557] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 3964.787558] ? nmi_handle+0xbd/0x210 [ 3964.787560] ? default_do_nmi+0x4b/0x1b0 [ 3964.787560] ? do_nmi+0x280/0x380 [ 3964.787561] ? end_repeat_nmi+0x16/0x1f [ 3964.787562] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3964.787563] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3964.787564] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3964.787566] [ 3964.787567] do_raw_spin_lock+0x177/0x190 [ 3964.787568] _raw_spin_lock+0x21/0x30 [ 3964.787569] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3964.787570] process_one_work+0x2c8/0x700 [ 3964.787571] worker_thread+0x296/0x6e0 [ 3964.787572] ? rescuer_thread+0x570/0x570 [ 3964.787573] kthread+0x1d1/0x200 [ 3964.787574] ? set_kthread_struct+0x70/0x70 [ 3964.787575] ret_from_fork+0x1f/0x30 [ 3964.789475] NMI backtrace for cpu 1 [ 3964.945314] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3964.952417] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3964.958398] Call Trace: [ 3964.959770] [ 3964.961002] ? dump_stack+0xbb/0x10e [ 3964.962862] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3964.965250] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3964.967941] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3964.970452] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3964.972789] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3964.975207] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 3964.977436] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 3964.979875] ? update_process_times+0x6d/0xc0 [ 3964.981651] ? tick_sched_handle.isra.18+0x27/0x90 [ 3964.984275] ? tick_sched_timer+0x53/0xb0 [ 3964.986068] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3964.988053] ? tick_sched_handle.isra.18+0x90/0x90 [ 3964.990013] ? hrtimer_interrupt+0x128/0x2c0 [ 3964.992077] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3964.994344] ? apic_timer_interrupt+0xf/0x20 [ 3964.996140] [ 3964.997520] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3964.999555] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3965.002220] ? do_raw_spin_lock+0x177/0x190 [ 3965.003785] ? _raw_spin_lock+0x21/0x30 [ 3965.005564] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3965.008048] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3965.010201] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3965.012210] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3965.014893] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3965.017912] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3965.020344] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3965.022886] ? __next_timer_interrupt+0x160/0x160 [ 3965.024600] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3965.027271] ? do_wait_intr+0xf0/0xf0 [ 3965.029275] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3965.032176] ? kthread+0x1d1/0x200 [ 3965.034807] ? set_kthread_struct+0x70/0x70 [ 3965.036647] ? ret_from_fork+0x1f/0x30 [ 3976.052026] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 3976.055735] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3976.100318] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3976.112969] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3976.123222] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3976.124951] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3976.153657] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3976.159302] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3976.163789] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 3976.171083] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 3976.176687] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 3976.183885] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 3976.194144] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 3976.201851] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3976.209666] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 3976.217660] Call Trace: [ 3976.219273] [ 3976.220018] ? show_regs.cold.9+0x22/0x2f [ 3976.228903] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3976.235184] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3976.237987] ? watchdog+0x40/0x40 [ 3976.239147] ? hrtimer_interrupt+0x128/0x2c0 [ 3976.244474] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3976.250073] ? apic_timer_interrupt+0xf/0x20 [ 3976.253416] [ 3976.254481] ? native_safe_halt+0x2e/0x50 [ 3976.259947] kvm_wait+0xcd/0xe0 [ 3976.263981] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3976.267692] do_raw_spin_lock+0x177/0x190 [ 3976.269889] _raw_spin_lock+0x21/0x30 [ 3976.273270] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3976.280558] ? do_raw_spin_unlock+0x75/0x190 [ 3976.282050] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3976.291852] ? do_wait_intr+0xf0/0xf0 [ 3976.297606] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3976.302157] mdc_reint+0x2d/0x110 [mdc] [ 3976.306822] mdc_unlink+0x658/0x870 [mdc] [ 3976.309661] lmv_unlink+0x34d/0xbd0 [lmv] [ 3976.319734] ll_unlink+0x24d/0x7f0 [lustre] [ 3976.321313] vfs_unlink+0x202/0x330 [ 3976.331477] do_unlinkat+0x2fa/0x460 [ 3976.335007] __x64_sys_unlink+0x37/0x50 [ 3976.337583] do_syscall_64+0xc1/0x440 [ 3976.340740] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3976.344044] RIP: 0033:0x7f151bdee10b [ 3976.349795] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3976.378441] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3976.385030] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 3976.389227] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 3976.394287] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3976.404228] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3976.409380] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3981.795775] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2051s! [ 3981.802617] Showing busy workqueues and worker pools: [ 3981.806561] workqueue events: flags=0x0 [ 3981.810919] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3981.817670] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 3981.826771] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3981.830232] pending: kfree_rcu_monitor [ 3981.831981] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3981.836994] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 3981.841905] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3981.845712] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 3981.849632] workqueue events_long: flags=0x0 [ 3981.852636] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3981.862131] pending: lu_objects_flush [obdclass] [ 3981.866840] workqueue events_unbound: flags=0x2 [ 3981.870852] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 3981.877343] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 3981.884914] workqueue events_power_efficient: flags=0x80 [ 3981.888412] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3981.893511] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 3981.898885] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3981.906385] pending: neigh_periodic_work [ 3981.910192] workqueue mm_percpu_wq: flags=0x8 [ 3981.915250] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3981.919841] pending: vmstat_update, lru_add_drain_per_cpu [ 3981.923938] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3981.931732] pending: vmstat_update [ 3981.936275] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3981.945124] pending: lru_add_drain_per_cpu [ 3981.949058] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3981.952676] pending: lru_add_drain_per_cpu BAR(48) [ 3981.961735] workqueue writeback: flags=0x4a [ 3981.965245] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3981.968773] pending: wb_workfn [ 3981.972828] workqueue xprtiod: flags=0xa [ 3981.974619] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 3981.985587] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 3981.996416] workqueue ipv6_addrconf: flags=0x40008 [ 3981.998179] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3982.010156] pending: addrconf_verify_work [ 3982.018889] workqueue knbd0-recv: flags=0x1a [ 3982.023540] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3982.031950] in-flight: 658:recv_work [ 3982.034874] workqueue ptlrpc_pinger: flags=0x2 [ 3982.037329] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3982.046388] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 3982.054395] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2051s workers=4 idle: 32984 37 8 [ 3982.061552] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3966s workers=2 idle: 661 [ 3992.055020] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 3992.057124] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3992.057925] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 3992.058095] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 3992.058110] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3992.058169] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3992.058175] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3992.058178] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3992.058193] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3992.058197] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3992.058202] RAX: 0000000000007e92 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3992.058204] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 3992.058207] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3992.058209] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 3992.058211] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 3992.058214] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 3992.058217] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3992.058219] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 3992.058228] Call Trace: [ 3992.058234] [ 3992.058240] ? show_regs.cold.9+0x22/0x2f [ 3992.058249] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3992.058256] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3992.058260] ? watchdog+0x40/0x40 [ 3992.058264] ? hrtimer_interrupt+0x128/0x2c0 [ 3992.058267] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3992.058275] ? apic_timer_interrupt+0xf/0x20 [ 3992.058279] [ 3992.058282] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3992.058288] do_raw_spin_lock+0x177/0x190 [ 3992.058292] _raw_spin_lock+0x21/0x30 [ 3992.058297] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3992.058588] ? do_raw_spin_unlock+0x75/0x190 [ 3992.058594] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3992.058781] ? do_wait_intr+0xf0/0xf0 [ 3992.058789] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3992.058992] mdc_close+0x53b/0xd70 [mdc] [ 3992.059031] lmv_close+0x18d/0x490 [lmv] [ 3992.059062] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3992.059221] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 3992.059327] ll_md_real_close+0xa5/0x2a0 [lustre] [ 3992.059407] ll_md_close+0x274/0x800 [lustre] [ 3992.059500] ll_file_release+0x128/0xc50 [lustre] [ 3992.059583] __fput+0x10d/0x440 [ 3992.059591] ____fput+0x12/0x20 [ 3992.059595] task_work_run+0xc6/0x110 [ 3992.059602] exit_to_usermode_loop+0x1dd/0x1f0 [ 3992.059610] do_syscall_64+0x3d6/0x440 [ 3992.059614] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3992.059620] RIP: 0033:0x7f151bee3cd8 [ 3992.059627] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 3992.059630] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 3992.059635] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 3992.059637] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 3992.059639] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 3992.059641] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 3992.059643] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3992.072571] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3992.072582] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3992.072584] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3992.072596] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3992.072600] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3992.072606] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 3992.072608] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 3992.072610] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 3992.072612] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 3992.072614] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3992.072617] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 3992.072619] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3992.072621] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3992.072630] Call Trace: [ 3992.072636] [ 3992.072641] ? show_regs.cold.9+0x22/0x2f [ 3992.072650] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3992.072658] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3992.072663] ? watchdog+0x40/0x40 [ 3992.072667] ? hrtimer_interrupt+0x128/0x2c0 [ 3992.072672] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3992.072678] ? apic_timer_interrupt+0xf/0x20 [ 3992.072683] [ 3992.072686] ? native_safe_halt+0x2e/0x50 [ 3992.072689] kvm_wait+0xcd/0xe0 [ 3992.072697] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3992.072705] do_raw_spin_lock+0x177/0x190 [ 3992.072710] _raw_spin_lock+0x21/0x30 [ 3992.072716] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3992.077237] Modules linked in: [ 3992.078967] obd_mod_free+0x46/0x140 [ptlrpc] [ 3992.095955] lustre(O) [ 3992.098739] mdc_commit_open+0x91/0xb0 [mdc] [ 3992.098787] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3992.101988] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3992.102071] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3992.102078] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3992.102086] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3992.104531] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3992.104547] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3992.104552] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3992.104556] RAX: 0000000000007cab RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 3992.104558] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 3992.104560] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 3992.104562] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 3992.104564] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 3992.104567] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 3992.104569] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3992.104571] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 3992.104580] Call Trace: [ 3992.104586] [ 3992.104591] ? show_regs.cold.9+0x22/0x2f [ 3992.104599] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3992.104605] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3992.104610] ? watchdog+0x40/0x40 [ 3992.104614] ? hrtimer_interrupt+0x128/0x2c0 [ 3992.104618] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3992.104626] ? apic_timer_interrupt+0xf/0x20 [ 3992.104631] [ 3992.104634] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3992.104639] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3992.104645] do_raw_spin_lock+0x177/0x190 [ 3992.104709] _raw_spin_lock+0x21/0x30 [ 3992.104714] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3992.110880] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3992.124010] process_one_work+0x2c8/0x700 [ 3992.131744] after_reply+0xe2c/0x1780 [ptlrpc] [ 3992.132030] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3992.132237] ? __next_timer_interrupt+0x160/0x160 [ 3992.132248] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3992.132463] ? do_wait_intr+0xf0/0xf0 [ 3992.132484] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3992.132689] kthread+0x1d1/0x200 [ 3992.132695] ? set_kthread_struct+0x70/0x70 [ 3992.132699] ret_from_fork+0x1f/0x30 [ 3992.503025] worker_thread+0x296/0x6e0 [ 3992.504176] ? rescuer_thread+0x570/0x570 [ 3992.505606] kthread+0x1d1/0x200 [ 3992.507003] ? set_kthread_struct+0x70/0x70 [ 3992.508858] ret_from_fork+0x1f/0x30 [ 4008.055412] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 4008.063045] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4008.096532] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4008.103004] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4008.116496] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4008.126818] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4008.137666] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4008.144175] RAX: 0000000000003b82 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4008.148276] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 4008.156554] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 4008.162615] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 4008.170602] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 4008.180875] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 4008.194479] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4008.211001] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 4008.240472] Call Trace: [ 4008.250536] [ 4008.253567] ? show_regs.cold.9+0x22/0x2f [ 4008.254957] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4008.266079] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4008.267599] ? watchdog+0x40/0x40 [ 4008.283437] ? hrtimer_interrupt+0x128/0x2c0 [ 4008.290620] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4008.296246] ? apic_timer_interrupt+0xf/0x20 [ 4008.308042] [ 4008.308883] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4008.310523] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4008.327011] do_raw_spin_lock+0x177/0x190 [ 4008.328508] _raw_spin_lock+0x21/0x30 [ 4008.344161] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4008.351423] ? do_raw_spin_unlock+0x75/0x190 [ 4008.362831] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4008.368835] ? do_wait_intr+0xf0/0xf0 [ 4008.370209] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4008.372221] mdc_reint+0x2d/0x110 [mdc] [ 4008.386720] mdc_unlink+0x658/0x870 [mdc] [ 4008.390609] lmv_unlink+0x34d/0xbd0 [lmv] [ 4008.398975] ll_unlink+0x24d/0x7f0 [lustre] [ 4008.405668] vfs_unlink+0x202/0x330 [ 4008.409712] do_unlinkat+0x2fa/0x460 [ 4008.412485] __x64_sys_unlink+0x37/0x50 [ 4008.413874] do_syscall_64+0xc1/0x440 [ 4008.415206] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4008.427289] RIP: 0033:0x7f151bdee10b [ 4008.428590] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4008.454132] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4008.464955] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 4008.475217] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 4008.484373] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4008.490970] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4008.501294] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4012.512061] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2081s! [ 4012.516765] Showing busy workqueues and worker pools: [ 4012.518862] workqueue events: flags=0x0 [ 4012.521364] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4012.528375] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 4012.536742] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4012.544088] pending: kfree_rcu_monitor [ 4012.545923] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4012.550489] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 4012.561040] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4012.566332] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4012.570772] workqueue events_long: flags=0x0 [ 4012.574792] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4012.579663] pending: lu_objects_flush [obdclass] [ 4012.583442] workqueue events_unbound: flags=0x2 [ 4012.586725] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4012.594701] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4012.607734] workqueue events_power_efficient: flags=0x80 [ 4012.615133] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4012.620850] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 4012.629670] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4012.635800] pending: neigh_periodic_work [ 4012.646941] workqueue mm_percpu_wq: flags=0x8 [ 4012.649514] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4012.653781] pending: vmstat_update, lru_add_drain_per_cpu [ 4012.657522] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4012.661042] pending: vmstat_update [ 4012.662706] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4012.665826] pending: lru_add_drain_per_cpu [ 4012.673513] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4012.679038] pending: lru_add_drain_per_cpu BAR(48) [ 4012.683483] workqueue writeback: flags=0x4a [ 4012.686017] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4012.689954] pending: wb_workfn [ 4012.692497] workqueue xprtiod: flags=0xa [ 4012.695067] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4012.698970] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4012.702971] workqueue ipv6_addrconf: flags=0x40008 [ 4012.705337] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4012.708563] pending: addrconf_verify_work [ 4012.711329] workqueue knbd0-recv: flags=0x1a [ 4012.713470] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4012.716204] in-flight: 658:recv_work [ 4012.718613] workqueue ptlrpc_pinger: flags=0x2 [ 4012.720910] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4012.724529] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4012.727925] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2082s workers=4 idle: 32984 37 8 [ 4012.732301] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3997s workers=2 idle: 661 [ 4020.056032] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4020.058750] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 4020.059306] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 4020.059314] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4020.059395] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4020.059401] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4020.059404] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4020.059419] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4020.059424] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4020.059428] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 4020.059430] RDX: 0000000000002c24 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 4020.059433] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 4020.059435] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 4020.059437] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4020.059440] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 4020.059443] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4020.059445] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4020.059454] Call Trace: [ 4020.059460] [ 4020.059466] ? show_regs.cold.9+0x22/0x2f [ 4020.059475] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4020.059483] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4020.059487] ? watchdog+0x40/0x40 [ 4020.059491] ? hrtimer_interrupt+0x128/0x2c0 [ 4020.059495] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4020.059504] ? apic_timer_interrupt+0xf/0x20 [ 4020.059507] [ 4020.059510] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4020.059515] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4020.059520] do_raw_spin_lock+0x177/0x190 [ 4020.059524] _raw_spin_lock+0x21/0x30 [ 4020.059529] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4020.059772] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4020.059832] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4020.059837] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4020.059842] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4020.060277] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4020.060285] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4020.060290] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4020.060294] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4020.060296] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 4020.060298] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 4020.060387] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 4020.060389] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 4020.060392] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 4020.060394] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4020.060396] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4020.060402] Call Trace: [ 4020.060405] [ 4020.060409] ? show_regs.cold.9+0x22/0x2f [ 4020.060418] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4020.060423] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4020.060427] ? watchdog+0x40/0x40 [ 4020.060431] ? hrtimer_interrupt+0x128/0x2c0 [ 4020.060435] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4020.060441] ? apic_timer_interrupt+0xf/0x20 [ 4020.060446] [ 4020.060449] ? native_safe_halt+0x2e/0x50 [ 4020.060453] kvm_wait+0xcd/0xe0 [ 4020.060462] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4020.060468] do_raw_spin_lock+0x177/0x190 [ 4020.060611] _raw_spin_lock+0x21/0x30 [ 4020.060616] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4020.060832] process_one_work+0x2c8/0x700 [ 4020.060839] worker_thread+0x296/0x6e0 [ 4020.060843] ? rescuer_thread+0x570/0x570 [ 4020.060847] kthread+0x1d1/0x200 [ 4020.060851] ? set_kthread_struct+0x70/0x70 [ 4020.060855] ret_from_fork+0x1f/0x30 [ 4020.066021] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4020.066209] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4020.066217] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4020.066221] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4020.066235] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4020.066239] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 4020.066243] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4020.066245] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 4020.066248] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4020.066250] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 4020.066252] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 4020.066256] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 4020.066259] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4020.066261] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 4020.066270] Call Trace: [ 4020.066274] [ 4020.066280] ? show_regs.cold.9+0x22/0x2f [ 4020.066290] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4020.066296] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4020.066315] ? watchdog+0x40/0x40 [ 4020.066394] ? hrtimer_interrupt+0x128/0x2c0 [ 4020.066400] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4020.066407] ? apic_timer_interrupt+0xf/0x20 [ 4020.066412] [ 4020.066415] ? native_safe_halt+0x2e/0x50 [ 4020.066420] kvm_wait+0xcd/0xe0 [ 4020.066430] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4020.066439] do_raw_spin_lock+0x177/0x190 [ 4020.066444] _raw_spin_lock+0x21/0x30 [ 4020.066449] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4020.066980] ? do_raw_spin_unlock+0x75/0x190 [ 4020.066988] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4020.067787] ? do_wait_intr+0xf0/0xf0 [ 4020.067796] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4020.070757] mdc_close+0x53b/0xd70 [mdc] [ 4020.070802] lmv_close+0x18d/0x490 [lmv] [ 4020.070831] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4020.080580] obd_mod_free+0x46/0x140 [ptlrpc] [ 4020.101220] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4020.101359] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4020.101440] ll_md_close+0x274/0x800 [lustre] [ 4020.101524] ll_file_release+0x128/0xc50 [lustre] [ 4020.101603] __fput+0x10d/0x440 [ 4020.101613] ____fput+0x12/0x20 [ 4020.101618] task_work_run+0xc6/0x110 [ 4020.101626] exit_to_usermode_loop+0x1dd/0x1f0 [ 4020.101634] do_syscall_64+0x3d6/0x440 [ 4020.101638] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4020.101647] RIP: 0033:0x7f151bee3cd8 [ 4020.101655] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4020.101660] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4020.101665] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 4020.101667] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 4020.101669] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4020.101671] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4020.101673] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4020.943316] mdc_commit_open+0x91/0xb0 [mdc] [ 4020.946985] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4020.950587] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4020.958752] after_reply+0xe2c/0x1780 [ptlrpc] [ 4020.966420] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4020.971905] ? __next_timer_interrupt+0x160/0x160 [ 4020.977614] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4020.983299] ? do_wait_intr+0xf0/0xf0 [ 4020.987039] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4020.991929] kthread+0x1d1/0x200 [ 4020.995947] ? set_kthread_struct+0x70/0x70 [ 4020.998619] ret_from_fork+0x1f/0x30 [ 4040.052429] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 4040.067357] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4040.114407] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4040.123099] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4040.130543] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4040.135046] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4040.153882] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4040.158502] RAX: 0000000000006148 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4040.166496] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 4040.175051] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 4040.184309] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 4040.191409] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 4040.197575] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 4040.202429] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4040.208584] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 4040.216503] Call Trace: [ 4040.220631] [ 4040.225707] ? show_regs.cold.9+0x22/0x2f [ 4040.232016] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4040.237908] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4040.243251] ? watchdog+0x40/0x40 [ 4040.252960] ? hrtimer_interrupt+0x128/0x2c0 [ 4040.254715] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4040.263699] ? apic_timer_interrupt+0xf/0x20 [ 4040.270985] [ 4040.274478] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4040.284231] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4040.291442] do_raw_spin_lock+0x177/0x190 [ 4040.294760] _raw_spin_lock+0x21/0x30 [ 4040.301047] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4040.306542] ? do_raw_spin_unlock+0x75/0x190 [ 4040.310556] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4040.320471] ? do_wait_intr+0xf0/0xf0 [ 4040.323556] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4040.326592] mdc_reint+0x2d/0x110 [mdc] [ 4040.330673] mdc_unlink+0x658/0x870 [mdc] [ 4040.333151] lmv_unlink+0x34d/0xbd0 [lmv] [ 4040.336867] ll_unlink+0x24d/0x7f0 [lustre] [ 4040.340781] vfs_unlink+0x202/0x330 [ 4040.346870] do_unlinkat+0x2fa/0x460 [ 4040.355327] __x64_sys_unlink+0x37/0x50 [ 4040.357965] do_syscall_64+0xc1/0x440 [ 4040.362395] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4040.373550] RIP: 0033:0x7f151bdee10b [ 4040.383551] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4040.406615] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4040.420901] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 4040.429644] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 4040.434529] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4040.442673] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4040.452907] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4043.232038] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2112s! [ 4043.235402] Showing busy workqueues and worker pools: [ 4043.237078] workqueue events: flags=0x0 [ 4043.238111] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4043.241203] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 4043.245956] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4043.249774] pending: kfree_rcu_monitor [ 4043.251802] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4043.255203] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 4043.260164] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4043.266108] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4043.271373] workqueue events_long: flags=0x0 [ 4043.281482] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4043.288311] pending: lu_objects_flush [obdclass] [ 4043.291390] workqueue events_unbound: flags=0x2 [ 4043.297500] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4043.305746] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4043.313342] workqueue events_power_efficient: flags=0x80 [ 4043.317262] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4043.325953] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 4043.334776] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4043.340307] pending: neigh_periodic_work [ 4043.346375] workqueue mm_percpu_wq: flags=0x8 [ 4043.352283] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4043.356937] pending: vmstat_update, lru_add_drain_per_cpu [ 4043.367667] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4043.376275] pending: vmstat_update [ 4043.380639] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4043.386198] pending: lru_add_drain_per_cpu [ 4043.388531] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4043.393622] pending: lru_add_drain_per_cpu BAR(48) [ 4043.401206] workqueue writeback: flags=0x4a [ 4043.405529] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4043.414757] pending: wb_workfn [ 4043.418443] workqueue xprtiod: flags=0xa [ 4043.425714] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4043.435528] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4043.443356] workqueue ipv6_addrconf: flags=0x40008 [ 4043.445751] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4043.450812] pending: addrconf_verify_work [ 4043.452853] workqueue knbd0-recv: flags=0x1a [ 4043.456335] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4043.466983] in-flight: 658:recv_work [ 4043.470714] workqueue ptlrpc_pinger: flags=0x2 [ 4043.473957] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4043.478935] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4043.489297] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2112s workers=4 idle: 32984 37 8 [ 4043.500639] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4028s workers=2 idle: 661 [ 4048.055015] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 4048.056573] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 4048.060466] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 4048.060480] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4048.060563] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4048.060568] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4048.060571] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4048.060585] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4048.060589] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4048.060593] RAX: 0000000000007f87 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4048.060596] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 4048.060599] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4048.060601] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 4048.060603] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 4048.060606] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 4048.060609] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4048.060611] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 4048.060620] Call Trace: [ 4048.060625] [ 4048.060630] ? show_regs.cold.9+0x22/0x2f [ 4048.060639] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4048.060645] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4048.060649] ? watchdog+0x40/0x40 [ 4048.060653] ? hrtimer_interrupt+0x128/0x2c0 [ 4048.060657] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4048.060664] ? apic_timer_interrupt+0xf/0x20 [ 4048.060669] [ 4048.060672] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4048.060679] do_raw_spin_lock+0x177/0x190 [ 4048.060684] _raw_spin_lock+0x21/0x30 [ 4048.060688] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4048.060972] ? do_raw_spin_unlock+0x75/0x190 [ 4048.060978] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4048.064060] ? do_wait_intr+0xf0/0xf0 [ 4048.064071] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4048.065504] mdc_close+0x53b/0xd70 [mdc] [ 4048.065553] lmv_close+0x18d/0x490 [lmv] [ 4048.065580] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4048.066730] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4048.066803] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4048.066809] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4048.066811] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4048.066824] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4048.066828] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4048.066833] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 4048.066841] RDX: 00000000000052d8 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 4048.066844] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 4048.066846] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 4048.066849] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4048.066860] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 4048.067546] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4048.067704] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4048.067795] ll_md_close+0x274/0x800 [lustre] [ 4048.067883] ll_file_release+0x128/0xc50 [lustre] [ 4048.067996] __fput+0x10d/0x440 [ 4048.068006] ____fput+0x12/0x20 [ 4048.068012] task_work_run+0xc6/0x110 [ 4048.068020] exit_to_usermode_loop+0x1dd/0x1f0 [ 4048.068028] do_syscall_64+0x3d6/0x440 [ 4048.070184] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4048.070207] RIP: 0033:0x7f151bee3cd8 [ 4048.070220] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4048.070225] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4048.070231] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 4048.070233] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 4048.070235] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4048.070236] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4048.070238] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4048.079661] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) [ 4048.090607] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4048.090628] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4048.090638] Call Trace: [ 4048.090646] [ 4048.090652] ? show_regs.cold.9+0x22/0x2f [ 4048.090666] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4048.090673] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4048.090678] ? watchdog+0x40/0x40 [ 4048.090682] ? hrtimer_interrupt+0x128/0x2c0 [ 4048.090687] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4048.090695] ? apic_timer_interrupt+0xf/0x20 [ 4048.090700] [ 4048.090702] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4048.090709] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4048.090714] do_raw_spin_lock+0x177/0x190 [ 4048.090718] _raw_spin_lock+0x21/0x30 [ 4048.090726] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4048.133874] mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4048.133964] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4048.133974] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4048.133986] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4048.142696] obd_mod_free+0x46/0x140 [ptlrpc] [ 4048.151564] [ 4048.154389] mdc_commit_open+0x91/0xb0 [mdc] [ 4048.154432] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4048.154728] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4048.154915] after_reply+0xe2c/0x1780 [ptlrpc] [ 4048.155120] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4048.155422] ? __next_timer_interrupt+0x160/0x160 [ 4048.155433] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4048.155689] ? do_wait_intr+0xf0/0xf0 [ 4048.155698] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4048.155900] kthread+0x1d1/0x200 [ 4048.155906] ? set_kthread_struct+0x70/0x70 [ 4048.155911] ret_from_fork+0x1f/0x30 [ 4048.673187] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4048.676718] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4048.700977] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4048.704334] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4048.708556] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 4048.712066] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 4048.714767] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 4048.722219] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 4048.724039] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 4048.731160] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4048.732565] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4048.735737] Call Trace: [ 4048.736426] [ 4048.737055] ? show_regs.cold.9+0x22/0x2f [ 4048.738383] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4048.740344] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4048.741647] ? watchdog+0x40/0x40 [ 4048.742868] ? hrtimer_interrupt+0x128/0x2c0 [ 4048.744377] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4048.746020] ? apic_timer_interrupt+0xf/0x20 [ 4048.747419] [ 4048.748277] ? native_safe_halt+0x2e/0x50 [ 4048.750886] kvm_wait+0xcd/0xe0 [ 4048.752717] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4048.755734] do_raw_spin_lock+0x177/0x190 [ 4048.757689] _raw_spin_lock+0x21/0x30 [ 4048.759374] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4048.761442] process_one_work+0x2c8/0x700 [ 4048.763236] worker_thread+0x296/0x6e0 [ 4048.764985] ? rescuer_thread+0x570/0x570 [ 4048.766742] kthread+0x1d1/0x200 [ 4048.768203] ? set_kthread_struct+0x70/0x70 [ 4048.770882] ret_from_fork+0x1f/0x30 [ 4068.052020] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 4068.058324] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4068.091074] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4068.101749] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4068.108149] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4068.112318] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4068.123535] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4068.125524] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4068.136720] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 4068.142751] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 4068.147197] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 4068.152698] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 4068.156827] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 4068.164717] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4068.169124] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 4068.173366] Call Trace: [ 4068.174715] [ 4068.178096] ? show_regs.cold.9+0x22/0x2f [ 4068.182389] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4068.186169] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4068.196284] ? watchdog+0x40/0x40 [ 4068.202762] ? hrtimer_interrupt+0x128/0x2c0 [ 4068.204319] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4068.212090] ? apic_timer_interrupt+0xf/0x20 [ 4068.218188] [ 4068.221200] ? native_safe_halt+0x2e/0x50 [ 4068.227105] kvm_wait+0xcd/0xe0 [ 4068.229133] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4068.234789] do_raw_spin_lock+0x177/0x190 [ 4068.237417] _raw_spin_lock+0x21/0x30 [ 4068.241321] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4068.244271] ? do_raw_spin_unlock+0x75/0x190 [ 4068.249606] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4068.255945] ? do_wait_intr+0xf0/0xf0 [ 4068.257397] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4068.264110] mdc_reint+0x2d/0x110 [mdc] [ 4068.269056] mdc_unlink+0x658/0x870 [mdc] [ 4068.270577] lmv_unlink+0x34d/0xbd0 [lmv] [ 4068.274232] ll_unlink+0x24d/0x7f0 [lustre] [ 4068.275696] vfs_unlink+0x202/0x330 [ 4068.278067] do_unlinkat+0x2fa/0x460 [ 4068.283542] __x64_sys_unlink+0x37/0x50 [ 4068.288112] do_syscall_64+0xc1/0x440 [ 4068.289465] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4068.294565] RIP: 0033:0x7f151bdee10b [ 4068.296575] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4068.310083] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4068.319820] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 4068.324220] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 4068.327780] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4068.331518] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4068.335244] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4073.954797] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2143s! [ 4073.962460] Showing busy workqueues and worker pools: [ 4073.971181] workqueue events: flags=0x0 [ 4073.979804] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4073.984513] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 4073.993885] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4074.004841] pending: kfree_rcu_monitor [ 4074.006413] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4074.010558] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 4074.023078] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4074.028139] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4074.036284] workqueue events_long: flags=0x0 [ 4074.044982] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4074.050064] pending: lu_objects_flush [obdclass] [ 4074.059045] workqueue events_unbound: flags=0x2 [ 4074.062014] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4074.067262] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4074.075247] workqueue events_power_efficient: flags=0x80 [ 4074.080393] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4074.086045] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 4074.092568] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4074.097573] pending: neigh_periodic_work [ 4074.100345] workqueue mm_percpu_wq: flags=0x8 [ 4074.103147] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4074.107003] pending: vmstat_update, lru_add_drain_per_cpu [ 4074.110955] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4074.114944] pending: vmstat_update [ 4074.117860] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4074.123464] pending: lru_add_drain_per_cpu [ 4074.126972] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4074.131071] pending: lru_add_drain_per_cpu BAR(48) [ 4074.134180] workqueue writeback: flags=0x4a [ 4074.138458] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4074.142818] pending: wb_workfn [ 4074.144422] workqueue xprtiod: flags=0xa [ 4074.146453] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4074.150987] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4074.154797] workqueue ipv6_addrconf: flags=0x40008 [ 4074.157052] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4074.160824] pending: addrconf_verify_work [ 4074.163178] workqueue knbd0-recv: flags=0x1a [ 4074.166081] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4074.167844] in-flight: 658:recv_work [ 4074.169379] workqueue ptlrpc_pinger: flags=0x2 [ 4074.171222] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4074.174327] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4074.178240] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2143s workers=4 idle: 32984 37 8 [ 4074.182100] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4058s workers=2 idle: 661 [ 4076.055012] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 4076.058045] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4076.058264] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 4076.058857] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 4076.058864] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4076.058930] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4076.058935] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4076.058939] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4076.058952] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4076.058956] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 4076.058960] RAX: 000000000000360d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4076.058963] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 4076.058965] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4076.058967] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 4076.058969] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 4076.058973] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 4076.058976] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4076.058978] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 4076.058986] Call Trace: [ 4076.058991] [ 4076.058995] ? show_regs.cold.9+0x22/0x2f [ 4076.059004] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4076.059009] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4076.059013] ? watchdog+0x40/0x40 [ 4076.059017] ? hrtimer_interrupt+0x128/0x2c0 [ 4076.059021] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4076.059029] ? apic_timer_interrupt+0xf/0x20 [ 4076.059033] [ 4076.059036] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4076.059053] do_raw_spin_lock+0x177/0x190 [ 4076.059057] _raw_spin_lock+0x21/0x30 [ 4076.059062] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4076.059380] ? do_raw_spin_unlock+0x75/0x190 [ 4076.059386] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4076.060150] ? do_wait_intr+0xf0/0xf0 [ 4076.060159] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4076.060379] mdc_close+0x53b/0xd70 [mdc] [ 4076.060420] lmv_close+0x18d/0x490 [lmv] [ 4076.060451] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4076.060867] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4076.060975] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4076.061055] ll_md_close+0x274/0x800 [lustre] [ 4076.061141] ll_file_release+0x128/0xc50 [lustre] [ 4076.061223] __fput+0x10d/0x440 [ 4076.061233] ____fput+0x12/0x20 [ 4076.061238] task_work_run+0xc6/0x110 [ 4076.061247] exit_to_usermode_loop+0x1dd/0x1f0 [ 4076.061255] do_syscall_64+0x3d6/0x440 [ 4076.061259] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4076.061267] RIP: 0033:0x7f151bee3cd8 [ 4076.061331] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4076.061336] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4076.061340] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 4076.061342] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 4076.061344] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4076.061346] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4076.061348] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4076.078093] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4076.078107] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4076.078110] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4076.078122] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4076.078126] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4076.078132] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 4076.078135] RDX: 0000000000001538 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 4076.078137] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 4076.078139] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 4076.078142] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4076.078145] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 4076.078148] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4076.078150] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4076.078158] Call Trace: [ 4076.078163] [ 4076.078170] ? show_regs.cold.9+0x22/0x2f [ 4076.078180] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4076.078187] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4076.078191] ? watchdog+0x40/0x40 [ 4076.078195] ? hrtimer_interrupt+0x128/0x2c0 [ 4076.078198] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4076.078205] ? apic_timer_interrupt+0xf/0x20 [ 4076.078210] [ 4076.078213] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4076.078218] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4076.078223] do_raw_spin_lock+0x177/0x190 [ 4076.078228] _raw_spin_lock+0x21/0x30 [ 4076.078236] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4076.078578] obd_mod_free+0x46/0x140 [ptlrpc] [ 4076.081978] Modules linked in: lustre(O) [ 4076.084700] mdc_commit_open+0x91/0xb0 [mdc] [ 4076.119418] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4076.119493] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4076.119500] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4076.119506] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4076.119860] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4076.119878] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4076.119882] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4076.119886] RAX: 0000000000002e90 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4076.119888] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 4076.119890] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 4076.119892] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 4076.119894] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 4076.119897] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 4076.119900] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4076.119902] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4076.119910] Call Trace: [ 4076.119915] [ 4076.119920] ? show_regs.cold.9+0x22/0x2f [ 4076.119929] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4076.119936] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4076.119940] ? watchdog+0x40/0x40 [ 4076.119945] ? hrtimer_interrupt+0x128/0x2c0 [ 4076.119949] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4076.119956] ? apic_timer_interrupt+0xf/0x20 [ 4076.119961] [ 4076.119964] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4076.119969] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4076.119974] do_raw_spin_lock+0x177/0x190 [ 4076.119979] _raw_spin_lock+0x21/0x30 [ 4076.119983] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4076.130421] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4076.135993] process_one_work+0x2c8/0x700 [ 4076.136015] worker_thread+0x296/0x6e0 [ 4076.136021] ? rescuer_thread+0x570/0x570 [ 4076.136025] kthread+0x1d1/0x200 [ 4076.136030] ? set_kthread_struct+0x70/0x70 [ 4076.136034] ret_from_fork+0x1f/0x30 [ 4076.629995] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4076.635452] after_reply+0xe2c/0x1780 [ptlrpc] [ 4076.641315] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4076.646893] ? __next_timer_interrupt+0x160/0x160 [ 4076.651623] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4076.653211] ? do_wait_intr+0xf0/0xf0 [ 4076.657394] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4076.661574] kthread+0x1d1/0x200 [ 4076.662713] ? set_kthread_struct+0x70/0x70 [ 4076.667275] ret_from_fork+0x1f/0x30 [ 4100.056260] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 4100.062361] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4100.110035] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4100.122044] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4100.128013] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4100.136757] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4100.157204] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4100.166808] RAX: 0000000000001f8c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4100.173819] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 4100.181803] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 4100.186122] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 4100.193784] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 4100.202109] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 4100.211737] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4100.219861] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 4100.228465] Call Trace: [ 4100.229750] [ 4100.233277] ? show_regs.cold.9+0x22/0x2f [ 4100.237493] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4100.244122] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4100.246278] ? watchdog+0x40/0x40 [ 4100.252109] ? hrtimer_interrupt+0x128/0x2c0 [ 4100.256880] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4100.262086] ? apic_timer_interrupt+0xf/0x20 [ 4100.268824] [ 4100.270132] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4100.277843] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4100.283195] do_raw_spin_lock+0x177/0x190 [ 4100.289523] _raw_spin_lock+0x21/0x30 [ 4100.295185] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4100.302523] ? do_raw_spin_unlock+0x75/0x190 [ 4100.309118] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4100.318947] ? do_wait_intr+0xf0/0xf0 [ 4100.325588] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4100.333900] mdc_reint+0x2d/0x110 [mdc] [ 4100.339687] mdc_unlink+0x658/0x870 [mdc] [ 4100.345093] lmv_unlink+0x34d/0xbd0 [lmv] [ 4100.348108] ll_unlink+0x24d/0x7f0 [lustre] [ 4100.357270] vfs_unlink+0x202/0x330 [ 4100.363095] do_unlinkat+0x2fa/0x460 [ 4100.371129] __x64_sys_unlink+0x37/0x50 [ 4100.377584] do_syscall_64+0xc1/0x440 [ 4100.388975] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4100.392739] RIP: 0033:0x7f151bdee10b [ 4100.398816] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4100.411400] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4100.420640] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 4100.430959] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 4100.433028] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4100.439622] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4100.452153] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4104.055306] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 4104.056788] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 4104.060613] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 4104.060627] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4104.060693] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4104.060698] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4104.060700] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4104.060715] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4104.060719] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4104.060723] RAX: 0000000000002499 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4104.060726] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 4104.060728] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4104.060729] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 4104.060732] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 4104.060735] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 4104.060738] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4104.060740] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 4104.060749] Call Trace: [ 4104.060755] [ 4104.060760] ? show_regs.cold.9+0x22/0x2f [ 4104.060769] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4104.060776] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4104.060780] ? watchdog+0x40/0x40 [ 4104.060784] ? hrtimer_interrupt+0x128/0x2c0 [ 4104.060789] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4104.060796] ? apic_timer_interrupt+0xf/0x20 [ 4104.060801] [ 4104.060804] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4104.060811] do_raw_spin_lock+0x177/0x190 [ 4104.060816] _raw_spin_lock+0x21/0x30 [ 4104.060821] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4104.062230] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) [ 4104.062259] ? do_raw_spin_unlock+0x75/0x190 [ 4104.062266] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4104.062473] ? do_wait_intr+0xf0/0xf0 [ 4104.062481] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4104.062703] obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4104.062756] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4104.062762] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4104.062764] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4104.062774] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4104.062778] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4104.062783] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 4104.062785] RDX: 0000000000002c0a RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 4104.062788] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 4104.062790] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 4104.062792] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4104.062795] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 4104.062797] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4104.062800] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4104.062807] Call Trace: [ 4104.062811] [ 4104.062814] ? show_regs.cold.9+0x22/0x2f [ 4104.062822] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4104.062828] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4104.062832] ? watchdog+0x40/0x40 [ 4104.062836] ? hrtimer_interrupt+0x128/0x2c0 [ 4104.062840] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4104.062847] ? apic_timer_interrupt+0xf/0x20 [ 4104.062851] [ 4104.062854] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4104.062859] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4104.062864] do_raw_spin_lock+0x177/0x190 [ 4104.062868] _raw_spin_lock+0x21/0x30 [ 4104.062873] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4104.063099] obd_mod_free+0x46/0x140 [ptlrpc] [ 4104.069768] mdc_commit_open+0x91/0xb0 [mdc] [ 4104.069810] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4104.070055] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4104.070360] after_reply+0xe2c/0x1780 [ptlrpc] [ 4104.070869] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) [ 4104.070777] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4104.070884] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4104.071691] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4104.071698] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4104.071703] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4104.072128] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4104.072139] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4104.072143] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4104.072148] RAX: 0000000000003110 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4104.072151] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 4104.072153] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 4104.072155] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 4104.072158] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 4104.072161] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 4104.072164] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4104.072167] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4104.072176] Call Trace: [ 4104.072181] [ 4104.072184] ? show_regs.cold.9+0x22/0x2f [ 4104.072194] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4104.072200] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4104.072205] ? watchdog+0x40/0x40 [ 4104.072209] ? hrtimer_interrupt+0x128/0x2c0 [ 4104.072213] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4104.072222] ? apic_timer_interrupt+0xf/0x20 [ 4104.072227] [ 4104.072230] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4104.072235] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4104.072240] do_raw_spin_lock+0x177/0x190 [ 4104.072245] _raw_spin_lock+0x21/0x30 [ 4104.072396] ? __next_timer_interrupt+0x160/0x160 [ 4104.072250] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4104.073209] process_one_work+0x2c8/0x700 [ 4104.073217] worker_thread+0x296/0x6e0 [ 4104.073221] ? rescuer_thread+0x570/0x570 [ 4104.073224] kthread+0x1d1/0x200 [ 4104.073228] ? set_kthread_struct+0x70/0x70 [ 4104.073232] ret_from_fork+0x1f/0x30 [ 4104.072406] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4104.073477] ? do_wait_intr+0xf0/0xf0 [ 4104.073486] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4104.080219] mdc_close+0x53b/0xd70 [mdc] [ 4104.083209] lmv_close+0x18d/0x490 [lmv] [ 4104.083243] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4104.130195] kthread+0x1d1/0x200 [ 4104.130214] ? set_kthread_struct+0x70/0x70 [ 4104.130219] ret_from_fork+0x1f/0x30 [ 4104.676659] BUG: workqueue lockup - pool [ 4104.678082] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4104.678204] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4104.678391] ll_md_close+0x274/0x800 [lustre] [ 4104.678595] ll_file_release+0x128/0xc50 [lustre] [ 4104.678683] __fput+0x10d/0x440 [ 4104.678692] ____fput+0x12/0x20 [ 4104.678697] task_work_run+0xc6/0x110 [ 4104.678706] exit_to_usermode_loop+0x1dd/0x1f0 [ 4104.678713] do_syscall_64+0x3d6/0x440 [ 4104.678717] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4104.678725] RIP: 0033:0x7f151bee3cd8 [ 4104.678733] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4104.678737] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4104.678742] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 4104.678744] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 4104.678746] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4104.678748] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4104.678750] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4104.787799] cpus=0-3 flags=0x4 nice=0 stuck for 2174s! [ 4104.790294] Showing busy workqueues and worker pools: [ 4104.792890] workqueue events: flags=0x0 [ 4104.795571] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4104.799034] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 4104.803276] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4104.806229] pending: kfree_rcu_monitor [ 4104.808505] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4104.812099] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 4104.817830] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4104.822553] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4104.826109] workqueue events_long: flags=0x0 [ 4104.827845] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4104.831488] pending: lu_objects_flush [obdclass] [ 4104.834407] workqueue events_unbound: flags=0x2 [ 4104.837189] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4104.840007] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4104.845273] workqueue events_power_efficient: flags=0x80 [ 4104.847722] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4104.851416] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 4104.855001] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4104.858090] pending: neigh_periodic_work [ 4104.860343] workqueue mm_percpu_wq: flags=0x8 [ 4104.862355] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4104.865910] pending: vmstat_update, lru_add_drain_per_cpu [ 4104.869280] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4104.875265] pending: vmstat_update [ 4104.877165] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4104.880703] pending: lru_add_drain_per_cpu [ 4104.883266] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4104.886558] pending: lru_add_drain_per_cpu BAR(48) [ 4104.888786] workqueue writeback: flags=0x4a [ 4104.890872] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4104.894148] pending: wb_workfn [ 4104.896371] workqueue xprtiod: flags=0xa [ 4104.898309] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4104.901319] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4104.904696] workqueue ipv6_addrconf: flags=0x40008 [ 4104.906489] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4104.910302] pending: addrconf_verify_work [ 4104.912443] workqueue knbd0-recv: flags=0x1a [ 4104.914653] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4104.917783] in-flight: 658:recv_work [ 4104.920451] workqueue ptlrpc_pinger: flags=0x2 [ 4104.923236] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4104.926366] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4104.929359] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2174s workers=4 idle: 32984 37 8 [ 4104.932704] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4089s workers=2 idle: 661 [ 4132.052158] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 4132.054979] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4132.055950] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 4132.059222] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 4132.059235] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4132.059305] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4132.059310] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4132.059313] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4132.059326] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4132.059331] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4132.059335] RAX: 0000000000007e94 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4132.059337] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 4132.059339] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4132.059341] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 4132.059343] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 4132.059346] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 4132.059349] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4132.059608] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 4132.059617] Call Trace: [ 4132.059622] [ 4132.059628] ? show_regs.cold.9+0x22/0x2f [ 4132.059638] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4132.059644] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4132.059648] ? watchdog+0x40/0x40 [ 4132.059653] ? hrtimer_interrupt+0x128/0x2c0 [ 4132.059656] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4132.059665] ? apic_timer_interrupt+0xf/0x20 [ 4132.059669] [ 4132.059672] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4132.059678] do_raw_spin_lock+0x177/0x190 [ 4132.059683] _raw_spin_lock+0x21/0x30 [ 4132.059725] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 4132.059731] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4132.059789] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4132.059793] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4132.059798] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4132.060415] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4132.060424] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4132.060428] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4132.060432] RAX: 0000000000007f0a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4132.060434] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 4132.060436] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 4132.060438] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 4132.060440] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 4132.060443] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 4132.060446] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4132.060448] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4132.060456] Call Trace: [ 4132.060460] [ 4132.060463] ? show_regs.cold.9+0x22/0x2f [ 4132.060471] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4132.060477] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4132.060481] ? watchdog+0x40/0x40 [ 4132.060486] ? hrtimer_interrupt+0x128/0x2c0 [ 4132.060491] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4132.059688] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4132.060500] ? apic_timer_interrupt+0xf/0x20 [ 4132.060507] [ 4132.060511] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4132.060515] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4132.060606] ? do_raw_spin_unlock+0x75/0x190 [ 4132.060612] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4132.060885] do_raw_spin_lock+0x177/0x190 [ 4132.060892] _raw_spin_lock+0x21/0x30 [ 4132.061026] ? do_wait_intr+0xf0/0xf0 [ 4132.060897] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4132.061490] process_one_work+0x2c8/0x700 [ 4132.061498] worker_thread+0x296/0x6e0 [ 4132.061502] ? rescuer_thread+0x570/0x570 [ 4132.061505] kthread+0x1d1/0x200 [ 4132.061509] ? set_kthread_struct+0x70/0x70 [ 4132.061513] ret_from_fork+0x1f/0x30 [ 4132.061034] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4132.061772] mdc_close+0x53b/0xd70 [mdc] [ 4132.061811] lmv_close+0x18d/0x490 [lmv] [ 4132.061837] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4132.061999] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4132.062122] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4132.062215] ll_md_close+0x274/0x800 [lustre] [ 4132.062297] ll_file_release+0x128/0xc50 [lustre] [ 4132.062648] __fput+0x10d/0x440 [ 4132.062658] ____fput+0x12/0x20 [ 4132.062662] task_work_run+0xc6/0x110 [ 4132.062670] exit_to_usermode_loop+0x1dd/0x1f0 [ 4132.062676] do_syscall_64+0x3d6/0x440 [ 4132.062681] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4132.062687] RIP: 0033:0x7f151bee3cd8 [ 4132.062695] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4132.062699] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4132.062705] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 4132.062707] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 4132.062710] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4132.062712] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4132.062715] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4132.081059] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4132.081072] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4132.081075] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4132.081090] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4132.081094] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4132.081099] RAX: 0000000000007f8b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4132.081102] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 4132.081104] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 4132.081107] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 4132.081118] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 4132.081121] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 4132.081124] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4132.081127] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 4132.081135] Call Trace: [ 4132.081140] [ 4132.081146] ? show_regs.cold.9+0x22/0x2f [ 4132.081155] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4132.081162] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4132.081166] ? watchdog+0x40/0x40 [ 4132.081170] ? hrtimer_interrupt+0x128/0x2c0 [ 4132.081175] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4132.081185] ? apic_timer_interrupt+0xf/0x20 [ 4132.081189] [ 4132.081192] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4132.081197] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4132.081202] do_raw_spin_lock+0x177/0x190 [ 4132.081207] _raw_spin_lock+0x21/0x30 [ 4132.081214] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4132.081681] ? do_raw_spin_unlock+0x75/0x190 [ 4132.081688] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4132.081882] ? do_wait_intr+0xf0/0xf0 [ 4132.081889] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4132.082194] mdc_reint+0x2d/0x110 [mdc] [ 4132.082232] mdc_unlink+0x658/0x870 [mdc] [ 4132.082257] lmv_unlink+0x34d/0xbd0 [lmv] [ 4132.082286] ll_unlink+0x24d/0x7f0 [lustre] [ 4132.082416] vfs_unlink+0x202/0x330 [ 4132.082425] do_unlinkat+0x2fa/0x460 [ 4132.082430] __x64_sys_unlink+0x37/0x50 [ 4132.082434] do_syscall_64+0xc1/0x440 [ 4132.082440] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4132.082448] RIP: 0033:0x7f151bdee10b [ 4132.082455] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4132.082459] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4132.082463] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 4132.082465] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 4132.082467] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4132.082469] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4132.082471] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4132.973436] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4133.036085] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4133.044954] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4133.058834] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4133.065218] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4133.085463] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4133.100195] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 4133.106312] RDX: 0000000000002be3 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 4133.116798] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 4133.124025] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 4133.131817] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4133.133853] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 4133.141445] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4133.145255] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4133.148294] Call Trace: [ 4133.148980] [ 4133.153827] ? show_regs.cold.9+0x22/0x2f [ 4133.158768] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4133.163217] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4133.168485] ? watchdog+0x40/0x40 [ 4133.173167] ? hrtimer_interrupt+0x128/0x2c0 [ 4133.178703] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4133.184274] ? apic_timer_interrupt+0xf/0x20 [ 4133.189088] [ 4133.190967] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4133.195528] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4133.198688] do_raw_spin_lock+0x177/0x190 [ 4133.211196] _raw_spin_lock+0x21/0x30 [ 4133.218146] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4133.223698] obd_mod_free+0x46/0x140 [ptlrpc] [ 4133.226726] mdc_commit_open+0x91/0xb0 [mdc] [ 4133.231290] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4133.238648] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4133.245150] after_reply+0xe2c/0x1780 [ptlrpc] [ 4133.249866] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4133.255318] ? __next_timer_interrupt+0x160/0x160 [ 4133.260216] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4133.265733] ? do_wait_intr+0xf0/0xf0 [ 4133.271160] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4133.277605] kthread+0x1d1/0x200 [ 4133.283466] ? set_kthread_struct+0x70/0x70 [ 4133.289514] ret_from_fork+0x1f/0x30 [ 4135.398522] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2204s! [ 4135.406126] Showing busy workqueues and worker pools: [ 4135.411024] workqueue events: flags=0x0 [ 4135.417774] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4135.424598] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 4135.441474] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4135.445236] pending: kfree_rcu_monitor [ 4135.453973] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4135.461381] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 4135.478172] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4135.491244] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4135.504208] workqueue events_long: flags=0x0 [ 4135.509714] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4135.517960] pending: lu_objects_flush [obdclass] [ 4135.526711] workqueue events_unbound: flags=0x2 [ 4135.531352] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4135.538178] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4135.555881] workqueue events_power_efficient: flags=0x80 [ 4135.563385] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4135.568460] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 4135.573935] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4135.577604] pending: neigh_periodic_work [ 4135.581857] workqueue mm_percpu_wq: flags=0x8 [ 4135.587303] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4135.592467] pending: vmstat_update, lru_add_drain_per_cpu [ 4135.599096] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4135.605725] pending: vmstat_update [ 4135.607193] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4135.618048] pending: lru_add_drain_per_cpu [ 4135.624338] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4135.638392] pending: lru_add_drain_per_cpu BAR(48) [ 4135.640329] workqueue writeback: flags=0x4a [ 4135.654321] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4135.659840] pending: wb_workfn [ 4135.664475] workqueue xprtiod: flags=0xa [ 4135.670954] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4135.679297] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4135.698848] workqueue ipv6_addrconf: flags=0x40008 [ 4135.707292] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4135.719722] pending: addrconf_verify_work [ 4135.721423] workqueue knbd0-recv: flags=0x1a [ 4135.722887] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4135.728856] in-flight: 658:recv_work [ 4135.753069] workqueue ptlrpc_pinger: flags=0x2 [ 4135.762674] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4135.764882] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4135.778958] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2205s workers=4 idle: 32984 37 8 [ 4135.793557] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4120s workers=2 idle: 661 [ 4144.688064] rcu: INFO: rcu_sched self-detected stall on CPU [ 4144.690430] rcu: 1-...!: (1314525 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 4144.695057] (t=2220041 jiffies g=249657 q=3072) [ 4144.696758] rcu: rcu_sched kthread starved for 2214993 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 4144.700950] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4144.704737] rcu: RCU grace-period kthread stack dump: [ 4144.706970] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4144.710469] Call Trace: [ 4144.711314] __schedule+0x351/0xcb0 [ 4144.712652] schedule+0xc0/0x180 [ 4144.714102] schedule_timeout+0xb4/0x190 [ 4144.715746] ? do_raw_spin_unlock+0x75/0x190 [ 4144.717497] ? __next_timer_interrupt+0x160/0x160 [ 4144.719185] ? prepare_to_swait_event+0xf3/0x140 [ 4144.720586] rcu_gp_kthread+0x8dd/0x15b0 [ 4144.721660] ? force_qs_rnp+0x2c0/0x2c0 [ 4144.722975] kthread+0x1d1/0x200 [ 4144.724238] ? set_kthread_struct+0x70/0x70 [ 4144.725319] ret_from_fork+0x1f/0x30 [ 4144.726361] rcu: Stack dump where RCU GP kthread last ran: [ 4144.728157] Sending NMI from CPU 1 to CPUs 3: [ 4144.729519] NMI backtrace for cpu 3 skipped: idling at native_safe_halt+0x2e/0x50 [ 4144.730474] NMI backtrace for cpu 1 [ 4144.733387] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4144.737467] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4144.740687] Call Trace: [ 4144.741722] [ 4144.742580] ? dump_stack+0xbb/0x10e [ 4144.744368] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4144.746711] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4144.748513] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4144.749839] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4144.751714] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4144.753579] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 4144.755480] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 4144.757248] ? update_process_times+0x6d/0xc0 [ 4144.758752] ? tick_sched_handle.isra.18+0x27/0x90 [ 4144.760074] ? tick_sched_timer+0x53/0xb0 [ 4144.761202] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4144.762363] ? tick_sched_handle.isra.18+0x90/0x90 [ 4144.763634] ? hrtimer_interrupt+0x128/0x2c0 [ 4144.764772] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4144.766363] ? apic_timer_interrupt+0xf/0x20 [ 4144.767647] [ 4144.768278] ? native_safe_halt+0x2e/0x50 [ 4144.769512] ? kvm_wait+0xcd/0xe0 [ 4144.770479] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4144.772177] ? do_raw_spin_lock+0x177/0x190 [ 4144.773449] ? _raw_spin_lock+0x21/0x30 [ 4144.774511] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4144.776167] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 4144.777613] ? mdc_commit_open+0x91/0xb0 [mdc] [ 4144.779164] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4144.780631] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4144.782483] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 4144.784106] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4144.785838] ? __next_timer_interrupt+0x160/0x160 [ 4144.787707] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4144.789357] ? do_wait_intr+0xf0/0xf0 [ 4144.790581] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4144.792297] ? kthread+0x1d1/0x200 [ 4144.793282] ? set_kthread_struct+0x70/0x70 [ 4144.794379] ? ret_from_fork+0x1f/0x30 [ 4160.057910] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 4160.058777] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4160.075552] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4160.077433] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4160.077441] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4160.077443] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4160.077458] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4160.077462] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4160.077467] RAX: 0000000000004f15 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4160.077469] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 4160.077472] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 4160.077474] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 4160.077476] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 4160.077480] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 4160.077482] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4160.077485] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 4160.077494] Call Trace: [ 4160.077499] [ 4160.077505] ? show_regs.cold.9+0x22/0x2f [ 4160.077514] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4160.077521] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4160.077526] ? watchdog+0x40/0x40 [ 4160.077530] ? hrtimer_interrupt+0x128/0x2c0 [ 4160.077534] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4160.077542] ? apic_timer_interrupt+0xf/0x20 [ 4160.077547] [ 4160.077549] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4160.077553] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4160.077558] do_raw_spin_lock+0x177/0x190 [ 4160.077562] _raw_spin_lock+0x21/0x30 [ 4160.077568] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4160.078123] ? do_raw_spin_unlock+0x75/0x190 [ 4160.078131] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4160.084144] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4160.084221] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4160.084227] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4160.084234] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4160.084499] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4160.084509] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4160.084514] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4160.084518] RAX: 0000000000005e8d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4160.084520] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 4160.084522] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 4160.084524] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 4160.084527] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 4160.084530] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 4160.084532] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4160.084534] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4160.084543] Call Trace: [ 4160.084549] [ 4160.084555] ? show_regs.cold.9+0x22/0x2f [ 4160.084564] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4160.084571] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4160.084574] ? watchdog+0x40/0x40 [ 4160.084579] ? hrtimer_interrupt+0x128/0x2c0 [ 4160.084583] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4160.084592] ? apic_timer_interrupt+0xf/0x20 [ 4160.084596] [ 4160.084599] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4160.084603] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4160.084608] do_raw_spin_lock+0x177/0x190 [ 4160.084612] _raw_spin_lock+0x21/0x30 [ 4160.084617] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4160.088105] process_one_work+0x2c8/0x700 [ 4160.088115] worker_thread+0x296/0x6e0 [ 4160.088118] ? rescuer_thread+0x570/0x570 [ 4160.088122] kthread+0x1d1/0x200 [ 4160.088126] ? set_kthread_struct+0x70/0x70 [ 4160.088129] ret_from_fork+0x1f/0x30 [ 4160.482006] ? do_wait_intr+0xf0/0xf0 [ 4160.483643] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4160.485805] mdc_reint+0x2d/0x110 [mdc] [ 4160.487474] mdc_unlink+0x658/0x870 [mdc] [ 4160.489794] lmv_unlink+0x34d/0xbd0 [lmv] [ 4160.491460] ll_unlink+0x24d/0x7f0 [lustre] [ 4160.493618] vfs_unlink+0x202/0x330 [ 4160.495184] do_unlinkat+0x2fa/0x460 [ 4160.496827] __x64_sys_unlink+0x37/0x50 [ 4160.498479] do_syscall_64+0xc1/0x440 [ 4160.500184] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4160.503499] RIP: 0033:0x7f151bdee10b [ 4160.504719] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4160.537264] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4160.546128] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 4160.554094] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 4160.563542] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4160.570030] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4160.577072] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4166.112883] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2235s! [ 4166.123073] Showing busy workqueues and worker pools: [ 4166.137228] workqueue events: flags=0x0 [ 4166.138451] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4166.152384] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 4166.160836] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4166.177357] pending: kfree_rcu_monitor [ 4166.178599] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4166.191364] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 4166.202369] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4166.213970] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4166.227550] workqueue events_long: flags=0x0 [ 4166.231518] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4166.237924] pending: lu_objects_flush [obdclass] [ 4166.250312] workqueue events_unbound: flags=0x2 [ 4166.253533] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4166.263099] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4166.273173] workqueue events_power_efficient: flags=0x80 [ 4166.284048] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4166.293801] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 4166.302518] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4166.315273] pending: neigh_periodic_work [ 4166.322355] workqueue mm_percpu_wq: flags=0x8 [ 4166.331507] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4166.343251] pending: vmstat_update, lru_add_drain_per_cpu [ 4166.345134] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4166.362620] pending: vmstat_update [ 4166.375262] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4166.392463] pending: lru_add_drain_per_cpu [ 4166.403531] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4166.416140] pending: lru_add_drain_per_cpu BAR(48) [ 4166.424480] workqueue writeback: flags=0x4a [ 4166.437833] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4166.450607] pending: wb_workfn [ 4166.454031] workqueue xprtiod: flags=0xa [ 4166.470657] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4166.478461] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4166.501619] workqueue ipv6_addrconf: flags=0x40008 [ 4166.510148] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4166.520662] pending: addrconf_verify_work [ 4166.535350] workqueue knbd0-recv: flags=0x1a [ 4166.542527] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4166.557182] in-flight: 658:recv_work [ 4166.566377] workqueue ptlrpc_pinger: flags=0x2 [ 4166.583587] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4166.598051] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4166.611363] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2236s workers=4 idle: 32984 37 8 [ 4166.625924] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4151s workers=2 idle: 661 [ 4172.055055] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 4172.057016] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 4172.069902] Modules linked in: [ 4172.072941] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4172.073032] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4172.073041] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4172.073044] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4172.073061] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4172.073066] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4172.073071] RAX: 000000000000351c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4172.073073] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 4172.073076] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4172.073078] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 4172.073081] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 4172.073085] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 4172.073088] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4172.073090] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 4172.073099] Call Trace: [ 4172.073105] [ 4172.073112] ? show_regs.cold.9+0x22/0x2f [ 4172.073122] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4172.073129] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4172.073133] ? watchdog+0x40/0x40 [ 4172.073138] ? hrtimer_interrupt+0x128/0x2c0 [ 4172.073142] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4172.073151] ? apic_timer_interrupt+0xf/0x20 [ 4172.073155] [ 4172.073158] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4172.073164] do_raw_spin_lock+0x177/0x190 [ 4172.073168] _raw_spin_lock+0x21/0x30 [ 4172.073173] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4172.073502] ? do_raw_spin_unlock+0x75/0x190 [ 4172.073508] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4172.073711] ? do_wait_intr+0xf0/0xf0 [ 4172.073732] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4172.079199] mdc_close+0x53b/0xd70 [mdc] [ 4172.079249] lmv_close+0x18d/0x490 [lmv] [ 4172.079280] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4172.079956] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4172.080951] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4172.081052] ll_md_close+0x274/0x800 [lustre] [ 4172.081140] ll_file_release+0x128/0xc50 [lustre] [ 4172.085084] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4172.087842] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4172.087852] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4172.087856] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4172.087874] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4172.087880] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4172.087884] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 4172.087886] RDX: 0000000000002d6f RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 4172.087889] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 4172.087891] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 4172.087894] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4172.087897] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 4172.087900] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4172.087903] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4172.087913] Call Trace: [ 4172.087920] [ 4172.087927] ? show_regs.cold.9+0x22/0x2f [ 4172.087938] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4172.087946] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4172.087951] ? watchdog+0x40/0x40 [ 4172.087955] ? hrtimer_interrupt+0x128/0x2c0 [ 4172.087959] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4172.087969] ? apic_timer_interrupt+0xf/0x20 [ 4172.087973] [ 4172.087977] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4172.087981] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4172.087987] do_raw_spin_lock+0x177/0x190 [ 4172.087993] _raw_spin_lock+0x21/0x30 [ 4172.088002] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4172.092121] obd_mod_free+0x46/0x140 [ptlrpc] [ 4172.095093] mdc_commit_open+0x91/0xb0 [mdc] [ 4172.095139] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4172.152092] __fput+0x10d/0x440 [ 4172.152120] ____fput+0x12/0x20 [ 4172.152125] task_work_run+0xc6/0x110 [ 4172.152135] exit_to_usermode_loop+0x1dd/0x1f0 [ 4172.152143] do_syscall_64+0x3d6/0x440 [ 4172.152159] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4172.152170] RIP: 0033:0x7f151bee3cd8 [ 4172.157442] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4172.170430] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4172.170868] after_reply+0xe2c/0x1780 [ptlrpc] [ 4172.171089] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4172.182052] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4172.182066] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 4172.182069] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 4172.182071] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4172.182073] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4172.182075] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4172.880567] ? __next_timer_interrupt+0x160/0x160 [ 4172.882817] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4172.885167] ? do_wait_intr+0xf0/0xf0 [ 4172.886866] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4172.889347] kthread+0x1d1/0x200 [ 4172.890767] ? set_kthread_struct+0x70/0x70 [ 4172.893373] ret_from_fork+0x1f/0x30 [ 4188.056065] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4188.058635] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4188.075148] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4188.080173] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4188.087232] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4188.091772] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4188.096071] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4188.109581] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4188.114942] RAX: 000000000000620b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4188.121215] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 4188.126118] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 4188.133675] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 4188.141618] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 4188.145846] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 4188.150146] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4188.152970] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4188.156326] Call Trace: [ 4188.158194] [ 4188.159015] ? show_regs.cold.9+0x22/0x2f [ 4188.161043] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4188.163531] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4188.166575] ? watchdog+0x40/0x40 [ 4188.167886] ? hrtimer_interrupt+0x128/0x2c0 [ 4188.169904] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4188.172393] ? apic_timer_interrupt+0xf/0x20 [ 4188.174271] [ 4188.175140] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4188.177680] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4188.180185] do_raw_spin_lock+0x177/0x190 [ 4188.182411] _raw_spin_lock+0x21/0x30 [ 4188.184138] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4188.187076] process_one_work+0x2c8/0x700 [ 4188.189610] worker_thread+0x296/0x6e0 [ 4188.193856] ? rescuer_thread+0x570/0x570 [ 4188.196803] kthread+0x1d1/0x200 [ 4188.198515] ? set_kthread_struct+0x70/0x70 [ 4188.201115] ret_from_fork+0x1f/0x30 [ 4192.053835] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 4192.060543] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4192.082131] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4192.087372] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4192.091095] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4192.094003] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4192.103887] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4192.108336] RAX: 0000000000006423 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4192.114001] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 4192.118773] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 4192.122035] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 4192.126248] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 4192.134799] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 4192.139746] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4192.142588] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 4192.145870] Call Trace: [ 4192.147251] [ 4192.149248] ? show_regs.cold.9+0x22/0x2f [ 4192.151486] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4192.154121] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4192.156152] ? watchdog+0x40/0x40 [ 4192.157847] ? hrtimer_interrupt+0x128/0x2c0 [ 4192.162559] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4192.167131] ? apic_timer_interrupt+0xf/0x20 [ 4192.173348] [ 4192.175908] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4192.186952] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4192.192104] do_raw_spin_lock+0x177/0x190 [ 4192.194615] _raw_spin_lock+0x21/0x30 [ 4192.197868] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4192.205540] ? do_raw_spin_unlock+0x75/0x190 [ 4192.212944] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4192.217647] ? do_wait_intr+0xf0/0xf0 [ 4192.221248] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4192.226154] mdc_reint+0x2d/0x110 [mdc] [ 4192.229812] mdc_unlink+0x658/0x870 [mdc] [ 4192.231929] lmv_unlink+0x34d/0xbd0 [lmv] [ 4192.234952] ll_unlink+0x24d/0x7f0 [lustre] [ 4192.240746] vfs_unlink+0x202/0x330 [ 4192.242715] do_unlinkat+0x2fa/0x460 [ 4192.246942] __x64_sys_unlink+0x37/0x50 [ 4192.251012] do_syscall_64+0xc1/0x440 [ 4192.254926] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4192.260981] RIP: 0033:0x7f151bdee10b [ 4192.262844] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4192.274151] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4192.282558] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 4192.296134] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 4192.306256] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4192.311021] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4192.316102] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4196.833580] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2266s! [ 4196.836298] Showing busy workqueues and worker pools: [ 4196.841862] workqueue events: flags=0x0 [ 4196.845989] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4196.855263] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 4196.867754] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4196.874228] pending: kfree_rcu_monitor [ 4196.879894] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4196.884692] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 4196.893721] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4196.909778] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4196.912124] workqueue events_long: flags=0x0 [ 4196.925801] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4196.934928] pending: lu_objects_flush [obdclass] [ 4196.940700] workqueue events_unbound: flags=0x2 [ 4196.946622] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4196.950743] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4196.961135] workqueue events_power_efficient: flags=0x80 [ 4196.965417] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4196.970060] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 4196.975489] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4196.980322] pending: neigh_periodic_work [ 4196.983100] workqueue mm_percpu_wq: flags=0x8 [ 4196.985896] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4196.989772] pending: vmstat_update, lru_add_drain_per_cpu [ 4196.993190] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4196.995516] pending: vmstat_update [ 4196.996908] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4196.999701] pending: lru_add_drain_per_cpu [ 4197.001488] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4197.008119] pending: lru_add_drain_per_cpu BAR(48) [ 4197.014405] workqueue writeback: flags=0x4a [ 4197.018774] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4197.021939] pending: wb_workfn [ 4197.023567] workqueue xprtiod: flags=0xa [ 4197.025266] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4197.028289] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4197.032283] workqueue ipv6_addrconf: flags=0x40008 [ 4197.034735] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4197.046076] pending: addrconf_verify_work [ 4197.050849] workqueue knbd0-recv: flags=0x1a [ 4197.055349] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4197.060095] in-flight: 658:recv_work [ 4197.063757] workqueue ptlrpc_pinger: flags=0x2 [ 4197.067704] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4197.074417] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4197.081329] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2266s workers=4 idle: 32984 37 8 [ 4197.089815] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4181s workers=2 idle: 661 [ 4200.055031] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 4200.057042] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 4200.062882] Modules linked in: lustre(O) [ 4200.077345] Modules linked in: [ 4200.077348] osp(O) ofd(O) [ 4200.078743] lustre(O) [ 4200.080664] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4200.080866] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4200.080873] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4200.080876] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4200.080888] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4200.080892] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4200.080897] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 4200.080900] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 4200.080902] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 4200.080904] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 4200.080907] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4200.080910] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 4200.080913] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4200.080973] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4200.080980] Call Trace: [ 4200.080985] [ 4200.080990] ? show_regs.cold.9+0x22/0x2f [ 4200.080999] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4200.081006] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4200.081009] ? watchdog+0x40/0x40 [ 4200.081014] ? hrtimer_interrupt+0x128/0x2c0 [ 4200.081018] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4200.081024] ? apic_timer_interrupt+0xf/0x20 [ 4200.081030] [ 4200.081033] ? native_safe_halt+0x2e/0x50 [ 4200.081038] kvm_wait+0xcd/0xe0 [ 4200.081045] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4200.081052] do_raw_spin_lock+0x177/0x190 [ 4200.081057] _raw_spin_lock+0x21/0x30 [ 4200.081062] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4200.081484] obd_mod_free+0x46/0x140 [ptlrpc] [ 4200.083249] osp(O) [ 4200.084327] mdc_commit_open+0x91/0xb0 [mdc] [ 4200.084372] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4200.084640] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4200.084852] after_reply+0xe2c/0x1780 [ptlrpc] [ 4200.085052] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4200.085263] ? __next_timer_interrupt+0x160/0x160 [ 4200.085273] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4200.085503] ? do_wait_intr+0xf0/0xf0 [ 4200.085512] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4200.085724] kthread+0x1d1/0x200 [ 4200.085729] ? set_kthread_struct+0x70/0x70 [ 4200.085733] ret_from_fork+0x1f/0x30 [ 4200.308038] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4200.343458] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4200.350358] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4200.356095] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4200.357407] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4200.366808] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 4200.370821] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4200.381278] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 4200.389787] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4200.394009] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 4200.398290] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 4200.404889] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 4200.414740] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4200.420091] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 4200.426652] Call Trace: [ 4200.429808] [ 4200.430937] ? show_regs.cold.9+0x22/0x2f [ 4200.433716] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4200.438051] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4200.442951] ? watchdog+0x40/0x40 [ 4200.444956] ? hrtimer_interrupt+0x128/0x2c0 [ 4200.453556] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4200.460264] ? apic_timer_interrupt+0xf/0x20 [ 4200.463415] [ 4200.464192] ? native_safe_halt+0x2e/0x50 [ 4200.466580] kvm_wait+0xcd/0xe0 [ 4200.467643] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4200.472032] do_raw_spin_lock+0x177/0x190 [ 4200.473926] _raw_spin_lock+0x21/0x30 [ 4200.479487] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4200.483566] ? do_raw_spin_unlock+0x75/0x190 [ 4200.485039] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4200.495015] ? do_wait_intr+0xf0/0xf0 [ 4200.501629] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4200.507803] mdc_close+0x53b/0xd70 [mdc] [ 4200.511009] lmv_close+0x18d/0x490 [lmv] [ 4200.514624] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4200.515991] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4200.524416] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4200.533791] ll_md_close+0x274/0x800 [lustre] [ 4200.534978] ll_file_release+0x128/0xc50 [lustre] [ 4200.538413] __fput+0x10d/0x440 [ 4200.539568] ____fput+0x12/0x20 [ 4200.541478] task_work_run+0xc6/0x110 [ 4200.542862] exit_to_usermode_loop+0x1dd/0x1f0 [ 4200.544515] do_syscall_64+0x3d6/0x440 [ 4200.547910] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4200.555006] RIP: 0033:0x7f151bee3cd8 [ 4200.557848] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4200.573036] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4200.577292] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 4200.585699] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 4200.592901] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4200.599053] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4200.609781] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4216.056027] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4216.062019] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4216.090515] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4216.098111] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4216.102985] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4216.107402] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4216.110556] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4216.121638] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4216.126416] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4216.130480] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 4216.134652] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 4216.140056] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 4216.147319] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 4216.154923] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 4216.158944] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4216.161120] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4216.173862] Call Trace: [ 4216.183171] [ 4216.187381] ? show_regs.cold.9+0x22/0x2f [ 4216.198506] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4216.211194] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4216.219438] ? watchdog+0x40/0x40 [ 4216.232976] ? hrtimer_interrupt+0x128/0x2c0 [ 4216.237363] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4216.245206] ? apic_timer_interrupt+0xf/0x20 [ 4216.255392] [ 4216.266145] ? native_safe_halt+0x2e/0x50 [ 4216.278119] kvm_wait+0xcd/0xe0 [ 4216.281593] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4216.293501] do_raw_spin_lock+0x177/0x190 [ 4216.301976] _raw_spin_lock+0x21/0x30 [ 4216.312077] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4216.313994] process_one_work+0x2c8/0x700 [ 4216.324889] worker_thread+0x296/0x6e0 [ 4216.329955] ? rescuer_thread+0x570/0x570 [ 4216.337175] kthread+0x1d1/0x200 [ 4216.340415] ? set_kthread_struct+0x70/0x70 [ 4216.345029] ret_from_fork+0x1f/0x30 [ 4224.054373] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 4224.063550] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4224.113343] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4224.123722] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4224.128508] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4224.133051] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4224.147389] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4224.156103] RAX: 0000000000007897 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4224.158666] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 4224.164766] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 4224.168315] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 4224.179424] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 4224.183636] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 4224.186823] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4224.189380] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 4224.201508] Call Trace: [ 4224.205033] [ 4224.207433] ? show_regs.cold.9+0x22/0x2f [ 4224.208509] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4224.214726] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4224.219237] ? watchdog+0x40/0x40 [ 4224.220890] ? hrtimer_interrupt+0x128/0x2c0 [ 4224.223736] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4224.232839] ? apic_timer_interrupt+0xf/0x20 [ 4224.239487] [ 4224.241692] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4224.247448] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4224.256325] do_raw_spin_lock+0x177/0x190 [ 4224.257789] _raw_spin_lock+0x21/0x30 [ 4224.260561] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4224.263480] ? do_raw_spin_unlock+0x75/0x190 [ 4224.265661] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4224.271914] ? do_wait_intr+0xf0/0xf0 [ 4224.273296] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4224.277515] mdc_reint+0x2d/0x110 [mdc] [ 4224.282750] mdc_unlink+0x658/0x870 [mdc] [ 4224.285500] lmv_unlink+0x34d/0xbd0 [lmv] [ 4224.290333] ll_unlink+0x24d/0x7f0 [lustre] [ 4224.295414] vfs_unlink+0x202/0x330 [ 4224.297858] do_unlinkat+0x2fa/0x460 [ 4224.307197] __x64_sys_unlink+0x37/0x50 [ 4224.309811] do_syscall_64+0xc1/0x440 [ 4224.312692] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4224.314148] RIP: 0033:0x7f151bdee10b [ 4224.316345] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4224.330918] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4224.338728] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 4224.342877] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 4224.348305] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4224.350653] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4224.362704] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4227.553198] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2296s! [ 4227.557292] Showing busy workqueues and worker pools: [ 4227.559459] workqueue events: flags=0x0 [ 4227.561599] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4227.565025] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 4227.569138] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4227.573020] pending: kfree_rcu_monitor [ 4227.575759] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4227.579755] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 4227.583852] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4227.587804] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4227.592506] workqueue events_long: flags=0x0 [ 4227.594356] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4227.596668] pending: lu_objects_flush [obdclass] [ 4227.599659] workqueue events_unbound: flags=0x2 [ 4227.602234] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4227.605649] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4227.611396] workqueue events_power_efficient: flags=0x80 [ 4227.614169] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4227.618639] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 4227.622432] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4227.627055] pending: neigh_periodic_work [ 4227.628484] workqueue mm_percpu_wq: flags=0x8 [ 4227.631799] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4227.637039] pending: vmstat_update, lru_add_drain_per_cpu [ 4227.641412] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4227.645140] pending: vmstat_update [ 4227.647026] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4227.653666] pending: lru_add_drain_per_cpu [ 4227.659253] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4227.665255] pending: lru_add_drain_per_cpu BAR(48) [ 4227.669255] workqueue writeback: flags=0x4a [ 4227.674393] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4227.678799] pending: wb_workfn [ 4227.680267] workqueue xprtiod: flags=0xa [ 4227.682727] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4227.688030] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4227.694692] workqueue ipv6_addrconf: flags=0x40008 [ 4227.700528] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4227.705406] pending: addrconf_verify_work [ 4227.708424] workqueue knbd0-recv: flags=0x1a [ 4227.712583] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4227.717648] in-flight: 658:recv_work [ 4227.719993] workqueue ptlrpc_pinger: flags=0x2 [ 4227.728497] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4227.732238] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4227.739200] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2297s workers=4 idle: 32984 37 8 [ 4227.745472] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4212s workers=2 idle: 661 [ 4228.055199] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 4228.057925] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 4228.058630] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4228.058750] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4228.058755] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4228.058758] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4228.058772] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4228.058776] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4228.058781] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 4228.058783] RDX: 000000000000523a RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 4228.058785] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 4228.058787] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 4228.058789] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4228.058792] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 4228.058794] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4228.058796] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4228.058804] Call Trace: [ 4228.058809] [ 4228.058814] ? show_regs.cold.9+0x22/0x2f [ 4228.058824] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4228.058830] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4228.058835] ? watchdog+0x40/0x40 [ 4228.058839] ? hrtimer_interrupt+0x128/0x2c0 [ 4228.058843] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4228.058851] ? apic_timer_interrupt+0xf/0x20 [ 4228.058855] [ 4228.058858] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4228.058862] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4228.058866] do_raw_spin_lock+0x177/0x190 [ 4228.058871] _raw_spin_lock+0x21/0x30 [ 4228.058877] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4228.059342] obd_mod_free+0x46/0x140 [ptlrpc] [ 4228.059536] mdc_commit_open+0x91/0xb0 [mdc] [ 4228.059571] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4228.059766] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4228.059959] after_reply+0xe2c/0x1780 [ptlrpc] [ 4228.060176] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4228.060383] ? __next_timer_interrupt+0x160/0x160 [ 4228.060395] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4228.060608] ? do_wait_intr+0xf0/0xf0 [ 4228.060616] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4228.060815] kthread+0x1d1/0x200 [ 4228.060821] ? set_kthread_struct+0x70/0x70 [ 4228.060825] ret_from_fork+0x1f/0x30 [ 4228.197220] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4228.218848] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4228.223538] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4228.227278] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4228.230188] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4228.238032] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4228.241686] RAX: 0000000000006af5 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4228.244857] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 4228.247116] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4228.250163] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 4228.253410] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 4228.256944] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 4228.260608] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4228.263230] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 4228.266509] Call Trace: [ 4228.267664] [ 4228.268546] ? show_regs.cold.9+0x22/0x2f [ 4228.270495] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4228.272759] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4228.274717] ? watchdog+0x40/0x40 [ 4228.276267] ? hrtimer_interrupt+0x128/0x2c0 [ 4228.278542] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4228.280886] ? apic_timer_interrupt+0xf/0x20 [ 4228.282840] [ 4228.283838] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4228.286221] do_raw_spin_lock+0x177/0x190 [ 4228.287990] _raw_spin_lock+0x21/0x30 [ 4228.289772] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4228.292440] ? do_raw_spin_unlock+0x75/0x190 [ 4228.294132] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4228.296825] ? do_wait_intr+0xf0/0xf0 [ 4228.298579] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4228.301004] mdc_close+0x53b/0xd70 [mdc] [ 4228.302841] lmv_close+0x18d/0x490 [lmv] [ 4228.304562] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4228.307440] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4228.309911] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4228.312014] ll_md_close+0x274/0x800 [lustre] [ 4228.313902] ll_file_release+0x128/0xc50 [lustre] [ 4228.316236] __fput+0x10d/0x440 [ 4228.317677] ____fput+0x12/0x20 [ 4228.319028] task_work_run+0xc6/0x110 [ 4228.320915] exit_to_usermode_loop+0x1dd/0x1f0 [ 4228.322946] do_syscall_64+0x3d6/0x440 [ 4228.324559] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4228.326833] RIP: 0033:0x7f151bee3cd8 [ 4228.328567] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4228.338594] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4228.341880] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 4228.346616] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 4228.349874] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4228.354842] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4228.359799] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4244.056128] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4244.060092] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4244.101045] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4244.113042] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4244.124913] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4244.146633] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4244.150864] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4244.174216] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4244.190747] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4244.193570] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 4244.217171] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 4244.228557] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 4244.235992] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 4244.242077] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 4244.258784] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4244.266839] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4244.276829] Call Trace: [ 4244.278039] [ 4244.279052] ? show_regs.cold.9+0x22/0x2f [ 4244.284646] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4244.295147] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4244.302338] ? watchdog+0x40/0x40 [ 4244.305094] ? hrtimer_interrupt+0x128/0x2c0 [ 4244.309443] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4244.312139] ? apic_timer_interrupt+0xf/0x20 [ 4244.320252] [ 4244.322626] ? native_safe_halt+0x2e/0x50 [ 4244.324856] kvm_wait+0xcd/0xe0 [ 4244.331758] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4244.337039] do_raw_spin_lock+0x177/0x190 [ 4244.338776] _raw_spin_lock+0x21/0x30 [ 4244.343524] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4244.353539] process_one_work+0x2c8/0x700 [ 4244.358785] worker_thread+0x296/0x6e0 [ 4244.363226] ? rescuer_thread+0x570/0x570 [ 4244.368843] kthread+0x1d1/0x200 [ 4244.375634] ? set_kthread_struct+0x70/0x70 [ 4244.383596] ret_from_fork+0x1f/0x30 [ 4252.052146] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 4252.059508] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4252.100442] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4252.105853] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4252.109573] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4252.113361] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4252.129717] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4252.136349] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4252.141671] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 4252.145919] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 4252.149004] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 4252.154587] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 4252.156775] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 4252.167797] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4252.176045] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 4252.188440] Call Trace: [ 4252.189495] [ 4252.191471] ? show_regs.cold.9+0x22/0x2f [ 4252.196548] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4252.211262] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4252.216622] ? watchdog+0x40/0x40 [ 4252.226203] ? hrtimer_interrupt+0x128/0x2c0 [ 4252.227770] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4252.244451] ? apic_timer_interrupt+0xf/0x20 [ 4252.249367] [ 4252.253565] ? native_safe_halt+0x2e/0x50 [ 4252.258480] kvm_wait+0xcd/0xe0 [ 4252.269211] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4252.271086] do_raw_spin_lock+0x177/0x190 [ 4252.285758] _raw_spin_lock+0x21/0x30 [ 4252.287144] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4252.295459] ? do_raw_spin_unlock+0x75/0x190 [ 4252.301384] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4252.307514] ? do_wait_intr+0xf0/0xf0 [ 4252.314072] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4252.317016] mdc_reint+0x2d/0x110 [mdc] [ 4252.323103] mdc_unlink+0x658/0x870 [mdc] [ 4252.326070] lmv_unlink+0x34d/0xbd0 [lmv] [ 4252.333446] ll_unlink+0x24d/0x7f0 [lustre] [ 4252.341070] vfs_unlink+0x202/0x330 [ 4252.346336] do_unlinkat+0x2fa/0x460 [ 4252.352901] __x64_sys_unlink+0x37/0x50 [ 4252.360903] do_syscall_64+0xc1/0x440 [ 4252.369289] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4252.376504] RIP: 0033:0x7f151bdee10b [ 4252.377833] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4252.393375] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4252.403162] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 4252.409260] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 4252.416096] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4252.426095] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4252.430508] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4256.059443] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 4256.061949] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 4256.063675] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4256.063751] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4256.063758] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4256.063761] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4256.063775] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4256.063780] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4256.063784] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 4256.063787] RDX: 0000000000007f00 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 4256.063789] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 4256.063791] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 4256.063793] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4256.063797] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 4256.063800] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4256.063802] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4256.063811] Call Trace: [ 4256.063816] [ 4256.063821] ? show_regs.cold.9+0x22/0x2f [ 4256.063841] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4256.063847] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4256.063851] ? watchdog+0x40/0x40 [ 4256.063855] ? hrtimer_interrupt+0x128/0x2c0 [ 4256.063860] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4256.063870] ? apic_timer_interrupt+0xf/0x20 [ 4256.063874] [ 4256.063877] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4256.063882] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4256.063886] do_raw_spin_lock+0x177/0x190 [ 4256.063891] _raw_spin_lock+0x21/0x30 [ 4256.063896] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4256.064403] obd_mod_free+0x46/0x140 [ptlrpc] [ 4256.064603] mdc_commit_open+0x91/0xb0 [mdc] [ 4256.064644] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4256.065257] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4256.065475] after_reply+0xe2c/0x1780 [ptlrpc] [ 4256.065677] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4256.066072] ? __next_timer_interrupt+0x160/0x160 [ 4256.066083] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4256.075218] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4256.075289] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4256.075297] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4256.075300] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4256.075313] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4256.075317] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 4256.075322] RAX: 0000000000007e86 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4256.075324] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 4256.075326] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4256.075329] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 4256.075331] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 4256.075335] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 4256.075338] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4256.075340] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 4256.075349] Call Trace: [ 4256.075354] [ 4256.075360] ? show_regs.cold.9+0x22/0x2f [ 4256.075369] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4256.075376] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4256.075380] ? watchdog+0x40/0x40 [ 4256.075384] ? hrtimer_interrupt+0x128/0x2c0 [ 4256.075387] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4256.075396] ? apic_timer_interrupt+0xf/0x20 [ 4256.075401] [ 4256.075404] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4256.075410] do_raw_spin_lock+0x177/0x190 [ 4256.075415] _raw_spin_lock+0x21/0x30 [ 4256.075419] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4256.075684] ? do_raw_spin_unlock+0x75/0x190 [ 4256.075689] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4256.075895] ? do_wait_intr+0xf0/0xf0 [ 4256.075902] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4256.076110] mdc_close+0x53b/0xd70 [mdc] [ 4256.076150] lmv_close+0x18d/0x490 [lmv] [ 4256.076180] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4256.076332] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4256.076438] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4256.076518] ll_md_close+0x274/0x800 [lustre] [ 4256.076601] ll_file_release+0x128/0xc50 [lustre] [ 4256.076684] __fput+0x10d/0x440 [ 4256.076692] ____fput+0x12/0x20 [ 4256.076696] task_work_run+0xc6/0x110 [ 4256.076703] exit_to_usermode_loop+0x1dd/0x1f0 [ 4256.076711] do_syscall_64+0x3d6/0x440 [ 4256.076716] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4256.076722] RIP: 0033:0x7f151bee3cd8 [ 4256.076729] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4256.076732] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4256.076736] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 4256.076739] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 4256.076741] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4256.076743] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4256.076745] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4256.583930] ? do_wait_intr+0xf0/0xf0 [ 4256.589268] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4256.592570] kthread+0x1d1/0x200 [ 4256.594215] ? set_kthread_struct+0x70/0x70 [ 4256.596707] ret_from_fork+0x1f/0x30 [ 4257.761217] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2327s! [ 4257.771568] Showing busy workqueues and worker pools: [ 4257.776013] workqueue events: flags=0x0 [ 4257.780491] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4257.790560] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 4257.800423] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4257.805980] pending: kfree_rcu_monitor [ 4257.809947] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4257.815352] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 4257.821985] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4257.831000] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4257.839228] workqueue events_long: flags=0x0 [ 4257.840336] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4257.851031] pending: lu_objects_flush [obdclass] [ 4257.861485] workqueue events_unbound: flags=0x2 [ 4257.865261] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4257.871148] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4257.880776] workqueue events_power_efficient: flags=0x80 [ 4257.886765] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4257.890279] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 4257.898210] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4257.908041] pending: neigh_periodic_work [ 4257.913166] workqueue mm_percpu_wq: flags=0x8 [ 4257.921407] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4257.929436] pending: vmstat_update, lru_add_drain_per_cpu [ 4257.933323] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4257.943604] pending: vmstat_update [ 4257.945706] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4257.950462] pending: lru_add_drain_per_cpu [ 4257.953985] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4257.960715] pending: lru_add_drain_per_cpu BAR(48) [ 4257.967318] workqueue writeback: flags=0x4a [ 4257.970772] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4257.984356] pending: wb_workfn [ 4257.987568] workqueue xprtiod: flags=0xa [ 4257.993743] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4257.997601] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4258.009369] workqueue ipv6_addrconf: flags=0x40008 [ 4258.013472] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4258.018604] pending: addrconf_verify_work [ 4258.024046] workqueue knbd0-recv: flags=0x1a [ 4258.026323] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4258.031268] in-flight: 658:recv_work [ 4258.036479] workqueue ptlrpc_pinger: flags=0x2 [ 4258.039834] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4258.051091] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4258.059075] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2327s workers=4 idle: 32984 37 8 [ 4258.068778] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4242s workers=2 idle: 661 [ 4272.056022] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4272.059403] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4272.093514] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4272.103224] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4272.111628] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4272.117714] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4272.120683] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4272.139149] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4272.143429] RAX: 0000000000005819 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4272.146362] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 4272.149696] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 4272.153426] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 4272.165105] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 4272.168164] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 4272.171397] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4272.173543] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4272.177285] Call Trace: [ 4272.178818] [ 4272.180047] ? show_regs.cold.9+0x22/0x2f [ 4272.182110] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4272.184515] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4272.186931] ? watchdog+0x40/0x40 [ 4272.188524] ? hrtimer_interrupt+0x128/0x2c0 [ 4272.191267] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4272.193442] ? apic_timer_interrupt+0xf/0x20 [ 4272.196049] [ 4272.197052] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4272.199492] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4272.202209] do_raw_spin_lock+0x177/0x190 [ 4272.204029] _raw_spin_lock+0x21/0x30 [ 4272.205525] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4272.208415] process_one_work+0x2c8/0x700 [ 4272.210797] worker_thread+0x296/0x6e0 [ 4272.212284] ? rescuer_thread+0x570/0x570 [ 4272.214433] kthread+0x1d1/0x200 [ 4272.215961] ? set_kthread_struct+0x70/0x70 [ 4272.218103] ret_from_fork+0x1f/0x30 [ 4284.052611] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 4284.055013] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 4284.057032] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 4284.057043] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4284.057108] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4284.057113] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4284.057126] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4284.057140] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4284.057144] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4284.057149] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4284.057151] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 4284.057153] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4284.057154] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 4284.057157] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 4284.057159] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 4284.057163] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4284.057165] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 4284.057173] Call Trace: [ 4284.057180] [ 4284.057185] ? show_regs.cold.9+0x22/0x2f [ 4284.057195] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4284.057201] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4284.057206] ? watchdog+0x40/0x40 [ 4284.057210] ? hrtimer_interrupt+0x128/0x2c0 [ 4284.057214] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4284.057222] ? apic_timer_interrupt+0xf/0x20 [ 4284.057227] [ 4284.057230] ? native_safe_halt+0x2e/0x50 [ 4284.057235] kvm_wait+0xcd/0xe0 [ 4284.057242] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4284.057250] do_raw_spin_lock+0x177/0x190 [ 4284.057255] _raw_spin_lock+0x21/0x30 [ 4284.057259] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4284.057523] ? do_raw_spin_unlock+0x75/0x190 [ 4284.057529] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4284.057737] ? do_wait_intr+0xf0/0xf0 [ 4284.057745] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4284.057955] mdc_close+0x53b/0xd70 [mdc] [ 4284.058000] lmv_close+0x18d/0x490 [lmv] [ 4284.058034] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4284.058190] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4284.058275] Modules linked in: lustre(O) [ 4284.058294] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4284.058376] ll_md_close+0x274/0x800 [lustre] [ 4284.058460] ll_file_release+0x128/0xc50 [lustre] [ 4284.058547] __fput+0x10d/0x440 [ 4284.058557] ____fput+0x12/0x20 [ 4284.058562] task_work_run+0xc6/0x110 [ 4284.058570] exit_to_usermode_loop+0x1dd/0x1f0 [ 4284.058577] do_syscall_64+0x3d6/0x440 [ 4284.058582] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4284.058589] RIP: 0033:0x7f151bee3cd8 [ 4284.058595] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4284.058599] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4284.058603] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 4284.058605] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 4284.058607] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4284.058609] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4284.058612] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4284.062953] Modules linked in: lustre(O) [ 4284.070489] osp(O) [ 4284.096726] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4284.096800] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4284.096807] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4284.096810] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4284.096824] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4284.096828] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4284.096833] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 4284.096835] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 4284.096837] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 4284.096840] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 4284.096842] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4284.096845] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 4284.096848] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4284.096851] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4284.096934] Call Trace: [ 4284.096940] [ 4284.096946] ? show_regs.cold.9+0x22/0x2f [ 4284.096956] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4284.096962] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4284.096966] ? watchdog+0x40/0x40 [ 4284.096970] ? hrtimer_interrupt+0x128/0x2c0 [ 4284.096974] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4284.096981] ? apic_timer_interrupt+0xf/0x20 [ 4284.096986] [ 4284.096989] ? native_safe_halt+0x2e/0x50 [ 4284.096993] kvm_wait+0xcd/0xe0 [ 4284.097000] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4284.097008] do_raw_spin_lock+0x177/0x190 [ 4284.097012] _raw_spin_lock+0x21/0x30 [ 4284.097018] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4284.097512] obd_mod_free+0x46/0x140 [ptlrpc] [ 4284.097806] mdc_commit_open+0x91/0xb0 [mdc] [ 4284.101981] ofd(O) [ 4284.106906] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4284.110904] lod(O) [ 4284.119710] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4284.119993] after_reply+0xe2c/0x1780 [ptlrpc] [ 4284.120193] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4284.120385] ? __next_timer_interrupt+0x160/0x160 [ 4284.120396] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4284.125004] mdt(O) [ 4284.127579] ? do_wait_intr+0xf0/0xf0 [ 4284.127599] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4284.130435] mdd(O) [ 4284.133239] kthread+0x1d1/0x200 [ 4284.135947] mgs(O) [ 4284.138471] ? set_kthread_struct+0x70/0x70 [ 4284.148347] lquota(O) [ 4284.150372] ret_from_fork+0x1f/0x30 [ 4284.509972] lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4284.534674] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4284.539739] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4284.546468] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4284.549751] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4284.556359] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 4284.559114] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4284.562578] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 4284.566883] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 4284.571196] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 4284.579069] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 4284.583573] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 4284.587042] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4284.590064] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 4284.594220] Call Trace: [ 4284.595157] [ 4284.595806] ? show_regs.cold.9+0x22/0x2f [ 4284.597733] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4284.603143] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4284.605489] ? watchdog+0x40/0x40 [ 4284.607385] ? hrtimer_interrupt+0x128/0x2c0 [ 4284.611597] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4284.615086] ? apic_timer_interrupt+0xf/0x20 [ 4284.617170] [ 4284.618212] ? native_safe_halt+0x2e/0x50 [ 4284.624118] kvm_wait+0xcd/0xe0 [ 4284.625450] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4284.631219] do_raw_spin_lock+0x177/0x190 [ 4284.636241] _raw_spin_lock+0x21/0x30 [ 4284.640275] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4284.646554] ? do_raw_spin_unlock+0x75/0x190 [ 4284.650300] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4284.653977] ? do_wait_intr+0xf0/0xf0 [ 4284.656126] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4284.658560] mdc_reint+0x2d/0x110 [mdc] [ 4284.660062] mdc_unlink+0x658/0x870 [mdc] [ 4284.664231] lmv_unlink+0x34d/0xbd0 [lmv] [ 4284.667696] ll_unlink+0x24d/0x7f0 [lustre] [ 4284.670835] vfs_unlink+0x202/0x330 [ 4284.673990] do_unlinkat+0x2fa/0x460 [ 4284.677526] __x64_sys_unlink+0x37/0x50 [ 4284.679892] do_syscall_64+0xc1/0x440 [ 4284.683088] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4284.685789] RIP: 0033:0x7f151bdee10b [ 4284.688704] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4284.711075] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4284.713558] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 4284.718325] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 4284.722304] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4284.726122] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4284.729293] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4288.481496] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2357s! [ 4288.494461] Showing busy workqueues and worker pools: [ 4288.501273] workqueue events: flags=0x0 [ 4288.503980] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4288.511407] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 4288.521098] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4288.525351] pending: kfree_rcu_monitor [ 4288.529267] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4288.540193] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 4288.548061] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4288.555949] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4288.573402] workqueue events_long: flags=0x0 [ 4288.580570] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4288.589140] pending: lu_objects_flush [obdclass] [ 4288.597979] workqueue events_unbound: flags=0x2 [ 4288.605452] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4288.614742] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4288.629250] workqueue events_power_efficient: flags=0x80 [ 4288.640081] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4288.651948] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 4288.660127] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4288.665413] pending: neigh_periodic_work [ 4288.671295] workqueue mm_percpu_wq: flags=0x8 [ 4288.676298] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4288.681950] pending: vmstat_update, lru_add_drain_per_cpu [ 4288.687134] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4288.694113] pending: vmstat_update [ 4288.697944] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4288.703066] pending: lru_add_drain_per_cpu [ 4288.705829] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4288.716232] pending: lru_add_drain_per_cpu BAR(48) [ 4288.721894] workqueue writeback: flags=0x4a [ 4288.726991] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4288.735636] pending: wb_workfn [ 4288.736943] workqueue xprtiod: flags=0xa [ 4288.742862] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4288.749328] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4288.763373] workqueue ipv6_addrconf: flags=0x40008 [ 4288.766115] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4288.769549] pending: addrconf_verify_work [ 4288.776076] workqueue knbd0-recv: flags=0x1a [ 4288.778288] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4288.783235] in-flight: 658:recv_work [ 4288.785341] workqueue ptlrpc_pinger: flags=0x2 [ 4288.789965] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4288.801106] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4288.808192] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2358s workers=4 idle: 32984 37 8 [ 4288.814127] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4273s workers=2 idle: 661 [ 4300.057717] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4300.067384] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4300.115286] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4300.130391] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4300.143521] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4300.152138] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4300.163370] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4300.181406] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4300.195645] RAX: 0000000000007e86 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4300.203288] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 4300.211332] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 4300.215841] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 4300.223398] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 4300.228774] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 4300.236189] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4300.241987] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4300.250822] Call Trace: [ 4300.252062] [ 4300.254313] ? show_regs.cold.9+0x22/0x2f [ 4300.257309] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4300.260745] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4300.265239] ? watchdog+0x40/0x40 [ 4300.267561] ? hrtimer_interrupt+0x128/0x2c0 [ 4300.269891] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4300.272214] ? apic_timer_interrupt+0xf/0x20 [ 4300.275654] [ 4300.279329] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4300.286525] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4300.294208] do_raw_spin_lock+0x177/0x190 [ 4300.299254] _raw_spin_lock+0x21/0x30 [ 4300.303144] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4300.309014] process_one_work+0x2c8/0x700 [ 4300.313517] worker_thread+0x296/0x6e0 [ 4300.316739] ? rescuer_thread+0x570/0x570 [ 4300.317949] kthread+0x1d1/0x200 [ 4300.319271] ? set_kthread_struct+0x70/0x70 [ 4300.320776] ret_from_fork+0x1f/0x30 [ 4312.055388] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 4312.059423] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 4312.063010] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4312.063089] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4312.063097] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4312.063099] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4312.063113] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4312.063118] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4312.063123] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 4312.063125] RDX: 0000000000006f83 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 4312.063128] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 4312.063130] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 4312.063132] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4312.063136] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 4312.063139] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4312.063141] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4312.063150] Call Trace: [ 4312.063156] [ 4312.063163] ? show_regs.cold.9+0x22/0x2f [ 4312.063173] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4312.063180] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4312.063185] ? watchdog+0x40/0x40 [ 4312.063189] ? hrtimer_interrupt+0x128/0x2c0 [ 4312.063193] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4312.063202] ? apic_timer_interrupt+0xf/0x20 [ 4312.063207] [ 4312.063209] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4312.063213] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4312.063219] do_raw_spin_lock+0x177/0x190 [ 4312.063223] _raw_spin_lock+0x21/0x30 [ 4312.063228] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4312.063522] obd_mod_free+0x46/0x140 [ptlrpc] [ 4312.063964] mdc_commit_open+0x91/0xb0 [mdc] [ 4312.064004] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4312.064224] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4312.064615] after_reply+0xe2c/0x1780 [ptlrpc] [ 4312.064977] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4312.065179] ? __next_timer_interrupt+0x160/0x160 [ 4312.065189] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4312.065567] ? do_wait_intr+0xf0/0xf0 [ 4312.065576] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4312.065957] kthread+0x1d1/0x200 [ 4312.065963] ? set_kthread_struct+0x70/0x70 [ 4312.065968] ret_from_fork+0x1f/0x30 [ 4312.374942] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4312.407547] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4312.416440] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4312.423425] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4312.426124] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4312.447123] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4312.459275] RAX: 0000000000007b06 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4312.461879] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 4312.482814] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4312.487050] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 4312.493857] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 4312.499968] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 4312.504808] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4312.510524] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 4312.516787] Call Trace: [ 4312.518978] [ 4312.520960] ? show_regs.cold.9+0x22/0x2f [ 4312.523562] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4312.525841] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4312.529903] ? watchdog+0x40/0x40 [ 4312.531804] ? hrtimer_interrupt+0x128/0x2c0 [ 4312.533941] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4312.536792] ? apic_timer_interrupt+0xf/0x20 [ 4312.540947] [ 4312.543004] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4312.548284] do_raw_spin_lock+0x177/0x190 [ 4312.551100] _raw_spin_lock+0x21/0x30 [ 4312.557166] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4312.560935] ? do_raw_spin_unlock+0x75/0x190 [ 4312.563510] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4312.570471] ? do_wait_intr+0xf0/0xf0 [ 4312.571588] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4312.577964] mdc_close+0x53b/0xd70 [mdc] [ 4312.581952] lmv_close+0x18d/0x490 [lmv] [ 4312.586604] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4312.590658] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4312.595898] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4312.599189] ll_md_close+0x274/0x800 [lustre] [ 4312.601808] ll_file_release+0x128/0xc50 [lustre] [ 4312.608184] __fput+0x10d/0x440 [ 4312.609999] ____fput+0x12/0x20 [ 4312.611932] task_work_run+0xc6/0x110 [ 4312.622088] exit_to_usermode_loop+0x1dd/0x1f0 [ 4312.625506] do_syscall_64+0x3d6/0x440 [ 4312.628869] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4312.635188] RIP: 0033:0x7f151bee3cd8 [ 4312.640802] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4312.657865] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4312.664008] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 4312.667730] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 4312.674748] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4312.677428] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4312.681138] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4316.052113] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 4316.054621] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4316.080147] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4316.084845] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4316.087785] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4316.089694] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4316.098465] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4316.101120] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4316.103070] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 4316.105472] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 4316.107800] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 4316.110358] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 4316.112916] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 4316.115558] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4316.117522] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 4316.120297] Call Trace: [ 4316.121049] [ 4316.122091] ? show_regs.cold.9+0x22/0x2f [ 4316.123696] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4316.125558] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4316.128138] ? watchdog+0x40/0x40 [ 4316.129502] ? hrtimer_interrupt+0x128/0x2c0 [ 4316.131539] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4316.133772] ? apic_timer_interrupt+0xf/0x20 [ 4316.135432] [ 4316.136569] ? native_safe_halt+0x2e/0x50 [ 4316.138395] kvm_wait+0xcd/0xe0 [ 4316.139815] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4316.142133] do_raw_spin_lock+0x177/0x190 [ 4316.143742] _raw_spin_lock+0x21/0x30 [ 4316.145665] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4316.148152] ? do_raw_spin_unlock+0x75/0x190 [ 4316.150097] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4316.152295] ? do_wait_intr+0xf0/0xf0 [ 4316.153934] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4316.156863] mdc_reint+0x2d/0x110 [mdc] [ 4316.158803] mdc_unlink+0x658/0x870 [mdc] [ 4316.160692] lmv_unlink+0x34d/0xbd0 [lmv] [ 4316.162951] ll_unlink+0x24d/0x7f0 [lustre] [ 4316.165371] vfs_unlink+0x202/0x330 [ 4316.167063] do_unlinkat+0x2fa/0x460 [ 4316.169126] __x64_sys_unlink+0x37/0x50 [ 4316.170789] do_syscall_64+0xc1/0x440 [ 4316.172921] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4316.175515] RIP: 0033:0x7f151bdee10b [ 4316.177211] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4316.186069] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4316.189520] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 4316.192578] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 4316.196110] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4316.199516] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4316.203101] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4319.206577] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2388s! [ 4319.217075] Showing busy workqueues and worker pools: [ 4319.229034] workqueue events: flags=0x0 [ 4319.235822] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4319.242778] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 4319.255406] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4319.265483] pending: kfree_rcu_monitor [ 4319.271992] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4319.274463] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 4319.279667] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4319.285679] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4319.292392] workqueue events_long: flags=0x0 [ 4319.296251] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4319.300677] pending: lu_objects_flush [obdclass] [ 4319.305545] workqueue events_unbound: flags=0x2 [ 4319.308755] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4319.313129] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4319.321338] workqueue events_power_efficient: flags=0x80 [ 4319.329268] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4319.331826] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 4319.334021] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4319.341974] pending: neigh_periodic_work [ 4319.347618] workqueue mm_percpu_wq: flags=0x8 [ 4319.352805] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4319.359988] pending: vmstat_update, lru_add_drain_per_cpu [ 4319.365676] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4319.370562] pending: vmstat_update [ 4319.373597] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4319.380107] pending: lru_add_drain_per_cpu [ 4319.385630] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4319.393531] pending: lru_add_drain_per_cpu BAR(48) [ 4319.400025] workqueue writeback: flags=0x4a [ 4319.401522] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4319.408439] pending: wb_workfn [ 4319.411462] workqueue xprtiod: flags=0xa [ 4319.417038] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4319.420159] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4319.427558] workqueue ipv6_addrconf: flags=0x40008 [ 4319.430876] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4319.435583] pending: addrconf_verify_work [ 4319.440513] workqueue knbd0-recv: flags=0x1a [ 4319.446559] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4319.448969] in-flight: 658:recv_work [ 4319.454811] workqueue ptlrpc_pinger: flags=0x2 [ 4319.457220] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4319.463276] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4319.469893] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2388s workers=4 idle: 32984 37 8 [ 4319.476481] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4304s workers=2 idle: 661 [ 4324.691631] rcu: INFO: rcu_sched self-detected stall on CPU [ 4324.697054] rcu: 1-...!: (1409234 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 4324.707987] (t=2400044 jiffies g=249657 q=3136) [ 4324.711560] rcu: rcu_sched kthread starved for 2394996 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 4324.723083] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4324.731558] rcu: RCU grace-period kthread stack dump: [ 4324.736527] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4324.742267] Call Trace: [ 4324.745412] __schedule+0x351/0xcb0 [ 4324.747638] schedule+0xc0/0x180 [ 4324.749358] schedule_timeout+0xb4/0x190 [ 4324.756390] ? do_raw_spin_unlock+0x75/0x190 [ 4324.759404] ? __next_timer_interrupt+0x160/0x160 [ 4324.765421] ? prepare_to_swait_event+0xf3/0x140 [ 4324.770760] rcu_gp_kthread+0x8dd/0x15b0 [ 4324.775803] ? force_qs_rnp+0x2c0/0x2c0 [ 4324.777746] kthread+0x1d1/0x200 [ 4324.779133] ? set_kthread_struct+0x70/0x70 [ 4324.786806] ret_from_fork+0x1f/0x30 [ 4324.788491] rcu: Stack dump where RCU GP kthread last ran: [ 4324.797182] Sending NMI from CPU 1 to CPUs 0: [ 4324.800347] NMI backtrace for cpu 0 [ 4324.800354] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4324.800356] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4324.800357] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4324.800359] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4324.800362] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 [ 4324.800364] RAX: 00000000000071f5 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4324.800366] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 4324.800367] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 4324.800368] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 4324.800370] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 4324.800371] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 4324.800372] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4324.800373] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 4324.800374] Call Trace: [ 4324.800375] [ 4324.800376] ? show_regs.cold.9+0x22/0x2f [ 4324.800377] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 4324.800378] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 4324.800379] ? nmi_handle+0xbd/0x210 [ 4324.800379] ? default_do_nmi+0x4b/0x1b0 [ 4324.800380] ? do_nmi+0x280/0x380 [ 4324.800381] ? end_repeat_nmi+0x16/0x1f [ 4324.800382] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4324.800383] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4324.800384] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4324.800385] [ 4324.800385] do_raw_spin_lock+0x177/0x190 [ 4324.800386] _raw_spin_lock+0x21/0x30 [ 4324.800387] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4324.800388] ? do_raw_spin_unlock+0x75/0x190 [ 4324.800389] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4324.800390] ? do_wait_intr+0xf0/0xf0 [ 4324.800391] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4324.800392] mdc_reint+0x2d/0x110 [mdc] [ 4324.800393] mdc_unlink+0x658/0x870 [mdc] [ 4324.800394] lmv_unlink+0x34d/0xbd0 [lmv] [ 4324.800395] ll_unlink+0x24d/0x7f0 [lustre] [ 4324.800396] vfs_unlink+0x202/0x330 [ 4324.800397] do_unlinkat+0x2fa/0x460 [ 4324.800397] __x64_sys_unlink+0x37/0x50 [ 4324.800398] do_syscall_64+0xc1/0x440 [ 4324.800399] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4324.800400] RIP: 0033:0x7f151bdee10b [ 4324.800401] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4324.800402] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4324.800404] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 4324.800406] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 4324.800407] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4324.800408] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4324.800409] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4324.800424] NMI backtrace for cpu 1 [ 4324.955255] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4324.959372] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4324.963865] Call Trace: [ 4324.965211] [ 4324.966400] ? dump_stack+0xbb/0x10e [ 4324.968400] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4324.970532] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4324.972193] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4324.974072] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4324.976192] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4324.978345] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 4324.979866] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 4324.981750] ? update_process_times+0x6d/0xc0 [ 4324.983419] ? tick_sched_handle.isra.18+0x27/0x90 [ 4324.984992] ? tick_sched_timer+0x53/0xb0 [ 4324.986177] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4324.988000] ? tick_sched_handle.isra.18+0x90/0x90 [ 4324.989812] ? hrtimer_interrupt+0x128/0x2c0 [ 4324.991559] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4324.993421] ? apic_timer_interrupt+0xf/0x20 [ 4324.994939] [ 4324.995963] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4324.998299] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4325.000592] ? do_raw_spin_lock+0x177/0x190 [ 4325.004247] ? _raw_spin_lock+0x21/0x30 [ 4325.005707] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4325.008098] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 4325.011949] ? mdc_commit_open+0x91/0xb0 [mdc] [ 4325.014276] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4325.016773] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4325.018578] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 4325.026328] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4325.031267] ? __next_timer_interrupt+0x160/0x160 [ 4325.034939] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4325.039506] ? do_wait_intr+0xf0/0xf0 [ 4325.042427] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4325.047527] ? kthread+0x1d1/0x200 [ 4325.049228] ? set_kthread_struct+0x70/0x70 [ 4325.050871] ? ret_from_fork+0x1f/0x30 [ 4328.057938] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4328.065832] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4328.081710] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4328.084812] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4328.087440] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4328.089502] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4328.091062] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4328.096422] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4328.104799] RAX: 0000000000007f0e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4328.114388] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 4328.124721] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 4328.127408] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 4328.134948] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 4328.144251] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 4328.149713] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4328.153403] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4328.157364] Call Trace: [ 4328.158987] [ 4328.160132] ? show_regs.cold.9+0x22/0x2f [ 4328.162394] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4328.166184] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4328.170220] ? watchdog+0x40/0x40 [ 4328.172136] ? hrtimer_interrupt+0x128/0x2c0 [ 4328.174510] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4328.177393] ? apic_timer_interrupt+0xf/0x20 [ 4328.179916] [ 4328.181001] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4328.184242] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4328.188260] do_raw_spin_lock+0x177/0x190 [ 4328.190306] _raw_spin_lock+0x21/0x30 [ 4328.191783] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4328.195477] process_one_work+0x2c8/0x700 [ 4328.197810] worker_thread+0x296/0x6e0 [ 4328.200244] ? rescuer_thread+0x570/0x570 [ 4328.203653] kthread+0x1d1/0x200 [ 4328.205523] ? set_kthread_struct+0x70/0x70 [ 4328.208272] ret_from_fork+0x1f/0x30 [ 4340.057029] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 4340.063349] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4340.108089] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4340.117754] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4340.124874] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4340.131110] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4340.149195] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4340.156459] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4340.161999] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 4340.167735] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4340.170533] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 4340.173309] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 4340.175934] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 4340.178831] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4340.180882] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 4340.183541] Call Trace: [ 4340.184454] [ 4340.185358] ? show_regs.cold.9+0x22/0x2f [ 4340.186890] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4340.188526] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4340.190253] ? watchdog+0x40/0x40 [ 4340.191416] ? hrtimer_interrupt+0x128/0x2c0 [ 4340.193048] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4340.194862] ? apic_timer_interrupt+0xf/0x20 [ 4340.196348] [ 4340.197100] ? native_safe_halt+0x2e/0x50 [ 4340.199855] kvm_wait+0xcd/0xe0 [ 4340.200929] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4340.204207] do_raw_spin_lock+0x177/0x190 [ 4340.206459] _raw_spin_lock+0x21/0x30 [ 4340.207865] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4340.210028] ? do_raw_spin_unlock+0x75/0x190 [ 4340.211544] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4340.214712] ? do_wait_intr+0xf0/0xf0 [ 4340.216299] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4340.218538] mdc_close+0x53b/0xd70 [mdc] [ 4340.223471] lmv_close+0x18d/0x490 [lmv] [ 4340.225836] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4340.233140] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4340.237016] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4340.242344] ll_md_close+0x274/0x800 [lustre] [ 4340.249041] ll_file_release+0x128/0xc50 [lustre] [ 4340.254381] __fput+0x10d/0x440 [ 4340.256755] ____fput+0x12/0x20 [ 4340.263834] task_work_run+0xc6/0x110 [ 4340.265431] exit_to_usermode_loop+0x1dd/0x1f0 [ 4340.268192] do_syscall_64+0x3d6/0x440 [ 4340.269339] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4340.271437] RIP: 0033:0x7f151bee3cd8 [ 4340.273010] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4340.281322] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4340.284000] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 4340.286056] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 4340.288355] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4340.292406] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4340.295324] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4349.924644] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2419s! [ 4349.928344] Showing busy workqueues and worker pools: [ 4349.934061] workqueue events: flags=0x0 [ 4349.935524] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4349.943514] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 4349.953158] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4349.959295] pending: kfree_rcu_monitor [ 4349.960793] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4349.966238] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 4349.977749] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4349.982339] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4349.988333] workqueue events_long: flags=0x0 [ 4349.994656] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4349.998796] pending: lu_objects_flush [obdclass] [ 4350.005062] workqueue events_unbound: flags=0x2 [ 4350.010505] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4350.014710] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4350.018754] workqueue events_power_efficient: flags=0x80 [ 4350.022651] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4350.027176] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 4350.030127] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4350.031929] pending: neigh_periodic_work [ 4350.033812] workqueue mm_percpu_wq: flags=0x8 [ 4350.035768] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4350.038775] pending: vmstat_update, lru_add_drain_per_cpu [ 4350.044294] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4350.048640] pending: vmstat_update [ 4350.052805] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4350.054381] pending: lru_add_drain_per_cpu [ 4350.058964] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4350.061195] pending: lru_add_drain_per_cpu BAR(48) [ 4350.063457] workqueue writeback: flags=0x4a [ 4350.065310] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4350.067709] pending: wb_workfn [ 4350.069200] workqueue xprtiod: flags=0xa [ 4350.070669] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4350.073438] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4350.076691] workqueue ipv6_addrconf: flags=0x40008 [ 4350.078500] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4350.081232] pending: addrconf_verify_work [ 4350.082838] workqueue knbd0-recv: flags=0x1a [ 4350.084465] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4350.086982] in-flight: 658:recv_work [ 4350.088469] workqueue ptlrpc_pinger: flags=0x2 [ 4350.090361] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4350.096627] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4350.101247] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2419s workers=4 idle: 32984 37 8 [ 4350.104493] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4334s workers=2 idle: 661 [ 4352.056546] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 4352.065951] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4352.104625] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4352.110191] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4352.114099] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4352.116432] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4352.133342] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4352.139378] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 4352.145759] RDX: 00000000000003b7 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 4352.149048] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 4352.156716] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 4352.161477] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4352.166133] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 4352.169687] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4352.174192] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4352.177495] Call Trace: [ 4352.178309] [ 4352.179212] ? show_regs.cold.9+0x22/0x2f [ 4352.181356] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4352.183560] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4352.185510] ? watchdog+0x40/0x40 [ 4352.187129] ? hrtimer_interrupt+0x128/0x2c0 [ 4352.188868] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4352.190664] ? apic_timer_interrupt+0xf/0x20 [ 4352.192459] [ 4352.193512] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4352.195522] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4352.197684] do_raw_spin_lock+0x177/0x190 [ 4352.199429] _raw_spin_lock+0x21/0x30 [ 4352.201188] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4352.203483] obd_mod_free+0x46/0x140 [ptlrpc] [ 4352.206968] mdc_commit_open+0x91/0xb0 [mdc] [ 4352.209250] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4352.214034] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4352.218216] after_reply+0xe2c/0x1780 [ptlrpc] [ 4352.220389] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4352.223862] ? __next_timer_interrupt+0x160/0x160 [ 4352.225676] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4352.230692] ? do_wait_intr+0xf0/0xf0 [ 4352.232173] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4352.234063] kthread+0x1d1/0x200 [ 4352.234936] ? set_kthread_struct+0x70/0x70 [ 4352.243522] ret_from_fork+0x1f/0x30 [ 4356.056409] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4356.060655] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4356.084681] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4356.088874] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4356.092020] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4356.095365] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4356.097986] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4356.107261] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4356.110394] RAX: 0000000000002c33 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4356.117071] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 4356.120798] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 4356.124639] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 4356.128387] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 4356.133993] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 4356.140871] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4356.144713] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4356.149439] Call Trace: [ 4356.150887] [ 4356.151696] ? show_regs.cold.9+0x22/0x2f [ 4356.154274] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4356.159033] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4356.162824] ? watchdog+0x40/0x40 [ 4356.164092] ? hrtimer_interrupt+0x128/0x2c0 [ 4356.166417] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4356.168841] ? apic_timer_interrupt+0xf/0x20 [ 4356.170088] [ 4356.170643] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4356.172412] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4356.174422] do_raw_spin_lock+0x177/0x190 [ 4356.176175] _raw_spin_lock+0x21/0x30 [ 4356.178127] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4356.181495] process_one_work+0x2c8/0x700 [ 4356.183527] worker_thread+0x296/0x6e0 [ 4356.188658] ? rescuer_thread+0x570/0x570 [ 4356.190328] kthread+0x1d1/0x200 [ 4356.191896] ? set_kthread_struct+0x70/0x70 [ 4356.193908] ret_from_fork+0x1f/0x30 [ 4368.057061] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 4368.060529] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4368.085849] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4368.093186] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4368.097094] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4368.101104] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4368.115218] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 4368.119715] RAX: 00000000000044f4 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4368.128344] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 4368.143038] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4368.146681] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 4368.151014] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 4368.154154] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 4368.158895] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4368.162013] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 4368.170536] Call Trace: [ 4368.172016] [ 4368.173207] ? show_regs.cold.9+0x22/0x2f [ 4368.175999] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4368.180049] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4368.184158] ? watchdog+0x40/0x40 [ 4368.186253] ? hrtimer_interrupt+0x128/0x2c0 [ 4368.188731] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4368.191327] ? apic_timer_interrupt+0xf/0x20 [ 4368.193645] [ 4368.195044] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4368.199862] do_raw_spin_lock+0x177/0x190 [ 4368.201993] _raw_spin_lock+0x21/0x30 [ 4368.204096] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4368.209699] ? do_raw_spin_unlock+0x75/0x190 [ 4368.213243] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4368.216682] ? do_wait_intr+0xf0/0xf0 [ 4368.218508] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4368.221058] mdc_close+0x53b/0xd70 [mdc] [ 4368.225412] lmv_close+0x18d/0x490 [lmv] [ 4368.227908] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4368.230739] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4368.234045] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4368.237084] ll_md_close+0x274/0x800 [lustre] [ 4368.239419] ll_file_release+0x128/0xc50 [lustre] [ 4368.244037] __fput+0x10d/0x440 [ 4368.245445] ____fput+0x12/0x20 [ 4368.249223] task_work_run+0xc6/0x110 [ 4368.250631] exit_to_usermode_loop+0x1dd/0x1f0 [ 4368.253236] do_syscall_64+0x3d6/0x440 [ 4368.255199] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4368.259580] RIP: 0033:0x7f151bee3cd8 [ 4368.261020] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4368.268861] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4368.271900] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 4368.274662] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 4368.279669] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4368.282759] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4368.290116] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4376.052763] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 4376.057228] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4376.109579] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4376.119985] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4376.128215] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4376.130210] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4376.137167] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4376.140485] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4376.143364] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 4376.146583] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 4376.149955] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 4376.154391] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 4376.157369] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 4376.160754] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4376.163492] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 4376.167521] Call Trace: [ 4376.168528] [ 4376.169204] ? show_regs.cold.9+0x22/0x2f [ 4376.170873] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4376.173098] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4376.174975] ? watchdog+0x40/0x40 [ 4376.176266] ? hrtimer_interrupt+0x128/0x2c0 [ 4376.178215] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4376.179820] ? apic_timer_interrupt+0xf/0x20 [ 4376.181334] [ 4376.182488] ? native_safe_halt+0x2e/0x50 [ 4376.184146] kvm_wait+0xcd/0xe0 [ 4376.185692] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4376.187274] do_raw_spin_lock+0x177/0x190 [ 4376.189220] _raw_spin_lock+0x21/0x30 [ 4376.190894] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4376.193923] ? do_raw_spin_unlock+0x75/0x190 [ 4376.195831] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4376.198665] ? do_wait_intr+0xf0/0xf0 [ 4376.200327] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4376.202980] mdc_reint+0x2d/0x110 [mdc] [ 4376.204517] mdc_unlink+0x658/0x870 [mdc] [ 4376.206526] lmv_unlink+0x34d/0xbd0 [lmv] [ 4376.208876] ll_unlink+0x24d/0x7f0 [lustre] [ 4376.211308] vfs_unlink+0x202/0x330 [ 4376.212723] do_unlinkat+0x2fa/0x460 [ 4376.214422] __x64_sys_unlink+0x37/0x50 [ 4376.216129] do_syscall_64+0xc1/0x440 [ 4376.218200] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4376.220280] RIP: 0033:0x7f151bdee10b [ 4376.221973] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4376.229619] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4376.233333] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 4376.236662] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 4376.239409] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4376.242350] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4376.245696] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4380.056581] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 4380.063014] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4380.104232] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4380.115100] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4380.120788] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4380.123949] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4380.133777] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4380.136350] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 4380.140721] RDX: 0000000000007f89 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 4380.145274] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 4380.153977] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 4380.159699] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4380.164424] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 4380.170386] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4380.174712] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4380.179898] Call Trace: [ 4380.181473] [ 4380.182807] ? show_regs.cold.9+0x22/0x2f [ 4380.185454] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4380.188689] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4380.190903] ? watchdog+0x40/0x40 [ 4380.197489] ? hrtimer_interrupt+0x128/0x2c0 [ 4380.199889] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4380.205366] ? apic_timer_interrupt+0xf/0x20 [ 4380.209494] [ 4380.212365] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4380.215636] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4380.218008] do_raw_spin_lock+0x177/0x190 [ 4380.220167] _raw_spin_lock+0x21/0x30 [ 4380.225794] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4380.229780] obd_mod_free+0x46/0x140 [ptlrpc] [ 4380.233049] mdc_commit_open+0x91/0xb0 [mdc] [ 4380.235800] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4380.239481] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4380.243820] after_reply+0xe2c/0x1780 [ptlrpc] [ 4380.246750] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4380.250390] ? __next_timer_interrupt+0x160/0x160 [ 4380.252855] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4380.255629] ? do_wait_intr+0xf0/0xf0 [ 4380.257569] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4380.261124] kthread+0x1d1/0x200 [ 4380.262242] ? set_kthread_struct+0x70/0x70 [ 4380.263867] ret_from_fork+0x1f/0x30 [ 4380.267696] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2449s! [ 4380.283376] Showing busy workqueues and worker pools: [ 4380.292843] workqueue events: flags=0x0 [ 4380.300023] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4380.308114] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 4380.319670] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4380.342475] pending: kfree_rcu_monitor [ 4380.351386] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4380.369409] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 4380.392272] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4380.397493] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4380.405097] workqueue events_long: flags=0x0 [ 4380.413260] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4380.421468] pending: lu_objects_flush [obdclass] [ 4380.428751] workqueue events_unbound: flags=0x2 [ 4380.433846] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4380.444392] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4380.469526] workqueue events_power_efficient: flags=0x80 [ 4380.471398] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4380.497301] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 4380.505950] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4380.523603] pending: neigh_periodic_work [ 4380.531058] workqueue mm_percpu_wq: flags=0x8 [ 4380.538915] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4380.551799] pending: vmstat_update, lru_add_drain_per_cpu [ 4380.553621] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4380.568795] pending: vmstat_update [ 4380.575976] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4380.578208] pending: lru_add_drain_per_cpu [ 4380.601130] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4380.620999] pending: lru_add_drain_per_cpu BAR(48) [ 4380.633141] workqueue writeback: flags=0x4a [ 4380.640607] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4380.650119] pending: wb_workfn [ 4380.660056] workqueue xprtiod: flags=0xa [ 4380.667731] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4380.682801] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4380.695016] workqueue ipv6_addrconf: flags=0x40008 [ 4380.700091] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4380.711207] pending: addrconf_verify_work [ 4380.720561] workqueue knbd0-recv: flags=0x1a [ 4380.725151] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4380.731837] in-flight: 658:recv_work [ 4380.733296] workqueue ptlrpc_pinger: flags=0x2 [ 4380.750181] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4380.765643] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4380.772907] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2450s workers=4 idle: 32984 37 8 [ 4380.790057] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4365s workers=2 idle: 661 [ 4384.058280] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4384.077988] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4384.176041] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4384.194968] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4384.201303] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4384.207820] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4384.217924] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4384.246402] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4384.256036] RAX: 000000000000480d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4384.268607] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 4384.284585] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 4384.294645] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 4384.303865] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 4384.315964] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 4384.330897] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4384.348279] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4384.362944] Call Trace: [ 4384.367000] [ 4384.371339] ? show_regs.cold.9+0x22/0x2f [ 4384.378088] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4384.382047] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4384.396314] ? watchdog+0x40/0x40 [ 4384.408456] ? hrtimer_interrupt+0x128/0x2c0 [ 4384.416057] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4384.421267] ? apic_timer_interrupt+0xf/0x20 [ 4384.427194] [ 4384.428799] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4384.434819] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4384.445402] do_raw_spin_lock+0x177/0x190 [ 4384.451861] _raw_spin_lock+0x21/0x30 [ 4384.455413] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4384.462442] process_one_work+0x2c8/0x700 [ 4384.463850] worker_thread+0x296/0x6e0 [ 4384.474915] ? rescuer_thread+0x570/0x570 [ 4384.482082] kthread+0x1d1/0x200 [ 4384.489716] ? set_kthread_struct+0x70/0x70 [ 4384.494888] ret_from_fork+0x1f/0x30 [ 4396.057044] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 4396.060217] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4396.089564] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4396.097893] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4396.109929] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4396.117321] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4396.147252] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4396.152951] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4396.160569] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 4396.170256] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4396.180410] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 4396.188864] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 4396.202230] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 4396.213515] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4396.219141] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 4396.225013] Call Trace: [ 4396.228214] [ 4396.229233] ? show_regs.cold.9+0x22/0x2f [ 4396.231328] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4396.233684] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4396.237994] ? watchdog+0x40/0x40 [ 4396.239206] ? hrtimer_interrupt+0x128/0x2c0 [ 4396.242298] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4396.247174] ? apic_timer_interrupt+0xf/0x20 [ 4396.251967] [ 4396.254157] ? native_safe_halt+0x2e/0x50 [ 4396.258523] kvm_wait+0xcd/0xe0 [ 4396.261255] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4396.263801] do_raw_spin_lock+0x177/0x190 [ 4396.266032] _raw_spin_lock+0x21/0x30 [ 4396.269010] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4396.271588] ? do_raw_spin_unlock+0x75/0x190 [ 4396.276159] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4396.278442] ? do_wait_intr+0xf0/0xf0 [ 4396.280388] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4396.283257] mdc_close+0x53b/0xd70 [mdc] [ 4396.285024] lmv_close+0x18d/0x490 [lmv] [ 4396.286669] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4396.288831] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4396.291727] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4396.293645] ll_md_close+0x274/0x800 [lustre] [ 4396.295264] ll_file_release+0x128/0xc50 [lustre] [ 4396.298142] __fput+0x10d/0x440 [ 4396.299805] ____fput+0x12/0x20 [ 4396.301072] task_work_run+0xc6/0x110 [ 4396.302598] exit_to_usermode_loop+0x1dd/0x1f0 [ 4396.307795] do_syscall_64+0x3d6/0x440 [ 4396.310864] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4396.313212] RIP: 0033:0x7f151bee3cd8 [ 4396.314869] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4396.339404] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4396.357232] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 4396.364149] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 4396.371189] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4396.375825] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4396.383196] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4408.052637] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 4408.055028] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 4408.058746] Modules linked in: [ 4408.063359] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4408.063450] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4408.063458] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4408.063462] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4408.063474] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4408.063478] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4408.063482] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 4408.063485] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 4408.063486] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 4408.063489] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 4408.063491] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4408.063494] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 4408.063497] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4408.063499] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4408.063506] Call Trace: [ 4408.063510] [ 4408.063516] ? show_regs.cold.9+0x22/0x2f [ 4408.063525] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4408.063530] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4408.063534] ? watchdog+0x40/0x40 [ 4408.063537] ? hrtimer_interrupt+0x128/0x2c0 [ 4408.063541] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4408.063547] ? apic_timer_interrupt+0xf/0x20 [ 4408.063551] [ 4408.063554] ? native_safe_halt+0x2e/0x50 [ 4408.063557] kvm_wait+0xcd/0xe0 [ 4408.063564] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4408.063571] do_raw_spin_lock+0x177/0x190 [ 4408.063574] _raw_spin_lock+0x21/0x30 [ 4408.063579] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4408.063859] obd_mod_free+0x46/0x140 [ptlrpc] [ 4408.064055] mdc_commit_open+0x91/0xb0 [mdc] [ 4408.064090] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4408.064276] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4408.064965] after_reply+0xe2c/0x1780 [ptlrpc] [ 4408.065254] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4408.065562] ? __next_timer_interrupt+0x160/0x160 [ 4408.065571] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4408.065763] ? do_wait_intr+0xf0/0xf0 [ 4408.065771] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4408.065968] kthread+0x1d1/0x200 [ 4408.065975] ? set_kthread_struct+0x70/0x70 [ 4408.065979] ret_from_fork+0x1f/0x30 [ 4408.348397] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4408.397745] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4408.408932] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4408.421690] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4408.429229] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4408.453879] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4408.460272] RAX: 0000000000007908 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4408.471593] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 4408.476005] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 4408.483747] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 4408.493597] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 4408.509809] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 4408.524667] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4408.537169] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 4408.548551] Call Trace: [ 4408.553623] [ 4408.556988] ? show_regs.cold.9+0x22/0x2f [ 4408.561238] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4408.573621] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4408.582626] ? watchdog+0x40/0x40 [ 4408.592253] ? hrtimer_interrupt+0x128/0x2c0 [ 4408.593817] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4408.604522] ? apic_timer_interrupt+0xf/0x20 [ 4408.609942] [ 4408.610993] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4408.615832] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4408.619624] do_raw_spin_lock+0x177/0x190 [ 4408.625770] _raw_spin_lock+0x21/0x30 [ 4408.629620] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4408.643502] ? do_raw_spin_unlock+0x75/0x190 [ 4408.654970] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4408.660506] ? do_wait_intr+0xf0/0xf0 [ 4408.662691] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4408.667886] mdc_reint+0x2d/0x110 [mdc] [ 4408.675405] mdc_unlink+0x658/0x870 [mdc] [ 4408.678163] lmv_unlink+0x34d/0xbd0 [lmv] [ 4408.680459] ll_unlink+0x24d/0x7f0 [lustre] [ 4408.685181] vfs_unlink+0x202/0x330 [ 4408.689547] do_unlinkat+0x2fa/0x460 [ 4408.692110] __x64_sys_unlink+0x37/0x50 [ 4408.694606] do_syscall_64+0xc1/0x440 [ 4408.697726] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4408.702182] RIP: 0033:0x7f151bdee10b [ 4408.706474] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4408.726117] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4408.737676] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 4408.749620] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 4408.762286] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4408.779322] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4408.789634] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4410.848751] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2480s! [ 4410.859878] Showing busy workqueues and worker pools: [ 4410.867170] workqueue events: flags=0x0 [ 4410.880723] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4410.891207] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 4410.915549] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4410.929790] pending: kfree_rcu_monitor [ 4410.934893] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4410.944369] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 4410.961115] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4410.974914] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4410.983985] workqueue events_long: flags=0x0 [ 4410.990161] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4410.994740] pending: lu_objects_flush [obdclass] [ 4410.998818] workqueue events_unbound: flags=0x2 [ 4411.004680] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4411.008549] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4411.016601] workqueue events_power_efficient: flags=0x80 [ 4411.023626] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4411.030528] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 4411.040699] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4411.049529] pending: neigh_periodic_work [ 4411.054095] workqueue mm_percpu_wq: flags=0x8 [ 4411.059688] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4411.078513] pending: vmstat_update, lru_add_drain_per_cpu [ 4411.085798] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4411.094472] pending: vmstat_update [ 4411.100498] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4411.116986] pending: lru_add_drain_per_cpu [ 4411.125342] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4411.133729] pending: lru_add_drain_per_cpu BAR(48) [ 4411.145217] workqueue writeback: flags=0x4a [ 4411.153216] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4411.162488] pending: wb_workfn [ 4411.166788] workqueue xprtiod: flags=0xa [ 4411.176574] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4411.185623] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4411.199133] workqueue ipv6_addrconf: flags=0x40008 [ 4411.205750] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4411.216510] pending: addrconf_verify_work [ 4411.225701] workqueue knbd0-recv: flags=0x1a [ 4411.232533] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4411.246723] in-flight: 658:recv_work [ 4411.256105] workqueue ptlrpc_pinger: flags=0x2 [ 4411.261144] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4411.266193] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4411.271586] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2480s workers=4 idle: 32984 37 8 [ 4411.275994] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4396s workers=2 idle: 661 [ 4412.057179] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 4412.062861] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4412.103657] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4412.113979] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4412.121119] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4412.128383] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4412.131801] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4412.150934] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4412.156962] RAX: 0000000000006087 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4412.165582] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 4412.174213] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 4412.182825] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 4412.186813] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 4412.195359] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 4412.201724] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4412.209671] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4412.212366] Call Trace: [ 4412.213163] [ 4412.216436] ? show_regs.cold.9+0x22/0x2f [ 4412.219857] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4412.223215] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4412.228793] ? watchdog+0x40/0x40 [ 4412.230093] ? hrtimer_interrupt+0x128/0x2c0 [ 4412.233347] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4412.236963] ? apic_timer_interrupt+0xf/0x20 [ 4412.241776] [ 4412.242696] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4412.246156] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4412.253708] do_raw_spin_lock+0x177/0x190 [ 4412.255195] _raw_spin_lock+0x21/0x30 [ 4412.256595] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4412.261868] process_one_work+0x2c8/0x700 [ 4412.265952] worker_thread+0x296/0x6e0 [ 4412.269465] ? rescuer_thread+0x570/0x570 [ 4412.272637] kthread+0x1d1/0x200 [ 4412.276460] ? set_kthread_struct+0x70/0x70 [ 4412.279004] ret_from_fork+0x1f/0x30 [ 4424.057792] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 4424.063475] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4424.099523] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4424.104824] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4424.110997] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4424.118744] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4424.133617] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4424.137816] RAX: 0000000000007a03 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4424.142763] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 4424.148562] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4424.159838] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 4424.165036] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 4424.168516] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 4424.171696] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4424.174992] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 4424.185678] Call Trace: [ 4424.186779] [ 4424.187515] ? show_regs.cold.9+0x22/0x2f [ 4424.189511] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4424.191172] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4424.194229] ? watchdog+0x40/0x40 [ 4424.197601] ? hrtimer_interrupt+0x128/0x2c0 [ 4424.200545] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4424.203736] ? apic_timer_interrupt+0xf/0x20 [ 4424.205789] [ 4424.206573] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4424.208473] do_raw_spin_lock+0x177/0x190 [ 4424.210156] _raw_spin_lock+0x21/0x30 [ 4424.211158] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4424.212576] ? do_raw_spin_unlock+0x75/0x190 [ 4424.214078] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4424.216535] ? do_wait_intr+0xf0/0xf0 [ 4424.217677] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4424.219878] mdc_close+0x53b/0xd70 [mdc] [ 4424.223475] lmv_close+0x18d/0x490 [lmv] [ 4424.224725] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4424.227021] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4424.229327] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4424.231391] ll_md_close+0x274/0x800 [lustre] [ 4424.233010] ll_file_release+0x128/0xc50 [lustre] [ 4424.235340] __fput+0x10d/0x440 [ 4424.236697] ____fput+0x12/0x20 [ 4424.238068] task_work_run+0xc6/0x110 [ 4424.239610] exit_to_usermode_loop+0x1dd/0x1f0 [ 4424.241600] do_syscall_64+0x3d6/0x440 [ 4424.243004] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4424.244867] RIP: 0033:0x7f151bee3cd8 [ 4424.245981] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4424.252523] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4424.254901] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 4424.257292] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 4424.262744] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4424.265570] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4424.268520] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4436.055179] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 4436.058679] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4436.063748] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 4436.083219] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4436.083234] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4436.083238] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4436.083254] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4436.083258] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 4436.083350] RAX: 0000000000007118 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4436.083353] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 4436.083355] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 4436.083357] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 4436.083360] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 4436.083364] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 4436.083366] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4436.083368] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 4436.083448] Call Trace: [ 4436.083453] [ 4436.083459] ? show_regs.cold.9+0x22/0x2f [ 4436.083470] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4436.083476] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4436.083480] ? watchdog+0x40/0x40 [ 4436.083485] ? hrtimer_interrupt+0x128/0x2c0 [ 4436.083489] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4436.083500] ? apic_timer_interrupt+0xf/0x20 [ 4436.083504] [ 4436.083507] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4436.083512] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4436.083517] do_raw_spin_lock+0x177/0x190 [ 4436.083522] _raw_spin_lock+0x21/0x30 [ 4436.083531] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4436.094431] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) [ 4436.101006] ? do_raw_spin_unlock+0x75/0x190 [ 4436.101040] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4436.101593] ? do_wait_intr+0xf0/0xf0 [ 4436.101601] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4436.101826] mdc_reint+0x2d/0x110 [mdc] [ 4436.101869] mdc_unlink+0x658/0x870 [mdc] [ 4436.101900] lmv_unlink+0x34d/0xbd0 [lmv] [ 4436.101933] ll_unlink+0x24d/0x7f0 [lustre] [ 4436.102060] vfs_unlink+0x202/0x330 [ 4436.102070] do_unlinkat+0x2fa/0x460 [ 4436.102075] __x64_sys_unlink+0x37/0x50 [ 4436.102078] do_syscall_64+0xc1/0x440 [ 4436.102084] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4436.102092] RIP: 0033:0x7f151bdee10b [ 4436.102100] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4436.102104] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4436.102110] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 4436.102113] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 4436.102115] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4436.102118] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4436.102120] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4436.344712] lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4436.389256] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4436.407154] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4436.412242] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4436.420754] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4436.443488] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4436.455056] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 4436.470892] RDX: 0000000000001280 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 4436.481294] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 4436.500245] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 4436.510327] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4436.519557] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 4436.529670] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4436.532446] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4436.539373] Call Trace: [ 4436.540661] [ 4436.541804] ? show_regs.cold.9+0x22/0x2f [ 4436.546554] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4436.549477] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4436.551999] ? watchdog+0x40/0x40 [ 4436.553961] ? hrtimer_interrupt+0x128/0x2c0 [ 4436.556355] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4436.559029] ? apic_timer_interrupt+0xf/0x20 [ 4436.560709] [ 4436.561831] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4436.565786] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4436.571296] do_raw_spin_lock+0x177/0x190 [ 4436.575424] _raw_spin_lock+0x21/0x30 [ 4436.578918] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4436.584516] obd_mod_free+0x46/0x140 [ptlrpc] [ 4436.591356] mdc_commit_open+0x91/0xb0 [mdc] [ 4436.594309] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4436.601775] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4436.609804] after_reply+0xe2c/0x1780 [ptlrpc] [ 4436.615362] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4436.620126] ? __next_timer_interrupt+0x160/0x160 [ 4436.623809] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4436.632889] ? do_wait_intr+0xf0/0xf0 [ 4436.635230] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4436.643786] kthread+0x1d1/0x200 [ 4436.654522] ? set_kthread_struct+0x70/0x70 [ 4436.657459] ret_from_fork+0x1f/0x30 [ 4440.058505] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 4440.070339] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4440.122325] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4440.134002] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4440.138927] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4440.142777] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4440.147170] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4440.160786] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4440.173505] RAX: 000000000000350d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4440.182006] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 4440.183539] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 4440.185372] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 4440.187101] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 4440.195552] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 4440.201228] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4440.202650] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4440.205680] Call Trace: [ 4440.206977] [ 4440.214118] ? show_regs.cold.9+0x22/0x2f [ 4440.219810] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4440.225151] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4440.227792] ? watchdog+0x40/0x40 [ 4440.228467] ? hrtimer_interrupt+0x128/0x2c0 [ 4440.230188] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4440.232679] ? apic_timer_interrupt+0xf/0x20 [ 4440.235594] [ 4440.236641] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4440.238453] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4440.241919] do_raw_spin_lock+0x177/0x190 [ 4440.243657] _raw_spin_lock+0x21/0x30 [ 4440.248510] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4440.251518] process_one_work+0x2c8/0x700 [ 4440.253795] worker_thread+0x296/0x6e0 [ 4440.257540] ? rescuer_thread+0x570/0x570 [ 4440.259897] kthread+0x1d1/0x200 [ 4440.261242] ? set_kthread_struct+0x70/0x70 [ 4440.263291] ret_from_fork+0x1f/0x30 [ 4441.569039] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2510s! [ 4441.574090] Showing busy workqueues and worker pools: [ 4441.576585] workqueue events: flags=0x0 [ 4441.578230] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4441.581912] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 4441.586143] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4441.589404] pending: kfree_rcu_monitor [ 4441.591730] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4441.595132] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 4441.600542] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4441.603711] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4441.607064] workqueue events_long: flags=0x0 [ 4441.609175] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4441.613819] pending: lu_objects_flush [obdclass] [ 4441.617627] workqueue events_unbound: flags=0x2 [ 4441.620047] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4441.623289] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4441.628702] workqueue events_power_efficient: flags=0x80 [ 4441.630516] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4441.634494] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 4441.640575] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4441.644078] pending: neigh_periodic_work [ 4441.646305] workqueue mm_percpu_wq: flags=0x8 [ 4441.648708] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4441.652392] pending: vmstat_update, lru_add_drain_per_cpu [ 4441.655483] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4441.661160] pending: vmstat_update [ 4441.664169] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4441.668266] pending: lru_add_drain_per_cpu [ 4441.670848] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4441.674239] pending: lru_add_drain_per_cpu BAR(48) [ 4441.676528] workqueue writeback: flags=0x4a [ 4441.678616] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4441.681761] pending: wb_workfn [ 4441.683566] workqueue xprtiod: flags=0xa [ 4441.684963] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4441.688765] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4441.699045] workqueue ipv6_addrconf: flags=0x40008 [ 4441.701235] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4441.703554] pending: addrconf_verify_work [ 4441.705257] workqueue knbd0-recv: flags=0x1a [ 4441.707034] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4441.710285] in-flight: 658:recv_work [ 4441.712500] workqueue ptlrpc_pinger: flags=0x2 [ 4441.714118] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4441.716771] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4441.719813] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2511s workers=4 idle: 32984 37 8 [ 4441.723706] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4426s workers=2 idle: 661 [ 4452.057033] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 4452.061542] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4452.093187] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4452.105178] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4452.111844] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4452.114837] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4452.128913] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4452.134242] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4452.137513] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 4452.140476] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4452.144000] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 4452.147786] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 4452.150971] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 4452.155370] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4452.158613] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 4452.162175] Call Trace: [ 4452.163386] [ 4452.164291] ? show_regs.cold.9+0x22/0x2f [ 4452.166172] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4452.168534] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4452.174241] ? watchdog+0x40/0x40 [ 4452.175483] ? hrtimer_interrupt+0x128/0x2c0 [ 4452.177831] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4452.180189] ? apic_timer_interrupt+0xf/0x20 [ 4452.186047] [ 4452.187699] ? native_safe_halt+0x2e/0x50 [ 4452.189470] kvm_wait+0xcd/0xe0 [ 4452.191194] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4452.193812] do_raw_spin_lock+0x177/0x190 [ 4452.198066] _raw_spin_lock+0x21/0x30 [ 4452.201457] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4452.208445] ? do_raw_spin_unlock+0x75/0x190 [ 4452.212528] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4452.216964] ? do_wait_intr+0xf0/0xf0 [ 4452.218833] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4452.221406] mdc_close+0x53b/0xd70 [mdc] [ 4452.223473] lmv_close+0x18d/0x490 [lmv] [ 4452.226466] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4452.228564] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4452.231071] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4452.233114] ll_md_close+0x274/0x800 [lustre] [ 4452.235085] ll_file_release+0x128/0xc50 [lustre] [ 4452.238156] __fput+0x10d/0x440 [ 4452.239542] ____fput+0x12/0x20 [ 4452.240854] task_work_run+0xc6/0x110 [ 4452.244114] exit_to_usermode_loop+0x1dd/0x1f0 [ 4452.247201] do_syscall_64+0x3d6/0x440 [ 4452.249248] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4452.251394] RIP: 0033:0x7f151bee3cd8 [ 4452.252998] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4452.260241] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4452.263517] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 4452.269650] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 4452.276824] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4452.280117] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4452.284594] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4464.056584] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 4464.062778] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4464.101922] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4464.106078] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4464.109909] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4464.111918] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4464.119704] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4464.123577] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 4464.126551] RDX: 0000000000001b47 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 4464.128917] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 4464.132282] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 4464.135225] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4464.138689] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 4464.142172] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4464.144353] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4464.147550] Call Trace: [ 4464.148528] [ 4464.149627] ? show_regs.cold.9+0x22/0x2f [ 4464.151676] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4464.157127] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4464.158930] ? watchdog+0x40/0x40 [ 4464.163711] ? hrtimer_interrupt+0x128/0x2c0 [ 4464.165387] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4464.170384] ? apic_timer_interrupt+0xf/0x20 [ 4464.173307] [ 4464.175428] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4464.179717] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4464.185479] do_raw_spin_lock+0x177/0x190 [ 4464.188265] _raw_spin_lock+0x21/0x30 [ 4464.190943] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4464.196403] obd_mod_free+0x46/0x140 [ptlrpc] [ 4464.198572] mdc_commit_open+0x91/0xb0 [mdc] [ 4464.202232] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4464.204451] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4464.207195] after_reply+0xe2c/0x1780 [ptlrpc] [ 4464.209297] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4464.211727] ? __next_timer_interrupt+0x160/0x160 [ 4464.213514] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4464.222320] ? do_wait_intr+0xf0/0xf0 [ 4464.225108] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4464.230580] kthread+0x1d1/0x200 [ 4464.232346] ? set_kthread_struct+0x70/0x70 [ 4464.237746] ret_from_fork+0x1f/0x30 [ 4468.052030] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 4468.056058] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 4468.056167] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4468.056239] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4468.056244] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4468.056246] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4468.056260] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4468.056264] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4468.056268] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4468.056271] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 4468.056273] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 4468.056275] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 4468.056277] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 4468.056280] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 4468.056283] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4468.056285] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 4468.056293] Call Trace: [ 4468.056298] [ 4468.056303] ? show_regs.cold.9+0x22/0x2f [ 4468.056313] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4468.056319] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4468.056322] ? watchdog+0x40/0x40 [ 4468.056326] ? hrtimer_interrupt+0x128/0x2c0 [ 4468.056330] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4468.056335] ? apic_timer_interrupt+0xf/0x20 [ 4468.056339] [ 4468.056341] ? native_safe_halt+0x2e/0x50 [ 4468.056345] kvm_wait+0xcd/0xe0 [ 4468.056351] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4468.056357] do_raw_spin_lock+0x177/0x190 [ 4468.056361] _raw_spin_lock+0x21/0x30 [ 4468.056365] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4468.062218] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4468.062305] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4468.062313] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4468.062322] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4468.062719] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4468.062728] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4468.062732] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 4468.062736] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4468.062739] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 4468.062741] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 4468.062743] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 4468.062745] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 4468.062748] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 4468.062751] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4468.062753] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4468.062760] Call Trace: [ 4468.062765] [ 4468.062771] ? show_regs.cold.9+0x22/0x2f [ 4468.062780] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4468.062787] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4468.062791] ? watchdog+0x40/0x40 [ 4468.062795] ? hrtimer_interrupt+0x128/0x2c0 [ 4468.062799] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4468.062805] ? apic_timer_interrupt+0xf/0x20 [ 4468.062809] [ 4468.062812] ? native_safe_halt+0x2e/0x50 [ 4468.062815] kvm_wait+0xcd/0xe0 [ 4468.062838] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4468.062847] do_raw_spin_lock+0x177/0x190 [ 4468.062861] _raw_spin_lock+0x21/0x30 [ 4468.062866] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4468.090510] ? do_raw_spin_unlock+0x75/0x190 [ 4468.090530] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4468.091045] ? do_wait_intr+0xf0/0xf0 [ 4468.091053] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4468.091248] mdc_reint+0x2d/0x110 [mdc] [ 4468.091288] mdc_unlink+0x658/0x870 [mdc] [ 4468.091317] lmv_unlink+0x34d/0xbd0 [lmv] [ 4468.091348] ll_unlink+0x24d/0x7f0 [lustre] [ 4468.091467] vfs_unlink+0x202/0x330 [ 4468.091476] do_unlinkat+0x2fa/0x460 [ 4468.091481] __x64_sys_unlink+0x37/0x50 [ 4468.091484] do_syscall_64+0xc1/0x440 [ 4468.091492] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4468.091500] RIP: 0033:0x7f151bdee10b [ 4468.091507] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4468.091511] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4468.091516] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 4468.091518] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 4468.091521] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4468.091522] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4468.091525] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4468.371270] process_one_work+0x2c8/0x700 [ 4468.373422] worker_thread+0x296/0x6e0 [ 4468.375176] ? rescuer_thread+0x570/0x570 [ 4468.377215] kthread+0x1d1/0x200 [ 4468.379288] ? set_kthread_struct+0x70/0x70 [ 4468.381195] ret_from_fork+0x1f/0x30 [ 4471.781461] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2541s! [ 4471.789971] Showing busy workqueues and worker pools: [ 4471.793475] workqueue events: flags=0x0 [ 4471.795376] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4471.800803] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 4471.806531] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4471.812490] pending: kfree_rcu_monitor [ 4471.814960] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4471.819600] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 4471.832112] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4471.839194] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4471.846595] workqueue events_long: flags=0x0 [ 4471.851252] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4471.864513] pending: lu_objects_flush [obdclass] [ 4471.868842] workqueue events_unbound: flags=0x2 [ 4471.872516] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4471.876412] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4471.881935] workqueue events_power_efficient: flags=0x80 [ 4471.885143] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4471.889521] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 4471.893585] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4471.897918] pending: neigh_periodic_work [ 4471.900220] workqueue mm_percpu_wq: flags=0x8 [ 4471.905079] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4471.910349] pending: vmstat_update, lru_add_drain_per_cpu [ 4471.912073] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4471.913722] pending: vmstat_update [ 4471.920277] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4471.938928] pending: lru_add_drain_per_cpu [ 4471.942634] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4471.951619] pending: lru_add_drain_per_cpu BAR(48) [ 4471.955637] workqueue writeback: flags=0x4a [ 4471.958046] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4471.962574] pending: wb_workfn [ 4471.964430] workqueue xprtiod: flags=0xa [ 4471.965919] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4471.969912] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4471.974884] workqueue ipv6_addrconf: flags=0x40008 [ 4471.977213] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4471.982199] pending: addrconf_verify_work [ 4471.986410] workqueue knbd0-recv: flags=0x1a [ 4471.988560] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4471.993375] in-flight: 658:recv_work [ 4471.995896] workqueue ptlrpc_pinger: flags=0x2 [ 4471.999019] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4472.003149] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4472.009626] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2541s workers=4 idle: 32984 37 8 [ 4472.024249] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4456s workers=2 idle: 661 [ 4480.057007] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 4480.062099] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4480.116407] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4480.125199] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4480.132233] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4480.136071] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4480.146103] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 4480.149502] RAX: 0000000000005584 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4480.152520] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 4480.162664] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4480.164551] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 4480.173495] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 4480.177101] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 4480.181913] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4480.185102] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 4480.188084] Call Trace: [ 4480.189368] [ 4480.190897] ? show_regs.cold.9+0x22/0x2f [ 4480.193519] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4480.196977] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4480.203026] ? watchdog+0x40/0x40 [ 4480.204679] ? hrtimer_interrupt+0x128/0x2c0 [ 4480.207080] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4480.211479] ? apic_timer_interrupt+0xf/0x20 [ 4480.214481] [ 4480.215557] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4480.225221] do_raw_spin_lock+0x177/0x190 [ 4480.227087] _raw_spin_lock+0x21/0x30 [ 4480.229500] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4480.236051] ? do_raw_spin_unlock+0x75/0x190 [ 4480.240516] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4480.242724] ? do_wait_intr+0xf0/0xf0 [ 4480.245024] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4480.247520] mdc_close+0x53b/0xd70 [mdc] [ 4480.249371] lmv_close+0x18d/0x490 [lmv] [ 4480.251291] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4480.253177] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4480.260091] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4480.264804] ll_md_close+0x274/0x800 [lustre] [ 4480.267452] ll_file_release+0x128/0xc50 [lustre] [ 4480.270233] __fput+0x10d/0x440 [ 4480.271943] ____fput+0x12/0x20 [ 4480.273538] task_work_run+0xc6/0x110 [ 4480.275677] exit_to_usermode_loop+0x1dd/0x1f0 [ 4480.278048] do_syscall_64+0x3d6/0x440 [ 4480.279740] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4480.282792] RIP: 0033:0x7f151bee3cd8 [ 4480.284824] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4480.294130] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4480.299713] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 4480.303433] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 4480.308455] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4480.312060] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4480.315418] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4492.055064] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 4492.064638] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4492.087595] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4492.092212] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4492.095050] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4492.096411] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4492.101735] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4492.103922] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 4492.106365] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 4492.108742] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 4492.111257] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 4492.113346] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4492.115330] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 4492.117634] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4492.119291] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4492.121608] Call Trace: [ 4492.122302] [ 4492.122903] ? show_regs.cold.9+0x22/0x2f [ 4492.124217] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4492.125626] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4492.127235] ? watchdog+0x40/0x40 [ 4492.128192] ? hrtimer_interrupt+0x128/0x2c0 [ 4492.129721] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4492.131327] ? apic_timer_interrupt+0xf/0x20 [ 4492.132649] [ 4492.134158] ? native_safe_halt+0x2e/0x50 [ 4492.136319] kvm_wait+0xcd/0xe0 [ 4492.138298] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4492.141241] do_raw_spin_lock+0x177/0x190 [ 4492.142455] _raw_spin_lock+0x21/0x30 [ 4492.143446] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4492.153345] obd_mod_free+0x46/0x140 [ptlrpc] [ 4492.158460] mdc_commit_open+0x91/0xb0 [mdc] [ 4492.162969] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4492.167471] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4492.169493] after_reply+0xe2c/0x1780 [ptlrpc] [ 4492.172779] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4492.176396] ? __next_timer_interrupt+0x160/0x160 [ 4492.181521] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4492.184331] ? do_wait_intr+0xf0/0xf0 [ 4492.188275] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4492.194079] kthread+0x1d1/0x200 [ 4492.195605] ? set_kthread_struct+0x70/0x70 [ 4492.197089] ret_from_fork+0x1f/0x30 [ 4496.054157] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 4496.059725] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 4496.059932] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4496.060004] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4496.060010] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4496.060013] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4496.060028] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4496.060032] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4496.060036] RAX: 0000000000006914 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4496.060039] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 4496.060041] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 4496.060043] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 4496.060045] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 4496.060048] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 4496.060051] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4496.060054] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 4496.060063] Call Trace: [ 4496.060069] [ 4496.060075] ? show_regs.cold.9+0x22/0x2f [ 4496.060085] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4496.060091] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4496.060096] ? watchdog+0x40/0x40 [ 4496.060100] ? hrtimer_interrupt+0x128/0x2c0 [ 4496.060105] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4496.060114] ? apic_timer_interrupt+0xf/0x20 [ 4496.060119] [ 4496.060122] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4496.060127] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4496.060132] do_raw_spin_lock+0x177/0x190 [ 4496.060137] _raw_spin_lock+0x21/0x30 [ 4496.060143] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4496.061854] ? do_raw_spin_unlock+0x75/0x190 [ 4496.061863] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4496.073935] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4496.074027] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4496.074034] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4496.074041] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4496.128839] ? do_wait_intr+0xf0/0xf0 [ 4496.128862] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4496.129156] mdc_reint+0x2d/0x110 [mdc] [ 4496.129200] mdc_unlink+0x658/0x870 [mdc] [ 4496.129225] lmv_unlink+0x34d/0xbd0 [lmv] [ 4496.129258] ll_unlink+0x24d/0x7f0 [lustre] [ 4496.129366] vfs_unlink+0x202/0x330 [ 4496.129374] do_unlinkat+0x2fa/0x460 [ 4496.129378] __x64_sys_unlink+0x37/0x50 [ 4496.129381] do_syscall_64+0xc1/0x440 [ 4496.129388] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4496.129397] RIP: 0033:0x7f151bdee10b [ 4496.129404] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4496.129408] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4496.129413] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 4496.129416] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 4496.129418] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4496.129421] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4496.129423] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4496.351795] [ 4496.352414] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4496.355637] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4496.364051] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4496.370150] RAX: 0000000000005b88 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4496.373066] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 4496.376039] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 4496.378836] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 4496.381876] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 4496.386073] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 4496.389997] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4496.392993] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4496.398019] Call Trace: [ 4496.399480] [ 4496.400803] ? show_regs.cold.9+0x22/0x2f [ 4496.402993] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4496.405300] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4496.408032] ? watchdog+0x40/0x40 [ 4496.409658] ? hrtimer_interrupt+0x128/0x2c0 [ 4496.412113] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4496.414360] ? apic_timer_interrupt+0xf/0x20 [ 4496.416350] [ 4496.417339] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4496.420221] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4496.423330] do_raw_spin_lock+0x177/0x190 [ 4496.425080] _raw_spin_lock+0x21/0x30 [ 4496.428295] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4496.431087] process_one_work+0x2c8/0x700 [ 4496.434129] worker_thread+0x296/0x6e0 [ 4496.438004] ? rescuer_thread+0x570/0x570 [ 4496.439021] kthread+0x1d1/0x200 [ 4496.442241] ? set_kthread_struct+0x70/0x70 [ 4496.446313] ret_from_fork+0x1f/0x30 [ 4502.497462] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2571s! [ 4502.511420] Showing busy workqueues and worker pools: [ 4502.514401] workqueue events: flags=0x0 [ 4502.516102] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4502.521902] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 4502.534935] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4502.538178] pending: kfree_rcu_monitor [ 4502.543321] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4502.556586] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 4502.565420] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4502.571611] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4502.579318] workqueue events_long: flags=0x0 [ 4502.583128] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4502.590258] pending: lu_objects_flush [obdclass] [ 4502.597390] workqueue events_unbound: flags=0x2 [ 4502.604265] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4502.608709] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4502.613972] workqueue events_power_efficient: flags=0x80 [ 4502.615790] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4502.621727] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 4502.626248] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4502.630332] pending: neigh_periodic_work [ 4502.635282] workqueue mm_percpu_wq: flags=0x8 [ 4502.640851] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4502.646289] pending: vmstat_update, lru_add_drain_per_cpu [ 4502.651222] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4502.655058] pending: vmstat_update [ 4502.660239] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4502.667558] pending: lru_add_drain_per_cpu [ 4502.670431] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4502.674192] pending: lru_add_drain_per_cpu BAR(48) [ 4502.678865] workqueue writeback: flags=0x4a [ 4502.682193] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4502.685901] pending: wb_workfn [ 4502.688286] workqueue xprtiod: flags=0xa [ 4502.690269] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4502.697357] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4502.705062] workqueue ipv6_addrconf: flags=0x40008 [ 4502.713312] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4502.723082] pending: addrconf_verify_work [ 4502.732144] workqueue knbd0-recv: flags=0x1a [ 4502.733392] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4502.745312] in-flight: 658:recv_work [ 4502.749164] workqueue ptlrpc_pinger: flags=0x2 [ 4502.754168] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4502.762163] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4502.765714] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2572s workers=4 idle: 32984 37 8 [ 4502.775162] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4487s workers=2 idle: 661 [ 4504.694357] rcu: INFO: rcu_sched self-detected stall on CPU [ 4504.700218] rcu: 1-...!: (1512464 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 4504.705133] (t=2580047 jiffies g=249657 q=3181) [ 4504.712143] rcu: rcu_sched kthread starved for 2574999 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 4504.722031] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4504.728675] rcu: RCU grace-period kthread stack dump: [ 4504.733867] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4504.741046] Call Trace: [ 4504.742705] __schedule+0x351/0xcb0 [ 4504.745265] schedule+0xc0/0x180 [ 4504.749178] schedule_timeout+0xb4/0x190 [ 4504.757316] ? do_raw_spin_unlock+0x75/0x190 [ 4504.763098] ? __next_timer_interrupt+0x160/0x160 [ 4504.765230] ? prepare_to_swait_event+0xf3/0x140 [ 4504.771776] rcu_gp_kthread+0x8dd/0x15b0 [ 4504.773788] ? force_qs_rnp+0x2c0/0x2c0 [ 4504.776794] kthread+0x1d1/0x200 [ 4504.778840] ? set_kthread_struct+0x70/0x70 [ 4504.784238] ret_from_fork+0x1f/0x30 [ 4504.790352] rcu: Stack dump where RCU GP kthread last ran: [ 4504.792380] Sending NMI from CPU 1 to CPUs 0: [ 4504.809259] INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 5.836 msecs [ 4504.809851] NMI backtrace for cpu 0 [ 4504.809856] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4504.809858] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4504.809860] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4504.809861] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4504.809863] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 [ 4504.809866] RAX: 0000000000004b94 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4504.809867] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 4504.809868] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 4504.809870] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 4504.809871] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 4504.809872] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 4504.809873] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4504.809874] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 4504.809875] Call Trace: [ 4504.809876] [ 4504.809877] ? show_regs.cold.9+0x22/0x2f [ 4504.809878] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 4504.809879] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 4504.809880] ? nmi_handle+0xbd/0x210 [ 4504.809881] ? default_do_nmi+0x4b/0x1b0 [ 4504.809882] ? do_nmi+0x280/0x380 [ 4504.809883] ? end_repeat_nmi+0x16/0x1f [ 4504.809883] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4504.809885] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4504.809885] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4504.809886] [ 4504.809887] do_raw_spin_lock+0x177/0x190 [ 4504.809888] _raw_spin_lock+0x21/0x30 [ 4504.809889] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4504.809890] ? do_raw_spin_unlock+0x75/0x190 [ 4504.809890] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4504.809892] ? do_wait_intr+0xf0/0xf0 [ 4504.809892] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4504.809893] mdc_reint+0x2d/0x110 [mdc] [ 4504.809894] mdc_unlink+0x658/0x870 [mdc] [ 4504.809895] lmv_unlink+0x34d/0xbd0 [lmv] [ 4504.809895] ll_unlink+0x24d/0x7f0 [lustre] [ 4504.809896] vfs_unlink+0x202/0x330 [ 4504.809897] do_unlinkat+0x2fa/0x460 [ 4504.809898] __x64_sys_unlink+0x37/0x50 [ 4504.809899] do_syscall_64+0xc1/0x440 [ 4504.809900] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4504.809901] RIP: 0033:0x7f151bdee10b [ 4504.809902] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4504.809904] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4504.809906] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 4504.809908] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 4504.809909] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4504.809910] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4504.809911] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4504.809917] NMI backtrace for cpu 1 [ 4505.073386] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4505.086124] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4505.093819] Call Trace: [ 4505.094764] [ 4505.095352] ? dump_stack+0xbb/0x10e [ 4505.099588] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4505.103331] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4505.109449] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4505.114257] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4505.120834] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4505.122791] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 4505.127062] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 4505.128574] ? update_process_times+0x6d/0xc0 [ 4505.131753] ? tick_sched_handle.isra.18+0x27/0x90 [ 4505.133788] ? tick_sched_timer+0x53/0xb0 [ 4505.136075] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4505.138668] ? tick_sched_handle.isra.18+0x90/0x90 [ 4505.143582] ? hrtimer_interrupt+0x128/0x2c0 [ 4505.146983] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4505.148860] ? apic_timer_interrupt+0xf/0x20 [ 4505.152155] [ 4505.154354] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4505.156232] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4505.159351] ? do_raw_spin_lock+0x177/0x190 [ 4505.162815] ? _raw_spin_lock+0x21/0x30 [ 4505.167947] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4505.174177] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 4505.179821] ? mdc_commit_open+0x91/0xb0 [mdc] [ 4505.181668] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4505.183684] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4505.188641] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 4505.199965] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4505.202022] ? __next_timer_interrupt+0x160/0x160 [ 4505.212170] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4505.218106] ? do_wait_intr+0xf0/0xf0 [ 4505.221516] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4505.225588] ? kthread+0x1d1/0x200 [ 4505.232023] ? set_kthread_struct+0x70/0x70 [ 4505.235882] ? ret_from_fork+0x1f/0x30 [ 4508.060132] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 4508.068089] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4508.095561] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4508.100882] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4508.105089] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4508.108529] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4508.119063] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 4508.123277] RAX: 0000000000007403 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4508.127028] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 4508.134631] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4508.139516] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 4508.149913] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 4508.158041] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 4508.167814] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4508.177806] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 4508.191910] Call Trace: [ 4508.195954] [ 4508.198853] ? show_regs.cold.9+0x22/0x2f [ 4508.206196] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4508.208816] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4508.211456] ? watchdog+0x40/0x40 [ 4508.213674] ? hrtimer_interrupt+0x128/0x2c0 [ 4508.217098] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4508.222900] ? apic_timer_interrupt+0xf/0x20 [ 4508.225045] [ 4508.225981] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4508.228892] do_raw_spin_lock+0x177/0x190 [ 4508.231760] _raw_spin_lock+0x21/0x30 [ 4508.233253] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4508.235945] ? do_raw_spin_unlock+0x75/0x190 [ 4508.237570] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4508.238970] ? do_wait_intr+0xf0/0xf0 [ 4508.240781] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4508.243363] mdc_close+0x53b/0xd70 [mdc] [ 4508.244908] lmv_close+0x18d/0x490 [lmv] [ 4508.246837] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4508.249550] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4508.251604] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4508.253570] ll_md_close+0x274/0x800 [lustre] [ 4508.255073] ll_file_release+0x128/0xc50 [lustre] [ 4508.257296] __fput+0x10d/0x440 [ 4508.258818] ____fput+0x12/0x20 [ 4508.260726] task_work_run+0xc6/0x110 [ 4508.262451] exit_to_usermode_loop+0x1dd/0x1f0 [ 4508.264698] do_syscall_64+0x3d6/0x440 [ 4508.266032] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4508.268416] RIP: 0033:0x7f151bee3cd8 [ 4508.270203] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4508.286971] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4508.293964] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 4508.302200] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 4508.309971] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4508.321714] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4508.328082] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4524.056017] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4524.059373] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4524.083549] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4524.090844] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4524.095669] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4524.098596] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4524.100913] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4524.108444] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4524.111840] RAX: 0000000000000dc4 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4524.114734] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 4524.117480] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 4524.120436] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 4524.123608] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 4524.126593] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 4524.130825] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4524.132738] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4524.135478] Call Trace: [ 4524.136440] [ 4524.137126] ? show_regs.cold.9+0x22/0x2f [ 4524.138581] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4524.142412] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4524.144122] ? watchdog+0x40/0x40 [ 4524.145803] ? hrtimer_interrupt+0x128/0x2c0 [ 4524.148593] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4524.150809] ? apic_timer_interrupt+0xf/0x20 [ 4524.153471] [ 4524.154967] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4524.158511] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4524.163003] do_raw_spin_lock+0x177/0x190 [ 4524.166133] _raw_spin_lock+0x21/0x30 [ 4524.168466] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4524.172056] process_one_work+0x2c8/0x700 [ 4524.174912] worker_thread+0x296/0x6e0 [ 4524.176980] ? rescuer_thread+0x570/0x570 [ 4524.179728] kthread+0x1d1/0x200 [ 4524.181124] ? set_kthread_struct+0x70/0x70 [ 4524.183874] ret_from_fork+0x1f/0x30 [ 4532.056477] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 4532.057705] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 4532.059508] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4532.059587] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4532.059600] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4532.059603] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4532.059616] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4532.059620] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4532.059625] RAX: 000000000000278d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4532.059627] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 4532.059629] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 4532.059631] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 4532.059634] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 4532.059637] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 4532.059640] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4532.059643] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 4532.059652] Call Trace: [ 4532.059657] [ 4532.059663] ? show_regs.cold.9+0x22/0x2f [ 4532.059673] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4532.059679] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4532.059683] ? watchdog+0x40/0x40 [ 4532.059688] ? hrtimer_interrupt+0x128/0x2c0 [ 4532.059692] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4532.059701] ? apic_timer_interrupt+0xf/0x20 [ 4532.059706] [ 4532.059709] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4532.059714] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4532.059719] do_raw_spin_lock+0x177/0x190 [ 4532.059725] _raw_spin_lock+0x21/0x30 [ 4532.059731] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4532.059998] ? do_raw_spin_unlock+0x75/0x190 [ 4532.060004] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4532.060211] ? do_wait_intr+0xf0/0xf0 [ 4532.060220] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4532.064875] mdc_reint+0x2d/0x110 [mdc] [ 4532.064915] mdc_unlink+0x658/0x870 [mdc] [ 4532.064944] lmv_unlink+0x34d/0xbd0 [lmv] [ 4532.064977] ll_unlink+0x24d/0x7f0 [lustre] [ 4532.065089] vfs_unlink+0x202/0x330 [ 4532.065099] do_unlinkat+0x2fa/0x460 [ 4532.065106] __x64_sys_unlink+0x37/0x50 [ 4532.065110] do_syscall_64+0xc1/0x440 [ 4532.065117] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4532.065126] RIP: 0033:0x7f151bdee10b [ 4532.065133] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4532.065137] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4532.065143] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 4532.065145] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 4532.065147] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4532.065149] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4532.065152] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4532.332367] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4532.374340] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4532.382015] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4532.386471] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4532.392252] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4532.402578] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4532.409657] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 4532.425368] RDX: 0000000000001e18 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 4532.430639] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 4532.440753] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 4532.446702] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4532.451925] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 4532.455356] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4532.462501] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4532.465892] Call Trace: [ 4532.466595] [ 4532.467221] ? show_regs.cold.9+0x22/0x2f [ 4532.471435] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4532.474926] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4532.478393] ? watchdog+0x40/0x40 [ 4532.479721] ? hrtimer_interrupt+0x128/0x2c0 [ 4532.481239] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4532.482979] ? apic_timer_interrupt+0xf/0x20 [ 4532.484540] [ 4532.486710] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4532.492972] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4532.497898] do_raw_spin_lock+0x177/0x190 [ 4532.504313] _raw_spin_lock+0x21/0x30 [ 4532.506563] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4532.518306] obd_mod_free+0x46/0x140 [ptlrpc] [ 4532.519879] mdc_commit_open+0x91/0xb0 [mdc] [ 4532.521197] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4532.528707] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4532.532530] after_reply+0xe2c/0x1780 [ptlrpc] [ 4532.538372] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4532.545929] ? __next_timer_interrupt+0x160/0x160 [ 4532.548631] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4532.550624] ? do_wait_intr+0xf0/0xf0 [ 4532.554182] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4532.556387] kthread+0x1d1/0x200 [ 4532.559515] ? set_kthread_struct+0x70/0x70 [ 4532.562088] ret_from_fork+0x1f/0x30 [ 4533.218858] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2602s! [ 4533.224141] Showing busy workqueues and worker pools: [ 4533.229537] workqueue events: flags=0x0 [ 4533.231330] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4533.234774] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 4533.239979] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4533.243796] pending: kfree_rcu_monitor [ 4533.246466] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4533.250786] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 4533.259316] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4533.265886] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4533.272071] workqueue events_long: flags=0x0 [ 4533.277448] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4533.284388] pending: lu_objects_flush [obdclass] [ 4533.287230] workqueue events_unbound: flags=0x2 [ 4533.289360] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4533.293508] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4533.299839] workqueue events_power_efficient: flags=0x80 [ 4533.307111] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4533.310848] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 4533.314669] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4533.317907] pending: neigh_periodic_work [ 4533.324301] workqueue mm_percpu_wq: flags=0x8 [ 4533.326524] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4533.333444] pending: vmstat_update, lru_add_drain_per_cpu [ 4533.341330] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4533.347926] pending: vmstat_update [ 4533.350927] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4533.354971] pending: lru_add_drain_per_cpu [ 4533.357790] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4533.362982] pending: lru_add_drain_per_cpu BAR(48) [ 4533.365667] workqueue writeback: flags=0x4a [ 4533.369034] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4533.376649] pending: wb_workfn [ 4533.378060] workqueue xprtiod: flags=0xa [ 4533.380357] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4533.382792] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4533.387827] workqueue ipv6_addrconf: flags=0x40008 [ 4533.390290] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4533.394544] pending: addrconf_verify_work [ 4533.397696] workqueue knbd0-recv: flags=0x1a [ 4533.399486] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4533.402930] in-flight: 658:recv_work [ 4533.405211] workqueue ptlrpc_pinger: flags=0x2 [ 4533.407157] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4533.411756] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4533.414696] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2602s workers=4 idle: 32984 37 8 [ 4533.419908] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4518s workers=2 idle: 661 [ 4536.057030] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 4536.065133] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4536.108007] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4536.120212] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4536.126896] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4536.129459] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4536.159652] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4536.166612] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4536.177980] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 4536.181909] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4536.186598] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 4536.197590] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 4536.208004] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 4536.214695] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4536.218585] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 4536.222601] Call Trace: [ 4536.224491] [ 4536.225802] ? show_regs.cold.9+0x22/0x2f [ 4536.232142] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4536.238544] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4536.242878] ? watchdog+0x40/0x40 [ 4536.247016] ? hrtimer_interrupt+0x128/0x2c0 [ 4536.251221] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4536.254594] ? apic_timer_interrupt+0xf/0x20 [ 4536.258554] [ 4536.260217] ? native_safe_halt+0x2e/0x50 [ 4536.262008] kvm_wait+0xcd/0xe0 [ 4536.264026] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4536.266695] do_raw_spin_lock+0x177/0x190 [ 4536.268200] _raw_spin_lock+0x21/0x30 [ 4536.269221] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4536.271119] ? do_raw_spin_unlock+0x75/0x190 [ 4536.272449] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4536.274651] ? do_wait_intr+0xf0/0xf0 [ 4536.276171] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4536.278217] mdc_close+0x53b/0xd70 [mdc] [ 4536.279844] lmv_close+0x18d/0x490 [lmv] [ 4536.281526] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4536.283858] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4536.286238] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4536.288409] ll_md_close+0x274/0x800 [lustre] [ 4536.290342] ll_file_release+0x128/0xc50 [lustre] [ 4536.292667] __fput+0x10d/0x440 [ 4536.293853] ____fput+0x12/0x20 [ 4536.295521] task_work_run+0xc6/0x110 [ 4536.296935] exit_to_usermode_loop+0x1dd/0x1f0 [ 4536.298707] do_syscall_64+0x3d6/0x440 [ 4536.305831] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4536.308751] RIP: 0033:0x7f151bee3cd8 [ 4536.310922] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4536.325662] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4536.334876] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 4536.341637] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 4536.347873] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4536.354501] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4536.358249] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4552.056592] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4552.064561] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4552.100164] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4552.109688] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4552.118912] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4552.124698] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4552.128332] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4552.155399] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4552.158801] RAX: 0000000000002073 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4552.164622] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 4552.168499] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 4552.175323] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 4552.179137] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 4552.185565] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 4552.190540] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4552.193559] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4552.198305] Call Trace: [ 4552.199571] [ 4552.205524] ? show_regs.cold.9+0x22/0x2f [ 4552.208063] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4552.210688] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4552.214489] ? watchdog+0x40/0x40 [ 4552.217116] ? hrtimer_interrupt+0x128/0x2c0 [ 4552.219222] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4552.223724] ? apic_timer_interrupt+0xf/0x20 [ 4552.227815] [ 4552.229472] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4552.232332] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4552.235301] do_raw_spin_lock+0x177/0x190 [ 4552.237851] _raw_spin_lock+0x21/0x30 [ 4552.239702] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4552.242575] process_one_work+0x2c8/0x700 [ 4552.244747] worker_thread+0x296/0x6e0 [ 4552.246553] ? rescuer_thread+0x570/0x570 [ 4552.248273] kthread+0x1d1/0x200 [ 4552.253280] ? set_kthread_struct+0x70/0x70 [ 4552.256374] ret_from_fork+0x1f/0x30 [ 4560.052027] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 4560.055008] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 4560.057057] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) [ 4560.061370] Modules linked in: [ 4560.061373] ksocklnd(O) lnet(O) [ 4560.071718] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) [ 4560.075275] libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4560.075318] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4560.075326] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4560.075328] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4560.075341] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4560.075345] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4560.075358] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4560.075360] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 4560.075362] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 4560.076552] fld(O) ptlrpc_gss(O) ptlrpc(O) [ 4560.086973] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 4560.086982] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 4560.086987] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 4560.086990] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4560.086992] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 4560.087001] Call Trace: [ 4560.098511] obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4560.098563] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4560.098571] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4560.098574] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4560.098587] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4560.098603] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4560.098608] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 4560.098610] RDX: 00000000000041e2 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 4560.104685] [ 4560.104698] ? show_regs.cold.9+0x22/0x2f [ 4560.104720] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4560.104727] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4560.104732] ? watchdog+0x40/0x40 [ 4560.104736] ? hrtimer_interrupt+0x128/0x2c0 [ 4560.104740] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4560.104754] ? apic_timer_interrupt+0xf/0x20 [ 4560.104759] [ 4560.104762] ? native_safe_halt+0x2e/0x50 [ 4560.104766] kvm_wait+0xcd/0xe0 [ 4560.104773] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4560.104781] do_raw_spin_lock+0x177/0x190 [ 4560.104785] _raw_spin_lock+0x21/0x30 [ 4560.104792] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4560.107571] ? do_raw_spin_unlock+0x75/0x190 [ 4560.107581] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4560.108035] ? do_wait_intr+0xf0/0xf0 [ 4560.108044] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4560.108239] mdc_reint+0x2d/0x110 [mdc] [ 4560.108275] mdc_unlink+0x658/0x870 [mdc] [ 4560.108301] lmv_unlink+0x34d/0xbd0 [lmv] [ 4560.108331] ll_unlink+0x24d/0x7f0 [lustre] [ 4560.108465] vfs_unlink+0x202/0x330 [ 4560.108472] do_unlinkat+0x2fa/0x460 [ 4560.108478] __x64_sys_unlink+0x37/0x50 [ 4560.108481] do_syscall_64+0xc1/0x440 [ 4560.108487] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4560.108496] RIP: 0033:0x7f151bdee10b [ 4560.108503] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4560.108507] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4560.108512] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 4560.108514] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 4560.108517] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4560.108519] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4560.108521] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4560.302110] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 4560.304254] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 4560.306378] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4560.308384] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 4560.311698] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4560.316777] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4560.318919] Call Trace: [ 4560.319572] [ 4560.322398] ? show_regs.cold.9+0x22/0x2f [ 4560.327980] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4560.334757] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4560.336480] ? watchdog+0x40/0x40 [ 4560.338410] ? hrtimer_interrupt+0x128/0x2c0 [ 4560.340482] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4560.346667] ? apic_timer_interrupt+0xf/0x20 [ 4560.349732] [ 4560.350744] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4560.356328] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4560.360469] do_raw_spin_lock+0x177/0x190 [ 4560.361637] _raw_spin_lock+0x21/0x30 [ 4560.362675] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4560.366745] obd_mod_free+0x46/0x140 [ptlrpc] [ 4560.373133] mdc_commit_open+0x91/0xb0 [mdc] [ 4560.376603] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4560.379914] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4560.398787] after_reply+0xe2c/0x1780 [ptlrpc] [ 4560.404171] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4560.408710] ? __next_timer_interrupt+0x160/0x160 [ 4560.417721] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4560.426388] ? do_wait_intr+0xf0/0xf0 [ 4560.431669] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4560.436014] kthread+0x1d1/0x200 [ 4560.438088] ? set_kthread_struct+0x70/0x70 [ 4560.442413] ret_from_fork+0x1f/0x30 [ 4563.937259] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2633s! [ 4563.943755] Showing busy workqueues and worker pools: [ 4563.947871] workqueue events: flags=0x0 [ 4563.950129] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4563.953289] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 4563.960938] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4563.966536] pending: kfree_rcu_monitor [ 4563.969323] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4563.974017] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 4563.982863] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4563.988673] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4563.993338] workqueue events_long: flags=0x0 [ 4563.995551] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4564.000840] pending: lu_objects_flush [obdclass] [ 4564.004608] workqueue events_unbound: flags=0x2 [ 4564.008047] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4564.015859] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4564.024213] workqueue events_power_efficient: flags=0x80 [ 4564.027900] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4564.033370] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 4564.037091] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4564.040834] pending: neigh_periodic_work [ 4564.043075] workqueue mm_percpu_wq: flags=0x8 [ 4564.045127] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4564.048553] pending: vmstat_update, lru_add_drain_per_cpu [ 4564.051642] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4564.055838] pending: vmstat_update [ 4564.057035] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 4564.061939] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4564.061962] pending: lru_add_drain_per_cpu [ 4564.061977] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4564.061985] pending: lru_add_drain_per_cpu BAR(48) [ 4564.062056] workqueue writeback: flags=0x4a [ 4564.062061] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4564.062069] pending: wb_workfn [ 4564.062090] workqueue xprtiod: flags=0xa [ 4564.062092] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4564.062098] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4564.062118] workqueue ipv6_addrconf: flags=0x40008 [ 4564.062121] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4564.062129] pending: addrconf_verify_work [ 4564.062142] workqueue knbd0-recv: flags=0x1a [ 4564.062145] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4564.062153] in-flight: 658:recv_work [ 4564.062300] workqueue ptlrpc_pinger: flags=0x2 [ 4564.062305] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4564.062311] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4564.062599] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2633s workers=4 idle: 32984 37 8 [ 4564.062611] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4548s workers=2 idle: 661 [ 4564.131334] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4564.153386] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4564.157256] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4564.160229] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4564.162444] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4564.169766] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4564.174117] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4564.176223] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 4564.178213] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4564.180451] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 4564.182983] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 4564.185295] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 4564.187791] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4564.189540] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 4564.191630] Call Trace: [ 4564.192437] [ 4564.193105] ? show_regs.cold.9+0x22/0x2f [ 4564.195257] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4564.196704] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4564.198122] ? watchdog+0x40/0x40 [ 4564.199116] ? hrtimer_interrupt+0x128/0x2c0 [ 4564.200394] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4564.202432] ? apic_timer_interrupt+0xf/0x20 [ 4564.204090] [ 4564.204639] ? native_safe_halt+0x2e/0x50 [ 4564.206445] kvm_wait+0xcd/0xe0 [ 4564.207329] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4564.208929] do_raw_spin_lock+0x177/0x190 [ 4564.210349] _raw_spin_lock+0x21/0x30 [ 4564.211706] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4564.214149] ? do_raw_spin_unlock+0x75/0x190 [ 4564.215611] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4564.218326] ? do_wait_intr+0xf0/0xf0 [ 4564.219864] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4564.222301] mdc_close+0x53b/0xd70 [mdc] [ 4564.223590] lmv_close+0x18d/0x490 [lmv] [ 4564.224934] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4564.226769] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4564.228693] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4564.230616] ll_md_close+0x274/0x800 [lustre] [ 4564.232333] ll_file_release+0x128/0xc50 [lustre] [ 4564.234245] __fput+0x10d/0x440 [ 4564.235311] ____fput+0x12/0x20 [ 4564.236569] task_work_run+0xc6/0x110 [ 4564.238259] exit_to_usermode_loop+0x1dd/0x1f0 [ 4564.240115] do_syscall_64+0x3d6/0x440 [ 4564.241504] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4564.243344] RIP: 0033:0x7f151bee3cd8 [ 4564.244506] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4564.250499] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4564.253636] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 4564.256073] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 4564.258310] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4564.260456] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4564.263465] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4580.057119] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4580.064945] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4580.108178] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4580.114644] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4580.123584] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4580.130868] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4580.133776] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4580.153106] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4580.159196] RAX: 0000000000007080 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4580.163207] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 4580.169586] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 4580.178460] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 4580.184918] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 4580.191300] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 4580.197783] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4580.199367] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4580.205576] Call Trace: [ 4580.206541] [ 4580.210557] ? show_regs.cold.9+0x22/0x2f [ 4580.212261] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4580.216893] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4580.220631] ? watchdog+0x40/0x40 [ 4580.222978] ? hrtimer_interrupt+0x128/0x2c0 [ 4580.224517] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4580.229355] ? apic_timer_interrupt+0xf/0x20 [ 4580.230837] [ 4580.231422] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4580.234351] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4580.236564] do_raw_spin_lock+0x177/0x190 [ 4580.238558] _raw_spin_lock+0x21/0x30 [ 4580.240066] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4580.242624] process_one_work+0x2c8/0x700 [ 4580.244778] worker_thread+0x296/0x6e0 [ 4580.246247] ? rescuer_thread+0x570/0x570 [ 4580.248208] kthread+0x1d1/0x200 [ 4580.249807] ? set_kthread_struct+0x70/0x70 [ 4580.251580] ret_from_fork+0x1f/0x30 [ 4588.055105] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 4588.059903] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4588.107995] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4588.117179] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4588.124610] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4588.128529] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4588.138150] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4588.141220] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 4588.143633] RDX: 0000000000002572 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 4588.148048] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 4588.155463] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 4588.162388] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4588.167741] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 4588.173314] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4588.178107] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4588.184479] Call Trace: [ 4588.187150] [ 4588.187950] ? show_regs.cold.9+0x22/0x2f [ 4588.189441] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4588.194268] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4588.196753] ? watchdog+0x40/0x40 [ 4588.201346] ? hrtimer_interrupt+0x128/0x2c0 [ 4588.210153] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4588.213991] ? apic_timer_interrupt+0xf/0x20 [ 4588.216410] [ 4588.218004] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4588.222953] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4588.233142] do_raw_spin_lock+0x177/0x190 [ 4588.236878] _raw_spin_lock+0x21/0x30 [ 4588.240925] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4588.246215] obd_mod_free+0x46/0x140 [ptlrpc] [ 4588.249739] mdc_commit_open+0x91/0xb0 [mdc] [ 4588.251313] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4588.257769] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4588.262044] after_reply+0xe2c/0x1780 [ptlrpc] [ 4588.266632] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4588.273901] ? __next_timer_interrupt+0x160/0x160 [ 4588.285287] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4588.288109] ? do_wait_intr+0xf0/0xf0 [ 4588.291075] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4588.300914] kthread+0x1d1/0x200 [ 4588.305820] ? set_kthread_struct+0x70/0x70 [ 4588.312414] ret_from_fork+0x1f/0x30 [ 4592.052104] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 4592.055362] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4592.058039] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 4592.077292] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4592.077307] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4592.077310] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4592.077325] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4592.077329] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4592.077336] RAX: 0000000000001645 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4592.077339] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 4592.077341] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 4592.077344] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 4592.077347] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 4592.077351] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 4592.077354] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4592.077357] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 4592.077366] Call Trace: [ 4592.077373] [ 4592.077379] ? show_regs.cold.9+0x22/0x2f [ 4592.077389] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4592.077396] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4592.077400] ? watchdog+0x40/0x40 [ 4592.077404] ? hrtimer_interrupt+0x128/0x2c0 [ 4592.077409] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4592.077418] ? apic_timer_interrupt+0xf/0x20 [ 4592.077423] [ 4592.077426] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4592.077432] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4592.077436] do_raw_spin_lock+0x177/0x190 [ 4592.077441] _raw_spin_lock+0x21/0x30 [ 4592.077449] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4592.078032] ? do_raw_spin_unlock+0x75/0x190 [ 4592.078038] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4592.078231] ? do_wait_intr+0xf0/0xf0 [ 4592.078238] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4592.078448] mdc_reint+0x2d/0x110 [mdc] [ 4592.078488] mdc_unlink+0x658/0x870 [mdc] [ 4592.078530] lmv_unlink+0x34d/0xbd0 [lmv] [ 4592.078561] ll_unlink+0x24d/0x7f0 [lustre] [ 4592.078686] vfs_unlink+0x202/0x330 [ 4592.078694] do_unlinkat+0x2fa/0x460 [ 4592.078698] __x64_sys_unlink+0x37/0x50 [ 4592.078701] do_syscall_64+0xc1/0x440 [ 4592.078708] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4592.078715] RIP: 0033:0x7f151bdee10b [ 4592.078722] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4592.078726] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4592.078730] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 4592.078732] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 4592.078734] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4592.078736] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4592.078739] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4592.397773] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4592.450495] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4592.467158] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4592.472386] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4592.481567] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4592.498692] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4592.504854] RAX: 0000000000007c31 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4592.513747] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 4592.524631] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4592.533473] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 4592.541692] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 4592.545860] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 4592.555230] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4592.558366] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 4592.563708] Call Trace: [ 4592.566222] [ 4592.567430] ? show_regs.cold.9+0x22/0x2f [ 4592.570232] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4592.572811] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4592.575740] ? watchdog+0x40/0x40 [ 4592.577435] ? hrtimer_interrupt+0x128/0x2c0 [ 4592.580136] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4592.582478] ? apic_timer_interrupt+0xf/0x20 [ 4592.585353] [ 4592.587204] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4592.590146] do_raw_spin_lock+0x177/0x190 [ 4592.592258] _raw_spin_lock+0x21/0x30 [ 4592.594256] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4592.598181] ? do_raw_spin_unlock+0x75/0x190 [ 4592.602916] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4592.608621] ? do_wait_intr+0xf0/0xf0 [ 4592.610965] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4592.624879] mdc_close+0x53b/0xd70 [mdc] [ 4592.630460] lmv_close+0x18d/0x490 [lmv] [ 4592.635543] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4592.649598] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4592.656299] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4592.667452] ll_md_close+0x274/0x800 [lustre] [ 4592.675569] ll_file_release+0x128/0xc50 [lustre] [ 4592.681600] __fput+0x10d/0x440 [ 4592.687694] ____fput+0x12/0x20 [ 4592.694487] task_work_run+0xc6/0x110 [ 4592.698301] exit_to_usermode_loop+0x1dd/0x1f0 [ 4592.707425] do_syscall_64+0x3d6/0x440 [ 4592.715468] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4592.730084] RIP: 0033:0x7f151bee3cd8 [ 4592.734491] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4592.749799] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4592.753355] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 4592.761854] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 4592.769696] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4592.784072] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4592.794285] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4594.145534] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2663s! [ 4594.149173] Showing busy workqueues and worker pools: [ 4594.151284] workqueue events: flags=0x0 [ 4594.153146] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4594.156143] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 4594.160515] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4594.163359] pending: kfree_rcu_monitor [ 4594.165051] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4594.167792] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 4594.172588] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4594.175891] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4594.179368] workqueue events_long: flags=0x0 [ 4594.181637] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4594.184293] pending: lu_objects_flush [obdclass] [ 4594.186791] workqueue events_unbound: flags=0x2 [ 4594.188579] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4594.191349] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4594.196109] workqueue events_power_efficient: flags=0x80 [ 4594.198318] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4594.201348] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 4594.204584] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4594.207648] pending: neigh_periodic_work [ 4594.209312] workqueue mm_percpu_wq: flags=0x8 [ 4594.211194] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4594.214113] pending: vmstat_update, lru_add_drain_per_cpu [ 4594.216556] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4594.219245] pending: vmstat_update [ 4594.221118] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4594.233829] pending: lru_add_drain_per_cpu [ 4594.236354] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4594.241352] pending: lru_add_drain_per_cpu BAR(48) [ 4594.246044] workqueue writeback: flags=0x4a [ 4594.248311] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4594.250672] pending: wb_workfn [ 4594.251982] workqueue xprtiod: flags=0xa [ 4594.253801] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4594.256778] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4594.261018] workqueue ipv6_addrconf: flags=0x40008 [ 4594.265131] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4594.272659] pending: addrconf_verify_work [ 4594.279151] workqueue knbd0-recv: flags=0x1a [ 4594.280400] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4594.285169] in-flight: 658:recv_work [ 4594.288467] workqueue ptlrpc_pinger: flags=0x2 [ 4594.294120] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4594.297603] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4594.301154] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2663s workers=4 idle: 32984 37 8 [ 4594.304661] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4579s workers=2 idle: 661 [ 4608.056016] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4608.058667] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4608.108608] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4608.116067] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4608.132987] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4608.147929] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4608.156258] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4608.176097] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4608.183969] RAX: 0000000000006c99 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4608.193742] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 4608.201208] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 4608.216305] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 4608.233092] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 4608.241849] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 4608.251175] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4608.259216] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4608.262987] Call Trace: [ 4608.264335] [ 4608.272192] ? show_regs.cold.9+0x22/0x2f [ 4608.276161] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4608.282164] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4608.288802] ? watchdog+0x40/0x40 [ 4608.291139] ? hrtimer_interrupt+0x128/0x2c0 [ 4608.297499] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4608.300934] ? apic_timer_interrupt+0xf/0x20 [ 4608.308445] [ 4608.309739] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4608.315220] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4608.319687] do_raw_spin_lock+0x177/0x190 [ 4608.323856] _raw_spin_lock+0x21/0x30 [ 4608.328186] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4608.335030] process_one_work+0x2c8/0x700 [ 4608.342184] worker_thread+0x296/0x6e0 [ 4608.347279] ? rescuer_thread+0x570/0x570 [ 4608.348910] kthread+0x1d1/0x200 [ 4608.350063] ? set_kthread_struct+0x70/0x70 [ 4608.359849] ret_from_fork+0x1f/0x30 [ 4616.055993] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 4616.066242] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4616.114312] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4616.125448] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4616.134570] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4616.136292] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4616.161357] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4616.166799] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 4616.172622] RDX: 0000000000002641 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 4616.181768] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 4616.187371] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 4616.193957] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4616.200035] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 4616.211713] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4616.220753] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4616.227624] Call Trace: [ 4616.230131] [ 4616.231896] ? show_regs.cold.9+0x22/0x2f [ 4616.235612] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4616.240066] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4616.244400] ? watchdog+0x40/0x40 [ 4616.246244] ? hrtimer_interrupt+0x128/0x2c0 [ 4616.250479] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4616.253055] ? apic_timer_interrupt+0xf/0x20 [ 4616.255937] [ 4616.258331] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4616.266166] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4616.274209] do_raw_spin_lock+0x177/0x190 [ 4616.277710] _raw_spin_lock+0x21/0x30 [ 4616.282944] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4616.289984] obd_mod_free+0x46/0x140 [ptlrpc] [ 4616.293950] mdc_commit_open+0x91/0xb0 [mdc] [ 4616.302297] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4616.316510] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4616.322437] after_reply+0xe2c/0x1780 [ptlrpc] [ 4616.329297] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4616.344530] ? __next_timer_interrupt+0x160/0x160 [ 4616.354667] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4616.363663] ? do_wait_intr+0xf0/0xf0 [ 4616.369177] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4616.376972] kthread+0x1d1/0x200 [ 4616.386162] ? set_kthread_struct+0x70/0x70 [ 4616.392597] ret_from_fork+0x1f/0x30 [ 4620.053166] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 4620.056960] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4620.058138] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 4620.079014] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4620.079033] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4620.079036] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4620.079051] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4620.079056] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4620.079062] RAX: 0000000000006a89 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4620.079064] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 4620.079067] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 4620.079069] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 4620.079071] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 4620.079075] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 4620.079077] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4620.079079] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 4620.079089] Call Trace: [ 4620.079100] [ 4620.079106] ? show_regs.cold.9+0x22/0x2f [ 4620.079116] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4620.079123] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4620.079128] ? watchdog+0x40/0x40 [ 4620.079133] ? hrtimer_interrupt+0x128/0x2c0 [ 4620.079137] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4620.079150] ? apic_timer_interrupt+0xf/0x20 [ 4620.079155] [ 4620.079159] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4620.079164] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4620.079168] do_raw_spin_lock+0x177/0x190 [ 4620.079173] _raw_spin_lock+0x21/0x30 [ 4620.079184] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4620.079520] ? do_raw_spin_unlock+0x75/0x190 [ 4620.079526] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4620.088571] Modules linked in: [ 4620.093919] ? do_wait_intr+0xf0/0xf0 [ 4620.093940] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4620.094450] mdc_reint+0x2d/0x110 [mdc] [ 4620.094495] mdc_unlink+0x658/0x870 [mdc] [ 4620.094524] lmv_unlink+0x34d/0xbd0 [lmv] [ 4620.094565] ll_unlink+0x24d/0x7f0 [lustre] [ 4620.094704] vfs_unlink+0x202/0x330 [ 4620.094714] do_unlinkat+0x2fa/0x460 [ 4620.094720] __x64_sys_unlink+0x37/0x50 [ 4620.094724] do_syscall_64+0xc1/0x440 [ 4620.094730] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4620.094741] RIP: 0033:0x7f151bdee10b [ 4620.094749] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4620.094757] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4620.094762] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 4620.094765] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 4620.094767] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4620.094769] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4620.094771] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4620.357915] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4620.425997] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4620.440136] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4620.446742] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4620.450350] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4620.458783] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4620.462006] RAX: 0000000000007d9c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4620.465310] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 4620.468510] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4620.471810] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 4620.477552] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 4620.481974] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 4620.486055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4620.488427] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 4620.491598] Call Trace: [ 4620.492850] [ 4620.493674] ? show_regs.cold.9+0x22/0x2f [ 4620.496782] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4620.505563] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4620.511129] ? watchdog+0x40/0x40 [ 4620.516514] ? hrtimer_interrupt+0x128/0x2c0 [ 4620.519245] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4620.522287] ? apic_timer_interrupt+0xf/0x20 [ 4620.528009] [ 4620.529671] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4620.535187] do_raw_spin_lock+0x177/0x190 [ 4620.539893] _raw_spin_lock+0x21/0x30 [ 4620.545876] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4620.552483] ? do_raw_spin_unlock+0x75/0x190 [ 4620.554056] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4620.559605] ? do_wait_intr+0xf0/0xf0 [ 4620.560933] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4620.575649] mdc_close+0x53b/0xd70 [mdc] [ 4620.580700] lmv_close+0x18d/0x490 [lmv] [ 4620.584572] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4620.590274] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4620.593628] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4620.596179] ll_md_close+0x274/0x800 [lustre] [ 4620.599509] ll_file_release+0x128/0xc50 [lustre] [ 4620.601265] __fput+0x10d/0x440 [ 4620.607585] ____fput+0x12/0x20 [ 4620.612241] task_work_run+0xc6/0x110 [ 4620.614082] exit_to_usermode_loop+0x1dd/0x1f0 [ 4620.619470] do_syscall_64+0x3d6/0x440 [ 4620.623198] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4620.626881] RIP: 0033:0x7f151bee3cd8 [ 4620.631117] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4620.648965] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4620.656502] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 4620.661601] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 4620.665857] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4620.670176] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4620.675175] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4624.358750] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2693s! [ 4624.369542] Showing busy workqueues and worker pools: [ 4624.378797] workqueue events: flags=0x0 [ 4624.386662] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4624.397505] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 4624.410560] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4624.422586] pending: kfree_rcu_monitor [ 4624.423705] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4624.430899] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 4624.441659] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4624.450887] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4624.460176] workqueue events_long: flags=0x0 [ 4624.467473] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4624.473899] pending: lu_objects_flush [obdclass] [ 4624.480770] workqueue events_unbound: flags=0x2 [ 4624.485830] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4624.491911] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4624.498102] workqueue events_power_efficient: flags=0x80 [ 4624.500640] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4624.505271] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 4624.508485] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4624.511610] pending: neigh_periodic_work [ 4624.514460] workqueue mm_percpu_wq: flags=0x8 [ 4624.516870] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4624.521144] pending: vmstat_update, lru_add_drain_per_cpu [ 4624.523877] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4624.530798] pending: vmstat_update [ 4624.538078] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4624.548632] pending: lru_add_drain_per_cpu [ 4624.553496] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4624.562258] pending: lru_add_drain_per_cpu BAR(48) [ 4624.564099] workqueue writeback: flags=0x4a [ 4624.576579] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4624.582915] pending: wb_workfn [ 4624.589877] workqueue xprtiod: flags=0xa [ 4624.592616] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4624.595797] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4624.604141] workqueue ipv6_addrconf: flags=0x40008 [ 4624.609493] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4624.615458] pending: addrconf_verify_work [ 4624.623530] workqueue knbd0-recv: flags=0x1a [ 4624.626130] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4624.630920] in-flight: 658:recv_work [ 4624.632846] workqueue ptlrpc_pinger: flags=0x2 [ 4624.634900] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4624.637680] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4624.640435] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2694s workers=4 idle: 32984 37 8 [ 4624.643144] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4609s workers=2 idle: 661 [ 4636.058383] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4636.071645] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4636.121943] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4636.126793] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4636.137754] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4636.143502] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4636.148918] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4636.160132] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4636.164697] RAX: 0000000000007185 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4636.171272] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 4636.174930] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 4636.178991] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 4636.183031] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 4636.186731] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 4636.195942] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4636.199069] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4636.203306] Call Trace: [ 4636.204451] [ 4636.205834] ? show_regs.cold.9+0x22/0x2f [ 4636.207698] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4636.211597] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4636.213493] ? watchdog+0x40/0x40 [ 4636.215393] ? hrtimer_interrupt+0x128/0x2c0 [ 4636.216999] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4636.218969] ? apic_timer_interrupt+0xf/0x20 [ 4636.221069] [ 4636.222375] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4636.224831] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4636.227110] do_raw_spin_lock+0x177/0x190 [ 4636.230936] _raw_spin_lock+0x21/0x30 [ 4636.233754] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4636.237372] process_one_work+0x2c8/0x700 [ 4636.238939] worker_thread+0x296/0x6e0 [ 4636.241104] ? rescuer_thread+0x570/0x570 [ 4636.243237] kthread+0x1d1/0x200 [ 4636.244801] ? set_kthread_struct+0x70/0x70 [ 4636.246866] ret_from_fork+0x1f/0x30 [ 4644.056364] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 4644.062214] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4644.089612] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4644.095057] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4644.099514] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4644.106709] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4644.129223] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4644.139175] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 4644.146322] RDX: 0000000000004fd9 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 4644.153706] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 4644.157584] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 4644.161588] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4644.165118] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 4644.168898] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4644.173625] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4644.178259] Call Trace: [ 4644.179257] [ 4644.180012] ? show_regs.cold.9+0x22/0x2f [ 4644.183423] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4644.186801] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4644.189778] ? watchdog+0x40/0x40 [ 4644.191888] ? hrtimer_interrupt+0x128/0x2c0 [ 4644.195580] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4644.199896] ? apic_timer_interrupt+0xf/0x20 [ 4644.202912] [ 4644.204581] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4644.208491] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4644.212338] do_raw_spin_lock+0x177/0x190 [ 4644.213852] _raw_spin_lock+0x21/0x30 [ 4644.216168] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4644.220210] obd_mod_free+0x46/0x140 [ptlrpc] [ 4644.230078] mdc_commit_open+0x91/0xb0 [mdc] [ 4644.241295] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4644.248466] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4644.254899] after_reply+0xe2c/0x1780 [ptlrpc] [ 4644.269602] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4644.271869] ? __next_timer_interrupt+0x160/0x160 [ 4644.273509] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4644.280695] ? do_wait_intr+0xf0/0xf0 [ 4644.284296] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4644.288759] kthread+0x1d1/0x200 [ 4644.290493] ? set_kthread_struct+0x70/0x70 [ 4644.293736] ret_from_fork+0x1f/0x30 [ 4648.057057] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 4648.060728] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4648.096674] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4648.105317] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4648.109498] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4648.111781] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4648.120621] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4648.123781] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4648.127326] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 4648.130304] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4648.133290] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 4648.137564] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 4648.146294] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 4648.152251] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4648.155813] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 4648.159521] Call Trace: [ 4648.161018] [ 4648.161739] ? show_regs.cold.9+0x22/0x2f [ 4648.163861] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4648.166006] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4648.172295] ? watchdog+0x40/0x40 [ 4648.175899] ? hrtimer_interrupt+0x128/0x2c0 [ 4648.178224] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4648.183421] ? apic_timer_interrupt+0xf/0x20 [ 4648.188519] [ 4648.189300] ? native_safe_halt+0x2e/0x50 [ 4648.190683] kvm_wait+0xcd/0xe0 [ 4648.197035] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4648.198851] do_raw_spin_lock+0x177/0x190 [ 4648.203076] _raw_spin_lock+0x21/0x30 [ 4648.207455] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4648.211922] ? do_raw_spin_unlock+0x75/0x190 [ 4648.214618] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4648.216944] ? do_wait_intr+0xf0/0xf0 [ 4648.218603] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4648.221211] mdc_close+0x53b/0xd70 [mdc] [ 4648.224531] lmv_close+0x18d/0x490 [lmv] [ 4648.227521] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4648.230669] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4648.236757] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4648.241820] ll_md_close+0x274/0x800 [lustre] [ 4648.245244] ll_file_release+0x128/0xc50 [lustre] [ 4648.248559] __fput+0x10d/0x440 [ 4648.250778] ____fput+0x12/0x20 [ 4648.255684] task_work_run+0xc6/0x110 [ 4648.257627] exit_to_usermode_loop+0x1dd/0x1f0 [ 4648.261930] do_syscall_64+0x3d6/0x440 [ 4648.265590] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4648.268204] RIP: 0033:0x7f151bee3cd8 [ 4648.271640] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4648.289169] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4648.296886] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 4648.301633] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 4648.305295] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4648.312795] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4648.317551] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4652.057165] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 4652.063082] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4652.092924] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4652.098297] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4652.102333] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4652.106490] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4652.121035] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 4652.133917] RAX: 0000000000001f17 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4652.151225] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 4652.158215] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 4652.166417] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 4652.172658] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 4652.178262] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 4652.181424] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4652.185566] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 4652.189229] Call Trace: [ 4652.190434] [ 4652.191454] ? show_regs.cold.9+0x22/0x2f [ 4652.193496] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4652.195800] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4652.198302] ? watchdog+0x40/0x40 [ 4652.201879] ? hrtimer_interrupt+0x128/0x2c0 [ 4652.204693] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4652.208705] ? apic_timer_interrupt+0xf/0x20 [ 4652.210837] [ 4652.212973] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4652.214788] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4652.218571] do_raw_spin_lock+0x177/0x190 [ 4652.220814] _raw_spin_lock+0x21/0x30 [ 4652.223558] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4652.232044] ? do_raw_spin_unlock+0x75/0x190 [ 4652.233693] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4652.238755] ? do_wait_intr+0xf0/0xf0 [ 4652.244433] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4652.246545] mdc_reint+0x2d/0x110 [mdc] [ 4652.249622] mdc_unlink+0x658/0x870 [mdc] [ 4652.251908] lmv_unlink+0x34d/0xbd0 [lmv] [ 4652.254402] ll_unlink+0x24d/0x7f0 [lustre] [ 4652.258887] vfs_unlink+0x202/0x330 [ 4652.260302] do_unlinkat+0x2fa/0x460 [ 4652.261659] __x64_sys_unlink+0x37/0x50 [ 4652.263991] do_syscall_64+0xc1/0x440 [ 4652.265916] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4652.275451] RIP: 0033:0x7f151bdee10b [ 4652.281431] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4652.304869] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4652.310901] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 4652.314423] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 4652.317741] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4652.321184] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4652.324444] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4655.072451] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2724s! [ 4655.083707] Showing busy workqueues and worker pools: [ 4655.085808] workqueue events: flags=0x0 [ 4655.088145] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4655.099763] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 4655.107099] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4655.111838] pending: kfree_rcu_monitor [ 4655.116812] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4655.124402] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 4655.133839] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4655.139601] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4655.147542] workqueue events_long: flags=0x0 [ 4655.151776] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4655.158478] pending: lu_objects_flush [obdclass] [ 4655.165692] workqueue events_unbound: flags=0x2 [ 4655.174812] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4655.178799] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4655.184510] workqueue events_power_efficient: flags=0x80 [ 4655.192241] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4655.195872] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 4655.198793] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4655.210542] pending: neigh_periodic_work [ 4655.215173] workqueue mm_percpu_wq: flags=0x8 [ 4655.220170] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4655.222467] pending: vmstat_update, lru_add_drain_per_cpu [ 4655.225079] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4655.229207] pending: vmstat_update [ 4655.230806] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4655.234335] pending: lru_add_drain_per_cpu [ 4655.237234] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4655.241216] pending: lru_add_drain_per_cpu BAR(48) [ 4655.244762] workqueue writeback: flags=0x4a [ 4655.251833] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4655.256644] pending: wb_workfn [ 4655.260851] workqueue xprtiod: flags=0xa [ 4655.264460] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4655.271493] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4655.277084] workqueue ipv6_addrconf: flags=0x40008 [ 4655.279702] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4655.282639] pending: addrconf_verify_work [ 4655.284825] workqueue knbd0-recv: flags=0x1a [ 4655.286765] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4655.289740] in-flight: 658:recv_work [ 4655.291583] workqueue ptlrpc_pinger: flags=0x2 [ 4655.293808] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4655.296693] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4655.305797] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2724s workers=4 idle: 32984 37 8 [ 4655.312983] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4640s workers=2 idle: 661 [ 4664.056059] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4664.063712] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4664.117609] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4664.130660] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4664.134994] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4664.150594] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4664.154322] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4664.177091] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4664.188208] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4664.198069] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 4664.206395] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 4664.212713] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 4664.226494] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 4664.237423] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 4664.255909] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4664.263178] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4664.277379] Call Trace: [ 4664.278312] [ 4664.283862] ? show_regs.cold.9+0x22/0x2f [ 4664.286053] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4664.288433] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4664.292801] ? watchdog+0x40/0x40 [ 4664.295410] ? hrtimer_interrupt+0x128/0x2c0 [ 4664.297097] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4664.298765] ? apic_timer_interrupt+0xf/0x20 [ 4664.305046] [ 4664.305899] ? native_safe_halt+0x2e/0x50 [ 4664.309204] kvm_wait+0xcd/0xe0 [ 4664.311436] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4664.315344] do_raw_spin_lock+0x177/0x190 [ 4664.318712] _raw_spin_lock+0x21/0x30 [ 4664.322463] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4664.326958] process_one_work+0x2c8/0x700 [ 4664.330423] worker_thread+0x296/0x6e0 [ 4664.332457] ? rescuer_thread+0x570/0x570 [ 4664.337046] kthread+0x1d1/0x200 [ 4664.340327] ? set_kthread_struct+0x70/0x70 [ 4664.343431] ret_from_fork+0x1f/0x30 [ 4672.055495] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 4672.058347] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4672.080098] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4672.084349] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4672.089230] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4672.091400] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4672.102580] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4672.106175] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 4672.109358] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 4672.112126] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 4672.114471] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 4672.117259] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4672.119721] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 4672.123628] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4672.126372] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4672.129914] Call Trace: [ 4672.131256] [ 4672.132293] ? show_regs.cold.9+0x22/0x2f [ 4672.134200] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4672.136165] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4672.138209] ? watchdog+0x40/0x40 [ 4672.140159] ? hrtimer_interrupt+0x128/0x2c0 [ 4672.141712] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4672.143521] ? apic_timer_interrupt+0xf/0x20 [ 4672.145806] [ 4672.146536] ? native_safe_halt+0x2e/0x50 [ 4672.148001] kvm_wait+0xcd/0xe0 [ 4672.149638] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4672.152453] do_raw_spin_lock+0x177/0x190 [ 4672.154654] _raw_spin_lock+0x21/0x30 [ 4672.156453] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4672.159078] obd_mod_free+0x46/0x140 [ptlrpc] [ 4672.161424] mdc_commit_open+0x91/0xb0 [mdc] [ 4672.163412] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4672.166321] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4672.169294] after_reply+0xe2c/0x1780 [ptlrpc] [ 4672.171797] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4672.175079] ? __next_timer_interrupt+0x160/0x160 [ 4672.177317] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4672.179898] ? do_wait_intr+0xf0/0xf0 [ 4672.181895] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4672.184706] kthread+0x1d1/0x200 [ 4672.186291] ? set_kthread_struct+0x70/0x70 [ 4672.188334] ret_from_fork+0x1f/0x30 [ 4676.057025] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 4676.059876] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4676.103534] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4676.111579] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4676.118609] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4676.121911] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4676.139520] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4676.147006] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4676.154097] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 4676.166613] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4676.176030] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 4676.183391] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 4676.185802] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 4676.194347] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4676.196285] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 4676.201772] Call Trace: [ 4676.203305] [ 4676.204500] ? show_regs.cold.9+0x22/0x2f [ 4676.206920] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4676.210095] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4676.212937] ? watchdog+0x40/0x40 [ 4676.214763] ? hrtimer_interrupt+0x128/0x2c0 [ 4676.218660] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4676.223382] ? apic_timer_interrupt+0xf/0x20 [ 4676.225560] [ 4676.226948] ? native_safe_halt+0x2e/0x50 [ 4676.229519] kvm_wait+0xcd/0xe0 [ 4676.231322] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4676.235437] do_raw_spin_lock+0x177/0x190 [ 4676.236568] _raw_spin_lock+0x21/0x30 [ 4676.237827] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4676.252391] ? do_raw_spin_unlock+0x75/0x190 [ 4676.256141] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4676.260550] ? do_wait_intr+0xf0/0xf0 [ 4676.262370] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4676.265357] mdc_close+0x53b/0xd70 [mdc] [ 4676.267815] lmv_close+0x18d/0x490 [lmv] [ 4676.270212] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4676.273822] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4676.276885] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4676.281251] ll_md_close+0x274/0x800 [lustre] [ 4676.284185] ll_file_release+0x128/0xc50 [lustre] [ 4676.287769] __fput+0x10d/0x440 [ 4676.289435] ____fput+0x12/0x20 [ 4676.291069] task_work_run+0xc6/0x110 [ 4676.293981] exit_to_usermode_loop+0x1dd/0x1f0 [ 4676.299429] do_syscall_64+0x3d6/0x440 [ 4676.301922] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4676.304602] RIP: 0033:0x7f151bee3cd8 [ 4676.307220] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4676.333176] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4676.339723] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 4676.342760] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 4676.345649] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4676.348400] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4676.351926] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4680.052010] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 4680.057662] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4680.086772] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4680.094177] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4680.101557] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4680.106019] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4680.124032] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4680.128649] RAX: 0000000000003764 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4680.133278] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 4680.137747] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 4680.143319] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 4680.148087] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 4680.153140] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 4680.158881] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4680.164588] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 4680.169967] Call Trace: [ 4680.171417] [ 4680.172995] ? show_regs.cold.9+0x22/0x2f [ 4680.175764] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4680.178719] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4680.181490] ? watchdog+0x40/0x40 [ 4680.183529] ? hrtimer_interrupt+0x128/0x2c0 [ 4680.186133] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4680.188719] ? apic_timer_interrupt+0xf/0x20 [ 4680.191062] [ 4680.192634] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4680.195364] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4680.198334] do_raw_spin_lock+0x177/0x190 [ 4680.200208] _raw_spin_lock+0x21/0x30 [ 4680.202007] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4680.205222] ? do_raw_spin_unlock+0x75/0x190 [ 4680.208216] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4680.210832] ? do_wait_intr+0xf0/0xf0 [ 4680.213272] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4680.216681] mdc_reint+0x2d/0x110 [mdc] [ 4680.219314] mdc_unlink+0x658/0x870 [mdc] [ 4680.222931] lmv_unlink+0x34d/0xbd0 [lmv] [ 4680.226014] ll_unlink+0x24d/0x7f0 [lustre] [ 4680.228418] vfs_unlink+0x202/0x330 [ 4680.230695] do_unlinkat+0x2fa/0x460 [ 4680.232946] __x64_sys_unlink+0x37/0x50 [ 4680.236484] do_syscall_64+0xc1/0x440 [ 4680.239120] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4680.242242] RIP: 0033:0x7f151bdee10b [ 4680.244355] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4680.255544] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4680.259827] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 4680.263347] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 4680.267671] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4680.274763] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4680.286509] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4684.697016] rcu: INFO: rcu_sched self-detected stall on CPU [ 4684.699510] rcu: 1-...!: (1616037 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 4684.704294] (t=2760050 jiffies g=249657 q=3201) [ 4684.706018] rcu: rcu_sched kthread starved for 2755002 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 4684.710461] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4684.714064] rcu: RCU grace-period kthread stack dump: [ 4684.716045] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4684.719431] Call Trace: [ 4684.720251] __schedule+0x351/0xcb0 [ 4684.722835] schedule+0xc0/0x180 [ 4684.724924] schedule_timeout+0xb4/0x190 [ 4684.727161] ? do_raw_spin_unlock+0x75/0x190 [ 4684.732046] ? __next_timer_interrupt+0x160/0x160 [ 4684.736259] ? prepare_to_swait_event+0xf3/0x140 [ 4684.740502] rcu_gp_kthread+0x8dd/0x15b0 [ 4684.744918] ? force_qs_rnp+0x2c0/0x2c0 [ 4684.747608] kthread+0x1d1/0x200 [ 4684.749400] ? set_kthread_struct+0x70/0x70 [ 4684.751598] ret_from_fork+0x1f/0x30 [ 4684.753176] rcu: Stack dump where RCU GP kthread last ran: [ 4684.763232] Sending NMI from CPU 1 to CPUs 2: [ 4684.771459] NMI backtrace for cpu 2 [ 4684.771466] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4684.771467] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4684.771469] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4684.771471] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4684.771473] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4684.771474] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 [ 4684.771476] RAX: 0000000000005187 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4684.771490] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 4684.771492] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 4684.771493] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 4684.771494] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 4684.771495] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 4684.771497] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4684.771498] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4684.771499] Call Trace: [ 4684.771500] [ 4684.771501] ? show_regs.cold.9+0x22/0x2f [ 4684.771502] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 4684.771503] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 4684.771504] ? nmi_handle+0xbd/0x210 [ 4684.771505] ? default_do_nmi+0x4b/0x1b0 [ 4684.771506] ? do_nmi+0x280/0x380 [ 4684.771507] ? end_repeat_nmi+0x16/0x1f [ 4684.771508] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4684.771509] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4684.771510] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4684.771512] [ 4684.771512] do_raw_spin_lock+0x177/0x190 [ 4684.771513] _raw_spin_lock+0x21/0x30 [ 4684.771514] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4684.771515] process_one_work+0x2c8/0x700 [ 4684.771516] worker_thread+0x296/0x6e0 [ 4684.771517] ? rescuer_thread+0x570/0x570 [ 4684.771518] kthread+0x1d1/0x200 [ 4684.771519] ? set_kthread_struct+0x70/0x70 [ 4684.771520] ret_from_fork+0x1f/0x30 [ 4684.771705] NMI backtrace for cpu 1 [ 4684.956358] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4684.961179] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4684.967346] Call Trace: [ 4684.968010] [ 4684.969078] ? dump_stack+0xbb/0x10e [ 4684.970889] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4684.972583] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4684.978057] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4684.980828] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4684.983910] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4684.985432] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 4684.986466] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 4684.987644] ? update_process_times+0x6d/0xc0 [ 4684.994333] ? tick_sched_handle.isra.18+0x27/0x90 [ 4684.995858] ? tick_sched_timer+0x53/0xb0 [ 4684.997448] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4684.999760] ? tick_sched_handle.isra.18+0x90/0x90 [ 4685.001071] ? hrtimer_interrupt+0x128/0x2c0 [ 4685.002959] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4685.009473] ? apic_timer_interrupt+0xf/0x20 [ 4685.015584] [ 4685.020146] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4685.025264] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4685.042231] ? do_raw_spin_lock+0x177/0x190 [ 4685.043813] ? _raw_spin_lock+0x21/0x30 [ 4685.045350] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4685.047215] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 4685.049073] ? mdc_commit_open+0x91/0xb0 [mdc] [ 4685.050256] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4685.051892] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4685.053410] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 4685.054725] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4685.056478] ? __next_timer_interrupt+0x160/0x160 [ 4685.058236] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4685.060108] ? do_wait_intr+0xf0/0xf0 [ 4685.061225] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4685.062783] ? kthread+0x1d1/0x200 [ 4685.063868] ? set_kthread_struct+0x70/0x70 [ 4685.065153] ? ret_from_fork+0x1f/0x30 [ 4685.803561] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2755s! [ 4685.810366] Showing busy workqueues and worker pools: [ 4685.817792] workqueue events: flags=0x0 [ 4685.819723] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4685.824852] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 4685.832459] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4685.837961] pending: kfree_rcu_monitor [ 4685.842917] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4685.848779] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 4685.859540] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4685.868486] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4685.873705] workqueue events_long: flags=0x0 [ 4685.876458] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4685.882388] pending: lu_objects_flush [obdclass] [ 4685.890810] workqueue events_unbound: flags=0x2 [ 4685.896821] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4685.902487] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4685.908588] workqueue events_power_efficient: flags=0x80 [ 4685.915462] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4685.917778] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 4685.927326] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4685.936157] pending: neigh_periodic_work [ 4685.939594] workqueue mm_percpu_wq: flags=0x8 [ 4685.942976] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4685.950348] pending: vmstat_update, lru_add_drain_per_cpu [ 4685.956043] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4685.960828] pending: vmstat_update [ 4685.962581] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4685.969140] pending: lru_add_drain_per_cpu [ 4685.972937] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4685.976471] pending: lru_add_drain_per_cpu BAR(48) [ 4685.979596] workqueue writeback: flags=0x4a [ 4685.982800] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4685.988400] pending: wb_workfn [ 4685.992949] workqueue xprtiod: flags=0xa [ 4685.996806] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4686.005348] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4686.013722] workqueue ipv6_addrconf: flags=0x40008 [ 4686.018975] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4686.023746] pending: addrconf_verify_work [ 4686.029502] workqueue knbd0-recv: flags=0x1a [ 4686.033070] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4686.039835] in-flight: 658:recv_work [ 4686.042821] workqueue ptlrpc_pinger: flags=0x2 [ 4686.047685] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4686.052862] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4686.059926] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2755s workers=4 idle: 32984 37 8 [ 4686.063511] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4670s workers=2 idle: 661 [ 4704.057085] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 4704.064237] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4704.091185] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4704.098907] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4704.103966] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4704.106635] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4704.115997] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4704.120548] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4704.131997] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 4704.141415] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4704.145213] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 4704.153975] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 4704.158782] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 4704.170607] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4704.177925] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 4704.184068] Call Trace: [ 4704.185591] [ 4704.188801] ? show_regs.cold.9+0x22/0x2f [ 4704.191070] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4704.193843] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4704.196223] ? watchdog+0x40/0x40 [ 4704.202217] ? hrtimer_interrupt+0x128/0x2c0 [ 4704.206878] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4704.212735] ? apic_timer_interrupt+0xf/0x20 [ 4704.222186] [ 4704.224308] ? native_safe_halt+0x2e/0x50 [ 4704.227687] kvm_wait+0xcd/0xe0 [ 4704.231405] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4704.233246] do_raw_spin_lock+0x177/0x190 [ 4704.235900] _raw_spin_lock+0x21/0x30 [ 4704.238254] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4704.248727] ? do_raw_spin_unlock+0x75/0x190 [ 4704.253309] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4704.255310] ? do_wait_intr+0xf0/0xf0 [ 4704.261727] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4704.267395] mdc_close+0x53b/0xd70 [mdc] [ 4704.269338] lmv_close+0x18d/0x490 [lmv] [ 4704.271670] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4704.280547] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4704.285591] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4704.290030] ll_md_close+0x274/0x800 [lustre] [ 4704.292993] ll_file_release+0x128/0xc50 [lustre] [ 4704.294812] __fput+0x10d/0x440 [ 4704.304613] ____fput+0x12/0x20 [ 4704.307250] task_work_run+0xc6/0x110 [ 4704.310162] exit_to_usermode_loop+0x1dd/0x1f0 [ 4704.315702] do_syscall_64+0x3d6/0x440 [ 4704.322976] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4704.331000] RIP: 0033:0x7f151bee3cd8 [ 4704.339129] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4704.354279] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4704.362421] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 4704.374870] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 4704.386348] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4704.391513] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4704.395793] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4712.053864] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 4712.056211] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4712.060256] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 4712.060271] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4712.060342] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4712.060348] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4712.060350] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4712.060365] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4712.060369] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4712.060373] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 4712.060376] RDX: 00000000000016e8 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 4712.060378] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 4712.060380] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 4712.060383] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4712.060385] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 4712.060388] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4712.060390] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4712.060399] Call Trace: [ 4712.060415] [ 4712.060420] ? show_regs.cold.9+0x22/0x2f [ 4712.060430] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4712.060436] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4712.060440] ? watchdog+0x40/0x40 [ 4712.060445] ? hrtimer_interrupt+0x128/0x2c0 [ 4712.060449] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4712.060456] ? apic_timer_interrupt+0xf/0x20 [ 4712.060461] [ 4712.060463] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4712.060468] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4712.060472] do_raw_spin_lock+0x177/0x190 [ 4712.060477] _raw_spin_lock+0x21/0x30 [ 4712.060492] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4712.060797] obd_mod_free+0x46/0x140 [ptlrpc] [ 4712.061003] mdc_commit_open+0x91/0xb0 [mdc] [ 4712.061044] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4712.061248] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4712.062037] after_reply+0xe2c/0x1780 [ptlrpc] [ 4712.062366] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4712.062951] ? __next_timer_interrupt+0x160/0x160 [ 4712.062961] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4712.064164] ? do_wait_intr+0xf0/0xf0 [ 4712.064173] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4712.064372] kthread+0x1d1/0x200 [ 4712.064378] ? set_kthread_struct+0x70/0x70 [ 4712.064382] ret_from_fork+0x1f/0x30 [ 4712.069453] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4712.069558] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4712.069566] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4712.069570] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4712.069584] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4712.069588] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4712.069593] RAX: 0000000000007f85 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4712.069596] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 4712.069599] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 4712.069601] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 4712.069604] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 4712.069607] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 4712.069610] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4712.069612] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 4712.069620] Call Trace: [ 4712.069626] [ 4712.069631] ? show_regs.cold.9+0x22/0x2f [ 4712.069640] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4712.069646] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4712.069651] ? watchdog+0x40/0x40 [ 4712.069655] ? hrtimer_interrupt+0x128/0x2c0 [ 4712.069659] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4712.069668] ? apic_timer_interrupt+0xf/0x20 [ 4712.069673] [ 4712.069676] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4712.069681] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4712.069685] do_raw_spin_lock+0x177/0x190 [ 4712.069690] _raw_spin_lock+0x21/0x30 [ 4712.069696] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4712.069985] ? do_raw_spin_unlock+0x75/0x190 [ 4712.069992] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4712.070211] ? do_wait_intr+0xf0/0xf0 [ 4712.070220] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4712.070723] mdc_reint+0x2d/0x110 [mdc] [ 4712.070761] mdc_unlink+0x658/0x870 [mdc] [ 4712.070788] lmv_unlink+0x34d/0xbd0 [lmv] [ 4712.070821] ll_unlink+0x24d/0x7f0 [lustre] [ 4712.070957] vfs_unlink+0x202/0x330 [ 4712.070966] do_unlinkat+0x2fa/0x460 [ 4712.070971] __x64_sys_unlink+0x37/0x50 [ 4712.070974] do_syscall_64+0xc1/0x440 [ 4712.070982] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4712.070991] RIP: 0033:0x7f151bdee10b [ 4712.071019] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4712.071024] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4712.071029] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 4712.071031] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 4712.071034] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4712.071037] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4712.071039] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4712.391880] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4712.415081] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4712.422147] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4712.429108] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4712.432578] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4712.436528] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4712.447404] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4712.450992] RAX: 0000000000007f87 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4712.454246] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 4712.466080] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 4712.470917] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 4712.474974] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 4712.483794] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 4712.488224] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4712.492820] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4712.495775] Call Trace: [ 4712.499043] [ 4712.499807] ? show_regs.cold.9+0x22/0x2f [ 4712.503660] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4712.506636] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4712.511234] ? watchdog+0x40/0x40 [ 4712.513237] ? hrtimer_interrupt+0x128/0x2c0 [ 4712.516554] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4712.519827] ? apic_timer_interrupt+0xf/0x20 [ 4712.521740] [ 4712.522788] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4712.525394] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4712.529744] do_raw_spin_lock+0x177/0x190 [ 4712.531846] _raw_spin_lock+0x21/0x30 [ 4712.533942] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4712.537859] process_one_work+0x2c8/0x700 [ 4712.539839] worker_thread+0x296/0x6e0 [ 4712.542155] ? rescuer_thread+0x570/0x570 [ 4712.545507] kthread+0x1d1/0x200 [ 4712.547241] ? set_kthread_struct+0x70/0x70 [ 4712.549181] ret_from_fork+0x1f/0x30 [ 4716.516728] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2785s! [ 4716.528029] Showing busy workqueues and worker pools: [ 4716.534569] workqueue events: flags=0x0 [ 4716.537546] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4716.542132] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 4716.549536] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4716.553187] pending: kfree_rcu_monitor [ 4716.556534] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4716.562245] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 4716.575921] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4716.582759] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4716.588986] workqueue events_long: flags=0x0 [ 4716.593188] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4716.602271] pending: lu_objects_flush [obdclass] [ 4716.610533] workqueue events_unbound: flags=0x2 [ 4716.614671] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4716.619653] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4716.626837] workqueue events_power_efficient: flags=0x80 [ 4716.628851] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4716.632163] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 4716.634614] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4716.636275] pending: neigh_periodic_work [ 4716.637447] workqueue mm_percpu_wq: flags=0x8 [ 4716.639378] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4716.641576] pending: vmstat_update, lru_add_drain_per_cpu [ 4716.643848] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4716.645879] pending: vmstat_update [ 4716.646868] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4716.648621] pending: lru_add_drain_per_cpu [ 4716.650027] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4716.655614] pending: lru_add_drain_per_cpu BAR(48) [ 4716.659891] workqueue writeback: flags=0x4a [ 4716.664374] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4716.670926] pending: wb_workfn [ 4716.675981] workqueue xprtiod: flags=0xa [ 4716.679856] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4716.684883] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4716.691799] workqueue ipv6_addrconf: flags=0x40008 [ 4716.694016] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4716.709816] pending: addrconf_verify_work [ 4716.723756] workqueue knbd0-recv: flags=0x1a [ 4716.725380] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4716.738299] in-flight: 658:recv_work [ 4716.746590] workqueue ptlrpc_pinger: flags=0x2 [ 4716.752689] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4716.760842] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4716.771200] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2786s workers=4 idle: 32984 37 8 [ 4716.784702] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4701s workers=2 idle: 661 [ 4732.058393] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 4732.061609] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4732.101912] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4732.107090] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4732.113198] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4732.116566] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4732.131310] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 4732.135626] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4732.139253] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 4732.143884] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4732.148957] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 4732.155546] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 4732.160114] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 4732.170223] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4732.178894] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 4732.186697] Call Trace: [ 4732.192377] [ 4732.196390] ? show_regs.cold.9+0x22/0x2f [ 4732.204642] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4732.214639] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4732.223031] ? watchdog+0x40/0x40 [ 4732.227691] ? hrtimer_interrupt+0x128/0x2c0 [ 4732.234581] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4732.241513] ? apic_timer_interrupt+0xf/0x20 [ 4732.247146] [ 4732.247939] ? native_safe_halt+0x2e/0x50 [ 4732.253543] kvm_wait+0xcd/0xe0 [ 4732.256486] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4732.267403] do_raw_spin_lock+0x177/0x190 [ 4732.273720] _raw_spin_lock+0x21/0x30 [ 4732.279336] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4732.286391] ? do_raw_spin_unlock+0x75/0x190 [ 4732.293433] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4732.299351] ? do_wait_intr+0xf0/0xf0 [ 4732.306735] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4732.312369] mdc_close+0x53b/0xd70 [mdc] [ 4732.322252] lmv_close+0x18d/0x490 [lmv] [ 4732.332194] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4732.337986] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4732.347739] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4732.353102] ll_md_close+0x274/0x800 [lustre] [ 4732.362139] ll_file_release+0x128/0xc50 [lustre] [ 4732.370001] __fput+0x10d/0x440 [ 4732.374513] ____fput+0x12/0x20 [ 4732.378112] task_work_run+0xc6/0x110 [ 4732.382584] exit_to_usermode_loop+0x1dd/0x1f0 [ 4732.386768] do_syscall_64+0x3d6/0x440 [ 4732.391351] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4732.394123] RIP: 0033:0x7f151bee3cd8 [ 4732.395973] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4732.404376] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4732.407339] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 4732.410956] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 4732.415374] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4732.424258] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4732.430037] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4740.056170] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4740.059608] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 4740.062783] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4740.062867] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4740.062874] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4740.062880] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4740.064846] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4740.064860] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4740.064864] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4740.064868] RAX: 0000000000000b43 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4740.064870] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 4740.064872] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 4740.064874] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 4740.064876] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 4740.064879] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 4740.064882] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4740.064883] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4740.066205] Call Trace: [ 4740.066227] [ 4740.066234] ? show_regs.cold.9+0x22/0x2f [ 4740.080957] Modules linked in: lustre(O) [ 4740.117345] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4740.117363] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4740.117367] ? watchdog+0x40/0x40 [ 4740.117443] ? hrtimer_interrupt+0x128/0x2c0 [ 4740.117448] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4740.117455] ? apic_timer_interrupt+0xf/0x20 [ 4740.117459] [ 4740.117464] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4740.117470] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4740.117642] do_raw_spin_lock+0x177/0x190 [ 4740.117648] _raw_spin_lock+0x21/0x30 [ 4740.117653] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4740.117928] process_one_work+0x2c8/0x700 [ 4740.117935] worker_thread+0x296/0x6e0 [ 4740.117940] ? rescuer_thread+0x570/0x570 [ 4740.117944] kthread+0x1d1/0x200 [ 4740.117947] ? set_kthread_struct+0x70/0x70 [ 4740.117951] ret_from_fork+0x1f/0x30 [ 4740.250665] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4740.289448] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4740.302068] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4740.318813] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4740.321802] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4740.335426] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4740.343800] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 4740.347499] RDX: 000000000000344f RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 4740.353343] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 4740.359059] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 4740.368327] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4740.379904] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 4740.384068] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4740.387748] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4740.395420] Call Trace: [ 4740.396378] [ 4740.397147] ? show_regs.cold.9+0x22/0x2f [ 4740.402602] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4740.406403] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4740.408606] ? watchdog+0x40/0x40 [ 4740.413931] ? hrtimer_interrupt+0x128/0x2c0 [ 4740.417677] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4740.419663] ? apic_timer_interrupt+0xf/0x20 [ 4740.421809] [ 4740.422844] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4740.425655] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4740.428500] do_raw_spin_lock+0x177/0x190 [ 4740.430399] _raw_spin_lock+0x21/0x30 [ 4740.435644] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4740.441053] obd_mod_free+0x46/0x140 [ptlrpc] [ 4740.444756] mdc_commit_open+0x91/0xb0 [mdc] [ 4740.448819] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4740.454297] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4740.464431] after_reply+0xe2c/0x1780 [ptlrpc] [ 4740.470827] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4740.474768] ? __next_timer_interrupt+0x160/0x160 [ 4740.484411] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4740.488791] ? do_wait_intr+0xf0/0xf0 [ 4740.492291] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4740.496567] kthread+0x1d1/0x200 [ 4740.498609] ? set_kthread_struct+0x70/0x70 [ 4740.506606] ret_from_fork+0x1f/0x30 [ 4744.052031] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 4744.056235] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4744.076989] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4744.082899] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4744.087524] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4744.090295] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4744.098892] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4744.102607] RAX: 0000000000006d0f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4744.106004] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 4744.109327] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 4744.113159] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 4744.116497] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 4744.119811] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 4744.124004] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4744.126879] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 4744.130142] Call Trace: [ 4744.131298] [ 4744.132301] ? show_regs.cold.9+0x22/0x2f [ 4744.134279] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4744.135757] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4744.138274] ? watchdog+0x40/0x40 [ 4744.139821] ? hrtimer_interrupt+0x128/0x2c0 [ 4744.141842] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4744.143610] ? apic_timer_interrupt+0xf/0x20 [ 4744.145431] [ 4744.146354] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4744.149230] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4744.151854] do_raw_spin_lock+0x177/0x190 [ 4744.157371] _raw_spin_lock+0x21/0x30 [ 4744.159298] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4744.161959] ? do_raw_spin_unlock+0x75/0x190 [ 4744.163625] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4744.168230] ? do_wait_intr+0xf0/0xf0 [ 4744.169699] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4744.174987] mdc_reint+0x2d/0x110 [mdc] [ 4744.176535] mdc_unlink+0x658/0x870 [mdc] [ 4744.181815] lmv_unlink+0x34d/0xbd0 [lmv] [ 4744.183388] ll_unlink+0x24d/0x7f0 [lustre] [ 4744.187426] vfs_unlink+0x202/0x330 [ 4744.189284] do_unlinkat+0x2fa/0x460 [ 4744.191815] __x64_sys_unlink+0x37/0x50 [ 4744.193785] do_syscall_64+0xc1/0x440 [ 4744.195651] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4744.198102] RIP: 0033:0x7f151bdee10b [ 4744.199797] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4744.208567] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4744.211860] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 4744.215345] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 4744.219115] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4744.223208] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4744.229536] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4747.237576] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2816s! [ 4747.240841] Showing busy workqueues and worker pools: [ 4747.242874] workqueue events: flags=0x0 [ 4747.244500] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4747.247254] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 4747.250573] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4747.253247] pending: kfree_rcu_monitor [ 4747.258196] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4747.260511] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 4747.265793] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4747.267761] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4747.271682] workqueue events_long: flags=0x0 [ 4747.273171] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4747.276504] pending: lu_objects_flush [obdclass] [ 4747.277943] workqueue events_unbound: flags=0x2 [ 4747.284584] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4747.286679] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4747.290022] workqueue events_power_efficient: flags=0x80 [ 4747.292224] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4747.294063] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 4747.296634] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4747.299642] pending: neigh_periodic_work [ 4747.302257] workqueue mm_percpu_wq: flags=0x8 [ 4747.304784] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4747.309169] pending: vmstat_update, lru_add_drain_per_cpu [ 4747.311318] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4747.313516] pending: vmstat_update [ 4747.314798] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4747.317319] pending: lru_add_drain_per_cpu [ 4747.318716] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4747.321059] pending: lru_add_drain_per_cpu BAR(48) [ 4747.322795] workqueue writeback: flags=0x4a [ 4747.324209] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4747.326521] pending: wb_workfn [ 4747.327646] workqueue xprtiod: flags=0xa [ 4747.328770] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4747.330981] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4747.333758] workqueue ipv6_addrconf: flags=0x40008 [ 4747.335100] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4747.337153] pending: addrconf_verify_work [ 4747.338540] workqueue knbd0-recv: flags=0x1a [ 4747.339853] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4747.341764] in-flight: 658:recv_work [ 4747.343073] workqueue ptlrpc_pinger: flags=0x2 [ 4747.344369] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4747.346038] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4747.348068] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2816s workers=4 idle: 32984 37 8 [ 4747.350030] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4732s workers=2 idle: 661 [ 4760.057265] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 4760.062629] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4760.116901] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4760.130523] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4760.137835] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4760.139533] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4760.172354] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4760.178347] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4760.183678] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 4760.191537] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4760.197659] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 4760.204325] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 4760.211706] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 4760.217860] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4760.221670] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 4760.228175] Call Trace: [ 4760.230327] [ 4760.232221] ? show_regs.cold.9+0x22/0x2f [ 4760.238549] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4760.250305] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4760.252222] ? watchdog+0x40/0x40 [ 4760.253307] ? hrtimer_interrupt+0x128/0x2c0 [ 4760.258226] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4760.262340] ? apic_timer_interrupt+0xf/0x20 [ 4760.265334] [ 4760.266624] ? native_safe_halt+0x2e/0x50 [ 4760.269877] kvm_wait+0xcd/0xe0 [ 4760.272354] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4760.276163] do_raw_spin_lock+0x177/0x190 [ 4760.279917] _raw_spin_lock+0x21/0x30 [ 4760.282815] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4760.287705] ? do_raw_spin_unlock+0x75/0x190 [ 4760.290560] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4760.294293] ? do_wait_intr+0xf0/0xf0 [ 4760.296730] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4760.302639] mdc_close+0x53b/0xd70 [mdc] [ 4760.306357] lmv_close+0x18d/0x490 [lmv] [ 4760.311282] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4760.318766] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4760.322409] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4760.328690] ll_md_close+0x274/0x800 [lustre] [ 4760.335172] ll_file_release+0x128/0xc50 [lustre] [ 4760.338440] __fput+0x10d/0x440 [ 4760.340327] ____fput+0x12/0x20 [ 4760.343028] task_work_run+0xc6/0x110 [ 4760.345247] exit_to_usermode_loop+0x1dd/0x1f0 [ 4760.347769] do_syscall_64+0x3d6/0x440 [ 4760.349849] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4760.354334] RIP: 0033:0x7f151bee3cd8 [ 4760.359300] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4760.367502] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4760.370818] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 4760.374401] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 4760.377697] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4760.380621] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4760.383877] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4768.055100] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 4768.056033] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4768.066195] Modules linked in: [ 4768.069058] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4768.069149] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4768.069157] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4768.069164] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4768.069476] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4768.069489] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4768.069494] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4768.069498] RAX: 0000000000000087 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4768.069504] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 4768.069506] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 4768.069508] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 4768.069511] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 4768.069514] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 4768.069516] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4768.069518] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4768.069526] Call Trace: [ 4768.069531] [ 4768.069541] ? show_regs.cold.9+0x22/0x2f [ 4768.069551] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4768.069558] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4768.069565] ? watchdog+0x40/0x40 [ 4768.069570] ? hrtimer_interrupt+0x128/0x2c0 [ 4768.069574] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4768.069582] ? apic_timer_interrupt+0xf/0x20 [ 4768.069586] [ 4768.069589] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4768.069594] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4768.069602] do_raw_spin_lock+0x177/0x190 [ 4768.069607] _raw_spin_lock+0x21/0x30 [ 4768.069612] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4768.069969] process_one_work+0x2c8/0x700 [ 4768.070018] worker_thread+0x296/0x6e0 [ 4768.070022] ? rescuer_thread+0x570/0x570 [ 4768.070026] kthread+0x1d1/0x200 [ 4768.070031] ? set_kthread_struct+0x70/0x70 [ 4768.070034] ret_from_fork+0x1f/0x30 [ 4768.185763] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4768.202363] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4768.207453] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4768.212300] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4768.215475] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4768.231630] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4768.239020] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 4768.242608] RDX: 00000000000021a9 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 4768.245960] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 4768.249371] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 4768.252618] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4768.258410] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 4768.265982] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4768.268918] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4768.273805] Call Trace: [ 4768.276739] [ 4768.278481] ? show_regs.cold.9+0x22/0x2f [ 4768.280754] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4768.284769] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4768.288139] ? watchdog+0x40/0x40 [ 4768.289600] ? hrtimer_interrupt+0x128/0x2c0 [ 4768.292336] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4768.295583] ? apic_timer_interrupt+0xf/0x20 [ 4768.298358] [ 4768.299385] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4768.301946] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4768.305502] do_raw_spin_lock+0x177/0x190 [ 4768.307428] _raw_spin_lock+0x21/0x30 [ 4768.309217] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4768.311695] obd_mod_free+0x46/0x140 [ptlrpc] [ 4768.316266] mdc_commit_open+0x91/0xb0 [mdc] [ 4768.319174] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4768.322374] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4768.324737] after_reply+0xe2c/0x1780 [ptlrpc] [ 4768.327294] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4768.329799] ? __next_timer_interrupt+0x160/0x160 [ 4768.331779] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4768.333526] ? do_wait_intr+0xf0/0xf0 [ 4768.334606] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4768.336645] kthread+0x1d1/0x200 [ 4768.337442] ? set_kthread_struct+0x70/0x70 [ 4768.338547] ret_from_fork+0x1f/0x30 [ 4772.055451] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 4772.060218] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4772.078981] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4772.082792] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4772.088348] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4772.090127] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4772.096519] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4772.099124] RAX: 0000000000006509 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4772.102540] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 4772.104473] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 4772.107109] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 4772.113334] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 4772.115293] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 4772.118933] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4772.120212] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 4772.123800] Call Trace: [ 4772.124418] [ 4772.126367] ? show_regs.cold.9+0x22/0x2f [ 4772.128258] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4772.129545] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4772.131360] ? watchdog+0x40/0x40 [ 4772.132199] ? hrtimer_interrupt+0x128/0x2c0 [ 4772.133357] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4772.135992] ? apic_timer_interrupt+0xf/0x20 [ 4772.138240] [ 4772.139109] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4772.143518] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4772.149554] do_raw_spin_lock+0x177/0x190 [ 4772.150485] _raw_spin_lock+0x21/0x30 [ 4772.151309] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4772.154138] ? do_raw_spin_unlock+0x75/0x190 [ 4772.155259] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4772.157139] ? do_wait_intr+0xf0/0xf0 [ 4772.158140] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4772.160728] mdc_reint+0x2d/0x110 [mdc] [ 4772.161868] mdc_unlink+0x658/0x870 [mdc] [ 4772.163284] lmv_unlink+0x34d/0xbd0 [lmv] [ 4772.164536] ll_unlink+0x24d/0x7f0 [lustre] [ 4772.166247] vfs_unlink+0x202/0x330 [ 4772.167017] do_unlinkat+0x2fa/0x460 [ 4772.167787] __x64_sys_unlink+0x37/0x50 [ 4772.168630] do_syscall_64+0xc1/0x440 [ 4772.171712] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4772.172931] RIP: 0033:0x7f151bdee10b [ 4772.174016] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4772.180153] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4772.184199] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 4772.187318] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 4772.191574] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4772.193523] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4772.195857] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4777.446033] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2846s! [ 4777.455454] Showing busy workqueues and worker pools: [ 4777.459395] workqueue events: flags=0x0 [ 4777.462138] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4777.468190] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 4777.482789] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4777.484735] pending: kfree_rcu_monitor [ 4777.486396] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4777.489462] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 4777.496386] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4777.500417] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4777.506030] workqueue events_long: flags=0x0 [ 4777.508664] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4777.513057] pending: lu_objects_flush [obdclass] [ 4777.517320] workqueue events_unbound: flags=0x2 [ 4777.519270] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4777.521559] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4777.526541] workqueue events_power_efficient: flags=0x80 [ 4777.529968] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4777.532780] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 4777.536448] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4777.540383] pending: neigh_periodic_work [ 4777.543261] workqueue mm_percpu_wq: flags=0x8 [ 4777.546179] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4777.551813] pending: vmstat_update, lru_add_drain_per_cpu [ 4777.555812] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4777.559513] pending: vmstat_update [ 4777.561342] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4777.568264] pending: lru_add_drain_per_cpu [ 4777.573169] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4777.578195] pending: lru_add_drain_per_cpu BAR(48) [ 4777.581241] workqueue writeback: flags=0x4a [ 4777.584512] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4777.588431] pending: wb_workfn [ 4777.590483] workqueue xprtiod: flags=0xa [ 4777.595512] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4777.599131] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4777.605647] workqueue ipv6_addrconf: flags=0x40008 [ 4777.609539] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4777.613980] pending: addrconf_verify_work [ 4777.616647] workqueue knbd0-recv: flags=0x1a [ 4777.620213] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4777.628325] in-flight: 658:recv_work [ 4777.632366] workqueue ptlrpc_pinger: flags=0x2 [ 4777.633718] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4777.638656] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4777.642362] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2847s workers=4 idle: 32984 37 8 [ 4777.650537] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4762s workers=2 idle: 661 [ 4788.057214] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 4788.067325] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4788.115984] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4788.125796] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4788.132319] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4788.135450] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4788.147874] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4788.154444] RAX: 0000000000005186 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4788.158641] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 4788.163812] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4788.168076] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 4788.174420] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 4788.179126] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 4788.183388] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4788.186726] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 4788.191351] Call Trace: [ 4788.192216] [ 4788.193294] ? show_regs.cold.9+0x22/0x2f [ 4788.195783] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4788.198527] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4788.201371] ? watchdog+0x40/0x40 [ 4788.203371] ? hrtimer_interrupt+0x128/0x2c0 [ 4788.205838] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4788.208265] ? apic_timer_interrupt+0xf/0x20 [ 4788.210845] [ 4788.211985] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4788.214385] do_raw_spin_lock+0x177/0x190 [ 4788.216935] _raw_spin_lock+0x21/0x30 [ 4788.222297] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4788.233434] ? do_raw_spin_unlock+0x75/0x190 [ 4788.236988] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4788.244171] ? do_wait_intr+0xf0/0xf0 [ 4788.248373] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4788.255942] mdc_close+0x53b/0xd70 [mdc] [ 4788.258216] lmv_close+0x18d/0x490 [lmv] [ 4788.261417] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4788.265043] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4788.268597] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4788.271723] ll_md_close+0x274/0x800 [lustre] [ 4788.274816] ll_file_release+0x128/0xc50 [lustre] [ 4788.277287] __fput+0x10d/0x440 [ 4788.278806] ____fput+0x12/0x20 [ 4788.280505] task_work_run+0xc6/0x110 [ 4788.282653] exit_to_usermode_loop+0x1dd/0x1f0 [ 4788.284792] do_syscall_64+0x3d6/0x440 [ 4788.287052] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4788.290276] RIP: 0033:0x7f151bee3cd8 [ 4788.292894] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4788.310438] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4788.317238] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 4788.323325] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 4788.328693] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4788.343276] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4788.350421] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4796.055946] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 4796.056978] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4796.065518] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4796.065609] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4796.065616] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4796.065619] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4796.065633] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4796.065638] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4796.065642] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 4796.065645] RDX: 000000000000630e RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 4796.065647] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 4796.065650] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 4796.065652] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4796.065656] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 4796.065659] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4796.065661] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4796.066049] Call Trace: [ 4796.070257] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4796.070352] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4796.070359] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4796.070366] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4796.119415] [ 4796.119428] ? show_regs.cold.9+0x22/0x2f [ 4796.119443] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4796.119449] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4796.119454] ? watchdog+0x40/0x40 [ 4796.119458] ? hrtimer_interrupt+0x128/0x2c0 [ 4796.119462] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4796.119470] ? apic_timer_interrupt+0xf/0x20 [ 4796.119475] [ 4796.119489] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4796.119497] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4796.119502] do_raw_spin_lock+0x177/0x190 [ 4796.119506] _raw_spin_lock+0x21/0x30 [ 4796.119514] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4796.127353] [ 4796.127365] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4796.127386] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4796.127391] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4796.127396] RAX: 000000000000616b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4796.127399] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 4796.127404] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 4796.127406] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 4796.127409] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 4796.127412] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 4796.127418] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4796.127420] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4796.127427] Call Trace: [ 4796.127432] [ 4796.127439] ? show_regs.cold.9+0x22/0x2f [ 4796.127450] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4796.127455] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4796.127460] ? watchdog+0x40/0x40 [ 4796.127464] ? hrtimer_interrupt+0x128/0x2c0 [ 4796.134203] obd_mod_free+0x46/0x140 [ptlrpc] [ 4796.136955] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4796.136981] ? apic_timer_interrupt+0xf/0x20 [ 4796.136986] [ 4796.136991] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4796.136998] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4796.137009] do_raw_spin_lock+0x177/0x190 [ 4796.137015] _raw_spin_lock+0x21/0x30 [ 4796.137020] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4796.137427] process_one_work+0x2c8/0x700 [ 4796.137434] worker_thread+0x296/0x6e0 [ 4796.137438] ? rescuer_thread+0x570/0x570 [ 4796.137441] kthread+0x1d1/0x200 [ 4796.137446] ? set_kthread_struct+0x70/0x70 [ 4796.137449] ret_from_fork+0x1f/0x30 [ 4796.446960] mdc_commit_open+0x91/0xb0 [mdc] [ 4796.448790] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4796.451035] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4796.453622] after_reply+0xe2c/0x1780 [ptlrpc] [ 4796.455645] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4796.457857] ? __next_timer_interrupt+0x160/0x160 [ 4796.459892] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4796.462092] ? do_wait_intr+0xf0/0xf0 [ 4796.463527] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4796.466424] kthread+0x1d1/0x200 [ 4796.468041] ? set_kthread_struct+0x70/0x70 [ 4796.474858] ret_from_fork+0x1f/0x30 [ 4804.052135] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 4804.063417] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4804.120363] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4804.125892] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4804.131213] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4804.133927] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4804.145724] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4804.150170] RAX: 0000000000007e08 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4804.156161] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 4804.163699] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 4804.167767] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 4804.171732] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 4804.179344] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 4804.185777] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4804.189741] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 4804.196096] Call Trace: [ 4804.197860] [ 4804.198758] ? show_regs.cold.9+0x22/0x2f [ 4804.202737] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4804.214050] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4804.219606] ? watchdog+0x40/0x40 [ 4804.224480] ? hrtimer_interrupt+0x128/0x2c0 [ 4804.229140] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4804.231678] ? apic_timer_interrupt+0xf/0x20 [ 4804.235930] [ 4804.236885] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4804.246126] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4804.249682] do_raw_spin_lock+0x177/0x190 [ 4804.251958] _raw_spin_lock+0x21/0x30 [ 4804.254041] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4804.256694] ? do_raw_spin_unlock+0x75/0x190 [ 4804.266917] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4804.273950] ? do_wait_intr+0xf0/0xf0 [ 4804.276310] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4804.279176] mdc_reint+0x2d/0x110 [mdc] [ 4804.287836] mdc_unlink+0x658/0x870 [mdc] [ 4804.290039] lmv_unlink+0x34d/0xbd0 [lmv] [ 4804.293409] ll_unlink+0x24d/0x7f0 [lustre] [ 4804.296103] vfs_unlink+0x202/0x330 [ 4804.297112] do_unlinkat+0x2fa/0x460 [ 4804.298402] __x64_sys_unlink+0x37/0x50 [ 4804.303940] do_syscall_64+0xc1/0x440 [ 4804.305303] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4804.312195] RIP: 0033:0x7f151bdee10b [ 4804.316408] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4804.337394] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4804.341206] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 4804.345972] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 4804.353311] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4804.362112] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4804.367302] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4808.162522] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2877s! [ 4808.165568] Showing busy workqueues and worker pools: [ 4808.173594] workqueue events: flags=0x0 [ 4808.175596] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4808.178560] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 4808.182545] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4808.185115] pending: kfree_rcu_monitor [ 4808.186884] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4808.189734] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 4808.193882] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4808.196693] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4808.199614] workqueue events_long: flags=0x0 [ 4808.201224] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4808.204294] pending: lu_objects_flush [obdclass] [ 4808.206774] workqueue events_unbound: flags=0x2 [ 4808.208541] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4808.211097] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4808.215141] workqueue events_power_efficient: flags=0x80 [ 4808.217505] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4808.220398] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 4808.223207] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4808.225752] pending: neigh_periodic_work [ 4808.227604] workqueue mm_percpu_wq: flags=0x8 [ 4808.229210] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4808.232039] pending: vmstat_update, lru_add_drain_per_cpu [ 4808.234593] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4808.236926] pending: vmstat_update [ 4808.238617] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4808.240808] pending: lru_add_drain_per_cpu [ 4808.242246] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4808.244696] pending: lru_add_drain_per_cpu BAR(48) [ 4808.246865] workqueue writeback: flags=0x4a [ 4808.248109] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4808.250730] pending: wb_workfn [ 4808.251918] workqueue xprtiod: flags=0xa [ 4808.253642] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4808.256034] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4808.259147] workqueue ipv6_addrconf: flags=0x40008 [ 4808.260833] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4808.263130] pending: addrconf_verify_work [ 4808.264779] workqueue knbd0-recv: flags=0x1a [ 4808.266168] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4808.268821] in-flight: 658:recv_work [ 4808.270534] workqueue ptlrpc_pinger: flags=0x2 [ 4808.272157] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4808.274083] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4808.276215] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2877s workers=4 idle: 32984 37 8 [ 4808.279161] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4793s workers=2 idle: 661 [ 4816.062552] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 4816.072055] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4816.115655] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4816.136336] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4816.140264] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4816.147004] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4816.166885] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4816.171695] RAX: 0000000000007da2 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4816.178637] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 4816.187642] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4816.196911] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 4816.203791] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 4816.209697] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 4816.217583] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4816.222745] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 4816.226832] Call Trace: [ 4816.227723] [ 4816.228449] ? show_regs.cold.9+0x22/0x2f [ 4816.232730] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4816.236673] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4816.240598] ? watchdog+0x40/0x40 [ 4816.244696] ? hrtimer_interrupt+0x128/0x2c0 [ 4816.247102] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4816.248536] ? apic_timer_interrupt+0xf/0x20 [ 4816.256954] [ 4816.260019] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4816.267925] do_raw_spin_lock+0x177/0x190 [ 4816.269541] _raw_spin_lock+0x21/0x30 [ 4816.275648] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4816.277815] ? do_raw_spin_unlock+0x75/0x190 [ 4816.287310] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4816.295561] ? do_wait_intr+0xf0/0xf0 [ 4816.298721] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4816.307789] mdc_close+0x53b/0xd70 [mdc] [ 4816.313253] lmv_close+0x18d/0x490 [lmv] [ 4816.317345] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4816.319391] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4816.333574] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4816.340658] ll_md_close+0x274/0x800 [lustre] [ 4816.349683] ll_file_release+0x128/0xc50 [lustre] [ 4816.352828] __fput+0x10d/0x440 [ 4816.355137] ____fput+0x12/0x20 [ 4816.363542] task_work_run+0xc6/0x110 [ 4816.368411] exit_to_usermode_loop+0x1dd/0x1f0 [ 4816.377217] do_syscall_64+0x3d6/0x440 [ 4816.379736] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4816.381440] RIP: 0033:0x7f151bee3cd8 [ 4816.385658] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4816.403703] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4816.408537] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 4816.418665] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 4816.428953] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4816.446941] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4816.458944] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4824.055170] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 4824.057040] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4824.059742] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4824.059820] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4824.059827] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4824.059830] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4824.059843] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4824.059848] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4824.059851] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 4824.059854] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 4824.059856] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 4824.059858] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 4824.059860] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4824.059862] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 4824.059865] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4824.059867] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4824.059875] Call Trace: [ 4824.059880] [ 4824.059886] ? show_regs.cold.9+0x22/0x2f [ 4824.059896] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4824.059902] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4824.059906] ? watchdog+0x40/0x40 [ 4824.059910] ? hrtimer_interrupt+0x128/0x2c0 [ 4824.059914] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4824.059920] ? apic_timer_interrupt+0xf/0x20 [ 4824.059925] [ 4824.059928] ? native_safe_halt+0x2e/0x50 [ 4824.059933] kvm_wait+0xcd/0xe0 [ 4824.060017] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4824.060150] do_raw_spin_lock+0x177/0x190 [ 4824.060155] _raw_spin_lock+0x21/0x30 [ 4824.060160] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4824.060658] obd_mod_free+0x46/0x140 [ptlrpc] [ 4824.060863] mdc_commit_open+0x91/0xb0 [mdc] [ 4824.060899] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4824.061445] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4824.061712] after_reply+0xe2c/0x1780 [ptlrpc] [ 4824.061906] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4824.062286] ? __next_timer_interrupt+0x160/0x160 [ 4824.062294] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4824.071282] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss [ 4824.154149] ? do_wait_intr+0xf0/0xf0 [ 4824.154170] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4824.169832] nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4824.169893] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4824.169903] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4824.169911] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4824.170235] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4824.170249] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4824.170253] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4824.170258] RAX: 0000000000001484 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4824.177751] kthread+0x1d1/0x200 [ 4824.177770] ? set_kthread_struct+0x70/0x70 [ 4824.177775] ret_from_fork+0x1f/0x30 [ 4824.591775] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 4824.598355] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 4824.606771] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 4824.610895] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 4824.619246] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 4824.633042] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4824.642842] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4824.646092] Call Trace: [ 4824.652282] [ 4824.654496] ? show_regs.cold.9+0x22/0x2f [ 4824.661215] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4824.663182] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4824.674772] ? watchdog+0x40/0x40 [ 4824.675989] ? hrtimer_interrupt+0x128/0x2c0 [ 4824.678190] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4824.679954] ? apic_timer_interrupt+0xf/0x20 [ 4824.684536] [ 4824.686431] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4824.690239] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4824.699063] do_raw_spin_lock+0x177/0x190 [ 4824.703194] _raw_spin_lock+0x21/0x30 [ 4824.709098] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4824.714911] process_one_work+0x2c8/0x700 [ 4824.716622] worker_thread+0x296/0x6e0 [ 4824.720784] ? rescuer_thread+0x570/0x570 [ 4824.722814] kthread+0x1d1/0x200 [ 4824.731335] ? set_kthread_struct+0x70/0x70 [ 4824.732857] ret_from_fork+0x1f/0x30 [ 4836.052620] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 4836.055345] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4836.076800] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4836.083534] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4836.089509] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4836.093647] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4836.105599] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4836.112373] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4836.114710] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 4836.129126] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 4836.137660] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 4836.141415] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 4836.145210] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 4836.149171] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4836.151647] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 4836.154595] Call Trace: [ 4836.155473] [ 4836.156392] ? show_regs.cold.9+0x22/0x2f [ 4836.158200] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4836.160329] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4836.162552] ? watchdog+0x40/0x40 [ 4836.164262] ? hrtimer_interrupt+0x128/0x2c0 [ 4836.166214] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4836.168531] ? apic_timer_interrupt+0xf/0x20 [ 4836.170600] [ 4836.171706] ? native_safe_halt+0x2e/0x50 [ 4836.173585] kvm_wait+0xcd/0xe0 [ 4836.174704] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4836.176369] do_raw_spin_lock+0x177/0x190 [ 4836.177860] _raw_spin_lock+0x21/0x30 [ 4836.179091] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4836.181238] ? do_raw_spin_unlock+0x75/0x190 [ 4836.182744] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4836.184691] ? do_wait_intr+0xf0/0xf0 [ 4836.186263] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4836.188594] mdc_reint+0x2d/0x110 [mdc] [ 4836.189971] mdc_unlink+0x658/0x870 [mdc] [ 4836.191721] lmv_unlink+0x34d/0xbd0 [lmv] [ 4836.193499] ll_unlink+0x24d/0x7f0 [lustre] [ 4836.195464] vfs_unlink+0x202/0x330 [ 4836.196693] do_unlinkat+0x2fa/0x460 [ 4836.198327] __x64_sys_unlink+0x37/0x50 [ 4836.199617] do_syscall_64+0xc1/0x440 [ 4836.200866] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4836.202655] RIP: 0033:0x7f151bdee10b [ 4836.203849] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4836.211004] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4836.214694] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 4836.221316] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 4836.225652] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4836.230244] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4836.234340] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4838.369158] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2907s! [ 4838.373131] Showing busy workqueues and worker pools: [ 4838.378427] workqueue events: flags=0x0 [ 4838.382742] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4838.387395] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 4838.395821] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4838.405783] pending: kfree_rcu_monitor [ 4838.407755] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4838.419464] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 4838.434701] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4838.446615] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4838.452236] workqueue events_long: flags=0x0 [ 4838.456238] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4838.461023] pending: lu_objects_flush [obdclass] [ 4838.465865] workqueue events_unbound: flags=0x2 [ 4838.470933] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4838.474827] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4838.486229] workqueue events_power_efficient: flags=0x80 [ 4838.497655] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4838.504729] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 4838.509182] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4838.512948] pending: neigh_periodic_work [ 4838.515291] workqueue mm_percpu_wq: flags=0x8 [ 4838.529063] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4838.536773] pending: vmstat_update, lru_add_drain_per_cpu [ 4838.546950] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4838.556692] pending: vmstat_update [ 4838.560473] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4838.576238] pending: lru_add_drain_per_cpu [ 4838.581483] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4838.589925] pending: lru_add_drain_per_cpu BAR(48) [ 4838.593574] workqueue writeback: flags=0x4a [ 4838.595198] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4838.601625] pending: wb_workfn [ 4838.607492] workqueue xprtiod: flags=0xa [ 4838.617251] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4838.624703] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4838.633919] workqueue ipv6_addrconf: flags=0x40008 [ 4838.641892] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4838.655738] pending: addrconf_verify_work [ 4838.663450] workqueue knbd0-recv: flags=0x1a [ 4838.665096] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4838.670550] in-flight: 658:recv_work [ 4838.680340] workqueue ptlrpc_pinger: flags=0x2 [ 4838.682878] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4838.692366] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4838.705365] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2908s workers=4 idle: 32984 37 8 [ 4838.714815] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4823s workers=2 idle: 661 [ 4844.057012] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 4844.063897] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4844.100551] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4844.111499] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4844.117933] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4844.121650] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4844.135131] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4844.142175] RAX: 00000000000004df RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4844.147979] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 4844.152945] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4844.160951] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 4844.163520] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 4844.165738] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 4844.174575] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4844.178838] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 4844.182575] Call Trace: [ 4844.183786] [ 4844.184966] ? show_regs.cold.9+0x22/0x2f [ 4844.186943] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4844.189477] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4844.191622] ? watchdog+0x40/0x40 [ 4844.193279] ? hrtimer_interrupt+0x128/0x2c0 [ 4844.195287] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4844.198248] ? apic_timer_interrupt+0xf/0x20 [ 4844.200554] [ 4844.201800] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4844.208052] do_raw_spin_lock+0x177/0x190 [ 4844.209990] _raw_spin_lock+0x21/0x30 [ 4844.212545] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4844.214611] ? do_raw_spin_unlock+0x75/0x190 [ 4844.216849] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4844.222841] ? do_wait_intr+0xf0/0xf0 [ 4844.225334] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4844.231459] mdc_close+0x53b/0xd70 [mdc] [ 4844.233018] lmv_close+0x18d/0x490 [lmv] [ 4844.239191] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4844.243060] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4844.252038] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4844.255243] ll_md_close+0x274/0x800 [lustre] [ 4844.260406] ll_file_release+0x128/0xc50 [lustre] [ 4844.266532] __fput+0x10d/0x440 [ 4844.271394] ____fput+0x12/0x20 [ 4844.272559] task_work_run+0xc6/0x110 [ 4844.273908] exit_to_usermode_loop+0x1dd/0x1f0 [ 4844.279734] do_syscall_64+0x3d6/0x440 [ 4844.281796] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4844.284470] RIP: 0033:0x7f151bee3cd8 [ 4844.285781] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4844.304338] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4844.312387] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 4844.315266] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 4844.318292] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4844.328936] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4844.338218] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4852.055022] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 4852.056055] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 4852.060501] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4852.060585] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4852.060592] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4852.060595] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4852.061040] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4852.061045] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4852.061050] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 4852.061202] RDX: 0000000000002c9a RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 4852.061204] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 4852.061206] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 4852.061208] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4852.061211] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 4852.061214] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4852.061216] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4852.061224] Call Trace: [ 4852.061229] [ 4852.061234] ? show_regs.cold.9+0x22/0x2f [ 4852.061243] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4852.061453] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4852.061458] ? watchdog+0x40/0x40 [ 4852.061462] ? hrtimer_interrupt+0x128/0x2c0 [ 4852.061465] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4852.061473] ? apic_timer_interrupt+0xf/0x20 [ 4852.061477] [ 4852.061480] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4852.061485] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4852.061490] do_raw_spin_lock+0x177/0x190 [ 4852.061495] _raw_spin_lock+0x21/0x30 [ 4852.061501] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4852.068379] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4852.068460] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4852.068467] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4852.068474] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4852.068752] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4852.068764] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4852.068768] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4852.068772] RAX: 0000000000005583 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4852.068775] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 4852.068777] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 4852.068779] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 4852.068782] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 4852.068785] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 4852.068788] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4852.068790] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4852.068800] Call Trace: [ 4852.068806] [ 4852.068811] ? show_regs.cold.9+0x22/0x2f [ 4852.068820] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4852.068827] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4852.068831] ? watchdog+0x40/0x40 [ 4852.068835] ? hrtimer_interrupt+0x128/0x2c0 [ 4852.068839] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4852.068846] ? apic_timer_interrupt+0xf/0x20 [ 4852.068851] [ 4852.068853] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4852.068857] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4852.068861] do_raw_spin_lock+0x177/0x190 [ 4852.068865] _raw_spin_lock+0x21/0x30 [ 4852.068869] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4852.069076] process_one_work+0x2c8/0x700 [ 4852.069084] worker_thread+0x296/0x6e0 [ 4852.069089] ? rescuer_thread+0x570/0x570 [ 4852.069093] kthread+0x1d1/0x200 [ 4852.069098] ? set_kthread_struct+0x70/0x70 [ 4852.069103] ret_from_fork+0x1f/0x30 [ 4852.708960] obd_mod_free+0x46/0x140 [ptlrpc] [ 4852.713760] mdc_commit_open+0x91/0xb0 [mdc] [ 4852.717627] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4852.723521] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4852.735985] after_reply+0xe2c/0x1780 [ptlrpc] [ 4852.737767] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4852.746805] ? __next_timer_interrupt+0x160/0x160 [ 4852.749509] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4852.756769] ? do_wait_intr+0xf0/0xf0 [ 4852.760312] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4852.764564] kthread+0x1d1/0x200 [ 4852.771735] ? set_kthread_struct+0x70/0x70 [ 4852.779952] ret_from_fork+0x1f/0x30 [ 4864.053199] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 4864.059239] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4864.101526] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4864.109417] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4864.119179] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4864.125998] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4864.148358] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4864.153997] RAX: 0000000000005c64 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4864.162741] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 4864.174749] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 4864.180517] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 4864.191375] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 4864.201359] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 4864.211526] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4864.217336] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 4864.223702] Call Trace: [ 4864.227030] [ 4864.228588] ? show_regs.cold.9+0x22/0x2f [ 4864.232392] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4864.238245] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4864.240429] ? watchdog+0x40/0x40 [ 4864.243453] ? hrtimer_interrupt+0x128/0x2c0 [ 4864.247770] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4864.253397] ? apic_timer_interrupt+0xf/0x20 [ 4864.258431] [ 4864.260441] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4864.263828] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4864.267669] do_raw_spin_lock+0x177/0x190 [ 4864.270089] _raw_spin_lock+0x21/0x30 [ 4864.277126] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4864.281898] ? do_raw_spin_unlock+0x75/0x190 [ 4864.284466] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4864.290685] ? do_wait_intr+0xf0/0xf0 [ 4864.292858] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4864.302059] mdc_reint+0x2d/0x110 [mdc] [ 4864.307430] mdc_unlink+0x658/0x870 [mdc] [ 4864.308771] lmv_unlink+0x34d/0xbd0 [lmv] [ 4864.315085] ll_unlink+0x24d/0x7f0 [lustre] [ 4864.319295] vfs_unlink+0x202/0x330 [ 4864.322359] do_unlinkat+0x2fa/0x460 [ 4864.324702] __x64_sys_unlink+0x37/0x50 [ 4864.327468] do_syscall_64+0xc1/0x440 [ 4864.329326] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4864.330808] RIP: 0033:0x7f151bdee10b [ 4864.331924] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4864.346485] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4864.354320] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 4864.364418] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 4864.368491] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4864.371800] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4864.373909] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4864.702878] rcu: INFO: rcu_sched self-detected stall on CPU [ 4864.708756] rcu: 1-...!: (1711578 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 4864.721228] (t=2940055 jiffies g=249657 q=3276) [ 4864.725798] rcu: rcu_sched kthread starved for 2935007 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 4864.751105] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4864.762836] rcu: RCU grace-period kthread stack dump: [ 4864.776081] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4864.800566] Call Trace: [ 4864.807392] __schedule+0x351/0xcb0 [ 4864.813181] schedule+0xc0/0x180 [ 4864.814387] schedule_timeout+0xb4/0x190 [ 4864.817772] ? do_raw_spin_unlock+0x75/0x190 [ 4864.819287] ? __next_timer_interrupt+0x160/0x160 [ 4864.829373] ? prepare_to_swait_event+0xf3/0x140 [ 4864.841595] rcu_gp_kthread+0x8dd/0x15b0 [ 4864.850542] ? force_qs_rnp+0x2c0/0x2c0 [ 4864.854117] kthread+0x1d1/0x200 [ 4864.855303] ? set_kthread_struct+0x70/0x70 [ 4864.864087] ret_from_fork+0x1f/0x30 [ 4864.866447] rcu: Stack dump where RCU GP kthread last ran: [ 4864.871070] Sending NMI from CPU 1 to CPUs 2: [ 4864.888084] NMI backtrace for cpu 2 [ 4864.888092] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4864.888094] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4864.888095] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4864.888097] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4864.888099] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4864.888101] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 [ 4864.888102] RAX: 0000000000004025 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4864.888104] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 4864.888105] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 4864.888106] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 4864.888107] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 4864.888109] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 4864.888110] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4864.888111] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4864.888112] Call Trace: [ 4864.888113] [ 4864.888114] ? show_regs.cold.9+0x22/0x2f [ 4864.888115] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 4864.888116] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 4864.888117] ? nmi_handle+0xbd/0x210 [ 4864.888117] ? default_do_nmi+0x4b/0x1b0 [ 4864.888118] ? do_nmi+0x280/0x380 [ 4864.888119] ? end_repeat_nmi+0x16/0x1f [ 4864.888120] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4864.888121] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4864.888123] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4864.888123] [ 4864.888124] do_raw_spin_lock+0x177/0x190 [ 4864.888125] _raw_spin_lock+0x21/0x30 [ 4864.888126] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4864.888127] process_one_work+0x2c8/0x700 [ 4864.888128] worker_thread+0x296/0x6e0 [ 4864.888129] ? rescuer_thread+0x570/0x570 [ 4864.888130] kthread+0x1d1/0x200 [ 4864.888130] ? set_kthread_struct+0x70/0x70 [ 4864.888131] ret_from_fork+0x1f/0x30 [ 4864.892232] NMI backtrace for cpu 1 [ 4865.145037] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4865.155415] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4865.171317] Call Trace: [ 4865.174456] [ 4865.176440] ? dump_stack+0xbb/0x10e [ 4865.179459] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4865.191647] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4865.201401] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4865.208198] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4865.213615] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4865.220424] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 4865.224166] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 4865.230105] ? update_process_times+0x6d/0xc0 [ 4865.233465] ? tick_sched_handle.isra.18+0x27/0x90 [ 4865.239336] ? tick_sched_timer+0x53/0xb0 [ 4865.242795] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4865.249309] ? tick_sched_handle.isra.18+0x90/0x90 [ 4865.250922] ? hrtimer_interrupt+0x128/0x2c0 [ 4865.258185] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4865.261840] ? apic_timer_interrupt+0xf/0x20 [ 4865.265226] [ 4865.269993] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4865.276318] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4865.284575] ? do_raw_spin_lock+0x177/0x190 [ 4865.287626] ? _raw_spin_lock+0x21/0x30 [ 4865.293636] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4865.301329] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 4865.304400] ? mdc_commit_open+0x91/0xb0 [mdc] [ 4865.310363] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4865.316832] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4865.324561] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 4865.328531] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4865.337183] ? __next_timer_interrupt+0x160/0x160 [ 4865.343757] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4865.345658] ? do_wait_intr+0xf0/0xf0 [ 4865.346873] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4865.364344] ? kthread+0x1d1/0x200 [ 4865.369790] ? set_kthread_struct+0x70/0x70 [ 4865.373869] ? ret_from_fork+0x1f/0x30 [ 4869.091031] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2938s! [ 4869.097954] Showing busy workqueues and worker pools: [ 4869.104312] workqueue events: flags=0x0 [ 4869.109206] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4869.118977] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 4869.127833] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4869.133097] pending: kfree_rcu_monitor [ 4869.135331] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4869.144078] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 4869.150285] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4869.154340] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4869.158611] workqueue events_long: flags=0x0 [ 4869.161025] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4869.164633] pending: lu_objects_flush [obdclass] [ 4869.167574] workqueue events_unbound: flags=0x2 [ 4869.170030] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4869.173476] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4869.179923] workqueue events_power_efficient: flags=0x80 [ 4869.182939] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4869.187047] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 4869.190593] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4869.194051] pending: neigh_periodic_work [ 4869.196149] workqueue mm_percpu_wq: flags=0x8 [ 4869.198262] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4869.201367] pending: vmstat_update, lru_add_drain_per_cpu [ 4869.204405] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4869.208396] pending: vmstat_update [ 4869.210242] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4869.213292] pending: lru_add_drain_per_cpu [ 4869.215447] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4869.218708] pending: lru_add_drain_per_cpu BAR(48) [ 4869.221257] workqueue writeback: flags=0x4a [ 4869.223275] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4869.226302] pending: wb_workfn [ 4869.228234] workqueue xprtiod: flags=0xa [ 4869.230373] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4869.233323] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4869.237344] workqueue ipv6_addrconf: flags=0x40008 [ 4869.239819] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4869.246489] pending: addrconf_verify_work [ 4869.252389] workqueue knbd0-recv: flags=0x1a [ 4869.257011] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4869.260645] in-flight: 658:recv_work [ 4869.262407] workqueue ptlrpc_pinger: flags=0x2 [ 4869.264364] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4869.265980] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4869.269327] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2938s workers=4 idle: 32984 37 8 [ 4869.273916] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4854s workers=2 idle: 661 [ 4872.064138] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 4872.074278] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4872.132179] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4872.143222] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4872.155508] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4872.164129] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4872.189641] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4872.192372] RAX: 0000000000003916 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4872.210178] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 4872.217631] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4872.224298] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 4872.233782] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 4872.256413] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 4872.259453] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4872.265502] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 4872.284344] Call Trace: [ 4872.286212] [ 4872.291677] ? show_regs.cold.9+0x22/0x2f [ 4872.293161] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4872.294904] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4872.313126] ? watchdog+0x40/0x40 [ 4872.316498] ? hrtimer_interrupt+0x128/0x2c0 [ 4872.321263] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4872.328021] ? apic_timer_interrupt+0xf/0x20 [ 4872.332131] [ 4872.334051] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4872.338106] do_raw_spin_lock+0x177/0x190 [ 4872.340538] _raw_spin_lock+0x21/0x30 [ 4872.343423] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4872.347217] ? do_raw_spin_unlock+0x75/0x190 [ 4872.354596] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4872.357976] ? do_wait_intr+0xf0/0xf0 [ 4872.361856] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4872.369220] mdc_close+0x53b/0xd70 [mdc] [ 4872.371963] lmv_close+0x18d/0x490 [lmv] [ 4872.373433] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4872.378646] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4872.383451] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4872.386264] ll_md_close+0x274/0x800 [lustre] [ 4872.391923] ll_file_release+0x128/0xc50 [lustre] [ 4872.394308] __fput+0x10d/0x440 [ 4872.396531] ____fput+0x12/0x20 [ 4872.398299] task_work_run+0xc6/0x110 [ 4872.400415] exit_to_usermode_loop+0x1dd/0x1f0 [ 4872.407539] do_syscall_64+0x3d6/0x440 [ 4872.410461] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4872.414407] RIP: 0033:0x7f151bee3cd8 [ 4872.416424] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4872.431855] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4872.448488] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 4872.456094] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 4872.470518] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4872.473176] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4872.481723] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4892.055069] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 4892.056128] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 4892.058322] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4892.058400] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4892.058407] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4892.058409] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4892.058424] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4892.058428] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4892.058433] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 4892.058435] RDX: 0000000000000b12 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 4892.058437] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 4892.058440] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 4892.058442] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4892.058445] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 4892.058448] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4892.058451] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4892.058460] Call Trace: [ 4892.058466] [ 4892.058471] ? show_regs.cold.9+0x22/0x2f [ 4892.058481] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4892.058488] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4892.058492] ? watchdog+0x40/0x40 [ 4892.058495] ? hrtimer_interrupt+0x128/0x2c0 [ 4892.058499] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4892.058507] ? apic_timer_interrupt+0xf/0x20 [ 4892.058511] [ 4892.058513] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4892.058518] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4892.058522] do_raw_spin_lock+0x177/0x190 [ 4892.058526] _raw_spin_lock+0x21/0x30 [ 4892.058532] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4892.058825] obd_mod_free+0x46/0x140 [ptlrpc] [ 4892.059018] mdc_commit_open+0x91/0xb0 [mdc] [ 4892.059060] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4892.059254] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4892.059445] after_reply+0xe2c/0x1780 [ptlrpc] [ 4892.059636] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4892.059838] ? __next_timer_interrupt+0x160/0x160 [ 4892.059848] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4892.060050] ? do_wait_intr+0xf0/0xf0 [ 4892.060059] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4892.060254] kthread+0x1d1/0x200 [ 4892.060260] ? set_kthread_struct+0x70/0x70 [ 4892.060263] ret_from_fork+0x1f/0x30 [ 4892.328263] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4892.365609] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4892.372372] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4892.379243] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4892.387150] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4892.393946] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4892.415322] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4892.424483] RAX: 0000000000007e8b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4892.433979] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 4892.437586] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 4892.442432] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 4892.445581] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 4892.456926] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 4892.466560] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4892.478853] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4892.498203] Call Trace: [ 4892.499692] [ 4892.500738] ? show_regs.cold.9+0x22/0x2f [ 4892.507071] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4892.511039] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4892.517486] ? watchdog+0x40/0x40 [ 4892.519812] ? hrtimer_interrupt+0x128/0x2c0 [ 4892.526394] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4892.530494] ? apic_timer_interrupt+0xf/0x20 [ 4892.534264] [ 4892.537502] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4892.542438] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4892.549194] do_raw_spin_lock+0x177/0x190 [ 4892.550625] _raw_spin_lock+0x21/0x30 [ 4892.554200] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4892.558051] process_one_work+0x2c8/0x700 [ 4892.560585] worker_thread+0x296/0x6e0 [ 4892.566179] ? rescuer_thread+0x570/0x570 [ 4892.569555] kthread+0x1d1/0x200 [ 4892.571702] ? set_kthread_struct+0x70/0x70 [ 4892.574511] ret_from_fork+0x1f/0x30 [ 4896.052453] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 4896.058523] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4896.099260] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4896.110357] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4896.117167] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4896.120164] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4896.130143] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4896.135510] RAX: 000000000000794c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4896.139531] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 4896.146917] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 4896.153019] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 4896.158735] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 4896.162402] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 4896.169065] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4896.177183] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 4896.185300] Call Trace: [ 4896.189047] [ 4896.189796] ? show_regs.cold.9+0x22/0x2f [ 4896.191241] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4896.198338] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4896.201173] ? watchdog+0x40/0x40 [ 4896.205131] ? hrtimer_interrupt+0x128/0x2c0 [ 4896.210156] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4896.212220] ? apic_timer_interrupt+0xf/0x20 [ 4896.218868] [ 4896.219587] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4896.221450] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4896.230137] do_raw_spin_lock+0x177/0x190 [ 4896.237478] _raw_spin_lock+0x21/0x30 [ 4896.245240] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4896.251177] ? do_raw_spin_unlock+0x75/0x190 [ 4896.256796] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4896.261439] ? do_wait_intr+0xf0/0xf0 [ 4896.265523] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4896.269405] mdc_reint+0x2d/0x110 [mdc] [ 4896.271814] mdc_unlink+0x658/0x870 [mdc] [ 4896.277193] lmv_unlink+0x34d/0xbd0 [lmv] [ 4896.281900] ll_unlink+0x24d/0x7f0 [lustre] [ 4896.289529] vfs_unlink+0x202/0x330 [ 4896.293854] do_unlinkat+0x2fa/0x460 [ 4896.295260] __x64_sys_unlink+0x37/0x50 [ 4896.302046] do_syscall_64+0xc1/0x440 [ 4896.304583] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4896.306243] RIP: 0033:0x7f151bdee10b [ 4896.310710] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4896.320199] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4896.324476] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 4896.338421] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 4896.352399] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4896.361136] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4896.369163] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4899.303946] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2968s! [ 4899.314226] Showing busy workqueues and worker pools: [ 4899.316163] workqueue events: flags=0x0 [ 4899.324930] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4899.330080] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 4899.344895] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4899.352177] pending: kfree_rcu_monitor [ 4899.355179] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4899.364330] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 4899.380462] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4899.384684] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4899.402864] workqueue events_long: flags=0x0 [ 4899.407812] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4899.422271] pending: lu_objects_flush [obdclass] [ 4899.430000] workqueue events_unbound: flags=0x2 [ 4899.438938] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4899.448672] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4899.464781] workqueue events_power_efficient: flags=0x80 [ 4899.475138] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4899.490790] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 4899.495266] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4899.502195] pending: neigh_periodic_work [ 4899.508474] workqueue mm_percpu_wq: flags=0x8 [ 4899.518416] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4899.523189] pending: vmstat_update, lru_add_drain_per_cpu [ 4899.525883] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4899.529552] pending: vmstat_update [ 4899.531852] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4899.536705] pending: lru_add_drain_per_cpu [ 4899.537822] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4899.544196] pending: lru_add_drain_per_cpu BAR(48) [ 4899.553993] workqueue writeback: flags=0x4a [ 4899.556289] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4899.558904] pending: wb_workfn [ 4899.563500] workqueue xprtiod: flags=0xa [ 4899.564855] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4899.568827] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4899.574852] workqueue ipv6_addrconf: flags=0x40008 [ 4899.576441] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4899.580384] pending: addrconf_verify_work [ 4899.581941] workqueue knbd0-recv: flags=0x1a [ 4899.584860] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4899.588756] in-flight: 658:recv_work [ 4899.590245] workqueue ptlrpc_pinger: flags=0x2 [ 4899.593469] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4899.596098] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4899.605352] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2969s workers=4 idle: 32984 37 8 [ 4899.612287] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4884s workers=2 idle: 661 [ 4900.057079] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 4900.059694] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4900.086798] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4900.095429] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4900.103457] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4900.105027] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4900.113068] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4900.117834] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4900.121614] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 4900.123961] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4900.126408] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 4900.136117] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 4900.140901] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 4900.145838] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4900.152063] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 4900.161896] Call Trace: [ 4900.163912] [ 4900.164498] ? show_regs.cold.9+0x22/0x2f [ 4900.166724] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4900.167912] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4900.170145] ? watchdog+0x40/0x40 [ 4900.171097] ? hrtimer_interrupt+0x128/0x2c0 [ 4900.173371] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4900.176802] ? apic_timer_interrupt+0xf/0x20 [ 4900.178810] [ 4900.181480] ? native_safe_halt+0x2e/0x50 [ 4900.183638] kvm_wait+0xcd/0xe0 [ 4900.184989] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4900.187373] do_raw_spin_lock+0x177/0x190 [ 4900.189082] _raw_spin_lock+0x21/0x30 [ 4900.192923] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4900.195419] ? do_raw_spin_unlock+0x75/0x190 [ 4900.197063] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4900.204105] ? do_wait_intr+0xf0/0xf0 [ 4900.206036] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4900.208585] mdc_close+0x53b/0xd70 [mdc] [ 4900.210543] lmv_close+0x18d/0x490 [lmv] [ 4900.212058] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4900.214557] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4900.216937] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4900.219289] ll_md_close+0x274/0x800 [lustre] [ 4900.221057] ll_file_release+0x128/0xc50 [lustre] [ 4900.222964] __fput+0x10d/0x440 [ 4900.224212] ____fput+0x12/0x20 [ 4900.225447] task_work_run+0xc6/0x110 [ 4900.226899] exit_to_usermode_loop+0x1dd/0x1f0 [ 4900.228912] do_syscall_64+0x3d6/0x440 [ 4900.233183] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4900.240830] RIP: 0033:0x7f151bee3cd8 [ 4900.244462] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4900.254913] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4900.257352] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 4900.264355] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 4900.267470] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4900.271124] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4900.273149] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4920.055659] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 4920.061656] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) [ 4920.061745] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 4920.061760] obdecho(O) [ 4920.066692] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4920.066844] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4920.066852] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4920.066868] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4920.067140] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4920.067150] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4920.067154] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4920.067159] RAX: 0000000000006a1a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4920.067162] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 4920.067163] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 4920.067166] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 4920.067168] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 4920.067171] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 4920.067174] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4920.067176] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4920.067184] Call Trace: [ 4920.067189] [ 4920.067195] ? show_regs.cold.9+0x22/0x2f [ 4920.067205] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4920.067211] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4920.067215] ? watchdog+0x40/0x40 [ 4920.067220] ? hrtimer_interrupt+0x128/0x2c0 [ 4920.067224] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4920.067232] ? apic_timer_interrupt+0xf/0x20 [ 4920.067236] [ 4920.067239] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4920.067244] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4920.067248] do_raw_spin_lock+0x177/0x190 [ 4920.067253] _raw_spin_lock+0x21/0x30 [ 4920.067257] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4920.082479] mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4920.082541] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4920.082548] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4920.082550] RIP: 0010:load_balance+0x725/0x1090 [ 4920.082562] Code: 48 83 00 01 9c 58 0f 1f 44 00 00 48 83 05 7a 6d dd 02 01 48 83 05 82 b9 dd 02 01 f6 c4 02 0f 85 22 03 00 00 48 8b 3c 24 57 9d <0f> 1f 44 00 00 8b 84 24 98 00 00 00 a8 02 74 23 83 e0 fd 48 83 05 [ 4920.082566] RSP: 0018:ffffb2cb80d24de0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4920.082571] RAX: 0000000000000002 RBX: ffff8ee547ffa870 RCX: ffff8ee5820b3440 [ 4920.082574] RDX: 0000000000000008 RSI: ffff8ee547ffa7c0 RDI: 0000000000000202 [ 4920.082576] RBP: ffffb2cb80d24e90 R08: ffff8ee5499927c0 R09: 0000000000000000 [ 4920.082578] R10: ffff8ee547ffa858 R11: 0000000000009e6a R12: ffff8ee547ffa7c0 [ 4920.082581] R13: ffff8ee582033e08 R14: 0000000000000009 R15: ffff8ee582033380 [ 4920.082583] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 4920.082586] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4920.082588] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4920.082598] Call Trace: [ 4920.082603] [ 4920.082608] ? show_regs.cold.9+0x22/0x2f [ 4920.082618] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4920.082624] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4920.082628] ? watchdog+0x40/0x40 [ 4920.082633] ? hrtimer_interrupt+0x128/0x2c0 [ 4920.082637] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4920.082644] ? apic_timer_interrupt+0xf/0x20 [ 4920.082649] ? apic_timer_interrupt+0xa/0x20 [ 4920.082655] ? load_balance+0x725/0x1090 [ 4920.082661] rebalance_domains+0x348/0x580 [ 4920.082666] ? kvm_clock_get_cycles+0x2c/0x50 [ 4920.082673] run_rebalance_domains+0x68/0xa0 [ 4920.082677] __do_softirq+0x134/0x4e8 [ 4920.082683] irq_exit_rcu+0x1c6/0x1e0 [ 4920.082688] irq_exit+0x12/0x30 [ 4920.082697] smp_apic_timer_interrupt+0xe0/0x270 [ 4920.082702] apic_timer_interrupt+0xf/0x20 [ 4920.082709] [ 4920.082710] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4920.082719] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4920.082722] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4920.082726] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 4920.082728] RDX: 0000000000001d03 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 4920.082730] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 4920.082732] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 4920.082733] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4920.082739] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4920.082743] do_raw_spin_lock+0x177/0x190 [ 4920.082747] _raw_spin_lock+0x21/0x30 [ 4920.082752] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4920.083359] obd_mod_free+0x46/0x140 [ptlrpc] [ 4920.083569] mdc_commit_open+0x91/0xb0 [mdc] [ 4920.083605] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4920.083800] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4920.084530] after_reply+0xe2c/0x1780 [ptlrpc] [ 4920.084727] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4920.086407] process_one_work+0x2c8/0x700 [ 4920.086422] worker_thread+0x296/0x6e0 [ 4920.086427] ? rescuer_thread+0x570/0x570 [ 4920.086431] kthread+0x1d1/0x200 [ 4920.086436] ? set_kthread_struct+0x70/0x70 [ 4920.086439] ret_from_fork+0x1f/0x30 [ 4920.624779] ? __next_timer_interrupt+0x160/0x160 [ 4920.626857] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4920.628447] ? do_wait_intr+0xf0/0xf0 [ 4920.631252] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4920.633930] kthread+0x1d1/0x200 [ 4920.635468] ? set_kthread_struct+0x70/0x70 [ 4920.637195] ret_from_fork+0x1f/0x30 [ 4924.052051] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 4924.057207] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4924.088939] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4924.096986] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4924.108918] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4924.110837] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4924.121548] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4924.128345] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4924.130298] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 4924.136635] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 4924.143741] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 4924.149667] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 4924.153630] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 4924.157817] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4924.159323] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 4924.164545] Call Trace: [ 4924.165726] [ 4924.166662] ? show_regs.cold.9+0x22/0x2f [ 4924.168395] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4924.170743] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4924.172689] ? watchdog+0x40/0x40 [ 4924.173952] ? hrtimer_interrupt+0x128/0x2c0 [ 4924.175803] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4924.177851] ? apic_timer_interrupt+0xf/0x20 [ 4924.179867] [ 4924.180803] ? native_safe_halt+0x2e/0x50 [ 4924.182583] kvm_wait+0xcd/0xe0 [ 4924.183830] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4924.193820] do_raw_spin_lock+0x177/0x190 [ 4924.200043] _raw_spin_lock+0x21/0x30 [ 4924.201791] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4924.205496] ? do_raw_spin_unlock+0x75/0x190 [ 4924.206682] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4924.208288] ? do_wait_intr+0xf0/0xf0 [ 4924.221967] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4924.226791] mdc_reint+0x2d/0x110 [mdc] [ 4924.235343] mdc_unlink+0x658/0x870 [mdc] [ 4924.236887] lmv_unlink+0x34d/0xbd0 [lmv] [ 4924.248587] ll_unlink+0x24d/0x7f0 [lustre] [ 4924.252847] vfs_unlink+0x202/0x330 [ 4924.259860] do_unlinkat+0x2fa/0x460 [ 4924.263940] __x64_sys_unlink+0x37/0x50 [ 4924.266816] do_syscall_64+0xc1/0x440 [ 4924.269921] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4924.277520] RIP: 0033:0x7f151bdee10b [ 4924.279604] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4924.290846] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4924.296755] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 4924.299916] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 4924.304647] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4924.312005] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4924.313993] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4928.058615] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 4928.067214] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4928.122656] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4928.134287] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4928.148047] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4928.153597] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4928.183881] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4928.192822] RAX: 000000000000438f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4928.201450] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 4928.208749] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4928.220525] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 4928.226077] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 4928.235835] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 4928.247926] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4928.268179] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 4928.284763] Call Trace: [ 4928.285641] [ 4928.286377] ? show_regs.cold.9+0x22/0x2f [ 4928.291156] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4928.295360] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4928.299968] ? watchdog+0x40/0x40 [ 4928.302054] ? hrtimer_interrupt+0x128/0x2c0 [ 4928.308638] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4928.316359] ? apic_timer_interrupt+0xf/0x20 [ 4928.318206] [ 4928.318978] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4928.320924] do_raw_spin_lock+0x177/0x190 [ 4928.323377] _raw_spin_lock+0x21/0x30 [ 4928.324756] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4928.331146] ? do_raw_spin_unlock+0x75/0x190 [ 4928.335207] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4928.341895] ? do_wait_intr+0xf0/0xf0 [ 4928.343948] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4928.351134] mdc_close+0x53b/0xd70 [mdc] [ 4928.357497] lmv_close+0x18d/0x490 [lmv] [ 4928.366088] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4928.374091] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4928.380305] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4928.384085] ll_md_close+0x274/0x800 [lustre] [ 4928.390234] ll_file_release+0x128/0xc50 [lustre] [ 4928.404176] __fput+0x10d/0x440 [ 4928.411044] ____fput+0x12/0x20 [ 4928.416059] task_work_run+0xc6/0x110 [ 4928.418111] exit_to_usermode_loop+0x1dd/0x1f0 [ 4928.427213] do_syscall_64+0x3d6/0x440 [ 4928.428611] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4928.439529] RIP: 0033:0x7f151bee3cd8 [ 4928.442244] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4928.475063] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4928.480521] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 4928.492762] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 4928.501810] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4928.514572] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4928.532925] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4930.016252] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2999s! [ 4930.020689] Showing busy workqueues and worker pools: [ 4930.025577] workqueue events: flags=0x0 [ 4930.027825] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4930.037646] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 4930.043056] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4930.050882] pending: kfree_rcu_monitor [ 4930.053217] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4930.063504] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 4930.076993] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4930.084855] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4930.098943] workqueue events_long: flags=0x0 [ 4930.103649] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4930.109275] pending: lu_objects_flush [obdclass] [ 4930.121341] workqueue events_unbound: flags=0x2 [ 4930.128580] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4930.134377] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4930.147244] workqueue events_power_efficient: flags=0x80 [ 4930.149075] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4930.165501] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 4930.174174] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4930.183548] pending: neigh_periodic_work [ 4930.186454] workqueue mm_percpu_wq: flags=0x8 [ 4930.189328] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4930.197435] pending: vmstat_update, lru_add_drain_per_cpu [ 4930.205393] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4930.219706] pending: vmstat_update [ 4930.232414] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4930.234905] pending: lru_add_drain_per_cpu [ 4930.245313] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4930.250799] pending: lru_add_drain_per_cpu BAR(48) [ 4930.260507] workqueue writeback: flags=0x4a [ 4930.268891] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4930.273309] pending: wb_workfn [ 4930.275781] workqueue xprtiod: flags=0xa [ 4930.280793] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4930.287524] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4930.297756] workqueue ipv6_addrconf: flags=0x40008 [ 4930.304315] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4930.309722] pending: addrconf_verify_work [ 4930.317487] workqueue knbd0-recv: flags=0x1a [ 4930.326615] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4930.334402] in-flight: 658:recv_work [ 4930.341012] workqueue ptlrpc_pinger: flags=0x2 [ 4930.348761] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4930.357719] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4930.368561] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2999s workers=4 idle: 32984 37 8 [ 4930.383610] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4915s workers=2 idle: 661 [ 4948.055281] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 4948.056136] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 4948.058882] Modules linked in: lustre(O) [ 4948.063872] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4948.064016] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4948.064024] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4948.064030] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4948.064434] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4948.064445] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4948.064449] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4948.064453] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4948.064455] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 4948.064458] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 4948.064460] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 4948.064462] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 4948.064464] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 4948.064467] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4948.064470] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4948.064478] Call Trace: [ 4948.064483] [ 4948.064488] ? show_regs.cold.9+0x22/0x2f [ 4948.064497] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4948.064503] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4948.064507] ? watchdog+0x40/0x40 [ 4948.064511] ? hrtimer_interrupt+0x128/0x2c0 [ 4948.064515] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4948.064522] ? apic_timer_interrupt+0xf/0x20 [ 4948.064526] [ 4948.064529] ? native_safe_halt+0x2e/0x50 [ 4948.064532] kvm_wait+0xcd/0xe0 [ 4948.064539] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4948.064546] do_raw_spin_lock+0x177/0x190 [ 4948.064550] _raw_spin_lock+0x21/0x30 [ 4948.064553] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4948.064983] process_one_work+0x2c8/0x700 [ 4948.064990] worker_thread+0x296/0x6e0 [ 4948.064995] ? rescuer_thread+0x570/0x570 [ 4948.064998] kthread+0x1d1/0x200 [ 4948.065002] ? set_kthread_struct+0x70/0x70 [ 4948.065006] ret_from_fork+0x1f/0x30 [ 4948.196746] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4948.219162] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4948.225188] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4948.229724] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4948.232300] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4948.246042] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4948.254428] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 4948.258394] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 4948.262272] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 4948.265738] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 4948.269825] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4948.274648] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 4948.280510] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4948.283394] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4948.287108] Call Trace: [ 4948.288437] [ 4948.289405] ? show_regs.cold.9+0x22/0x2f [ 4948.291745] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4948.294408] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4948.296713] ? watchdog+0x40/0x40 [ 4948.298381] ? hrtimer_interrupt+0x128/0x2c0 [ 4948.300552] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4948.303113] ? apic_timer_interrupt+0xf/0x20 [ 4948.305458] [ 4948.306584] ? native_safe_halt+0x2e/0x50 [ 4948.309901] kvm_wait+0xcd/0xe0 [ 4948.313463] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4948.316386] do_raw_spin_lock+0x177/0x190 [ 4948.319222] _raw_spin_lock+0x21/0x30 [ 4948.321114] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4948.326509] obd_mod_free+0x46/0x140 [ptlrpc] [ 4948.330488] mdc_commit_open+0x91/0xb0 [mdc] [ 4948.332827] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4948.337246] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4948.341679] after_reply+0xe2c/0x1780 [ptlrpc] [ 4948.344435] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4948.346516] ? __next_timer_interrupt+0x160/0x160 [ 4948.350294] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4948.353473] ? do_wait_intr+0xf0/0xf0 [ 4948.355392] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4948.357148] kthread+0x1d1/0x200 [ 4948.359359] ? set_kthread_struct+0x70/0x70 [ 4948.362301] ret_from_fork+0x1f/0x30 [ 4956.052589] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 4956.056855] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4956.060184] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 4956.075758] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4956.075769] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4956.078319] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4956.078393] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4956.078399] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4956.078402] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4956.078414] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4956.078417] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4956.078421] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4956.078423] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 4956.078425] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4956.078427] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 4956.078429] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 4956.078432] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 4956.078434] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4956.078436] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 4956.078443] Call Trace: [ 4956.078447] [ 4956.078452] ? show_regs.cold.9+0x22/0x2f [ 4956.078460] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4956.078466] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4956.078469] ? watchdog+0x40/0x40 [ 4956.078473] ? hrtimer_interrupt+0x128/0x2c0 [ 4956.078476] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4956.078484] ? apic_timer_interrupt+0xf/0x20 [ 4956.078488] [ 4956.078490] ? native_safe_halt+0x2e/0x50 [ 4956.078493] kvm_wait+0xcd/0xe0 [ 4956.078501] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4956.078509] do_raw_spin_lock+0x177/0x190 [ 4956.078513] _raw_spin_lock+0x21/0x30 [ 4956.078516] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4956.078797] ? do_raw_spin_unlock+0x75/0x190 [ 4956.078802] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4956.078999] ? do_wait_intr+0xf0/0xf0 [ 4956.079007] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4956.079207] mdc_close+0x53b/0xd70 [mdc] [ 4956.079242] lmv_close+0x18d/0x490 [lmv] [ 4956.079264] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4956.079409] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4956.083628] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4956.083649] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4956.083653] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 4956.083659] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4956.083661] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 4956.083664] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 4956.083666] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 4956.083669] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 4956.083672] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 4956.083675] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4956.083678] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 4956.083685] Call Trace: [ 4956.083690] [ 4956.083695] ? show_regs.cold.9+0x22/0x2f [ 4956.083705] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4956.083710] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4956.083715] ? watchdog+0x40/0x40 [ 4956.083719] ? hrtimer_interrupt+0x128/0x2c0 [ 4956.083723] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4956.083730] ? apic_timer_interrupt+0xf/0x20 [ 4956.083734] [ 4956.083737] ? native_safe_halt+0x2e/0x50 [ 4956.083741] kvm_wait+0xcd/0xe0 [ 4956.083748] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4956.083756] do_raw_spin_lock+0x177/0x190 [ 4956.083760] _raw_spin_lock+0x21/0x30 [ 4956.083766] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4956.084150] ? do_raw_spin_unlock+0x75/0x190 [ 4956.084156] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4956.084388] ? do_wait_intr+0xf0/0xf0 [ 4956.084394] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4956.084619] mdc_reint+0x2d/0x110 [mdc] [ 4956.084654] mdc_unlink+0x658/0x870 [mdc] [ 4956.084681] lmv_unlink+0x34d/0xbd0 [lmv] [ 4956.084710] ll_unlink+0x24d/0x7f0 [lustre] [ 4956.087413] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4956.087545] ll_md_close+0x274/0x800 [lustre] [ 4956.087633] ll_file_release+0x128/0xc50 [lustre] [ 4956.087721] __fput+0x10d/0x440 [ 4956.087731] ____fput+0x12/0x20 [ 4956.087736] task_work_run+0xc6/0x110 [ 4956.087744] exit_to_usermode_loop+0x1dd/0x1f0 [ 4956.087752] do_syscall_64+0x3d6/0x440 [ 4956.087756] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4956.087764] RIP: 0033:0x7f151bee3cd8 [ 4956.087771] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4956.087774] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4956.087779] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 4956.087780] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 4956.087783] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4956.087784] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4956.087786] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4956.496829] vfs_unlink+0x202/0x330 [ 4956.502814] do_unlinkat+0x2fa/0x460 [ 4956.504643] __x64_sys_unlink+0x37/0x50 [ 4956.519855] do_syscall_64+0xc1/0x440 [ 4956.525260] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4956.527717] RIP: 0033:0x7f151bdee10b [ 4956.534303] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4956.550339] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4956.561662] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 4956.566320] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 4956.570651] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4956.576742] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4956.581302] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4960.750467] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3030s! [ 4960.757730] Showing busy workqueues and worker pools: [ 4960.763417] workqueue events: flags=0x0 [ 4960.768608] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4960.776162] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 4960.788676] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4960.798396] pending: kfree_rcu_monitor [ 4960.800776] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4960.805132] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 4960.810407] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4960.813649] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4960.817692] workqueue events_long: flags=0x0 [ 4960.819816] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4960.823283] pending: lu_objects_flush [obdclass] [ 4960.826493] workqueue events_unbound: flags=0x2 [ 4960.828845] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4960.832313] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4960.837524] workqueue events_power_efficient: flags=0x80 [ 4960.840038] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4960.843100] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 4960.846105] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4960.849044] pending: neigh_periodic_work [ 4960.851142] workqueue mm_percpu_wq: flags=0x8 [ 4960.853115] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4960.856214] pending: vmstat_update, lru_add_drain_per_cpu [ 4960.858881] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4960.861794] pending: vmstat_update [ 4960.863553] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4960.866603] pending: lru_add_drain_per_cpu [ 4960.868596] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4960.871543] pending: lru_add_drain_per_cpu BAR(48) [ 4960.873688] workqueue writeback: flags=0x4a [ 4960.875625] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4960.878401] pending: wb_workfn [ 4960.879795] workqueue xprtiod: flags=0xa [ 4960.881320] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4960.883548] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4960.886782] workqueue ipv6_addrconf: flags=0x40008 [ 4960.888865] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4960.891735] pending: addrconf_verify_work [ 4960.894423] workqueue knbd0-recv: flags=0x1a [ 4960.900090] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4960.904912] in-flight: 658:recv_work [ 4960.908231] workqueue ptlrpc_pinger: flags=0x2 [ 4960.910401] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4960.913183] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4960.915881] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3030s workers=4 idle: 32984 37 8 [ 4960.919225] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4945s workers=2 idle: 661 [ 4976.055111] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 4976.056010] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 4976.058049] Modules linked in: [ 4976.060641] Modules linked in: [ 4976.060643] lustre(O) osp(O) ofd(O) [ 4976.062150] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4976.062224] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4976.062231] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4976.062238] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4976.063162] lod(O) [ 4976.064503] [ 4976.088457] mdt(O) mdd(O) [ 4976.092837] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4976.097775] mgs(O) [ 4976.100244] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4976.100255] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4976.100262] RAX: 000000000000002e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4976.100264] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 4976.100266] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 4976.100268] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 4976.100270] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 4976.100274] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 4976.101559] lquota(O) [ 4976.102181] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4976.102185] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4976.102194] Call Trace: [ 4976.102201] [ 4976.102206] ? show_regs.cold.9+0x22/0x2f [ 4976.102220] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4976.102227] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4976.102231] ? watchdog+0x40/0x40 [ 4976.102236] ? hrtimer_interrupt+0x128/0x2c0 [ 4976.102240] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4976.102248] ? apic_timer_interrupt+0xf/0x20 [ 4976.102253] [ 4976.102256] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4976.102263] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4976.103705] lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4976.103776] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4976.103784] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4976.103787] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4976.103800] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4976.103803] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4976.103808] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 4976.103810] RDX: 00000000000021df RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 4976.103813] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 4976.103815] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 4976.103817] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4976.103821] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 4976.103823] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4976.103826] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 4976.103834] Call Trace: [ 4976.103838] [ 4976.103843] ? show_regs.cold.9+0x22/0x2f [ 4976.103853] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4976.103859] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4976.103863] ? watchdog+0x40/0x40 [ 4976.103867] ? hrtimer_interrupt+0x128/0x2c0 [ 4976.106511] do_raw_spin_lock+0x177/0x190 [ 4976.108000] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4976.116300] _raw_spin_lock+0x21/0x30 [ 4976.119673] ? apic_timer_interrupt+0xf/0x20 [ 4976.119693] [ 4976.119698] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4976.119705] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4976.119709] do_raw_spin_lock+0x177/0x190 [ 4976.119715] _raw_spin_lock+0x21/0x30 [ 4976.119722] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4976.120144] obd_mod_free+0x46/0x140 [ptlrpc] [ 4976.120372] mdc_commit_open+0x91/0xb0 [mdc] [ 4976.120410] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4976.120658] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4976.120944] after_reply+0xe2c/0x1780 [ptlrpc] [ 4976.121154] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4976.121359] ? __next_timer_interrupt+0x160/0x160 [ 4976.121369] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4976.124311] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4976.124613] process_one_work+0x2c8/0x700 [ 4976.124622] worker_thread+0x296/0x6e0 [ 4976.124627] ? rescuer_thread+0x570/0x570 [ 4976.124631] kthread+0x1d1/0x200 [ 4976.124636] ? set_kthread_struct+0x70/0x70 [ 4976.124639] ret_from_fork+0x1f/0x30 [ 4976.314273] ? do_wait_intr+0xf0/0xf0 [ 4976.315631] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4976.317570] kthread+0x1d1/0x200 [ 4976.318988] ? set_kthread_struct+0x70/0x70 [ 4976.320650] ret_from_fork+0x1f/0x30 [ 4984.057285] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 4984.062611] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4984.083613] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4984.089069] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4984.094619] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4984.098302] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4984.106331] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4984.109376] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4984.112433] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 4984.115436] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4984.118608] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 4984.121632] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 4984.124583] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 4984.127932] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4984.130264] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 4984.133050] Call Trace: [ 4984.134155] [ 4984.134989] ? show_regs.cold.9+0x22/0x2f [ 4984.137241] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4984.139204] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4984.141188] ? watchdog+0x40/0x40 [ 4984.142600] ? hrtimer_interrupt+0x128/0x2c0 [ 4984.144454] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4984.146475] ? apic_timer_interrupt+0xf/0x20 [ 4984.150073] [ 4984.150944] ? native_safe_halt+0x2e/0x50 [ 4984.152942] kvm_wait+0xcd/0xe0 [ 4984.154269] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4984.156380] do_raw_spin_lock+0x177/0x190 [ 4984.158310] _raw_spin_lock+0x21/0x30 [ 4984.159904] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4984.162405] ? do_raw_spin_unlock+0x75/0x190 [ 4984.164682] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4984.167214] ? do_wait_intr+0xf0/0xf0 [ 4984.168851] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4984.171641] mdc_close+0x53b/0xd70 [mdc] [ 4984.173424] lmv_close+0x18d/0x490 [lmv] [ 4984.175475] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4984.178059] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 4984.180660] ll_md_real_close+0xa5/0x2a0 [lustre] [ 4984.182908] ll_md_close+0x274/0x800 [lustre] [ 4984.184873] ll_file_release+0x128/0xc50 [lustre] [ 4984.187209] __fput+0x10d/0x440 [ 4984.188403] ____fput+0x12/0x20 [ 4984.189786] task_work_run+0xc6/0x110 [ 4984.191332] exit_to_usermode_loop+0x1dd/0x1f0 [ 4984.193381] do_syscall_64+0x3d6/0x440 [ 4984.194889] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4984.196925] RIP: 0033:0x7f151bee3cd8 [ 4984.198501] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 4984.206647] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 4984.209664] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 4984.212684] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 4984.215781] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4984.220630] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4984.222581] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4988.054299] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 4988.061916] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4988.098905] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4988.106513] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4988.116980] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4988.126697] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4988.136919] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4988.139957] RAX: 0000000000007d0c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 4988.142687] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 4988.145120] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 4988.147718] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 4988.150276] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 4988.157681] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 4988.161233] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4988.165583] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 4988.169025] Call Trace: [ 4988.170064] [ 4988.170948] ? show_regs.cold.9+0x22/0x2f [ 4988.172509] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4988.174423] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4988.176341] ? watchdog+0x40/0x40 [ 4988.177617] ? hrtimer_interrupt+0x128/0x2c0 [ 4988.179223] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4988.181086] ? apic_timer_interrupt+0xf/0x20 [ 4988.182860] [ 4988.183788] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4988.185999] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4988.188043] do_raw_spin_lock+0x177/0x190 [ 4988.189792] _raw_spin_lock+0x21/0x30 [ 4988.191234] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4988.193612] ? do_raw_spin_unlock+0x75/0x190 [ 4988.195265] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4988.197247] ? do_wait_intr+0xf0/0xf0 [ 4988.198509] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4988.200878] mdc_reint+0x2d/0x110 [mdc] [ 4988.202154] mdc_unlink+0x658/0x870 [mdc] [ 4988.203507] lmv_unlink+0x34d/0xbd0 [lmv] [ 4988.205197] ll_unlink+0x24d/0x7f0 [lustre] [ 4988.218387] vfs_unlink+0x202/0x330 [ 4988.220049] do_unlinkat+0x2fa/0x460 [ 4988.221554] __x64_sys_unlink+0x37/0x50 [ 4988.222612] do_syscall_64+0xc1/0x440 [ 4988.224821] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4988.227239] RIP: 0033:0x7f151bdee10b [ 4988.228994] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4988.236410] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4988.239346] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 4988.242199] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 4988.244878] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 4988.247174] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 4988.249867] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4990.944736] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3060s! [ 4990.946695] Showing busy workqueues and worker pools: [ 4990.950130] workqueue events: flags=0x0 [ 4990.951831] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4990.954061] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 4990.959878] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4990.962930] pending: kfree_rcu_monitor [ 4990.964819] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4990.967519] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 4990.971583] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4990.975117] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 4990.982075] workqueue events_long: flags=0x0 [ 4990.984455] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4990.987366] pending: lu_objects_flush [obdclass] [ 4990.990282] workqueue events_unbound: flags=0x2 [ 4990.994282] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 4990.996046] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 4991.005318] workqueue events_power_efficient: flags=0x80 [ 4991.007577] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4991.012738] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 4991.014627] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4991.018030] pending: neigh_periodic_work [ 4991.022398] workqueue mm_percpu_wq: flags=0x8 [ 4991.024379] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4991.030238] pending: vmstat_update, lru_add_drain_per_cpu [ 4991.033283] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4991.037606] pending: vmstat_update [ 4991.039267] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4991.042138] pending: lru_add_drain_per_cpu [ 4991.043989] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4991.048223] pending: lru_add_drain_per_cpu BAR(48) [ 4991.050596] workqueue writeback: flags=0x4a [ 4991.052606] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 4991.057905] pending: wb_workfn [ 4991.059481] workqueue xprtiod: flags=0xa [ 4991.061650] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 4991.070219] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 4991.074392] workqueue ipv6_addrconf: flags=0x40008 [ 4991.076916] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4991.081385] pending: addrconf_verify_work [ 4991.083958] workqueue knbd0-recv: flags=0x1a [ 4991.085464] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4991.090442] in-flight: 658:recv_work [ 4991.092333] workqueue ptlrpc_pinger: flags=0x2 [ 4991.095226] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4991.098348] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 4991.101669] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3060s workers=4 idle: 32984 37 8 [ 4991.105807] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4975s workers=2 idle: 661 [ 5004.055018] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 5004.058643] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5004.059949] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5004.139134] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5004.139154] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5004.139159] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5004.139179] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5004.139183] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5004.139190] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 5004.139193] RDX: 00000000000048f3 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 5004.139195] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 5004.139197] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 5004.139200] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5004.139203] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 5004.139206] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5004.139209] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5004.139217] Call Trace: [ 5004.139224] [ 5004.139233] ? show_regs.cold.9+0x22/0x2f [ 5004.139246] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5004.139253] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5004.139259] ? watchdog+0x40/0x40 [ 5004.139263] ? hrtimer_interrupt+0x128/0x2c0 [ 5004.139267] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5004.139279] ? apic_timer_interrupt+0xf/0x20 [ 5004.139284] [ 5004.139351] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5004.139359] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5004.139363] do_raw_spin_lock+0x177/0x190 [ 5004.139369] _raw_spin_lock+0x21/0x30 [ 5004.139380] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5004.145150] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5004.145230] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5004.145237] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5004.145243] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5004.155003] obd_mod_free+0x46/0x140 [ptlrpc] [ 5004.155479] mdc_commit_open+0x91/0xb0 [mdc] [ 5004.155681] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5004.161190] [ 5004.161204] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5004.161223] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5004.161227] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5004.161236] RAX: 0000000000006d0b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5004.161239] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 5004.161241] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 5004.161243] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 5004.161246] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 5004.161255] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 5004.161258] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5004.161261] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5004.161268] Call Trace: [ 5004.161274] [ 5004.161279] ? show_regs.cold.9+0x22/0x2f [ 5004.161295] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5004.161306] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5004.161310] ? watchdog+0x40/0x40 [ 5004.161315] ? hrtimer_interrupt+0x128/0x2c0 [ 5004.161319] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5004.161326] ? apic_timer_interrupt+0xf/0x20 [ 5004.161335] [ 5004.161338] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5004.161346] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5004.161352] do_raw_spin_lock+0x177/0x190 [ 5004.161357] _raw_spin_lock+0x21/0x30 [ 5004.161362] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5004.161737] process_one_work+0x2c8/0x700 [ 5004.161744] worker_thread+0x296/0x6e0 [ 5004.161748] ? rescuer_thread+0x570/0x570 [ 5004.161751] kthread+0x1d1/0x200 [ 5004.161756] ? set_kthread_struct+0x70/0x70 [ 5004.161759] ret_from_fork+0x1f/0x30 [ 5004.368456] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5004.371295] after_reply+0xe2c/0x1780 [ptlrpc] [ 5004.374290] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5004.377824] ? __next_timer_interrupt+0x160/0x160 [ 5004.380250] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5004.383207] ? do_wait_intr+0xf0/0xf0 [ 5004.385880] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5004.390695] kthread+0x1d1/0x200 [ 5004.391753] ? set_kthread_struct+0x70/0x70 [ 5004.395143] ret_from_fork+0x1f/0x30 [ 5012.062076] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 5012.068393] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5012.110040] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5012.118718] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5012.128036] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5012.132966] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5012.149636] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5012.160304] RAX: 0000000000005008 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5012.171114] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 5012.178194] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5012.184621] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 5012.187513] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 5012.193375] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 5012.199357] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5012.208433] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 5012.215020] Call Trace: [ 5012.218382] [ 5012.220201] ? show_regs.cold.9+0x22/0x2f [ 5012.223241] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5012.231246] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5012.234019] ? watchdog+0x40/0x40 [ 5012.237833] ? hrtimer_interrupt+0x128/0x2c0 [ 5012.239377] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5012.245430] ? apic_timer_interrupt+0xf/0x20 [ 5012.250404] [ 5012.254642] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5012.257836] do_raw_spin_lock+0x177/0x190 [ 5012.260170] _raw_spin_lock+0x21/0x30 [ 5012.263669] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5012.266228] ? do_raw_spin_unlock+0x75/0x190 [ 5012.278047] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5012.284921] ? do_wait_intr+0xf0/0xf0 [ 5012.287667] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5012.291038] mdc_close+0x53b/0xd70 [mdc] [ 5012.293048] lmv_close+0x18d/0x490 [lmv] [ 5012.296968] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5012.300915] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5012.304303] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5012.308708] ll_md_close+0x274/0x800 [lustre] [ 5012.311688] ll_file_release+0x128/0xc50 [lustre] [ 5012.317756] __fput+0x10d/0x440 [ 5012.318538] ____fput+0x12/0x20 [ 5012.319582] task_work_run+0xc6/0x110 [ 5012.325937] exit_to_usermode_loop+0x1dd/0x1f0 [ 5012.335255] do_syscall_64+0x3d6/0x440 [ 5012.336596] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5012.338336] RIP: 0033:0x7f151bee3cd8 [ 5012.347131] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5012.371345] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5012.385767] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 5012.401345] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 5012.404288] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5012.409786] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5012.416699] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5016.052013] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 5016.055881] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5016.072955] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5016.077192] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5016.080611] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5016.082651] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5016.088960] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5016.091781] RAX: 00000000000054c6 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5016.094592] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 5016.097024] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 5016.099825] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 5016.102482] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 5016.104475] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 5016.106691] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5016.108124] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 5016.111069] Call Trace: [ 5016.111838] [ 5016.112699] ? show_regs.cold.9+0x22/0x2f [ 5016.114863] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5016.116771] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5016.118132] ? watchdog+0x40/0x40 [ 5016.119617] ? hrtimer_interrupt+0x128/0x2c0 [ 5016.120868] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5016.123545] ? apic_timer_interrupt+0xf/0x20 [ 5016.128128] [ 5016.130094] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5016.142327] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5016.146751] do_raw_spin_lock+0x177/0x190 [ 5016.148627] _raw_spin_lock+0x21/0x30 [ 5016.150341] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5016.154717] ? do_raw_spin_unlock+0x75/0x190 [ 5016.162055] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5016.167863] ? do_wait_intr+0xf0/0xf0 [ 5016.169229] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5016.179101] mdc_reint+0x2d/0x110 [mdc] [ 5016.182942] mdc_unlink+0x658/0x870 [mdc] [ 5016.186453] lmv_unlink+0x34d/0xbd0 [lmv] [ 5016.187856] ll_unlink+0x24d/0x7f0 [lustre] [ 5016.189782] vfs_unlink+0x202/0x330 [ 5016.191862] do_unlinkat+0x2fa/0x460 [ 5016.193881] __x64_sys_unlink+0x37/0x50 [ 5016.205194] do_syscall_64+0xc1/0x440 [ 5016.206460] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5016.210063] RIP: 0033:0x7f151bdee10b [ 5016.212013] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5016.232913] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5016.239010] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 5016.242227] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 5016.245618] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5016.249796] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5016.255879] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5021.155460] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3090s! [ 5021.164745] Showing busy workqueues and worker pools: [ 5021.168944] workqueue events: flags=0x0 [ 5021.172311] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5021.187808] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 5021.197226] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5021.206675] pending: kfree_rcu_monitor [ 5021.210425] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5021.226836] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 5021.239669] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5021.246852] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5021.254554] workqueue events_long: flags=0x0 [ 5021.258018] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5021.263542] pending: lu_objects_flush [obdclass] [ 5021.267965] workqueue events_unbound: flags=0x2 [ 5021.270583] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5021.279767] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5021.287471] workqueue events_power_efficient: flags=0x80 [ 5021.291375] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5021.300965] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 5021.307436] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5021.310560] pending: neigh_periodic_work [ 5021.313756] workqueue mm_percpu_wq: flags=0x8 [ 5021.319575] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5021.325011] pending: vmstat_update, lru_add_drain_per_cpu [ 5021.327112] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5021.332721] pending: vmstat_update [ 5021.336836] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5021.340534] pending: lru_add_drain_per_cpu [ 5021.342137] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5021.345984] pending: lru_add_drain_per_cpu BAR(48) [ 5021.349683] workqueue writeback: flags=0x4a [ 5021.352218] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5021.359214] pending: wb_workfn [ 5021.361840] workqueue xprtiod: flags=0xa [ 5021.365156] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5021.372306] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5021.378820] workqueue ipv6_addrconf: flags=0x40008 [ 5021.383246] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5021.385902] pending: addrconf_verify_work [ 5021.387760] workqueue knbd0-recv: flags=0x1a [ 5021.389162] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5021.396111] in-flight: 658:recv_work [ 5021.398954] workqueue ptlrpc_pinger: flags=0x2 [ 5021.402315] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5021.405592] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5021.409014] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3090s workers=4 idle: 32984 37 8 [ 5021.413988] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5006s workers=2 idle: 661 [ 5032.055056] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 5032.056026] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5032.059701] Modules linked in: lustre(O) [ 5032.063590] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5032.063752] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5032.063763] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5032.063772] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5032.064118] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5032.064132] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5032.064136] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5032.064141] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5032.064143] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 5032.064145] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 5032.064147] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 5032.064149] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 5032.064153] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 5032.064155] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5032.064157] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5032.064165] Call Trace: [ 5032.064169] [ 5032.064175] ? show_regs.cold.9+0x22/0x2f [ 5032.064185] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5032.064192] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5032.064196] ? watchdog+0x40/0x40 [ 5032.064201] ? hrtimer_interrupt+0x128/0x2c0 [ 5032.064205] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5032.064212] ? apic_timer_interrupt+0xf/0x20 [ 5032.064216] [ 5032.064219] ? native_safe_halt+0x2e/0x50 [ 5032.064222] kvm_wait+0xcd/0xe0 [ 5032.064229] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5032.064305] do_raw_spin_lock+0x177/0x190 [ 5032.064312] _raw_spin_lock+0x21/0x30 [ 5032.064317] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5032.064590] process_one_work+0x2c8/0x700 [ 5032.064673] worker_thread+0x296/0x6e0 [ 5032.064677] ? rescuer_thread+0x570/0x570 [ 5032.064685] kthread+0x1d1/0x200 [ 5032.064689] ? set_kthread_struct+0x70/0x70 [ 5032.064693] ret_from_fork+0x1f/0x30 [ 5032.292813] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5032.312077] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5032.317133] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5032.322159] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5032.324207] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5032.337184] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5032.343914] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 5032.347577] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 5032.351022] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 5032.354530] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 5032.357549] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5032.360505] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 5032.363039] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5032.365552] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5032.368879] Call Trace: [ 5032.370071] [ 5032.370981] ? show_regs.cold.9+0x22/0x2f [ 5032.372884] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5032.375022] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5032.378743] ? watchdog+0x40/0x40 [ 5032.380259] ? hrtimer_interrupt+0x128/0x2c0 [ 5032.382649] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5032.384859] ? apic_timer_interrupt+0xf/0x20 [ 5032.387833] [ 5032.389125] ? native_safe_halt+0x2e/0x50 [ 5032.391869] kvm_wait+0xcd/0xe0 [ 5032.393215] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5032.397019] do_raw_spin_lock+0x177/0x190 [ 5032.399223] _raw_spin_lock+0x21/0x30 [ 5032.403225] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5032.412101] obd_mod_free+0x46/0x140 [ptlrpc] [ 5032.415520] mdc_commit_open+0x91/0xb0 [mdc] [ 5032.417033] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5032.426184] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5032.430142] after_reply+0xe2c/0x1780 [ptlrpc] [ 5032.434773] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5032.441510] ? __next_timer_interrupt+0x160/0x160 [ 5032.444584] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5032.447649] ? do_wait_intr+0xf0/0xf0 [ 5032.449645] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5032.455976] kthread+0x1d1/0x200 [ 5032.460560] ? set_kthread_struct+0x70/0x70 [ 5032.464598] ret_from_fork+0x1f/0x30 [ 5040.060415] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 5040.066789] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5040.100789] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5040.110120] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5040.116575] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5040.119941] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5040.133610] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5040.139709] RAX: 0000000000007d8f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5040.146077] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 5040.156056] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5040.158782] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 5040.162154] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 5040.168471] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 5040.173374] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5040.175794] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 5040.185031] Call Trace: [ 5040.186004] [ 5040.186746] ? show_regs.cold.9+0x22/0x2f [ 5040.191857] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5040.197476] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5040.207268] ? watchdog+0x40/0x40 [ 5040.208915] ? hrtimer_interrupt+0x128/0x2c0 [ 5040.213250] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5040.219354] ? apic_timer_interrupt+0xf/0x20 [ 5040.220294] [ 5040.220767] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5040.231847] do_raw_spin_lock+0x177/0x190 [ 5040.233269] _raw_spin_lock+0x21/0x30 [ 5040.236635] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5040.241628] ? do_raw_spin_unlock+0x75/0x190 [ 5040.243254] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5040.249645] ? do_wait_intr+0xf0/0xf0 [ 5040.251052] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5040.254529] mdc_close+0x53b/0xd70 [mdc] [ 5040.256002] lmv_close+0x18d/0x490 [lmv] [ 5040.259096] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5040.264055] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5040.272174] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5040.276255] ll_md_close+0x274/0x800 [lustre] [ 5040.277984] ll_file_release+0x128/0xc50 [lustre] [ 5040.282092] __fput+0x10d/0x440 [ 5040.283234] ____fput+0x12/0x20 [ 5040.289934] task_work_run+0xc6/0x110 [ 5040.292157] exit_to_usermode_loop+0x1dd/0x1f0 [ 5040.298056] do_syscall_64+0x3d6/0x440 [ 5040.303024] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5040.311727] RIP: 0033:0x7f151bee3cd8 [ 5040.317826] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5040.335713] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5040.345669] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 5040.348275] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 5040.354864] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5040.358692] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5040.361952] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5044.705061] rcu: INFO: rcu_sched self-detected stall on CPU [ 5044.707960] rcu: 1-...!: (1810928 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 5044.713896] (t=3120058 jiffies g=249657 q=3331) [ 5044.716497] rcu: rcu_sched kthread starved for 3115010 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 5044.724505] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5044.728305] rcu: RCU grace-period kthread stack dump: [ 5044.730759] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5044.735664] Call Trace: [ 5044.736984] __schedule+0x351/0xcb0 [ 5044.738657] schedule+0xc0/0x180 [ 5044.739713] schedule_timeout+0xb4/0x190 [ 5044.741506] ? do_raw_spin_unlock+0x75/0x190 [ 5044.743460] ? __next_timer_interrupt+0x160/0x160 [ 5044.745294] ? prepare_to_swait_event+0xf3/0x140 [ 5044.747512] rcu_gp_kthread+0x8dd/0x15b0 [ 5044.749560] ? force_qs_rnp+0x2c0/0x2c0 [ 5044.751348] kthread+0x1d1/0x200 [ 5044.752764] ? set_kthread_struct+0x70/0x70 [ 5044.755342] ret_from_fork+0x1f/0x30 [ 5044.756801] rcu: Stack dump where RCU GP kthread last ran: [ 5044.759219] Sending NMI from CPU 1 to CPUs 3: [ 5044.760989] NMI backtrace for cpu 3 skipped: idling at native_safe_halt+0x2e/0x50 [ 5044.764011] NMI backtrace for cpu 1 [ 5044.766069] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5044.776655] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5044.785240] Call Trace: [ 5044.787395] [ 5044.789549] ? dump_stack+0xbb/0x10e [ 5044.790799] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5044.797717] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5044.803064] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5044.805987] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5044.810115] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5044.812722] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 5044.814499] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 5044.816091] ? update_process_times+0x6d/0xc0 [ 5044.817866] ? tick_sched_handle.isra.18+0x27/0x90 [ 5044.823036] ? tick_sched_timer+0x53/0xb0 [ 5044.829837] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5044.831521] ? tick_sched_handle.isra.18+0x90/0x90 [ 5044.833718] ? hrtimer_interrupt+0x128/0x2c0 [ 5044.835652] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5044.838936] ? apic_timer_interrupt+0xf/0x20 [ 5044.840810] [ 5044.841724] ? native_safe_halt+0x2e/0x50 [ 5044.843427] ? kvm_wait+0xcd/0xe0 [ 5044.844724] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5044.846624] ? do_raw_spin_lock+0x177/0x190 [ 5044.847867] ? _raw_spin_lock+0x21/0x30 [ 5044.849692] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5044.852224] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5044.854457] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5044.856395] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5044.858756] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5044.861342] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5044.863977] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5044.866393] ? __next_timer_interrupt+0x160/0x160 [ 5044.868186] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5044.870082] ? do_wait_intr+0xf0/0xf0 [ 5044.871348] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5044.873577] ? kthread+0x1d1/0x200 [ 5044.874900] ? set_kthread_struct+0x70/0x70 [ 5044.876643] ? ret_from_fork+0x1f/0x30 [ 5048.052077] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 5048.056063] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5048.082031] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5048.087523] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5048.091999] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5048.094810] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5048.105356] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5048.113349] RAX: 0000000000005526 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5048.120073] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 5048.127215] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 5048.134546] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 5048.140180] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 5048.144201] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 5048.147594] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5048.150195] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 5048.153024] Call Trace: [ 5048.154190] [ 5048.154962] ? show_regs.cold.9+0x22/0x2f [ 5048.156995] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5048.158868] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5048.161030] ? watchdog+0x40/0x40 [ 5048.162494] ? hrtimer_interrupt+0x128/0x2c0 [ 5048.164542] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5048.166525] ? apic_timer_interrupt+0xf/0x20 [ 5048.168671] [ 5048.169528] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5048.171500] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5048.173661] do_raw_spin_lock+0x177/0x190 [ 5048.175334] _raw_spin_lock+0x21/0x30 [ 5048.176818] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5048.178752] ? do_raw_spin_unlock+0x75/0x190 [ 5048.180493] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5048.182609] ? do_wait_intr+0xf0/0xf0 [ 5048.183703] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5048.186881] mdc_reint+0x2d/0x110 [mdc] [ 5048.188542] mdc_unlink+0x658/0x870 [mdc] [ 5048.190257] lmv_unlink+0x34d/0xbd0 [lmv] [ 5048.192083] ll_unlink+0x24d/0x7f0 [lustre] [ 5048.193745] vfs_unlink+0x202/0x330 [ 5048.194749] do_unlinkat+0x2fa/0x460 [ 5048.196041] __x64_sys_unlink+0x37/0x50 [ 5048.197750] do_syscall_64+0xc1/0x440 [ 5048.199037] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5048.201705] RIP: 0033:0x7f151bdee10b [ 5048.205075] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5048.222419] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5048.228345] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 5048.231752] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 5048.235038] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5048.238472] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5048.242265] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5051.883167] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3121s! [ 5051.892993] Showing busy workqueues and worker pools: [ 5051.896396] workqueue events: flags=0x0 [ 5051.898490] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5051.902856] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 5051.907704] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5051.911089] pending: kfree_rcu_monitor [ 5051.913044] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5051.916558] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 5051.922551] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5051.926512] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5051.931323] workqueue events_long: flags=0x0 [ 5051.933496] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5051.937211] pending: lu_objects_flush [obdclass] [ 5051.940469] workqueue events_unbound: flags=0x2 [ 5051.942829] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5051.946008] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5051.952600] workqueue events_power_efficient: flags=0x80 [ 5051.955855] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5051.959739] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 5051.962749] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5051.966733] pending: neigh_periodic_work [ 5051.968803] workqueue mm_percpu_wq: flags=0x8 [ 5051.971455] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5051.974841] pending: vmstat_update, lru_add_drain_per_cpu [ 5051.977890] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5051.981853] pending: vmstat_update [ 5051.983958] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5051.987410] pending: lru_add_drain_per_cpu [ 5051.989594] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5051.993330] pending: lru_add_drain_per_cpu BAR(48) [ 5051.995957] workqueue writeback: flags=0x4a [ 5051.998014] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5052.001794] pending: wb_workfn [ 5052.004321] workqueue xprtiod: flags=0xa [ 5052.006374] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5052.009386] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5052.013526] workqueue ipv6_addrconf: flags=0x40008 [ 5052.015611] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5052.017696] pending: addrconf_verify_work [ 5052.019780] workqueue knbd0-recv: flags=0x1a [ 5052.022270] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5052.025066] in-flight: 658:recv_work [ 5052.027228] workqueue ptlrpc_pinger: flags=0x2 [ 5052.029492] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5052.041015] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5052.048394] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3121s workers=4 idle: 32984 37 8 [ 5052.057814] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5036s workers=2 idle: 661 [ 5060.060969] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5060.068631] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5060.132489] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5060.141898] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5060.146177] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5060.150260] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5060.153388] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5060.162171] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5060.172561] RAX: 0000000000005a8d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5060.182879] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 5060.190522] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 5060.196392] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 5060.204768] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 5060.213412] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 5060.221605] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5060.229850] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5060.233789] Call Trace: [ 5060.240618] [ 5060.242614] ? show_regs.cold.9+0x22/0x2f [ 5060.248801] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5060.261335] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5060.268353] ? watchdog+0x40/0x40 [ 5060.271561] ? hrtimer_interrupt+0x128/0x2c0 [ 5060.278653] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5060.285651] ? apic_timer_interrupt+0xf/0x20 [ 5060.292394] [ 5060.293834] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5060.295368] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5060.301404] do_raw_spin_lock+0x177/0x190 [ 5060.303925] _raw_spin_lock+0x21/0x30 [ 5060.307553] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5060.311926] process_one_work+0x2c8/0x700 [ 5060.315463] worker_thread+0x296/0x6e0 [ 5060.316782] ? rescuer_thread+0x570/0x570 [ 5060.319791] kthread+0x1d1/0x200 [ 5060.320949] ? set_kthread_struct+0x70/0x70 [ 5060.323685] ret_from_fork+0x1f/0x30 [ 5072.055054] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 5072.057226] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5072.058045] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 5072.109466] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5072.109480] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5072.109483] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5072.109496] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5072.109499] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5072.109505] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 5072.109507] RDX: 0000000000003e64 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 5072.109508] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 5072.109511] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 5072.109513] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5072.109515] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 5072.109517] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5072.109519] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5072.109527] Call Trace: [ 5072.109533] [ 5072.109538] ? show_regs.cold.9+0x22/0x2f [ 5072.109547] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5072.109552] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5072.109556] ? watchdog+0x40/0x40 [ 5072.109559] ? hrtimer_interrupt+0x128/0x2c0 [ 5072.109563] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5072.109570] ? apic_timer_interrupt+0xf/0x20 [ 5072.109574] [ 5072.109576] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5072.109581] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5072.109585] do_raw_spin_lock+0x177/0x190 [ 5072.109589] _raw_spin_lock+0x21/0x30 [ 5072.109596] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5072.113543] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5072.113630] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5072.113638] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5072.113640] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5072.113655] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5072.113659] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5072.113663] RAX: 0000000000006758 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5072.113665] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 5072.113667] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5072.113670] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 5072.113672] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 5072.113675] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 5072.113678] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5072.113679] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 5072.113688] Call Trace: [ 5072.113693] [ 5072.113698] ? show_regs.cold.9+0x22/0x2f [ 5072.113707] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5072.113713] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5072.113717] ? watchdog+0x40/0x40 [ 5072.113721] ? hrtimer_interrupt+0x128/0x2c0 [ 5072.113726] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5072.113733] ? apic_timer_interrupt+0xf/0x20 [ 5072.113738] [ 5072.113741] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5072.113748] do_raw_spin_lock+0x177/0x190 [ 5072.113753] _raw_spin_lock+0x21/0x30 [ 5072.113757] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5072.114043] ? do_raw_spin_unlock+0x75/0x190 [ 5072.114049] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5072.114270] ? do_wait_intr+0xf0/0xf0 [ 5072.114277] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5072.114489] mdc_close+0x53b/0xd70 [mdc] [ 5072.114529] lmv_close+0x18d/0x490 [lmv] [ 5072.114557] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5072.114724] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5072.114842] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5072.114922] ll_md_close+0x274/0x800 [lustre] [ 5072.115018] ll_file_release+0x128/0xc50 [lustre] [ 5072.115094] __fput+0x10d/0x440 [ 5072.115103] ____fput+0x12/0x20 [ 5072.115107] task_work_run+0xc6/0x110 [ 5072.115114] exit_to_usermode_loop+0x1dd/0x1f0 [ 5072.115120] do_syscall_64+0x3d6/0x440 [ 5072.115124] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5072.115130] RIP: 0033:0x7f151bee3cd8 [ 5072.115137] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5072.115140] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5072.115144] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 5072.115146] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 5072.115148] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5072.115150] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5072.115152] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5072.518933] obd_mod_free+0x46/0x140 [ptlrpc] [ 5072.524818] mdc_commit_open+0x91/0xb0 [mdc] [ 5072.531074] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5072.537493] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5072.541731] after_reply+0xe2c/0x1780 [ptlrpc] [ 5072.551331] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5072.560481] ? __next_timer_interrupt+0x160/0x160 [ 5072.569428] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5072.578697] ? do_wait_intr+0xf0/0xf0 [ 5072.580101] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5072.589833] kthread+0x1d1/0x200 [ 5072.591075] ? set_kthread_struct+0x70/0x70 [ 5072.602882] ret_from_fork+0x1f/0x30 [ 5080.052848] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 5080.074879] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5080.127267] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5080.137208] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5080.142289] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5080.146924] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5080.154097] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5080.159726] RAX: 0000000000003d41 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5080.164312] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 5080.170684] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 5080.175712] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 5080.180037] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 5080.183261] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 5080.187167] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5080.190877] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 5080.198331] Call Trace: [ 5080.205036] [ 5080.207074] ? show_regs.cold.9+0x22/0x2f [ 5080.211758] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5080.219362] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5080.220813] ? watchdog+0x40/0x40 [ 5080.228874] ? hrtimer_interrupt+0x128/0x2c0 [ 5080.234656] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5080.237957] ? apic_timer_interrupt+0xf/0x20 [ 5080.240882] [ 5080.249022] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5080.253280] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5080.261004] do_raw_spin_lock+0x177/0x190 [ 5080.264932] _raw_spin_lock+0x21/0x30 [ 5080.269368] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5080.275836] ? do_raw_spin_unlock+0x75/0x190 [ 5080.277683] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5080.280711] ? do_wait_intr+0xf0/0xf0 [ 5080.282731] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5080.286158] mdc_reint+0x2d/0x110 [mdc] [ 5080.288869] mdc_unlink+0x658/0x870 [mdc] [ 5080.291041] lmv_unlink+0x34d/0xbd0 [lmv] [ 5080.293558] ll_unlink+0x24d/0x7f0 [lustre] [ 5080.296275] vfs_unlink+0x202/0x330 [ 5080.299128] do_unlinkat+0x2fa/0x460 [ 5080.301926] __x64_sys_unlink+0x37/0x50 [ 5080.304867] do_syscall_64+0xc1/0x440 [ 5080.307088] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5080.311097] RIP: 0033:0x7f151bdee10b [ 5080.314249] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5080.326991] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5080.336233] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 5080.343345] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 5080.352213] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5080.354322] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5080.371924] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5082.080983] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3151s! [ 5082.094118] Showing busy workqueues and worker pools: [ 5082.100942] workqueue events: flags=0x0 [ 5082.110171] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5082.126514] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 5082.137059] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5082.146998] pending: kfree_rcu_monitor [ 5082.152407] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5082.159168] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 5082.175885] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5082.179907] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5082.187728] workqueue events_long: flags=0x0 [ 5082.190931] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5082.196199] pending: lu_objects_flush [obdclass] [ 5082.200059] workqueue events_unbound: flags=0x2 [ 5082.209327] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5082.217101] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5082.229143] workqueue events_power_efficient: flags=0x80 [ 5082.236519] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5082.243372] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 5082.250065] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5082.255583] pending: neigh_periodic_work [ 5082.261613] workqueue mm_percpu_wq: flags=0x8 [ 5082.266422] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5082.272985] pending: vmstat_update, lru_add_drain_per_cpu [ 5082.278913] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5082.283090] pending: vmstat_update [ 5082.286444] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5082.293572] pending: lru_add_drain_per_cpu [ 5082.299319] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5082.303998] pending: lru_add_drain_per_cpu BAR(48) [ 5082.311524] workqueue writeback: flags=0x4a [ 5082.314071] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5082.323673] pending: wb_workfn [ 5082.326575] workqueue xprtiod: flags=0xa [ 5082.332044] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5082.337263] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5082.346962] workqueue ipv6_addrconf: flags=0x40008 [ 5082.360374] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5082.376535] pending: addrconf_verify_work [ 5082.383927] workqueue knbd0-recv: flags=0x1a [ 5082.390610] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5082.400031] in-flight: 658:recv_work [ 5082.401317] workqueue ptlrpc_pinger: flags=0x2 [ 5082.402833] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5082.411945] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5082.425621] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3151s workers=4 idle: 32984 37 8 [ 5082.433945] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5067s workers=2 idle: 661 [ 5088.056181] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5088.065511] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5088.118751] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5088.126715] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5088.130497] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5088.133727] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5088.136692] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5088.145569] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5088.148948] RAX: 000000000000695c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5088.156956] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 5088.159579] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 5088.168685] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 5088.175375] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 5088.178488] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 5088.182250] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5088.184625] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5088.188346] Call Trace: [ 5088.189554] [ 5088.191306] ? show_regs.cold.9+0x22/0x2f [ 5088.193185] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5088.195310] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5088.197341] ? watchdog+0x40/0x40 [ 5088.198677] ? hrtimer_interrupt+0x128/0x2c0 [ 5088.200499] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5088.202655] ? apic_timer_interrupt+0xf/0x20 [ 5088.204565] [ 5088.205435] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5088.207638] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5088.209969] do_raw_spin_lock+0x177/0x190 [ 5088.211874] _raw_spin_lock+0x21/0x30 [ 5088.213479] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5088.216180] process_one_work+0x2c8/0x700 [ 5088.217871] worker_thread+0x296/0x6e0 [ 5088.219609] ? rescuer_thread+0x570/0x570 [ 5088.221690] kthread+0x1d1/0x200 [ 5088.223206] ? set_kthread_struct+0x70/0x70 [ 5088.224984] ret_from_fork+0x1f/0x30 [ 5100.058532] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 5100.061077] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 5100.061451] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5100.061522] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5100.061528] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5100.061530] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5100.061544] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5100.061548] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5100.061552] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 5100.061554] RDX: 000000000000226b RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 5100.061557] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 5100.061559] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 5100.061561] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5100.061564] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 5100.061566] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5100.061568] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5100.061575] Call Trace: [ 5100.061580] [ 5100.062858] ? show_regs.cold.9+0x22/0x2f [ 5100.062874] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5100.062881] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5100.062887] ? watchdog+0x40/0x40 [ 5100.062891] ? hrtimer_interrupt+0x128/0x2c0 [ 5100.062896] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5100.062905] ? apic_timer_interrupt+0xf/0x20 [ 5100.062909] [ 5100.062914] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5100.062920] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5100.062925] do_raw_spin_lock+0x177/0x190 [ 5100.062931] _raw_spin_lock+0x21/0x30 [ 5100.062940] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5100.063239] obd_mod_free+0x46/0x140 [ptlrpc] [ 5100.063439] mdc_commit_open+0x91/0xb0 [mdc] [ 5100.063479] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5100.064183] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5100.064388] after_reply+0xe2c/0x1780 [ptlrpc] [ 5100.064599] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5100.064819] ? __next_timer_interrupt+0x160/0x160 [ 5100.064827] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5100.065011] ? do_wait_intr+0xf0/0xf0 [ 5100.065020] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5100.065225] kthread+0x1d1/0x200 [ 5100.065231] ? set_kthread_struct+0x70/0x70 [ 5100.065234] ret_from_fork+0x1f/0x30 [ 5100.316060] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5100.356419] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5100.367872] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5100.372439] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5100.385527] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5100.419146] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 5100.426248] RAX: 0000000000007a9f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5100.428442] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 5100.430636] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5100.444542] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 5100.453439] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 5100.459304] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 5100.468103] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5100.470930] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 5100.475120] Call Trace: [ 5100.479209] [ 5100.479937] ? show_regs.cold.9+0x22/0x2f [ 5100.483260] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5100.487428] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5100.492994] ? watchdog+0x40/0x40 [ 5100.498563] ? hrtimer_interrupt+0x128/0x2c0 [ 5100.500912] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5100.505032] ? apic_timer_interrupt+0xf/0x20 [ 5100.510424] [ 5100.512627] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5100.520155] do_raw_spin_lock+0x177/0x190 [ 5100.523059] _raw_spin_lock+0x21/0x30 [ 5100.525135] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5100.529335] ? do_raw_spin_unlock+0x75/0x190 [ 5100.533916] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5100.539298] ? do_wait_intr+0xf0/0xf0 [ 5100.543698] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5100.548153] mdc_close+0x53b/0xd70 [mdc] [ 5100.551588] lmv_close+0x18d/0x490 [lmv] [ 5100.560916] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5100.566380] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5100.573135] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5100.575254] ll_md_close+0x274/0x800 [lustre] [ 5100.577430] ll_file_release+0x128/0xc50 [lustre] [ 5100.579437] __fput+0x10d/0x440 [ 5100.587231] ____fput+0x12/0x20 [ 5100.588372] task_work_run+0xc6/0x110 [ 5100.591755] exit_to_usermode_loop+0x1dd/0x1f0 [ 5100.602400] do_syscall_64+0x3d6/0x440 [ 5100.606376] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5100.610502] RIP: 0033:0x7f151bee3cd8 [ 5100.613729] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5100.624478] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5100.631173] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 5100.638376] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 5100.644853] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5100.647888] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5100.659523] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5108.052936] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 5108.057037] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5108.098043] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5108.108739] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5108.121356] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5108.125400] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5108.148244] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5108.152945] RAX: 0000000000004413 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5108.156594] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 5108.160660] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 5108.164705] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 5108.168688] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 5108.172045] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 5108.176069] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5108.178976] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 5108.182876] Call Trace: [ 5108.184184] [ 5108.185877] ? show_regs.cold.9+0x22/0x2f [ 5108.188053] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5108.190881] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5108.193427] ? watchdog+0x40/0x40 [ 5108.195456] ? hrtimer_interrupt+0x128/0x2c0 [ 5108.198004] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5108.200160] ? apic_timer_interrupt+0xf/0x20 [ 5108.202152] [ 5108.203485] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5108.205742] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5108.208344] do_raw_spin_lock+0x177/0x190 [ 5108.210904] _raw_spin_lock+0x21/0x30 [ 5108.213475] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5108.218698] ? do_raw_spin_unlock+0x75/0x190 [ 5108.220091] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5108.221922] ? do_wait_intr+0xf0/0xf0 [ 5108.223250] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5108.231365] mdc_reint+0x2d/0x110 [mdc] [ 5108.233019] mdc_unlink+0x658/0x870 [mdc] [ 5108.235120] lmv_unlink+0x34d/0xbd0 [lmv] [ 5108.237282] ll_unlink+0x24d/0x7f0 [lustre] [ 5108.239717] vfs_unlink+0x202/0x330 [ 5108.241317] do_unlinkat+0x2fa/0x460 [ 5108.244442] __x64_sys_unlink+0x37/0x50 [ 5108.247403] do_syscall_64+0xc1/0x440 [ 5108.250744] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5108.253924] RIP: 0033:0x7f151bdee10b [ 5108.255985] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5108.267159] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5108.271759] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 5108.276127] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 5108.280050] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5108.283646] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5108.287257] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5112.814312] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3182s! [ 5112.825157] Showing busy workqueues and worker pools: [ 5112.832917] workqueue events: flags=0x0 [ 5112.836257] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5112.845722] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 5112.856530] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5112.865207] pending: kfree_rcu_monitor [ 5112.869611] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5112.880718] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 5112.892018] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5112.904216] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5112.918774] workqueue events_long: flags=0x0 [ 5112.920448] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5112.933930] pending: lu_objects_flush [obdclass] [ 5112.939179] workqueue events_unbound: flags=0x2 [ 5112.941299] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5112.948605] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5112.959232] workqueue events_power_efficient: flags=0x80 [ 5112.965559] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5112.974952] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 5112.985051] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5112.991079] pending: neigh_periodic_work [ 5112.995714] workqueue mm_percpu_wq: flags=0x8 [ 5113.000678] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5113.005896] pending: vmstat_update, lru_add_drain_per_cpu [ 5113.015347] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5113.020263] pending: vmstat_update [ 5113.026194] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5113.031447] pending: lru_add_drain_per_cpu [ 5113.033923] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5113.037630] pending: lru_add_drain_per_cpu BAR(48) [ 5113.040397] workqueue writeback: flags=0x4a [ 5113.042308] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5113.045285] pending: wb_workfn [ 5113.046477] workqueue xprtiod: flags=0xa [ 5113.048131] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5113.050297] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5113.053273] workqueue ipv6_addrconf: flags=0x40008 [ 5113.055092] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5113.057767] pending: addrconf_verify_work [ 5113.059377] workqueue knbd0-recv: flags=0x1a [ 5113.061124] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5113.063905] in-flight: 658:recv_work [ 5113.065538] workqueue ptlrpc_pinger: flags=0x2 [ 5113.067423] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5113.069728] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5113.072415] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3182s workers=4 idle: 32984 37 8 [ 5113.082490] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5097s workers=2 idle: 661 [ 5116.056028] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5116.060118] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5116.104225] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5116.112941] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5116.118300] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5116.125852] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5116.131349] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5116.142099] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5116.145160] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5116.147943] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 5116.150771] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 5116.153702] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 5116.156786] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 5116.164884] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 5116.173160] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5116.179807] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5116.184965] Call Trace: [ 5116.188152] [ 5116.189836] ? show_regs.cold.9+0x22/0x2f [ 5116.192666] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5116.195044] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5116.200296] ? watchdog+0x40/0x40 [ 5116.203400] ? hrtimer_interrupt+0x128/0x2c0 [ 5116.206939] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5116.210044] ? apic_timer_interrupt+0xf/0x20 [ 5116.211859] [ 5116.216820] ? native_safe_halt+0x2e/0x50 [ 5116.221022] kvm_wait+0xcd/0xe0 [ 5116.222937] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5116.228587] do_raw_spin_lock+0x177/0x190 [ 5116.230386] _raw_spin_lock+0x21/0x30 [ 5116.234350] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5116.237554] process_one_work+0x2c8/0x700 [ 5116.239417] worker_thread+0x296/0x6e0 [ 5116.241551] ? rescuer_thread+0x570/0x570 [ 5116.248110] kthread+0x1d1/0x200 [ 5116.250684] ? set_kthread_struct+0x70/0x70 [ 5116.256192] ret_from_fork+0x1f/0x30 [ 5128.055015] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 5128.057052] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 5128.058564] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5128.058650] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5128.058657] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5128.058659] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5128.058673] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5128.058678] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5128.058683] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 5128.058685] RDX: 0000000000000cad RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 5128.058687] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 5128.058702] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 5128.058704] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5128.058708] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 5128.058711] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5128.058713] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5128.058789] Call Trace: [ 5128.058795] [ 5128.058801] ? show_regs.cold.9+0x22/0x2f [ 5128.058812] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5128.058819] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5128.058824] ? watchdog+0x40/0x40 [ 5128.058828] ? hrtimer_interrupt+0x128/0x2c0 [ 5128.058832] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5128.058842] ? apic_timer_interrupt+0xf/0x20 [ 5128.058847] [ 5128.058850] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5128.058854] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5128.058859] do_raw_spin_lock+0x177/0x190 [ 5128.058863] _raw_spin_lock+0x21/0x30 [ 5128.058879] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5128.059187] obd_mod_free+0x46/0x140 [ptlrpc] [ 5128.059382] mdc_commit_open+0x91/0xb0 [mdc] [ 5128.059426] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5128.059705] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5128.060041] after_reply+0xe2c/0x1780 [ptlrpc] [ 5128.060248] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5128.060442] ? __next_timer_interrupt+0x160/0x160 [ 5128.060461] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5128.060669] ? do_wait_intr+0xf0/0xf0 [ 5128.060679] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5128.061181] kthread+0x1d1/0x200 [ 5128.061188] ? set_kthread_struct+0x70/0x70 [ 5128.061192] ret_from_fork+0x1f/0x30 [ 5128.361120] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5128.391613] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5128.399246] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5128.414363] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5128.417959] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5128.424628] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5128.428938] RAX: 0000000000007e0d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5128.445217] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 5128.453044] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5128.458502] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 5128.462718] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 5128.471285] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 5128.479033] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5128.487323] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 5128.489388] Call Trace: [ 5128.490487] [ 5128.491289] ? show_regs.cold.9+0x22/0x2f [ 5128.493033] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5128.494988] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5128.496717] ? watchdog+0x40/0x40 [ 5128.498086] ? hrtimer_interrupt+0x128/0x2c0 [ 5128.499980] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5128.502077] ? apic_timer_interrupt+0xf/0x20 [ 5128.503965] [ 5128.504837] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5128.507037] do_raw_spin_lock+0x177/0x190 [ 5128.508738] _raw_spin_lock+0x21/0x30 [ 5128.510384] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5128.519084] ? do_raw_spin_unlock+0x75/0x190 [ 5128.523182] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5128.525158] ? do_wait_intr+0xf0/0xf0 [ 5128.533203] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5128.539476] mdc_close+0x53b/0xd70 [mdc] [ 5128.541047] lmv_close+0x18d/0x490 [lmv] [ 5128.542430] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5128.548950] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5128.552485] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5128.555373] ll_md_close+0x274/0x800 [lustre] [ 5128.557124] ll_file_release+0x128/0xc50 [lustre] [ 5128.561427] __fput+0x10d/0x440 [ 5128.566554] ____fput+0x12/0x20 [ 5128.572596] task_work_run+0xc6/0x110 [ 5128.574624] exit_to_usermode_loop+0x1dd/0x1f0 [ 5128.582031] do_syscall_64+0x3d6/0x440 [ 5128.584471] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5128.589387] RIP: 0033:0x7f151bee3cd8 [ 5128.591555] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5128.606180] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5128.609525] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 5128.619598] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 5128.627908] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5128.630266] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5128.642193] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5140.052202] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 5140.055364] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5140.073971] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5140.078439] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5140.081647] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5140.083766] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5140.090268] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5140.093403] RAX: 000000000000648f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5140.096773] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 5140.099821] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 5140.103965] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 5140.107112] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 5140.109986] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 5140.113396] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5140.116403] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 5140.118516] Call Trace: [ 5140.119503] [ 5140.120432] ? show_regs.cold.9+0x22/0x2f [ 5140.121964] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5140.124154] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5140.125896] ? watchdog+0x40/0x40 [ 5140.127396] ? hrtimer_interrupt+0x128/0x2c0 [ 5140.129099] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5140.131049] ? apic_timer_interrupt+0xf/0x20 [ 5140.132884] [ 5140.133811] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5140.136149] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5140.138377] do_raw_spin_lock+0x177/0x190 [ 5140.139990] _raw_spin_lock+0x21/0x30 [ 5140.141480] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5140.143784] ? do_raw_spin_unlock+0x75/0x190 [ 5140.145685] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5140.147945] ? do_wait_intr+0xf0/0xf0 [ 5140.149601] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5140.151511] mdc_reint+0x2d/0x110 [mdc] [ 5140.152941] mdc_unlink+0x658/0x870 [mdc] [ 5140.154585] lmv_unlink+0x34d/0xbd0 [lmv] [ 5140.156253] ll_unlink+0x24d/0x7f0 [lustre] [ 5140.157947] vfs_unlink+0x202/0x330 [ 5140.159365] do_unlinkat+0x2fa/0x460 [ 5140.160472] __x64_sys_unlink+0x37/0x50 [ 5140.161773] do_syscall_64+0xc1/0x440 [ 5140.163147] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5140.165161] RIP: 0033:0x7f151bdee10b [ 5140.166737] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5140.174643] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5140.177458] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 5140.179809] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 5140.182690] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5140.185645] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5140.188461] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5143.526376] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3212s! [ 5143.539680] Showing busy workqueues and worker pools: [ 5143.549026] workqueue events: flags=0x0 [ 5143.555470] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5143.558449] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 5143.568308] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5143.583378] pending: kfree_rcu_monitor [ 5143.586105] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5143.596669] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 5143.614253] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5143.620544] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5143.624459] workqueue events_long: flags=0x0 [ 5143.627570] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5143.634033] pending: lu_objects_flush [obdclass] [ 5143.637012] workqueue events_unbound: flags=0x2 [ 5143.641075] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5143.648235] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5143.665282] workqueue events_power_efficient: flags=0x80 [ 5143.671105] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5143.675406] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 5143.679486] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5143.686691] pending: neigh_periodic_work [ 5143.690063] workqueue mm_percpu_wq: flags=0x8 [ 5143.691958] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5143.695352] pending: vmstat_update, lru_add_drain_per_cpu [ 5143.698271] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5143.701283] pending: vmstat_update [ 5143.702728] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5143.708378] pending: lru_add_drain_per_cpu [ 5143.710699] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5143.715070] pending: lru_add_drain_per_cpu BAR(48) [ 5143.719031] workqueue writeback: flags=0x4a [ 5143.721541] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5143.724382] pending: wb_workfn [ 5143.730670] workqueue xprtiod: flags=0xa [ 5143.732953] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5143.736252] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5143.739730] workqueue ipv6_addrconf: flags=0x40008 [ 5143.742078] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5143.745231] pending: addrconf_verify_work [ 5143.747389] workqueue knbd0-recv: flags=0x1a [ 5143.749136] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5143.752309] in-flight: 658:recv_work [ 5143.754164] workqueue ptlrpc_pinger: flags=0x2 [ 5143.755591] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5143.759459] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5143.763371] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3213s workers=4 idle: 32984 37 8 [ 5143.767488] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5128s workers=2 idle: 661 [ 5144.056027] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5144.061970] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5144.119278] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5144.130368] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5144.140915] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5144.146526] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5144.149641] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5144.181941] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5144.194150] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5144.196797] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 5144.202642] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 5144.212705] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 5144.217402] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 5144.220956] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 5144.229782] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5144.231703] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5144.243865] Call Trace: [ 5144.244746] [ 5144.250596] ? show_regs.cold.9+0x22/0x2f [ 5144.257996] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5144.260758] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5144.266746] ? watchdog+0x40/0x40 [ 5144.270918] ? hrtimer_interrupt+0x128/0x2c0 [ 5144.280912] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5144.282424] ? apic_timer_interrupt+0xf/0x20 [ 5144.297975] [ 5144.302358] ? native_safe_halt+0x2e/0x50 [ 5144.303837] kvm_wait+0xcd/0xe0 [ 5144.313643] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5144.315433] do_raw_spin_lock+0x177/0x190 [ 5144.325030] _raw_spin_lock+0x21/0x30 [ 5144.330812] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5144.343436] process_one_work+0x2c8/0x700 [ 5144.346986] worker_thread+0x296/0x6e0 [ 5144.355498] ? rescuer_thread+0x570/0x570 [ 5144.361574] kthread+0x1d1/0x200 [ 5144.362757] ? set_kthread_struct+0x70/0x70 [ 5144.365610] ret_from_fork+0x1f/0x30 [ 5156.057029] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 5156.058103] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 5156.062498] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5156.062582] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5156.062589] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5156.062592] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5156.062607] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5156.062611] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5156.062617] RAX: 000000000000308b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5156.062619] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 5156.062621] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5156.062624] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 5156.062626] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 5156.062629] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 5156.062633] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5156.062635] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 5156.062644] Call Trace: [ 5156.062650] [ 5156.062655] ? show_regs.cold.9+0x22/0x2f [ 5156.062665] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5156.062671] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5156.062674] ? watchdog+0x40/0x40 [ 5156.062679] ? hrtimer_interrupt+0x128/0x2c0 [ 5156.062683] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5156.062691] ? apic_timer_interrupt+0xf/0x20 [ 5156.062695] [ 5156.062698] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5156.062705] do_raw_spin_lock+0x177/0x190 [ 5156.062710] _raw_spin_lock+0x21/0x30 [ 5156.062714] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5156.063216] ? do_raw_spin_unlock+0x75/0x190 [ 5156.063223] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5156.063465] ? do_wait_intr+0xf0/0xf0 [ 5156.063473] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5156.063868] mdc_close+0x53b/0xd70 [mdc] [ 5156.063910] lmv_close+0x18d/0x490 [lmv] [ 5156.068918] Modules linked in: lustre(O) osp(O) [ 5156.088502] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5156.088681] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5156.088817] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5156.088915] ll_md_close+0x274/0x800 [lustre] [ 5156.089012] ll_file_release+0x128/0xc50 [lustre] [ 5156.089229] __fput+0x10d/0x440 [ 5156.089244] ____fput+0x12/0x20 [ 5156.089249] task_work_run+0xc6/0x110 [ 5156.089256] exit_to_usermode_loop+0x1dd/0x1f0 [ 5156.089264] do_syscall_64+0x3d6/0x440 [ 5156.089272] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5156.089280] RIP: 0033:0x7f151bee3cd8 [ 5156.089291] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5156.089296] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5156.089301] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 5156.089303] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 5156.089305] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5156.089307] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5156.089310] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5156.263780] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5156.301682] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5156.310770] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5156.318815] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5156.324662] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5156.335784] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5156.339788] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 5156.344513] RDX: 00000000000051b0 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 5156.348875] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 5156.352231] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 5156.358871] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5156.364206] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 5156.369918] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5156.374781] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5156.378926] Call Trace: [ 5156.380389] [ 5156.381704] ? show_regs.cold.9+0x22/0x2f [ 5156.383881] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5156.386388] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5156.389491] ? watchdog+0x40/0x40 [ 5156.391186] ? hrtimer_interrupt+0x128/0x2c0 [ 5156.393498] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5156.398385] ? apic_timer_interrupt+0xf/0x20 [ 5156.400561] [ 5156.401340] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5156.403511] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5156.407181] do_raw_spin_lock+0x177/0x190 [ 5156.409628] _raw_spin_lock+0x21/0x30 [ 5156.411291] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5156.417971] obd_mod_free+0x46/0x140 [ptlrpc] [ 5156.420472] mdc_commit_open+0x91/0xb0 [mdc] [ 5156.421721] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5156.424795] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5156.429201] after_reply+0xe2c/0x1780 [ptlrpc] [ 5156.432188] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5156.434771] ? __next_timer_interrupt+0x160/0x160 [ 5156.436429] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5156.438837] ? do_wait_intr+0xf0/0xf0 [ 5156.441679] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5156.445825] kthread+0x1d1/0x200 [ 5156.447589] ? set_kthread_struct+0x70/0x70 [ 5156.449718] ret_from_fork+0x1f/0x30 [ 5168.053383] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 5168.056504] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5168.090238] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5168.102082] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5168.111518] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5168.118781] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5168.128387] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5168.133203] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5168.138392] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 5168.143247] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 5168.147289] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 5168.150815] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 5168.158284] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 5168.164870] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5168.169883] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 5168.173277] Call Trace: [ 5168.174779] [ 5168.181287] ? show_regs.cold.9+0x22/0x2f [ 5168.186394] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5168.193650] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5168.200821] ? watchdog+0x40/0x40 [ 5168.204362] ? hrtimer_interrupt+0x128/0x2c0 [ 5168.207403] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5168.215614] ? apic_timer_interrupt+0xf/0x20 [ 5168.219832] [ 5168.221324] ? native_safe_halt+0x2e/0x50 [ 5168.222508] kvm_wait+0xcd/0xe0 [ 5168.229111] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5168.230933] do_raw_spin_lock+0x177/0x190 [ 5168.235225] _raw_spin_lock+0x21/0x30 [ 5168.238083] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5168.241978] ? do_raw_spin_unlock+0x75/0x190 [ 5168.243858] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5168.247989] ? do_wait_intr+0xf0/0xf0 [ 5168.252323] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5168.258978] mdc_reint+0x2d/0x110 [mdc] [ 5168.261608] mdc_unlink+0x658/0x870 [mdc] [ 5168.267068] lmv_unlink+0x34d/0xbd0 [lmv] [ 5168.270492] ll_unlink+0x24d/0x7f0 [lustre] [ 5168.275624] vfs_unlink+0x202/0x330 [ 5168.283887] do_unlinkat+0x2fa/0x460 [ 5168.288258] __x64_sys_unlink+0x37/0x50 [ 5168.291887] do_syscall_64+0xc1/0x440 [ 5168.295189] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5168.300508] RIP: 0033:0x7f151bdee10b [ 5168.303811] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5168.314008] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5168.319135] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 5168.326868] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 5168.332390] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5168.335079] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5168.342247] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5172.057949] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5172.064626] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5172.100776] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5172.108364] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5172.115603] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5172.121590] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5172.123556] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5172.131773] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5172.136293] RAX: 000000000000267f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5172.141984] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 5172.148742] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 5172.154129] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 5172.158745] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 5172.163600] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 5172.165751] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5172.175396] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5172.178636] Call Trace: [ 5172.179549] [ 5172.180758] ? show_regs.cold.9+0x22/0x2f [ 5172.182192] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5172.183865] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5172.188625] ? watchdog+0x40/0x40 [ 5172.192451] ? hrtimer_interrupt+0x128/0x2c0 [ 5172.194190] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5172.196807] ? apic_timer_interrupt+0xf/0x20 [ 5172.201604] [ 5172.203481] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5172.208377] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5172.210401] do_raw_spin_lock+0x177/0x190 [ 5172.222202] _raw_spin_lock+0x21/0x30 [ 5172.225729] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5172.233494] process_one_work+0x2c8/0x700 [ 5172.237614] worker_thread+0x296/0x6e0 [ 5172.239756] ? rescuer_thread+0x570/0x570 [ 5172.242557] kthread+0x1d1/0x200 [ 5172.245135] ? set_kthread_struct+0x70/0x70 [ 5172.247148] ret_from_fork+0x1f/0x30 [ 5174.241055] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3243s! [ 5174.245081] Showing busy workqueues and worker pools: [ 5174.247762] workqueue events: flags=0x0 [ 5174.253442] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5174.257944] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 5174.262553] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5174.264689] pending: kfree_rcu_monitor [ 5174.266540] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5174.268709] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 5174.272348] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5174.277210] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5174.281065] workqueue events_long: flags=0x0 [ 5174.282563] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5174.285052] pending: lu_objects_flush [obdclass] [ 5174.287924] workqueue events_unbound: flags=0x2 [ 5174.289846] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5174.292489] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5174.297571] workqueue events_power_efficient: flags=0x80 [ 5174.300033] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5174.302209] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 5174.304391] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5174.306704] pending: neigh_periodic_work [ 5174.308496] workqueue mm_percpu_wq: flags=0x8 [ 5174.309740] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5174.312466] pending: vmstat_update, lru_add_drain_per_cpu [ 5174.318530] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5174.325594] pending: vmstat_update [ 5174.329442] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5174.333574] pending: lru_add_drain_per_cpu [ 5174.339674] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5174.345306] pending: lru_add_drain_per_cpu BAR(48) [ 5174.347542] workqueue writeback: flags=0x4a [ 5174.350346] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5174.353785] pending: wb_workfn [ 5174.357490] workqueue xprtiod: flags=0xa [ 5174.360043] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5174.368638] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5174.384095] workqueue ipv6_addrconf: flags=0x40008 [ 5174.387573] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5174.392803] pending: addrconf_verify_work [ 5174.393909] workqueue knbd0-recv: flags=0x1a [ 5174.394932] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5174.402385] in-flight: 658:recv_work [ 5174.406136] workqueue ptlrpc_pinger: flags=0x2 [ 5174.408379] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5174.412388] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5174.415152] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3243s workers=4 idle: 32984 37 8 [ 5174.418736] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5159s workers=2 idle: 661 [ 5184.055100] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 5184.060676] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5184.065080] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 5184.101318] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5184.101333] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5184.101335] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5184.101349] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5184.101354] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5184.101360] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 5184.101362] RDX: 000000000000379d RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 5184.101364] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 5184.101366] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 5184.101368] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5184.101371] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 5184.101374] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5184.101377] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5184.101386] Call Trace: [ 5184.101391] [ 5184.101398] ? show_regs.cold.9+0x22/0x2f [ 5184.101407] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5184.101413] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5184.101417] ? watchdog+0x40/0x40 [ 5184.101421] ? hrtimer_interrupt+0x128/0x2c0 [ 5184.101425] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5184.101435] ? apic_timer_interrupt+0xf/0x20 [ 5184.101440] [ 5184.101454] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5184.101460] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5184.101465] do_raw_spin_lock+0x177/0x190 [ 5184.101470] _raw_spin_lock+0x21/0x30 [ 5184.101478] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5184.103345] obd_mod_free+0x46/0x140 [ptlrpc] [ 5184.103594] mdc_commit_open+0x91/0xb0 [mdc] [ 5184.103636] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5184.105428] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5184.106571] after_reply+0xe2c/0x1780 [ptlrpc] [ 5184.118186] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5184.118266] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5184.118274] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5184.118277] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5184.126301] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5184.126726] ? __next_timer_interrupt+0x160/0x160 [ 5184.126738] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5184.127330] ? do_wait_intr+0xf0/0xf0 [ 5184.127340] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5184.127617] kthread+0x1d1/0x200 [ 5184.127624] ? set_kthread_struct+0x70/0x70 [ 5184.127628] ret_from_fork+0x1f/0x30 [ 5184.361716] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5184.378490] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5184.381140] RAX: 0000000000007f86 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5184.389196] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 5184.392181] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5184.397179] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 5184.400503] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 5184.407497] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 5184.411718] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5184.415065] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 5184.417535] Call Trace: [ 5184.425597] [ 5184.428299] ? show_regs.cold.9+0x22/0x2f [ 5184.430544] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5184.440370] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5184.443057] ? watchdog+0x40/0x40 [ 5184.444572] ? hrtimer_interrupt+0x128/0x2c0 [ 5184.448438] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5184.450048] ? apic_timer_interrupt+0xf/0x20 [ 5184.456176] [ 5184.457100] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5184.460169] do_raw_spin_lock+0x177/0x190 [ 5184.462721] _raw_spin_lock+0x21/0x30 [ 5184.465616] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5184.474990] ? do_raw_spin_unlock+0x75/0x190 [ 5184.478262] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5184.480888] ? do_wait_intr+0xf0/0xf0 [ 5184.482673] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5184.486858] mdc_close+0x53b/0xd70 [mdc] [ 5184.488289] lmv_close+0x18d/0x490 [lmv] [ 5184.494905] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5184.501136] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5184.507104] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5184.512503] ll_md_close+0x274/0x800 [lustre] [ 5184.516065] ll_file_release+0x128/0xc50 [lustre] [ 5184.519339] __fput+0x10d/0x440 [ 5184.520646] ____fput+0x12/0x20 [ 5184.521831] task_work_run+0xc6/0x110 [ 5184.522957] exit_to_usermode_loop+0x1dd/0x1f0 [ 5184.525931] do_syscall_64+0x3d6/0x440 [ 5184.528110] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5184.530562] RIP: 0033:0x7f151bee3cd8 [ 5184.532576] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5184.547639] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5184.550845] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 5184.555344] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 5184.559277] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5184.565661] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5184.568396] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5200.052404] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 5200.056065] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5200.057844] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss [ 5200.062387] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5200.062912] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5200.062919] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5200.062926] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5200.063407] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5200.063419] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5200.063427] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5200.063432] RAX: 000000000000688c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5200.063435] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 5200.063437] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 5200.063507] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 5200.063510] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 5200.063513] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 5200.063521] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5200.063523] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5200.063531] Call Trace: [ 5200.063536] [ 5200.063542] ? show_regs.cold.9+0x22/0x2f [ 5200.063551] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5200.063558] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5200.063563] ? watchdog+0x40/0x40 [ 5200.063567] ? hrtimer_interrupt+0x128/0x2c0 [ 5200.063571] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5200.063830] ? apic_timer_interrupt+0xf/0x20 [ 5200.063839] [ 5200.063843] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5200.063850] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5200.063856] do_raw_spin_lock+0x177/0x190 [ 5200.063862] _raw_spin_lock+0x21/0x30 [ 5200.063867] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5200.064183] process_one_work+0x2c8/0x700 [ 5200.064190] worker_thread+0x296/0x6e0 [ 5200.064195] ? rescuer_thread+0x570/0x570 [ 5200.064199] kthread+0x1d1/0x200 [ 5200.064204] ? set_kthread_struct+0x70/0x70 [ 5200.064208] ret_from_fork+0x1f/0x30 [ 5200.214955] nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5200.225849] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5200.231956] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5200.239440] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5200.242235] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5200.251092] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5200.255138] RAX: 0000000000006c5b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5200.261330] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 5200.268321] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 5200.273202] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 5200.277724] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 5200.281093] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 5200.284847] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5200.288945] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 5200.293544] Call Trace: [ 5200.295189] [ 5200.296362] ? show_regs.cold.9+0x22/0x2f [ 5200.298175] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5200.302386] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5200.306254] ? watchdog+0x40/0x40 [ 5200.311196] ? hrtimer_interrupt+0x128/0x2c0 [ 5200.315855] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5200.317613] ? apic_timer_interrupt+0xf/0x20 [ 5200.322489] [ 5200.323471] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5200.327176] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5200.336843] do_raw_spin_lock+0x177/0x190 [ 5200.343618] _raw_spin_lock+0x21/0x30 [ 5200.346880] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5200.350828] ? do_raw_spin_unlock+0x75/0x190 [ 5200.356151] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5200.359431] ? do_wait_intr+0xf0/0xf0 [ 5200.361383] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5200.364251] mdc_reint+0x2d/0x110 [mdc] [ 5200.367254] mdc_unlink+0x658/0x870 [mdc] [ 5200.371238] lmv_unlink+0x34d/0xbd0 [lmv] [ 5200.374480] ll_unlink+0x24d/0x7f0 [lustre] [ 5200.377890] vfs_unlink+0x202/0x330 [ 5200.378782] do_unlinkat+0x2fa/0x460 [ 5200.379604] __x64_sys_unlink+0x37/0x50 [ 5200.383467] do_syscall_64+0xc1/0x440 [ 5200.385410] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5200.387357] RIP: 0033:0x7f151bdee10b [ 5200.392290] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5200.413974] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5200.421077] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 5200.428961] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 5200.433111] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5200.436097] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5200.442867] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5204.449042] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3273s! [ 5204.454281] Showing busy workqueues and worker pools: [ 5204.457836] workqueue events: flags=0x0 [ 5204.459250] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5204.465863] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 5204.470504] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5204.476509] pending: kfree_rcu_monitor [ 5204.479390] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5204.482361] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 5204.486736] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5204.489360] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5204.493063] workqueue events_long: flags=0x0 [ 5204.494894] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5204.498677] pending: lu_objects_flush [obdclass] [ 5204.501409] workqueue events_unbound: flags=0x2 [ 5204.503798] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5204.506501] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5204.514160] workqueue events_power_efficient: flags=0x80 [ 5204.518103] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5204.521762] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 5204.525776] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5204.528920] pending: neigh_periodic_work [ 5204.530792] workqueue mm_percpu_wq: flags=0x8 [ 5204.533229] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5204.536054] pending: vmstat_update, lru_add_drain_per_cpu [ 5204.539267] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5204.544326] pending: vmstat_update [ 5204.546278] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5204.550952] pending: lru_add_drain_per_cpu [ 5204.553349] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5204.557096] pending: lru_add_drain_per_cpu BAR(48) [ 5204.560946] workqueue writeback: flags=0x4a [ 5204.563347] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5204.565888] pending: wb_workfn [ 5204.567156] workqueue xprtiod: flags=0xa [ 5204.568574] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5204.573905] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5204.576974] workqueue ipv6_addrconf: flags=0x40008 [ 5204.579152] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5204.584618] pending: addrconf_verify_work [ 5204.590138] workqueue knbd0-recv: flags=0x1a [ 5204.595126] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5204.599426] in-flight: 658:recv_work [ 5204.601272] workqueue ptlrpc_pinger: flags=0x2 [ 5204.603206] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5204.605537] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5204.612550] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3274s workers=4 idle: 32984 37 8 [ 5204.618958] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5189s workers=2 idle: 661 [ 5212.058371] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 5212.062260] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 5212.070295] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5212.070371] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5212.070379] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5212.070382] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5212.070409] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5212.070414] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5212.070418] RAX: 0000000000002896 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5212.070421] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 5212.070423] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5212.070425] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 5212.070428] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 5212.070431] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 5212.070434] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5212.070436] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 5212.070445] Call Trace: [ 5212.070451] [ 5212.070459] ? show_regs.cold.9+0x22/0x2f [ 5212.070469] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5212.070476] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5212.070480] ? watchdog+0x40/0x40 [ 5212.070485] ? hrtimer_interrupt+0x128/0x2c0 [ 5212.070488] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5212.070590] ? apic_timer_interrupt+0xf/0x20 [ 5212.070597] [ 5212.070600] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5212.070607] do_raw_spin_lock+0x177/0x190 [ 5212.070613] _raw_spin_lock+0x21/0x30 [ 5212.070617] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5212.070940] ? do_raw_spin_unlock+0x75/0x190 [ 5212.070946] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5212.071151] ? do_wait_intr+0xf0/0xf0 [ 5212.071159] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5212.071361] mdc_close+0x53b/0xd70 [mdc] [ 5212.071404] lmv_close+0x18d/0x490 [lmv] [ 5212.071439] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5212.081223] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5212.081303] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5212.081311] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5212.081314] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5212.081328] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5212.081332] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5212.081337] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 5212.081339] RDX: 000000000000288f RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 5212.081341] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 5212.081344] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 5212.081347] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5212.081350] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 5212.081353] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5212.081356] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5212.081365] Call Trace: [ 5212.081370] [ 5212.081376] ? show_regs.cold.9+0x22/0x2f [ 5212.081387] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5212.081394] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5212.081399] ? watchdog+0x40/0x40 [ 5212.081404] ? hrtimer_interrupt+0x128/0x2c0 [ 5212.081408] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5212.081417] ? apic_timer_interrupt+0xf/0x20 [ 5212.081422] [ 5212.081426] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5212.081431] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5212.081436] do_raw_spin_lock+0x177/0x190 [ 5212.081441] _raw_spin_lock+0x21/0x30 [ 5212.081447] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5212.129388] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5212.134480] obd_mod_free+0x46/0x140 [ptlrpc] [ 5212.138351] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5212.138470] ll_md_close+0x274/0x800 [lustre] [ 5212.138553] ll_file_release+0x128/0xc50 [lustre] [ 5212.138645] __fput+0x10d/0x440 [ 5212.138656] ____fput+0x12/0x20 [ 5212.138661] task_work_run+0xc6/0x110 [ 5212.138668] exit_to_usermode_loop+0x1dd/0x1f0 [ 5212.138676] do_syscall_64+0x3d6/0x440 [ 5212.138681] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5212.143925] mdc_commit_open+0x91/0xb0 [mdc] [ 5212.154004] RIP: 0033:0x7f151bee3cd8 [ 5212.154027] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5212.154031] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5212.154038] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 5212.154040] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 5212.154042] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5212.154044] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5212.154047] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5212.544921] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5212.548904] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5212.552516] after_reply+0xe2c/0x1780 [ptlrpc] [ 5212.556220] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5212.560160] ? __next_timer_interrupt+0x160/0x160 [ 5212.563168] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5212.568730] ? do_wait_intr+0xf0/0xf0 [ 5212.570763] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5212.573687] kthread+0x1d1/0x200 [ 5212.575529] ? set_kthread_struct+0x70/0x70 [ 5212.577468] ret_from_fork+0x1f/0x30 [ 5224.708047] rcu: INFO: rcu_sched self-detected stall on CPU [ 5224.710723] rcu: 1-...!: (1922487 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 5224.715121] (t=3300061 jiffies g=249657 q=3349) [ 5224.716663] rcu: rcu_sched kthread starved for 3295013 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 5224.720079] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5224.723690] rcu: RCU grace-period kthread stack dump: [ 5224.725332] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5224.728959] Call Trace: [ 5224.729752] __schedule+0x351/0xcb0 [ 5224.730740] schedule+0xc0/0x180 [ 5224.731693] schedule_timeout+0xb4/0x190 [ 5224.732741] ? do_raw_spin_unlock+0x75/0x190 [ 5224.734092] ? __next_timer_interrupt+0x160/0x160 [ 5224.735979] ? prepare_to_swait_event+0xf3/0x140 [ 5224.737526] rcu_gp_kthread+0x8dd/0x15b0 [ 5224.738858] ? force_qs_rnp+0x2c0/0x2c0 [ 5224.740358] kthread+0x1d1/0x200 [ 5224.741588] ? set_kthread_struct+0x70/0x70 [ 5224.743186] ret_from_fork+0x1f/0x30 [ 5224.744333] rcu: Stack dump where RCU GP kthread last ran: [ 5224.745807] Sending NMI from CPU 1 to CPUs 2: [ 5224.746991] NMI backtrace for cpu 2 skipped: idling at native_safe_halt+0x2e/0x50 [ 5224.747954] NMI backtrace for cpu 1 [ 5224.750604] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5224.755364] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5224.759035] Call Trace: [ 5224.759765] [ 5224.760394] ? dump_stack+0xbb/0x10e [ 5224.761674] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5224.763603] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5224.765397] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5224.767660] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5224.770242] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5224.772366] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 5224.774030] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 5224.776456] ? update_process_times+0x6d/0xc0 [ 5224.778349] ? tick_sched_handle.isra.18+0x27/0x90 [ 5224.780250] ? tick_sched_timer+0x53/0xb0 [ 5224.781887] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5224.783821] ? tick_sched_handle.isra.18+0x90/0x90 [ 5224.785980] ? hrtimer_interrupt+0x128/0x2c0 [ 5224.787570] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5224.789947] ? apic_timer_interrupt+0xf/0x20 [ 5224.791708] [ 5224.792748] ? native_safe_halt+0x2e/0x50 [ 5224.794481] ? kvm_wait+0xcd/0xe0 [ 5224.795732] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5224.798052] ? do_raw_spin_lock+0x177/0x190 [ 5224.799906] ? _raw_spin_lock+0x21/0x30 [ 5224.801867] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5224.804307] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5224.806698] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5224.808872] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5224.811547] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5224.814101] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5224.816287] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5224.818612] ? __next_timer_interrupt+0x160/0x160 [ 5224.820699] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5224.822692] ? do_wait_intr+0xf0/0xf0 [ 5224.824177] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5224.826786] ? kthread+0x1d1/0x200 [ 5224.828382] ? set_kthread_struct+0x70/0x70 [ 5224.830190] ? ret_from_fork+0x1f/0x30 [ 5232.055310] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 5232.059470] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5232.084322] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5232.089984] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5232.094765] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5232.097421] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5232.108750] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5232.113955] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5232.119008] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 5232.123276] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 5232.130306] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 5232.134045] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 5232.138656] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 5232.142948] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5232.146264] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 5232.149412] Call Trace: [ 5232.150966] [ 5232.151972] ? show_regs.cold.9+0x22/0x2f [ 5232.153930] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5232.156651] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5232.158848] ? watchdog+0x40/0x40 [ 5232.160450] ? hrtimer_interrupt+0x128/0x2c0 [ 5232.162964] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5232.165809] ? apic_timer_interrupt+0xf/0x20 [ 5232.167915] [ 5232.168973] ? native_safe_halt+0x2e/0x50 [ 5232.172168] kvm_wait+0xcd/0xe0 [ 5232.173846] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5232.178825] do_raw_spin_lock+0x177/0x190 [ 5232.183749] _raw_spin_lock+0x21/0x30 [ 5232.185742] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5232.189445] ? do_raw_spin_unlock+0x75/0x190 [ 5232.191974] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5232.195785] ? do_wait_intr+0xf0/0xf0 [ 5232.197859] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5232.201216] mdc_reint+0x2d/0x110 [mdc] [ 5232.202840] mdc_unlink+0x658/0x870 [mdc] [ 5232.204908] lmv_unlink+0x34d/0xbd0 [lmv] [ 5232.207484] ll_unlink+0x24d/0x7f0 [lustre] [ 5232.209681] vfs_unlink+0x202/0x330 [ 5232.211734] do_unlinkat+0x2fa/0x460 [ 5232.214085] __x64_sys_unlink+0x37/0x50 [ 5232.216074] do_syscall_64+0xc1/0x440 [ 5232.218067] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5232.219890] RIP: 0033:0x7f151bdee10b [ 5232.221611] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5232.232122] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5232.235523] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 5232.239791] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 5232.243257] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5232.250155] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5232.253835] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5234.657486] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3304s! [ 5234.665483] Showing busy workqueues and worker pools: [ 5234.667861] workqueue events: flags=0x0 [ 5234.669163] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5234.679453] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 5234.688297] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5234.696844] pending: kfree_rcu_monitor [ 5234.702341] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5234.710068] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 5234.717031] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5234.719365] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5234.728995] workqueue events_long: flags=0x0 [ 5234.737994] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5234.745348] pending: lu_objects_flush [obdclass] [ 5234.753069] workqueue events_unbound: flags=0x2 [ 5234.754881] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5234.759579] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5234.766859] workqueue events_power_efficient: flags=0x80 [ 5234.769870] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5234.773284] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 5234.777210] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5234.780774] pending: neigh_periodic_work [ 5234.782701] workqueue mm_percpu_wq: flags=0x8 [ 5234.785160] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5234.788263] pending: vmstat_update, lru_add_drain_per_cpu [ 5234.790742] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5234.793650] pending: vmstat_update [ 5234.795052] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5234.798106] pending: lru_add_drain_per_cpu [ 5234.800245] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5234.803132] pending: lru_add_drain_per_cpu BAR(48) [ 5234.806983] workqueue writeback: flags=0x4a [ 5234.808837] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5234.811687] pending: wb_workfn [ 5234.813209] workqueue xprtiod: flags=0xa [ 5234.814942] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5234.817504] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5234.821580] workqueue ipv6_addrconf: flags=0x40008 [ 5234.823251] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5234.825416] pending: addrconf_verify_work [ 5234.827044] workqueue knbd0-recv: flags=0x1a [ 5234.828249] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5234.830614] in-flight: 658:recv_work [ 5234.831834] workqueue ptlrpc_pinger: flags=0x2 [ 5234.833175] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5234.835048] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5234.837195] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3304s workers=4 idle: 32984 37 8 [ 5234.839723] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5219s workers=2 idle: 661 [ 5240.057013] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 5240.062017] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5240.088403] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5240.092189] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5240.094992] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5240.096718] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5240.102169] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5240.104352] RAX: 000000000000508a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5240.106487] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 5240.108497] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5240.111008] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 5240.112989] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 5240.115123] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 5240.120226] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5240.123316] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 5240.127488] Call Trace: [ 5240.128888] [ 5240.130310] ? show_regs.cold.9+0x22/0x2f [ 5240.132164] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5240.134797] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5240.137479] ? watchdog+0x40/0x40 [ 5240.139281] ? hrtimer_interrupt+0x128/0x2c0 [ 5240.140472] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5240.144070] ? apic_timer_interrupt+0xf/0x20 [ 5240.146170] [ 5240.147327] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5240.152124] do_raw_spin_lock+0x177/0x190 [ 5240.157997] _raw_spin_lock+0x21/0x30 [ 5240.159082] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5240.162158] ? do_raw_spin_unlock+0x75/0x190 [ 5240.164920] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5240.167635] ? do_wait_intr+0xf0/0xf0 [ 5240.170053] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5240.172914] mdc_close+0x53b/0xd70 [mdc] [ 5240.176334] lmv_close+0x18d/0x490 [lmv] [ 5240.178181] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5240.181287] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5240.183959] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5240.185971] ll_md_close+0x274/0x800 [lustre] [ 5240.187888] ll_file_release+0x128/0xc50 [lustre] [ 5240.190293] __fput+0x10d/0x440 [ 5240.191755] ____fput+0x12/0x20 [ 5240.193493] task_work_run+0xc6/0x110 [ 5240.195620] exit_to_usermode_loop+0x1dd/0x1f0 [ 5240.197795] do_syscall_64+0x3d6/0x440 [ 5240.199938] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5240.202178] RIP: 0033:0x7f151bee3cd8 [ 5240.203844] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5240.212971] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5240.216494] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 5240.220574] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 5240.224092] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5240.227423] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5240.231162] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5252.055106] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 5252.056162] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5252.058717] Modules linked in: [ 5252.063430] Modules linked in: [ 5252.063432] lustre(O) osp(O) [ 5252.064558] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5252.064634] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5252.064642] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5252.064648] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5252.070294] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5252.070380] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5252.070388] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5252.070391] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5252.070406] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5252.070410] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5252.070414] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 5252.070417] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 5252.070419] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 5252.070421] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 5252.070423] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5252.070426] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 5252.070429] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5252.070432] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5252.070441] Call Trace: [ 5252.070446] [ 5252.070453] ? show_regs.cold.9+0x22/0x2f [ 5252.070464] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5252.070471] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5252.070475] ? watchdog+0x40/0x40 [ 5252.070479] ? hrtimer_interrupt+0x128/0x2c0 [ 5252.070483] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5252.070490] ? apic_timer_interrupt+0xf/0x20 [ 5252.070495] [ 5252.070498] ? native_safe_halt+0x2e/0x50 [ 5252.070502] kvm_wait+0xcd/0xe0 [ 5252.070509] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5252.070530] do_raw_spin_lock+0x177/0x190 [ 5252.070534] _raw_spin_lock+0x21/0x30 [ 5252.070539] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5252.070825] obd_mod_free+0x46/0x140 [ptlrpc] [ 5252.071224] mdc_commit_open+0x91/0xb0 [mdc] [ 5252.071276] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5252.071477] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5252.071736] after_reply+0xe2c/0x1780 [ptlrpc] [ 5252.071942] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5252.072308] ? __next_timer_interrupt+0x160/0x160 [ 5252.072319] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5252.073182] ? do_wait_intr+0xf0/0xf0 [ 5252.074917] [ 5252.074924] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5252.074940] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5252.074944] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5252.074949] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5252.074952] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 5252.074954] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 5252.074957] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 5252.074959] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 5252.074963] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 5252.074966] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5252.074968] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5252.074977] Call Trace: [ 5252.074981] [ 5252.074985] ? show_regs.cold.9+0x22/0x2f [ 5252.074995] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5252.107773] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5252.108310] kthread+0x1d1/0x200 [ 5252.108319] ? set_kthread_struct+0x70/0x70 [ 5252.108324] ret_from_fork+0x1f/0x30 [ 5252.265886] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5252.267430] ? watchdog+0x40/0x40 [ 5252.268580] ? hrtimer_interrupt+0x128/0x2c0 [ 5252.270651] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5252.272365] ? apic_timer_interrupt+0xf/0x20 [ 5252.273835] [ 5252.274683] ? native_safe_halt+0x2e/0x50 [ 5252.276465] kvm_wait+0xcd/0xe0 [ 5252.277477] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5252.279403] do_raw_spin_lock+0x177/0x190 [ 5252.280841] _raw_spin_lock+0x21/0x30 [ 5252.282223] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5252.284290] process_one_work+0x2c8/0x700 [ 5252.285840] worker_thread+0x296/0x6e0 [ 5252.287250] ? rescuer_thread+0x570/0x570 [ 5252.288564] kthread+0x1d1/0x200 [ 5252.289559] ? set_kthread_struct+0x70/0x70 [ 5252.291012] ret_from_fork+0x1f/0x30 [ 5260.053214] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 5260.059750] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5260.098809] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5260.108769] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5260.118239] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5260.123320] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5260.143173] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5260.148379] RAX: 0000000000003f0b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5260.157220] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 5260.172561] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 5260.178446] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 5260.186971] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 5260.193808] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 5260.202478] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5260.210791] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 5260.215350] Call Trace: [ 5260.216684] [ 5260.219933] ? show_regs.cold.9+0x22/0x2f [ 5260.222159] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5260.226639] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5260.229920] ? watchdog+0x40/0x40 [ 5260.233734] ? hrtimer_interrupt+0x128/0x2c0 [ 5260.237883] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5260.244226] ? apic_timer_interrupt+0xf/0x20 [ 5260.251962] [ 5260.254952] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5260.264844] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5260.266563] do_raw_spin_lock+0x177/0x190 [ 5260.270795] _raw_spin_lock+0x21/0x30 [ 5260.278007] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5260.280802] ? do_raw_spin_unlock+0x75/0x190 [ 5260.286713] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5260.294500] ? do_wait_intr+0xf0/0xf0 [ 5260.301539] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5260.304780] mdc_reint+0x2d/0x110 [mdc] [ 5260.307643] mdc_unlink+0x658/0x870 [mdc] [ 5260.310461] lmv_unlink+0x34d/0xbd0 [lmv] [ 5260.313156] ll_unlink+0x24d/0x7f0 [lustre] [ 5260.316216] vfs_unlink+0x202/0x330 [ 5260.319508] do_unlinkat+0x2fa/0x460 [ 5260.325808] __x64_sys_unlink+0x37/0x50 [ 5260.328141] do_syscall_64+0xc1/0x440 [ 5260.332362] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5260.334338] RIP: 0033:0x7f151bdee10b [ 5260.337751] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5260.359216] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5260.369770] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 5260.372664] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 5260.377374] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5260.385471] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5260.394140] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5264.871235] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3334s! [ 5264.876808] Showing busy workqueues and worker pools: [ 5264.878734] workqueue events: flags=0x0 [ 5264.880717] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5264.883089] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 5264.888138] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5264.895267] pending: kfree_rcu_monitor [ 5264.899750] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5264.905723] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 5264.916729] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5264.920178] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5264.923817] workqueue events_long: flags=0x0 [ 5264.925684] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5264.928724] pending: lu_objects_flush [obdclass] [ 5264.936153] workqueue events_unbound: flags=0x2 [ 5264.939975] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5264.944449] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5264.953711] workqueue events_power_efficient: flags=0x80 [ 5264.956809] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5264.960108] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 5264.963776] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5264.967131] pending: neigh_periodic_work [ 5264.969298] workqueue mm_percpu_wq: flags=0x8 [ 5264.971936] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5264.978689] pending: vmstat_update, lru_add_drain_per_cpu [ 5264.981628] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5264.985273] pending: vmstat_update [ 5264.986950] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5264.988896] pending: lru_add_drain_per_cpu [ 5264.990881] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5264.994362] pending: lru_add_drain_per_cpu BAR(48) [ 5264.996503] workqueue writeback: flags=0x4a [ 5264.998949] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5265.002276] pending: wb_workfn [ 5265.003840] workqueue xprtiod: flags=0xa [ 5265.005441] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5265.008337] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5265.010797] workqueue ipv6_addrconf: flags=0x40008 [ 5265.012243] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5265.015497] pending: addrconf_verify_work [ 5265.017509] workqueue knbd0-recv: flags=0x1a [ 5265.023833] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5265.031461] in-flight: 658:recv_work [ 5265.034362] workqueue ptlrpc_pinger: flags=0x2 [ 5265.040510] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5265.050383] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5265.057747] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3334s workers=4 idle: 32984 37 8 [ 5265.062485] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5249s workers=2 idle: 661 [ 5268.058669] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 5268.073484] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5268.112471] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5268.117849] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5268.121749] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5268.124060] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5268.130191] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 5268.133244] RAX: 0000000000000883 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5268.136495] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 5268.139297] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5268.144253] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 5268.148444] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 5268.150772] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 5268.154579] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5268.160588] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 5268.166213] Call Trace: [ 5268.168280] [ 5268.169385] ? show_regs.cold.9+0x22/0x2f [ 5268.170627] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5268.172426] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5268.174140] ? watchdog+0x40/0x40 [ 5268.175272] ? hrtimer_interrupt+0x128/0x2c0 [ 5268.176671] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5268.178289] ? apic_timer_interrupt+0xf/0x20 [ 5268.179676] [ 5268.180644] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5268.182664] do_raw_spin_lock+0x177/0x190 [ 5268.184243] _raw_spin_lock+0x21/0x30 [ 5268.185689] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5268.188168] ? do_raw_spin_unlock+0x75/0x190 [ 5268.189395] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5268.191608] ? do_wait_intr+0xf0/0xf0 [ 5268.193059] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5268.195554] mdc_close+0x53b/0xd70 [mdc] [ 5268.200747] lmv_close+0x18d/0x490 [lmv] [ 5268.202138] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5268.205243] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5268.210829] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5268.218423] ll_md_close+0x274/0x800 [lustre] [ 5268.224416] ll_file_release+0x128/0xc50 [lustre] [ 5268.229634] __fput+0x10d/0x440 [ 5268.234939] ____fput+0x12/0x20 [ 5268.239244] task_work_run+0xc6/0x110 [ 5268.243124] exit_to_usermode_loop+0x1dd/0x1f0 [ 5268.247427] do_syscall_64+0x3d6/0x440 [ 5268.253514] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5268.260394] RIP: 0033:0x7f151bee3cd8 [ 5268.261431] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5268.283927] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5268.290340] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 5268.295723] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 5268.300414] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5268.303829] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5268.307713] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5280.055618] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 5280.061826] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5280.064552] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5280.090063] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5280.090080] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5280.090083] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5280.090100] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5280.090104] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5280.090111] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 5280.090113] RDX: 0000000000007936 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 5280.090116] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 5280.090118] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 5280.090121] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5280.090125] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 5280.090128] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5280.090130] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5280.090140] Call Trace: [ 5280.090286] [ 5280.096810] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5280.096893] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5280.096899] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5280.096906] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5280.097248] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5280.097259] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5280.097263] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5280.097268] RAX: 0000000000003b2f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5280.097270] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 5280.097272] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 5280.097275] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 5280.097278] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 5280.097281] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 5280.097283] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5280.097286] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5280.097295] Call Trace: [ 5280.097299] [ 5280.097305] ? show_regs.cold.9+0x22/0x2f [ 5280.097314] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5280.097320] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5280.097324] ? watchdog+0x40/0x40 [ 5280.097328] ? hrtimer_interrupt+0x128/0x2c0 [ 5280.097333] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5280.097341] ? apic_timer_interrupt+0xf/0x20 [ 5280.097346] [ 5280.097349] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5280.097354] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5280.097359] do_raw_spin_lock+0x177/0x190 [ 5280.097364] _raw_spin_lock+0x21/0x30 [ 5280.097368] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5280.097648] process_one_work+0x2c8/0x700 [ 5280.097655] worker_thread+0x296/0x6e0 [ 5280.097659] ? rescuer_thread+0x570/0x570 [ 5280.097663] kthread+0x1d1/0x200 [ 5280.097667] ? set_kthread_struct+0x70/0x70 [ 5280.097671] ret_from_fork+0x1f/0x30 [ 5280.338149] ? show_regs.cold.9+0x22/0x2f [ 5280.339391] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5280.341266] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5280.343181] ? watchdog+0x40/0x40 [ 5280.344630] ? hrtimer_interrupt+0x128/0x2c0 [ 5280.346778] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5280.348821] ? apic_timer_interrupt+0xf/0x20 [ 5280.350419] [ 5280.351078] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5280.355417] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5280.357330] do_raw_spin_lock+0x177/0x190 [ 5280.359885] _raw_spin_lock+0x21/0x30 [ 5280.361124] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5280.362929] obd_mod_free+0x46/0x140 [ptlrpc] [ 5280.375733] mdc_commit_open+0x91/0xb0 [mdc] [ 5280.377353] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5280.379279] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5280.382992] after_reply+0xe2c/0x1780 [ptlrpc] [ 5280.384401] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5280.386137] ? __next_timer_interrupt+0x160/0x160 [ 5280.387613] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5280.388838] ? do_wait_intr+0xf0/0xf0 [ 5280.391457] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5280.393622] kthread+0x1d1/0x200 [ 5280.394750] ? set_kthread_struct+0x70/0x70 [ 5280.397475] ret_from_fork+0x1f/0x30 [ 5292.052189] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 5292.056336] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5292.083822] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5292.090831] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5292.096023] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5292.099454] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5292.112143] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5292.116145] RAX: 00000000000056a9 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5292.119387] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 5292.122426] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 5292.125862] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 5292.129129] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 5292.132378] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 5292.136009] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5292.138556] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 5292.142562] Call Trace: [ 5292.144213] [ 5292.145300] ? show_regs.cold.9+0x22/0x2f [ 5292.149287] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5292.152130] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5292.154322] ? watchdog+0x40/0x40 [ 5292.156202] ? hrtimer_interrupt+0x128/0x2c0 [ 5292.159082] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5292.161371] ? apic_timer_interrupt+0xf/0x20 [ 5292.163831] [ 5292.164942] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5292.167407] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5292.170155] do_raw_spin_lock+0x177/0x190 [ 5292.172114] _raw_spin_lock+0x21/0x30 [ 5292.174109] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5292.177131] ? do_raw_spin_unlock+0x75/0x190 [ 5292.179205] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5292.185397] ? do_wait_intr+0xf0/0xf0 [ 5292.189307] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5292.192829] mdc_reint+0x2d/0x110 [mdc] [ 5292.194958] mdc_unlink+0x658/0x870 [mdc] [ 5292.197098] lmv_unlink+0x34d/0xbd0 [lmv] [ 5292.199260] ll_unlink+0x24d/0x7f0 [lustre] [ 5292.201489] vfs_unlink+0x202/0x330 [ 5292.203368] do_unlinkat+0x2fa/0x460 [ 5292.205236] __x64_sys_unlink+0x37/0x50 [ 5292.207370] do_syscall_64+0xc1/0x440 [ 5292.209369] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5292.211850] RIP: 0033:0x7f151bdee10b [ 5292.215034] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5292.230931] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5292.236154] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 5292.240982] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 5292.248850] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5292.256364] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5292.261043] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5295.073198] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3364s! [ 5295.081429] Showing busy workqueues and worker pools: [ 5295.084461] workqueue events: flags=0x0 [ 5295.087569] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5295.096004] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 5295.101265] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5295.107192] pending: kfree_rcu_monitor [ 5295.108487] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5295.115707] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 5295.120702] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5295.125180] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5295.132586] workqueue events_long: flags=0x0 [ 5295.134173] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5295.143332] pending: lu_objects_flush [obdclass] [ 5295.149196] workqueue events_unbound: flags=0x2 [ 5295.156102] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5295.160963] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5295.170065] workqueue events_power_efficient: flags=0x80 [ 5295.173401] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5295.181546] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 5295.185502] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5295.191567] pending: neigh_periodic_work [ 5295.193759] workqueue mm_percpu_wq: flags=0x8 [ 5295.195165] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5295.200036] pending: vmstat_update, lru_add_drain_per_cpu [ 5295.202101] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5295.207616] pending: vmstat_update [ 5295.209907] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5295.214025] pending: lru_add_drain_per_cpu [ 5295.219653] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5295.223154] pending: lru_add_drain_per_cpu BAR(48) [ 5295.226206] workqueue writeback: flags=0x4a [ 5295.229255] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5295.231607] pending: wb_workfn [ 5295.235610] workqueue xprtiod: flags=0xa [ 5295.239331] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5295.249376] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5295.259260] workqueue ipv6_addrconf: flags=0x40008 [ 5295.261352] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5295.263923] pending: addrconf_verify_work [ 5295.266724] workqueue knbd0-recv: flags=0x1a [ 5295.268275] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5295.276607] in-flight: 658:recv_work [ 5295.280581] workqueue ptlrpc_pinger: flags=0x2 [ 5295.288137] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5295.290310] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5295.293672] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3364s workers=4 idle: 32984 37 8 [ 5295.301773] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5280s workers=2 idle: 661 [ 5296.057025] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 5296.060126] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5296.078325] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5296.082794] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5296.086652] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5296.089386] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5296.095955] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 5296.099258] RAX: 00000000000029fe RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5296.102138] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 5296.104585] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5296.107537] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 5296.110237] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 5296.113045] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 5296.116145] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5296.118253] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 5296.120879] Call Trace: [ 5296.122134] [ 5296.122854] ? show_regs.cold.9+0x22/0x2f [ 5296.124398] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5296.126355] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5296.128331] ? watchdog+0x40/0x40 [ 5296.129598] ? hrtimer_interrupt+0x128/0x2c0 [ 5296.131342] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5296.133169] ? apic_timer_interrupt+0xf/0x20 [ 5296.135002] [ 5296.135670] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5296.138038] do_raw_spin_lock+0x177/0x190 [ 5296.139542] _raw_spin_lock+0x21/0x30 [ 5296.141056] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5296.143452] ? do_raw_spin_unlock+0x75/0x190 [ 5296.145276] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5296.148152] ? do_wait_intr+0xf0/0xf0 [ 5296.149607] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5296.152202] mdc_close+0x53b/0xd70 [mdc] [ 5296.153880] lmv_close+0x18d/0x490 [lmv] [ 5296.155448] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5296.158028] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5296.160430] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5296.162435] ll_md_close+0x274/0x800 [lustre] [ 5296.164311] ll_file_release+0x128/0xc50 [lustre] [ 5296.167017] __fput+0x10d/0x440 [ 5296.168346] ____fput+0x12/0x20 [ 5296.169593] task_work_run+0xc6/0x110 [ 5296.171125] exit_to_usermode_loop+0x1dd/0x1f0 [ 5296.173197] do_syscall_64+0x3d6/0x440 [ 5296.174511] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5296.176290] RIP: 0033:0x7f151bee3cd8 [ 5296.177607] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5296.184924] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5296.187669] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 5296.190466] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 5296.193185] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5296.196416] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5296.198496] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5308.055029] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 5308.056021] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5308.058020] Modules linked in: [ 5308.075136] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5308.076414] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5308.076481] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5308.076488] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5308.076491] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5308.076506] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5308.076510] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5308.076515] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 5308.076517] RDX: 0000000000001d64 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 5308.076520] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 5308.076523] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 5308.076525] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5308.076528] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 5308.076531] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5308.076533] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5308.076541] Call Trace: [ 5308.076546] [ 5308.076553] ? show_regs.cold.9+0x22/0x2f [ 5308.076563] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5308.076570] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5308.076575] ? watchdog+0x40/0x40 [ 5308.076579] ? hrtimer_interrupt+0x128/0x2c0 [ 5308.076583] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5308.076591] ? apic_timer_interrupt+0xf/0x20 [ 5308.076596] [ 5308.076599] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5308.076603] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5308.076607] do_raw_spin_lock+0x177/0x190 [ 5308.076611] _raw_spin_lock+0x21/0x30 [ 5308.076617] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5308.076924] obd_mod_free+0x46/0x140 [ptlrpc] [ 5308.077128] mdc_commit_open+0x91/0xb0 [mdc] [ 5308.077166] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5308.077360] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5308.077554] after_reply+0xe2c/0x1780 [ptlrpc] [ 5308.077770] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5308.115368] [ 5308.152383] ? __next_timer_interrupt+0x160/0x160 [ 5308.152406] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5308.161910] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5308.161923] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5308.161931] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5308.162227] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5308.162239] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5308.162244] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5308.162248] RAX: 0000000000005b0f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5308.162250] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 5308.162253] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 5308.162255] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 5308.162258] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 5308.162261] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 5308.162264] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5308.162267] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5308.162277] Call Trace: [ 5308.162282] [ 5308.162288] ? show_regs.cold.9+0x22/0x2f [ 5308.162299] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5308.162307] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5308.162312] ? watchdog+0x40/0x40 [ 5308.162316] ? hrtimer_interrupt+0x128/0x2c0 [ 5308.162320] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5308.162328] ? apic_timer_interrupt+0xf/0x20 [ 5308.162333] [ 5308.162336] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5308.162341] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5308.162346] do_raw_spin_lock+0x177/0x190 [ 5308.162351] _raw_spin_lock+0x21/0x30 [ 5308.162356] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5308.162942] process_one_work+0x2c8/0x700 [ 5308.162952] worker_thread+0x296/0x6e0 [ 5308.162957] ? rescuer_thread+0x570/0x570 [ 5308.162961] kthread+0x1d1/0x200 [ 5308.162965] ? set_kthread_struct+0x70/0x70 [ 5308.162969] ret_from_fork+0x1f/0x30 [ 5308.563893] ? do_wait_intr+0xf0/0xf0 [ 5308.565867] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5308.569196] kthread+0x1d1/0x200 [ 5308.570686] ? set_kthread_struct+0x70/0x70 [ 5308.574519] ret_from_fork+0x1f/0x30 [ 5320.052066] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 5320.056650] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5320.084751] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5320.095144] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5320.100725] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5320.103315] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5320.118888] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 5320.123474] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5320.127152] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 5320.130637] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 5320.134291] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 5320.138837] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 5320.146086] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 5320.149937] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5320.152239] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 5320.157081] Call Trace: [ 5320.158217] [ 5320.158957] ? show_regs.cold.9+0x22/0x2f [ 5320.161744] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5320.164065] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5320.167731] ? watchdog+0x40/0x40 [ 5320.168871] ? hrtimer_interrupt+0x128/0x2c0 [ 5320.178369] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5320.183953] ? apic_timer_interrupt+0xf/0x20 [ 5320.186809] [ 5320.190853] ? native_safe_halt+0x2e/0x50 [ 5320.197325] kvm_wait+0xcd/0xe0 [ 5320.201981] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5320.205398] do_raw_spin_lock+0x177/0x190 [ 5320.211394] _raw_spin_lock+0x21/0x30 [ 5320.212881] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5320.215159] ? do_raw_spin_unlock+0x75/0x190 [ 5320.216758] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5320.219749] ? do_wait_intr+0xf0/0xf0 [ 5320.221180] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5320.228870] mdc_reint+0x2d/0x110 [mdc] [ 5320.236121] mdc_unlink+0x658/0x870 [mdc] [ 5320.238240] lmv_unlink+0x34d/0xbd0 [lmv] [ 5320.247212] ll_unlink+0x24d/0x7f0 [lustre] [ 5320.248866] vfs_unlink+0x202/0x330 [ 5320.255780] do_unlinkat+0x2fa/0x460 [ 5320.259250] __x64_sys_unlink+0x37/0x50 [ 5320.263355] do_syscall_64+0xc1/0x440 [ 5320.267761] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5320.273067] RIP: 0033:0x7f151bdee10b [ 5320.274961] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5320.284410] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5320.287921] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 5320.291430] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 5320.295597] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5320.299309] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5320.303149] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5324.057049] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 5324.060785] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5324.081919] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5324.088799] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5324.094915] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5324.097770] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5324.105214] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5324.108136] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5324.111906] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 5324.114730] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5324.119108] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 5324.121766] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 5324.124282] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 5324.129082] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5324.130840] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 5324.134804] Call Trace: [ 5324.137154] [ 5324.137823] ? show_regs.cold.9+0x22/0x2f [ 5324.139816] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5324.143099] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5324.149777] ? watchdog+0x40/0x40 [ 5324.153228] ? hrtimer_interrupt+0x128/0x2c0 [ 5324.154799] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5324.157718] ? apic_timer_interrupt+0xf/0x20 [ 5324.159866] [ 5324.162557] ? native_safe_halt+0x2e/0x50 [ 5324.164395] kvm_wait+0xcd/0xe0 [ 5324.166776] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5324.168870] do_raw_spin_lock+0x177/0x190 [ 5324.171812] _raw_spin_lock+0x21/0x30 [ 5324.173494] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5324.175593] ? do_raw_spin_unlock+0x75/0x190 [ 5324.177144] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5324.178878] ? do_wait_intr+0xf0/0xf0 [ 5324.180252] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5324.182529] mdc_close+0x53b/0xd70 [mdc] [ 5324.183788] lmv_close+0x18d/0x490 [lmv] [ 5324.185299] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5324.187690] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5324.190632] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5324.192876] ll_md_close+0x274/0x800 [lustre] [ 5324.197353] ll_file_release+0x128/0xc50 [lustre] [ 5324.202930] __fput+0x10d/0x440 [ 5324.210367] ____fput+0x12/0x20 [ 5324.215711] task_work_run+0xc6/0x110 [ 5324.223319] exit_to_usermode_loop+0x1dd/0x1f0 [ 5324.230027] do_syscall_64+0x3d6/0x440 [ 5324.237179] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5324.239198] RIP: 0033:0x7f151bee3cd8 [ 5324.246344] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5324.267566] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5324.270105] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 5324.272802] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 5324.275840] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5324.278199] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5324.280350] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5325.792063] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3395s! [ 5325.796520] Showing busy workqueues and worker pools: [ 5325.799024] workqueue events: flags=0x0 [ 5325.800737] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5325.803833] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 5325.808150] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5325.811617] pending: kfree_rcu_monitor [ 5325.813304] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5325.816444] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 5325.820512] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5325.823456] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5325.826621] workqueue events_long: flags=0x0 [ 5325.828547] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5325.831084] pending: lu_objects_flush [obdclass] [ 5325.833477] workqueue events_unbound: flags=0x2 [ 5325.835293] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5325.838037] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5325.843048] workqueue events_power_efficient: flags=0x80 [ 5325.846051] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5325.849082] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 5325.852020] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5325.854917] pending: neigh_periodic_work [ 5325.856725] workqueue mm_percpu_wq: flags=0x8 [ 5325.858824] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5325.861473] pending: vmstat_update, lru_add_drain_per_cpu [ 5325.863475] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5325.866591] pending: vmstat_update [ 5325.867841] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5325.869513] pending: lru_add_drain_per_cpu [ 5325.870866] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5325.872836] pending: lru_add_drain_per_cpu BAR(48) [ 5325.874366] workqueue writeback: flags=0x4a [ 5325.876123] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5325.878716] pending: wb_workfn [ 5325.880172] workqueue xprtiod: flags=0xa [ 5325.881863] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5325.883634] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5325.886047] workqueue ipv6_addrconf: flags=0x40008 [ 5325.887433] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5325.889294] pending: addrconf_verify_work [ 5325.890494] workqueue knbd0-recv: flags=0x1a [ 5325.891816] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5325.893376] in-flight: 658:recv_work [ 5325.894480] workqueue ptlrpc_pinger: flags=0x2 [ 5325.895588] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5325.897148] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5325.898904] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3395s workers=4 idle: 32984 37 8 [ 5325.900826] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5310s workers=2 idle: 661 [ 5336.055108] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 5336.058415] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5336.059113] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5336.059194] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5336.059201] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5336.059203] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5336.059216] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5336.059220] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5336.059225] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 5336.059227] RDX: 0000000000007f88 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 5336.059230] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 5336.059232] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 5336.059234] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5336.059237] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 5336.059240] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5336.059242] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5336.059252] Call Trace: [ 5336.059259] [ 5336.059264] ? show_regs.cold.9+0x22/0x2f [ 5336.059274] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5336.059280] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5336.059285] ? watchdog+0x40/0x40 [ 5336.059289] ? hrtimer_interrupt+0x128/0x2c0 [ 5336.059373] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5336.059381] ? apic_timer_interrupt+0xf/0x20 [ 5336.059386] [ 5336.059389] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5336.059394] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5336.059398] do_raw_spin_lock+0x177/0x190 [ 5336.059402] _raw_spin_lock+0x21/0x30 [ 5336.059408] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5336.059705] obd_mod_free+0x46/0x140 [ptlrpc] [ 5336.059915] mdc_commit_open+0x91/0xb0 [mdc] [ 5336.059952] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5336.060165] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5336.061397] after_reply+0xe2c/0x1780 [ptlrpc] [ 5336.067788] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5336.067891] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5336.067898] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5336.067908] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5336.068640] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5336.068653] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5336.068658] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5336.068663] RAX: 0000000000007f0b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5336.068666] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 5336.068668] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 5336.068670] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 5336.068673] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 5336.068680] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 5336.068684] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5336.068686] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5336.068696] Call Trace: [ 5336.068702] [ 5336.068707] ? show_regs.cold.9+0x22/0x2f [ 5336.068717] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5336.068723] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5336.068728] ? watchdog+0x40/0x40 [ 5336.068732] ? hrtimer_interrupt+0x128/0x2c0 [ 5336.068736] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5336.068743] ? apic_timer_interrupt+0xf/0x20 [ 5336.068747] [ 5336.068750] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5336.068755] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5336.068761] do_raw_spin_lock+0x177/0x190 [ 5336.068765] _raw_spin_lock+0x21/0x30 [ 5336.068770] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5336.068980] process_one_work+0x2c8/0x700 [ 5336.068988] worker_thread+0x296/0x6e0 [ 5336.068993] ? rescuer_thread+0x570/0x570 [ 5336.068996] kthread+0x1d1/0x200 [ 5336.069001] ? set_kthread_struct+0x70/0x70 [ 5336.069004] ret_from_fork+0x1f/0x30 [ 5336.458730] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5336.471554] ? __next_timer_interrupt+0x160/0x160 [ 5336.477534] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5336.483409] ? do_wait_intr+0xf0/0xf0 [ 5336.488022] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5336.491534] kthread+0x1d1/0x200 [ 5336.496915] ? set_kthread_struct+0x70/0x70 [ 5336.498389] ret_from_fork+0x1f/0x30 [ 5352.052029] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 5352.057039] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 5352.061012] Modules linked in: [ 5352.064890] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5352.064962] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5352.064969] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5352.064973] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5352.064985] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5352.064989] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5352.064994] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5352.064996] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 5352.064998] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5352.065000] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 5352.065003] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 5352.065006] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 5352.065009] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5352.065013] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 5352.065021] Call Trace: [ 5352.065027] [ 5352.065033] ? show_regs.cold.9+0x22/0x2f [ 5352.065043] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5352.065049] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5352.065053] ? watchdog+0x40/0x40 [ 5352.065057] ? hrtimer_interrupt+0x128/0x2c0 [ 5352.065061] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5352.065067] ? apic_timer_interrupt+0xf/0x20 [ 5352.065072] [ 5352.065075] ? native_safe_halt+0x2e/0x50 [ 5352.065079] kvm_wait+0xcd/0xe0 [ 5352.065086] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5352.065095] do_raw_spin_lock+0x177/0x190 [ 5352.065099] _raw_spin_lock+0x21/0x30 [ 5352.065102] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5352.070351] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5352.070426] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5352.070433] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5352.070435] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5352.070448] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5352.070452] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5352.070457] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5352.070459] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 5352.070462] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 5352.070464] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 5352.070466] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 5352.070470] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 5352.070473] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5352.070475] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 5352.070484] Call Trace: [ 5352.070489] [ 5352.070495] ? show_regs.cold.9+0x22/0x2f [ 5352.070504] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5352.070510] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5352.070514] ? watchdog+0x40/0x40 [ 5352.070519] ? hrtimer_interrupt+0x128/0x2c0 [ 5352.070523] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5352.070530] ? apic_timer_interrupt+0xf/0x20 [ 5352.070534] [ 5352.070537] ? native_safe_halt+0x2e/0x50 [ 5352.070903] kvm_wait+0xcd/0xe0 [ 5352.118623] ? do_raw_spin_unlock+0x75/0x190 [ 5352.118643] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5352.118909] ? do_wait_intr+0xf0/0xf0 [ 5352.118918] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5352.126369] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5352.132374] mdc_close+0x53b/0xd70 [mdc] [ 5352.132423] lmv_close+0x18d/0x490 [lmv] [ 5352.132452] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5352.132671] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5352.132790] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5352.137593] do_raw_spin_lock+0x177/0x190 [ 5352.146397] ll_md_close+0x274/0x800 [lustre] [ 5352.146579] ll_file_release+0x128/0xc50 [lustre] [ 5352.146658] __fput+0x10d/0x440 [ 5352.146670] ____fput+0x12/0x20 [ 5352.146674] task_work_run+0xc6/0x110 [ 5352.146682] exit_to_usermode_loop+0x1dd/0x1f0 [ 5352.146688] do_syscall_64+0x3d6/0x440 [ 5352.146692] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5352.146701] RIP: 0033:0x7f151bee3cd8 [ 5352.146710] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5352.146714] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5352.146719] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 5352.146721] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 5352.146723] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5352.146725] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5352.146727] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5352.585674] _raw_spin_lock+0x21/0x30 [ 5352.588032] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5352.596787] ? do_raw_spin_unlock+0x75/0x190 [ 5352.599439] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5352.607794] ? do_wait_intr+0xf0/0xf0 [ 5352.610850] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5352.620357] mdc_reint+0x2d/0x110 [mdc] [ 5352.625772] mdc_unlink+0x658/0x870 [mdc] [ 5352.629388] lmv_unlink+0x34d/0xbd0 [lmv] [ 5352.635018] ll_unlink+0x24d/0x7f0 [lustre] [ 5352.639596] vfs_unlink+0x202/0x330 [ 5352.644390] do_unlinkat+0x2fa/0x460 [ 5352.648746] __x64_sys_unlink+0x37/0x50 [ 5352.659398] do_syscall_64+0xc1/0x440 [ 5352.665699] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5352.672400] RIP: 0033:0x7f151bdee10b [ 5352.679060] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5352.708818] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5352.712080] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 5352.719423] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 5352.726010] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5352.733751] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5352.739570] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5356.006129] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3425s! [ 5356.010371] Showing busy workqueues and worker pools: [ 5356.012733] workqueue events: flags=0x0 [ 5356.015306] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5356.018612] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 5356.023608] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5356.027552] pending: kfree_rcu_monitor [ 5356.030744] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5356.033474] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 5356.039787] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5356.042110] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5356.046298] workqueue events_long: flags=0x0 [ 5356.048834] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5356.053929] pending: lu_objects_flush [obdclass] [ 5356.057698] workqueue events_unbound: flags=0x2 [ 5356.062183] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5356.065693] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5356.070934] workqueue events_power_efficient: flags=0x80 [ 5356.073579] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5356.078982] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 5356.083432] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5356.089015] pending: neigh_periodic_work [ 5356.092270] workqueue mm_percpu_wq: flags=0x8 [ 5356.095316] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5356.100420] pending: vmstat_update, lru_add_drain_per_cpu [ 5356.107404] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5356.112955] pending: vmstat_update [ 5356.115646] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5356.119419] pending: lru_add_drain_per_cpu [ 5356.122249] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5356.125793] pending: lru_add_drain_per_cpu BAR(48) [ 5356.128744] workqueue writeback: flags=0x4a [ 5356.131189] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5356.134388] pending: wb_workfn [ 5356.136399] workqueue xprtiod: flags=0xa [ 5356.137921] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5356.140803] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5356.145245] workqueue ipv6_addrconf: flags=0x40008 [ 5356.148658] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5356.151743] pending: addrconf_verify_work [ 5356.154210] workqueue knbd0-recv: flags=0x1a [ 5356.156337] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5356.159619] in-flight: 658:recv_work [ 5356.161056] workqueue ptlrpc_pinger: flags=0x2 [ 5356.162759] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5356.164999] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5356.169055] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3425s workers=4 idle: 32984 37 8 [ 5356.174055] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5340s workers=2 idle: 661 [ 5364.055026] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 5364.056046] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5364.061567] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5364.061647] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5364.061655] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5364.061657] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5364.061669] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5364.061673] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5364.061678] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 5364.061680] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 5364.061683] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 5364.061685] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 5364.061688] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5364.061691] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 5364.061693] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5364.061695] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5364.061704] Call Trace: [ 5364.061710] [ 5364.061715] ? show_regs.cold.9+0x22/0x2f [ 5364.061724] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5364.061731] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5364.061735] ? watchdog+0x40/0x40 [ 5364.061738] ? hrtimer_interrupt+0x128/0x2c0 [ 5364.061742] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5364.061748] ? apic_timer_interrupt+0xf/0x20 [ 5364.061753] [ 5364.061755] ? native_safe_halt+0x2e/0x50 [ 5364.061759] kvm_wait+0xcd/0xe0 [ 5364.061766] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5364.061773] do_raw_spin_lock+0x177/0x190 [ 5364.061778] _raw_spin_lock+0x21/0x30 [ 5364.061783] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5364.062225] obd_mod_free+0x46/0x140 [ptlrpc] [ 5364.065090] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5364.065172] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5364.065178] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5364.065183] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5364.065427] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5364.065435] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5364.065439] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5364.065444] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5364.065446] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 5364.065448] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 5364.065450] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 5364.065452] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 5364.065455] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 5364.065458] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5364.065461] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5364.065468] Call Trace: [ 5364.065473] [ 5364.065477] ? show_regs.cold.9+0x22/0x2f [ 5364.065487] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5364.065493] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5364.065497] ? watchdog+0x40/0x40 [ 5364.065501] ? hrtimer_interrupt+0x128/0x2c0 [ 5364.065504] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5364.065509] ? apic_timer_interrupt+0xf/0x20 [ 5364.065514] [ 5364.065516] ? native_safe_halt+0x2e/0x50 [ 5364.065520] kvm_wait+0xcd/0xe0 [ 5364.065526] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5364.065533] do_raw_spin_lock+0x177/0x190 [ 5364.065538] _raw_spin_lock+0x21/0x30 [ 5364.065542] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5364.065743] process_one_work+0x2c8/0x700 [ 5364.065750] worker_thread+0x296/0x6e0 [ 5364.065754] ? rescuer_thread+0x570/0x570 [ 5364.065757] kthread+0x1d1/0x200 [ 5364.065761] ? set_kthread_struct+0x70/0x70 [ 5364.065764] ret_from_fork+0x1f/0x30 [ 5364.387841] mdc_commit_open+0x91/0xb0 [mdc] [ 5364.389648] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5364.392519] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5364.394850] after_reply+0xe2c/0x1780 [ptlrpc] [ 5364.397265] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5364.399625] ? __next_timer_interrupt+0x160/0x160 [ 5364.401778] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5364.406314] ? do_wait_intr+0xf0/0xf0 [ 5364.408665] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5364.411746] kthread+0x1d1/0x200 [ 5364.414142] ? set_kthread_struct+0x70/0x70 [ 5364.415684] ret_from_fork+0x1f/0x30 [ 5380.059410] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 5380.065573] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5380.091882] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5380.097429] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5380.101343] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5380.103814] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5380.112427] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5380.117261] RAX: 0000000000000aa3 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5380.121431] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 5380.125340] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5380.131123] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 5380.135243] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 5380.138444] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 5380.141646] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5380.144608] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 5380.147809] Call Trace: [ 5380.148612] [ 5380.149502] ? show_regs.cold.9+0x22/0x2f [ 5380.151196] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5380.155404] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5380.157508] ? watchdog+0x40/0x40 [ 5380.158701] ? hrtimer_interrupt+0x128/0x2c0 [ 5380.160820] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5380.162661] ? apic_timer_interrupt+0xf/0x20 [ 5380.164419] [ 5380.165266] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5380.167988] do_raw_spin_lock+0x177/0x190 [ 5380.169713] _raw_spin_lock+0x21/0x30 [ 5380.171348] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5380.174177] ? do_raw_spin_unlock+0x75/0x190 [ 5380.175996] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5380.178119] ? do_wait_intr+0xf0/0xf0 [ 5380.179734] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5380.182345] mdc_close+0x53b/0xd70 [mdc] [ 5380.184074] lmv_close+0x18d/0x490 [lmv] [ 5380.185526] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5380.188036] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5380.190397] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5380.192288] ll_md_close+0x274/0x800 [lustre] [ 5380.196130] ll_file_release+0x128/0xc50 [lustre] [ 5380.199443] __fput+0x10d/0x440 [ 5380.200822] ____fput+0x12/0x20 [ 5380.203669] task_work_run+0xc6/0x110 [ 5380.205731] exit_to_usermode_loop+0x1dd/0x1f0 [ 5380.207539] do_syscall_64+0x3d6/0x440 [ 5380.209307] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5380.211687] RIP: 0033:0x7f151bee3cd8 [ 5380.213415] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5380.222694] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5380.226524] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 5380.231476] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 5380.238029] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5380.247117] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5380.261795] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5384.053816] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 5384.057406] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5384.088339] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5384.093531] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5384.101412] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5384.113238] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5384.135341] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5384.148654] RAX: 00000000000077c7 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5384.150803] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 5384.154566] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 5384.157358] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 5384.160257] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 5384.162941] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 5384.166103] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5384.167799] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 5384.172804] Call Trace: [ 5384.175682] [ 5384.177103] ? show_regs.cold.9+0x22/0x2f [ 5384.178633] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5384.181231] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5384.184369] ? watchdog+0x40/0x40 [ 5384.189253] ? hrtimer_interrupt+0x128/0x2c0 [ 5384.191139] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5384.193398] ? apic_timer_interrupt+0xf/0x20 [ 5384.195137] [ 5384.195841] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5384.197785] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5384.199566] do_raw_spin_lock+0x177/0x190 [ 5384.201082] _raw_spin_lock+0x21/0x30 [ 5384.203286] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5384.207368] ? do_raw_spin_unlock+0x75/0x190 [ 5384.209193] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5384.211622] ? do_wait_intr+0xf0/0xf0 [ 5384.215294] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5384.220143] mdc_reint+0x2d/0x110 [mdc] [ 5384.223571] mdc_unlink+0x658/0x870 [mdc] [ 5384.226215] lmv_unlink+0x34d/0xbd0 [lmv] [ 5384.229173] ll_unlink+0x24d/0x7f0 [lustre] [ 5384.231507] vfs_unlink+0x202/0x330 [ 5384.233404] do_unlinkat+0x2fa/0x460 [ 5384.235234] __x64_sys_unlink+0x37/0x50 [ 5384.236785] do_syscall_64+0xc1/0x440 [ 5384.238104] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5384.241312] RIP: 0033:0x7f151bdee10b [ 5384.242686] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5384.252496] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5384.255573] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 5384.258262] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 5384.260982] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5384.264758] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5384.267008] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5386.210147] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3455s! [ 5386.214212] Showing busy workqueues and worker pools: [ 5386.221994] workqueue events: flags=0x0 [ 5386.224745] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5386.228383] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 5386.232600] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5386.237715] pending: kfree_rcu_monitor [ 5386.240052] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5386.244181] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 5386.250333] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5386.253993] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5386.258176] workqueue events_long: flags=0x0 [ 5386.259707] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5386.266462] pending: lu_objects_flush [obdclass] [ 5386.268465] workqueue events_unbound: flags=0x2 [ 5386.270580] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5386.274551] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5386.279453] workqueue events_power_efficient: flags=0x80 [ 5386.282078] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5386.284295] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 5386.287543] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5386.291696] pending: neigh_periodic_work [ 5386.296295] workqueue mm_percpu_wq: flags=0x8 [ 5386.298321] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5386.300547] pending: vmstat_update, lru_add_drain_per_cpu [ 5386.304325] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5386.307742] pending: vmstat_update [ 5386.310117] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5386.314918] pending: lru_add_drain_per_cpu [ 5386.316937] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5386.319342] pending: lru_add_drain_per_cpu BAR(48) [ 5386.321378] workqueue writeback: flags=0x4a [ 5386.322756] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5386.325551] pending: wb_workfn [ 5386.331170] workqueue xprtiod: flags=0xa [ 5386.334108] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5386.338967] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5386.345422] workqueue ipv6_addrconf: flags=0x40008 [ 5386.348445] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5386.351399] pending: addrconf_verify_work [ 5386.354295] workqueue knbd0-recv: flags=0x1a [ 5386.356141] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5386.359812] in-flight: 658:recv_work [ 5386.364116] workqueue ptlrpc_pinger: flags=0x2 [ 5386.366013] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5386.369195] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5386.371400] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3455s workers=4 idle: 32984 37 8 [ 5386.376318] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5371s workers=2 idle: 661 [ 5392.055116] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 5392.056165] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5392.066939] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5392.067027] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5392.067034] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5392.067037] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5392.067051] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5392.067055] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5392.067059] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 5392.067062] RDX: 00000000000035b8 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 5392.067065] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 5392.067068] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 5392.067070] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5392.067074] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 5392.067077] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5392.067079] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5392.067087] Call Trace: [ 5392.067093] [ 5392.067097] ? show_regs.cold.9+0x22/0x2f [ 5392.067169] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5392.067177] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5392.067181] ? watchdog+0x40/0x40 [ 5392.067186] ? hrtimer_interrupt+0x128/0x2c0 [ 5392.067190] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5392.067199] ? apic_timer_interrupt+0xf/0x20 [ 5392.067203] [ 5392.067207] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5392.067211] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5392.067216] do_raw_spin_lock+0x177/0x190 [ 5392.067220] _raw_spin_lock+0x21/0x30 [ 5392.067226] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5392.075721] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5392.075800] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5392.075808] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5392.108767] obd_mod_free+0x46/0x140 [ptlrpc] [ 5392.109066] mdc_commit_open+0x91/0xb0 [mdc] [ 5392.109104] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5392.109302] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5392.109500] after_reply+0xe2c/0x1780 [ptlrpc] [ 5392.109699] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5392.118579] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5392.122736] ? __next_timer_interrupt+0x160/0x160 [ 5392.122824] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5392.123623] ? do_wait_intr+0xf0/0xf0 [ 5392.123634] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5392.124026] kthread+0x1d1/0x200 [ 5392.124033] ? set_kthread_struct+0x70/0x70 [ 5392.124036] ret_from_fork+0x1f/0x30 [ 5392.305474] [ 5392.306479] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5392.311365] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5392.321673] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5392.324360] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5392.327104] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 5392.329542] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 5392.332335] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 5392.334575] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 5392.336968] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 5392.339717] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5392.341634] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5392.344101] Call Trace: [ 5392.344961] [ 5392.345640] ? show_regs.cold.9+0x22/0x2f [ 5392.346855] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5392.348356] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5392.349858] ? watchdog+0x40/0x40 [ 5392.351123] ? hrtimer_interrupt+0x128/0x2c0 [ 5392.352843] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5392.354649] ? apic_timer_interrupt+0xf/0x20 [ 5392.356249] [ 5392.357001] ? native_safe_halt+0x2e/0x50 [ 5392.358436] kvm_wait+0xcd/0xe0 [ 5392.359543] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5392.361699] do_raw_spin_lock+0x177/0x190 [ 5392.363479] _raw_spin_lock+0x21/0x30 [ 5392.365255] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5392.367976] process_one_work+0x2c8/0x700 [ 5392.370260] worker_thread+0x296/0x6e0 [ 5392.374693] ? rescuer_thread+0x570/0x570 [ 5392.378075] kthread+0x1d1/0x200 [ 5392.381985] ? set_kthread_struct+0x70/0x70 [ 5392.384581] ret_from_fork+0x1f/0x30 [ 5404.711026] rcu: INFO: rcu_sched self-detected stall on CPU [ 5404.714109] rcu: 1-...!: (2039548 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 5404.720620] (t=3480064 jiffies g=249657 q=3356) [ 5404.723683] rcu: rcu_sched kthread starved for 3475016 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 5404.730439] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5404.735326] rcu: RCU grace-period kthread stack dump: [ 5404.744391] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5404.754701] Call Trace: [ 5404.755990] __schedule+0x351/0xcb0 [ 5404.758254] schedule+0xc0/0x180 [ 5404.760160] schedule_timeout+0xb4/0x190 [ 5404.766198] ? do_raw_spin_unlock+0x75/0x190 [ 5404.767805] ? __next_timer_interrupt+0x160/0x160 [ 5404.769796] ? prepare_to_swait_event+0xf3/0x140 [ 5404.771689] rcu_gp_kthread+0x8dd/0x15b0 [ 5404.773110] ? force_qs_rnp+0x2c0/0x2c0 [ 5404.780102] kthread+0x1d1/0x200 [ 5404.781735] ? set_kthread_struct+0x70/0x70 [ 5404.784048] ret_from_fork+0x1f/0x30 [ 5404.786014] rcu: Stack dump where RCU GP kthread last ran: [ 5404.791459] NMI backtrace for cpu 1 [ 5404.793084] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5404.803493] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5404.811059] Call Trace: [ 5404.812187] [ 5404.813236] ? dump_stack+0xbb/0x10e [ 5404.815165] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5404.818216] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5404.824156] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5404.828400] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5404.831363] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5404.835599] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 5404.839909] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 5404.842564] ? update_process_times+0x6d/0xc0 [ 5404.847784] ? tick_sched_handle.isra.18+0x27/0x90 [ 5404.849275] ? tick_sched_timer+0x53/0xb0 [ 5404.853207] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5404.858687] ? tick_sched_handle.isra.18+0x90/0x90 [ 5404.863240] ? hrtimer_interrupt+0x128/0x2c0 [ 5404.867855] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5404.871067] ? apic_timer_interrupt+0xf/0x20 [ 5404.874770] [ 5404.876147] ? native_safe_halt+0x2e/0x50 [ 5404.879114] ? kvm_wait+0xcd/0xe0 [ 5404.881222] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5404.883975] ? do_raw_spin_lock+0x177/0x190 [ 5404.886188] ? _raw_spin_lock+0x21/0x30 [ 5404.888280] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5404.890885] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5404.896064] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5404.901838] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5404.909038] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5404.916262] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5404.921612] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5404.925917] ? __next_timer_interrupt+0x160/0x160 [ 5404.929072] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5404.932199] ? do_wait_intr+0xf0/0xf0 [ 5404.935137] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5404.940296] ? kthread+0x1d1/0x200 [ 5404.943815] ? set_kthread_struct+0x70/0x70 [ 5404.946059] ? ret_from_fork+0x1f/0x30 [ 5404.949249] NMI backtrace for cpu 1 [ 5404.951315] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5404.961479] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5404.966187] Call Trace: [ 5404.967295] [ 5404.968362] ? dump_stack+0xbb/0x10e [ 5404.970186] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5404.971967] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5404.976939] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5404.980306] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5404.983091] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5404.984447] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 5404.988981] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 5404.994469] ? update_process_times+0x6d/0xc0 [ 5405.004880] ? tick_sched_handle.isra.18+0x27/0x90 [ 5405.008175] ? tick_sched_timer+0x53/0xb0 [ 5405.014335] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5405.018187] ? tick_sched_handle.isra.18+0x90/0x90 [ 5405.021357] ? hrtimer_interrupt+0x128/0x2c0 [ 5405.024927] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5405.027886] ? apic_timer_interrupt+0xf/0x20 [ 5405.029563] [ 5405.030739] ? native_safe_halt+0x2e/0x50 [ 5405.032476] ? kvm_wait+0xcd/0xe0 [ 5405.034883] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5405.040494] ? do_raw_spin_lock+0x177/0x190 [ 5405.045018] ? _raw_spin_lock+0x21/0x30 [ 5405.053202] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5405.062983] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5405.071977] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5405.077743] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5405.084472] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5405.090154] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5405.096342] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5405.100059] ? __next_timer_interrupt+0x160/0x160 [ 5405.103051] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5405.105995] ? do_wait_intr+0xf0/0xf0 [ 5405.108237] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5405.111713] ? kthread+0x1d1/0x200 [ 5405.113733] ? set_kthread_struct+0x70/0x70 [ 5405.116912] ? ret_from_fork+0x1f/0x30 [ 5408.057286] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 5408.061960] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5408.096056] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5408.113255] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5408.123149] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5408.127824] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5408.142916] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5408.146960] RAX: 0000000000004709 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5408.154965] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 5408.160048] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5408.164301] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 5408.174814] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 5408.182306] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 5408.193945] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5408.197652] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 5408.203555] Call Trace: [ 5408.204715] [ 5408.206750] ? show_regs.cold.9+0x22/0x2f [ 5408.211994] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5408.213218] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5408.218104] ? watchdog+0x40/0x40 [ 5408.223912] ? hrtimer_interrupt+0x128/0x2c0 [ 5408.235458] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5408.244581] ? apic_timer_interrupt+0xf/0x20 [ 5408.245968] [ 5408.247965] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5408.253512] do_raw_spin_lock+0x177/0x190 [ 5408.255312] _raw_spin_lock+0x21/0x30 [ 5408.258495] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5408.263507] ? do_raw_spin_unlock+0x75/0x190 [ 5408.270214] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5408.275348] ? do_wait_intr+0xf0/0xf0 [ 5408.278997] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5408.284160] mdc_close+0x53b/0xd70 [mdc] [ 5408.290880] lmv_close+0x18d/0x490 [lmv] [ 5408.292352] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5408.303934] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5408.307160] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5408.309463] ll_md_close+0x274/0x800 [lustre] [ 5408.313900] ll_file_release+0x128/0xc50 [lustre] [ 5408.317862] __fput+0x10d/0x440 [ 5408.319916] ____fput+0x12/0x20 [ 5408.321114] task_work_run+0xc6/0x110 [ 5408.331112] exit_to_usermode_loop+0x1dd/0x1f0 [ 5408.335952] do_syscall_64+0x3d6/0x440 [ 5408.341793] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5408.347787] RIP: 0033:0x7f151bee3cd8 [ 5408.351953] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5408.377172] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5408.383119] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 5408.391239] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 5408.394437] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5408.400924] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5408.404133] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5412.054683] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 5412.060837] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5412.085108] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5412.091034] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5412.095897] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5412.099283] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5412.109986] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5412.114508] RAX: 0000000000007590 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5412.118791] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 5412.122471] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 5412.126185] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 5412.129777] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 5412.132976] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 5412.136744] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5412.138776] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 5412.141996] Call Trace: [ 5412.143308] [ 5412.144275] ? show_regs.cold.9+0x22/0x2f [ 5412.146133] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5412.148661] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5412.150877] ? watchdog+0x40/0x40 [ 5412.153016] ? hrtimer_interrupt+0x128/0x2c0 [ 5412.155031] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5412.157923] ? apic_timer_interrupt+0xf/0x20 [ 5412.160109] [ 5412.161115] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5412.163913] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5412.166038] do_raw_spin_lock+0x177/0x190 [ 5412.167232] _raw_spin_lock+0x21/0x30 [ 5412.168386] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5412.170848] ? do_raw_spin_unlock+0x75/0x190 [ 5412.172070] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5412.174159] ? do_wait_intr+0xf0/0xf0 [ 5412.175319] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5412.177283] mdc_reint+0x2d/0x110 [mdc] [ 5412.178629] mdc_unlink+0x658/0x870 [mdc] [ 5412.180917] lmv_unlink+0x34d/0xbd0 [lmv] [ 5412.183031] ll_unlink+0x24d/0x7f0 [lustre] [ 5412.185287] vfs_unlink+0x202/0x330 [ 5412.186714] do_unlinkat+0x2fa/0x460 [ 5412.187833] __x64_sys_unlink+0x37/0x50 [ 5412.189409] do_syscall_64+0xc1/0x440 [ 5412.191316] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5412.193651] RIP: 0033:0x7f151bdee10b [ 5412.195347] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5412.205247] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5412.208434] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 5412.211966] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 5412.215909] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5412.219261] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5412.228648] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5416.417057] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3485s! [ 5416.420444] Showing busy workqueues and worker pools: [ 5416.427533] workqueue events: flags=0x0 [ 5416.430996] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5416.437025] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 5416.446348] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5416.455083] pending: kfree_rcu_monitor [ 5416.457173] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5416.464049] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 5416.477879] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5416.482466] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5416.486430] workqueue events_long: flags=0x0 [ 5416.488893] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5416.492779] pending: lu_objects_flush [obdclass] [ 5416.495491] workqueue events_unbound: flags=0x2 [ 5416.497288] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5416.502468] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5416.512238] workqueue events_power_efficient: flags=0x80 [ 5416.516290] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5416.520480] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 5416.524796] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5416.526924] pending: neigh_periodic_work [ 5416.534578] workqueue mm_percpu_wq: flags=0x8 [ 5416.537086] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5416.542371] pending: vmstat_update, lru_add_drain_per_cpu [ 5416.546247] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5416.553068] pending: vmstat_update [ 5416.554821] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5416.558009] pending: lru_add_drain_per_cpu [ 5416.560262] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5416.563308] pending: lru_add_drain_per_cpu BAR(48) [ 5416.566089] workqueue writeback: flags=0x4a [ 5416.567889] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5416.570767] pending: wb_workfn [ 5416.572281] workqueue xprtiod: flags=0xa [ 5416.573982] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5416.576785] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5416.580064] workqueue ipv6_addrconf: flags=0x40008 [ 5416.582046] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5416.584774] pending: addrconf_verify_work [ 5416.586466] workqueue knbd0-recv: flags=0x1a [ 5416.588243] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5416.590696] in-flight: 658:recv_work [ 5416.591789] workqueue ptlrpc_pinger: flags=0x2 [ 5416.593042] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5416.595268] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5416.597501] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3486s workers=4 idle: 32984 37 8 [ 5416.600625] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5401s workers=2 idle: 661 [ 5420.059417] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5420.063703] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5420.095440] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5420.110280] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5420.119902] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5420.126599] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5420.129001] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5420.148029] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5420.152142] RAX: 000000000000101b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5420.165074] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 5420.169872] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 5420.173623] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 5420.177005] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 5420.183921] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 5420.197151] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5420.200690] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5420.203181] Call Trace: [ 5420.204166] [ 5420.205011] ? show_regs.cold.9+0x22/0x2f [ 5420.206451] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5420.209005] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5420.210423] ? watchdog+0x40/0x40 [ 5420.211313] ? hrtimer_interrupt+0x128/0x2c0 [ 5420.212559] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5420.214132] ? apic_timer_interrupt+0xf/0x20 [ 5420.215453] [ 5420.216130] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5420.225595] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5420.227088] do_raw_spin_lock+0x177/0x190 [ 5420.230399] _raw_spin_lock+0x21/0x30 [ 5420.233348] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5420.239328] process_one_work+0x2c8/0x700 [ 5420.244222] worker_thread+0x296/0x6e0 [ 5420.246210] ? rescuer_thread+0x570/0x570 [ 5420.249444] kthread+0x1d1/0x200 [ 5420.251412] ? set_kthread_struct+0x70/0x70 [ 5420.253331] ret_from_fork+0x1f/0x30 [ 5432.055116] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 5432.058575] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5432.078337] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5432.083679] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5432.088237] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5432.090300] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5432.100907] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 5432.105103] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 5432.107731] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 5432.111277] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 5432.115659] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 5432.118639] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5432.123081] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 5432.127323] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5432.130284] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5432.134514] Call Trace: [ 5432.135888] [ 5432.137177] ? show_regs.cold.9+0x22/0x2f [ 5432.139486] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5432.141994] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5432.144858] ? watchdog+0x40/0x40 [ 5432.147670] ? hrtimer_interrupt+0x128/0x2c0 [ 5432.153627] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5432.160666] ? apic_timer_interrupt+0xf/0x20 [ 5432.165661] [ 5432.171543] ? native_safe_halt+0x2e/0x50 [ 5432.176264] kvm_wait+0xcd/0xe0 [ 5432.180201] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5432.183716] do_raw_spin_lock+0x177/0x190 [ 5432.185348] _raw_spin_lock+0x21/0x30 [ 5432.186654] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5432.189274] obd_mod_free+0x46/0x140 [ptlrpc] [ 5432.191238] mdc_commit_open+0x91/0xb0 [mdc] [ 5432.192664] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5432.194816] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5432.197194] after_reply+0xe2c/0x1780 [ptlrpc] [ 5432.199687] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5432.205231] ? __next_timer_interrupt+0x160/0x160 [ 5432.208897] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5432.213801] ? do_wait_intr+0xf0/0xf0 [ 5432.217266] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5432.222285] kthread+0x1d1/0x200 [ 5432.225170] ? set_kthread_struct+0x70/0x70 [ 5432.226969] ret_from_fork+0x1f/0x30 [ 5436.058938] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 5436.069480] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5436.162174] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5436.174646] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5436.179241] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5436.192403] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5436.212666] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 5436.216666] RAX: 0000000000007e0e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5436.221073] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 5436.225630] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5436.229754] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 5436.238883] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 5436.255930] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 5436.267238] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5436.274957] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 5436.287206] Call Trace: [ 5436.290554] [ 5436.293510] ? show_regs.cold.9+0x22/0x2f [ 5436.301623] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5436.304516] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5436.309175] ? watchdog+0x40/0x40 [ 5436.315276] ? hrtimer_interrupt+0x128/0x2c0 [ 5436.320063] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5436.327124] ? apic_timer_interrupt+0xf/0x20 [ 5436.329045] [ 5436.332180] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5436.338387] do_raw_spin_lock+0x177/0x190 [ 5436.342313] _raw_spin_lock+0x21/0x30 [ 5436.344448] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5436.353712] ? do_raw_spin_unlock+0x75/0x190 [ 5436.357157] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5436.365634] ? do_wait_intr+0xf0/0xf0 [ 5436.372305] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5436.376472] mdc_close+0x53b/0xd70 [mdc] [ 5436.385020] lmv_close+0x18d/0x490 [lmv] [ 5436.389069] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5436.393170] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5436.398910] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5436.403126] ll_md_close+0x274/0x800 [lustre] [ 5436.405670] ll_file_release+0x128/0xc50 [lustre] [ 5436.407689] __fput+0x10d/0x440 [ 5436.408914] ____fput+0x12/0x20 [ 5436.410443] task_work_run+0xc6/0x110 [ 5436.411866] exit_to_usermode_loop+0x1dd/0x1f0 [ 5436.413744] do_syscall_64+0x3d6/0x440 [ 5436.416634] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5436.418156] RIP: 0033:0x7f151bee3cd8 [ 5436.421555] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5436.431771] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5436.434931] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 5436.437876] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 5436.441656] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5436.445558] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5436.449178] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5444.057097] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 5444.063963] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5444.098084] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5444.108908] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5444.115671] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5444.119255] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5444.126664] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5444.130377] RAX: 0000000000005784 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5444.132278] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 5444.135104] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 5444.137545] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 5444.140991] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 5444.143716] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 5444.151548] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5444.155159] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 5444.158812] Call Trace: [ 5444.160024] [ 5444.161017] ? show_regs.cold.9+0x22/0x2f [ 5444.163097] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5444.165158] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5444.167480] ? watchdog+0x40/0x40 [ 5444.169092] ? hrtimer_interrupt+0x128/0x2c0 [ 5444.174111] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5444.178357] ? apic_timer_interrupt+0xf/0x20 [ 5444.180209] [ 5444.185999] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5444.190229] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5444.197637] do_raw_spin_lock+0x177/0x190 [ 5444.202075] _raw_spin_lock+0x21/0x30 [ 5444.207019] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5444.211120] ? do_raw_spin_unlock+0x75/0x190 [ 5444.214756] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5444.218441] ? do_wait_intr+0xf0/0xf0 [ 5444.220614] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5444.223572] mdc_reint+0x2d/0x110 [mdc] [ 5444.225147] mdc_unlink+0x658/0x870 [mdc] [ 5444.226467] lmv_unlink+0x34d/0xbd0 [lmv] [ 5444.227978] ll_unlink+0x24d/0x7f0 [lustre] [ 5444.229636] vfs_unlink+0x202/0x330 [ 5444.230976] do_unlinkat+0x2fa/0x460 [ 5444.232639] __x64_sys_unlink+0x37/0x50 [ 5444.234116] do_syscall_64+0xc1/0x440 [ 5444.235750] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5444.238924] RIP: 0033:0x7f151bdee10b [ 5444.240228] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5444.251524] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5444.257118] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 5444.264012] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 5444.268771] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5444.272475] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5444.278440] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5446.624288] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3516s! [ 5446.634685] Showing busy workqueues and worker pools: [ 5446.643805] workqueue events: flags=0x0 [ 5446.645997] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5446.649024] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 5446.655655] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5446.658183] pending: kfree_rcu_monitor [ 5446.660837] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5446.663730] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 5446.670528] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5446.674864] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5446.679090] workqueue events_long: flags=0x0 [ 5446.681288] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5446.685047] pending: lu_objects_flush [obdclass] [ 5446.687452] workqueue events_unbound: flags=0x2 [ 5446.690569] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5446.693666] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5446.703368] workqueue events_power_efficient: flags=0x80 [ 5446.709347] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5446.713505] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 5446.718028] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5446.725594] pending: neigh_periodic_work [ 5446.728535] workqueue mm_percpu_wq: flags=0x8 [ 5446.730519] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5446.734627] pending: vmstat_update, lru_add_drain_per_cpu [ 5446.738096] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5446.741994] pending: vmstat_update [ 5446.744050] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5446.747902] pending: lru_add_drain_per_cpu [ 5446.750430] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5446.755380] pending: lru_add_drain_per_cpu BAR(48) [ 5446.758327] workqueue writeback: flags=0x4a [ 5446.760620] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5446.764639] pending: wb_workfn [ 5446.766664] workqueue xprtiod: flags=0xa [ 5446.769054] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5446.772900] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5446.777419] workqueue ipv6_addrconf: flags=0x40008 [ 5446.780696] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5446.784534] pending: addrconf_verify_work [ 5446.787314] workqueue knbd0-recv: flags=0x1a [ 5446.790290] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5446.794296] in-flight: 658:recv_work [ 5446.796588] workqueue ptlrpc_pinger: flags=0x2 [ 5446.799487] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5446.803512] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5446.807447] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3516s workers=4 idle: 32984 37 8 [ 5446.811298] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5431s workers=2 idle: 661 [ 5448.059594] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5448.063338] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5448.089481] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5448.096920] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5448.105237] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5448.110298] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5448.113493] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5448.129444] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5448.135186] RAX: 00000000000060e2 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5448.143423] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 5448.147774] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 5448.158628] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 5448.170881] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 5448.178771] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 5448.182083] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5448.190384] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5448.195614] Call Trace: [ 5448.197772] [ 5448.198895] ? show_regs.cold.9+0x22/0x2f [ 5448.204840] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5448.207247] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5448.208936] ? watchdog+0x40/0x40 [ 5448.212097] ? hrtimer_interrupt+0x128/0x2c0 [ 5448.217577] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5448.219797] ? apic_timer_interrupt+0xf/0x20 [ 5448.225055] [ 5448.227253] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5448.228735] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5448.231787] do_raw_spin_lock+0x177/0x190 [ 5448.236079] _raw_spin_lock+0x21/0x30 [ 5448.237653] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5448.239718] process_one_work+0x2c8/0x700 [ 5448.241363] worker_thread+0x296/0x6e0 [ 5448.242842] ? rescuer_thread+0x570/0x570 [ 5448.244582] kthread+0x1d1/0x200 [ 5448.246369] ? set_kthread_struct+0x70/0x70 [ 5448.248084] ret_from_fork+0x1f/0x30 [ 5460.055043] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 5460.058137] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5460.094791] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5460.112626] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5460.122771] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5460.125807] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5460.145024] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5460.156393] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 5460.160779] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 5460.164426] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 5460.166954] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 5460.171032] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5460.174699] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 5460.179383] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5460.182272] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5460.185417] Call Trace: [ 5460.186601] [ 5460.187827] ? show_regs.cold.9+0x22/0x2f [ 5460.189690] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5460.192066] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5460.194599] ? watchdog+0x40/0x40 [ 5460.196374] ? hrtimer_interrupt+0x128/0x2c0 [ 5460.198440] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5460.200762] ? apic_timer_interrupt+0xf/0x20 [ 5460.203013] [ 5460.203776] ? native_safe_halt+0x2e/0x50 [ 5460.205196] kvm_wait+0xcd/0xe0 [ 5460.207475] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5460.210322] do_raw_spin_lock+0x177/0x190 [ 5460.212283] _raw_spin_lock+0x21/0x30 [ 5460.213559] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5460.215611] obd_mod_free+0x46/0x140 [ptlrpc] [ 5460.218137] mdc_commit_open+0x91/0xb0 [mdc] [ 5460.220636] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5460.223492] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5460.226672] after_reply+0xe2c/0x1780 [ptlrpc] [ 5460.229452] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5460.231767] ? __next_timer_interrupt+0x160/0x160 [ 5460.234560] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5460.236474] ? do_wait_intr+0xf0/0xf0 [ 5460.238285] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5460.241041] kthread+0x1d1/0x200 [ 5460.242627] ? set_kthread_struct+0x70/0x70 [ 5460.244926] ret_from_fork+0x1f/0x30 [ 5464.057494] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 5464.069207] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5464.089673] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5464.094268] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5464.097612] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5464.099595] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5464.106492] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5464.109381] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5464.111856] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 5464.114679] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5464.117425] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 5464.120241] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 5464.122815] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 5464.125945] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5464.128178] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 5464.130576] Call Trace: [ 5464.131528] [ 5464.132249] ? show_regs.cold.9+0x22/0x2f [ 5464.133511] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5464.134996] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5464.136514] ? watchdog+0x40/0x40 [ 5464.137563] ? hrtimer_interrupt+0x128/0x2c0 [ 5464.138878] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5464.140565] ? apic_timer_interrupt+0xf/0x20 [ 5464.142238] [ 5464.142993] ? native_safe_halt+0x2e/0x50 [ 5464.145224] kvm_wait+0xcd/0xe0 [ 5464.146562] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5464.148175] do_raw_spin_lock+0x177/0x190 [ 5464.149411] _raw_spin_lock+0x21/0x30 [ 5464.150313] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5464.152919] ? do_raw_spin_unlock+0x75/0x190 [ 5464.155496] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5464.158438] ? do_wait_intr+0xf0/0xf0 [ 5464.160460] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5464.164208] mdc_close+0x53b/0xd70 [mdc] [ 5464.166137] lmv_close+0x18d/0x490 [lmv] [ 5464.168364] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5464.174355] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5464.177950] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5464.180480] ll_md_close+0x274/0x800 [lustre] [ 5464.183332] ll_file_release+0x128/0xc50 [lustre] [ 5464.185800] __fput+0x10d/0x440 [ 5464.187119] ____fput+0x12/0x20 [ 5464.188448] task_work_run+0xc6/0x110 [ 5464.190267] exit_to_usermode_loop+0x1dd/0x1f0 [ 5464.192264] do_syscall_64+0x3d6/0x440 [ 5464.202044] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5464.204156] RIP: 0033:0x7f151bee3cd8 [ 5464.205839] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5464.214741] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5464.217794] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 5464.220424] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 5464.222468] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5464.225345] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5464.228234] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5472.052137] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 5472.069334] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5472.108434] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5472.115498] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5472.120989] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5472.125449] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5472.146828] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 5472.157161] RAX: 0000000000007a14 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5472.165563] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 5472.171486] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 5472.176837] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 5472.182302] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 5472.196812] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 5472.207961] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5472.213351] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 5472.219516] Call Trace: [ 5472.223994] [ 5472.225754] ? show_regs.cold.9+0x22/0x2f [ 5472.229560] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5472.232688] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5472.239503] ? watchdog+0x40/0x40 [ 5472.241764] ? hrtimer_interrupt+0x128/0x2c0 [ 5472.252572] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5472.254346] ? apic_timer_interrupt+0xf/0x20 [ 5472.259279] [ 5472.264164] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5472.267682] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5472.274352] do_raw_spin_lock+0x177/0x190 [ 5472.279784] _raw_spin_lock+0x21/0x30 [ 5472.285557] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5472.295855] ? do_raw_spin_unlock+0x75/0x190 [ 5472.304204] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5472.311681] ? do_wait_intr+0xf0/0xf0 [ 5472.313486] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5472.316544] mdc_reint+0x2d/0x110 [mdc] [ 5472.320938] mdc_unlink+0x658/0x870 [mdc] [ 5472.325071] lmv_unlink+0x34d/0xbd0 [lmv] [ 5472.326665] ll_unlink+0x24d/0x7f0 [lustre] [ 5472.329739] vfs_unlink+0x202/0x330 [ 5472.334380] do_unlinkat+0x2fa/0x460 [ 5472.337561] __x64_sys_unlink+0x37/0x50 [ 5472.340672] do_syscall_64+0xc1/0x440 [ 5472.345221] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5472.347541] RIP: 0033:0x7f151bdee10b [ 5472.352179] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5472.365116] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5472.374631] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 5472.380789] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 5472.393067] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5472.399735] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5472.407376] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5476.056117] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5476.060710] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5476.090546] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5476.098387] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5476.103921] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5476.109256] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5476.110980] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5476.122925] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5476.129572] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5476.133447] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 5476.137296] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 5476.142284] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 5476.146572] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 5476.151541] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 5476.155462] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5476.158474] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5476.169254] Call Trace: [ 5476.170224] [ 5476.170939] ? show_regs.cold.9+0x22/0x2f [ 5476.173477] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5476.180724] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5476.183112] ? watchdog+0x40/0x40 [ 5476.185292] ? hrtimer_interrupt+0x128/0x2c0 [ 5476.187872] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5476.189613] ? apic_timer_interrupt+0xf/0x20 [ 5476.198717] [ 5476.205317] ? native_safe_halt+0x2e/0x50 [ 5476.212381] kvm_wait+0xcd/0xe0 [ 5476.218233] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5476.226378] do_raw_spin_lock+0x177/0x190 [ 5476.232400] _raw_spin_lock+0x21/0x30 [ 5476.234309] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5476.241591] process_one_work+0x2c8/0x700 [ 5476.248780] worker_thread+0x296/0x6e0 [ 5476.250006] ? rescuer_thread+0x570/0x570 [ 5476.256996] kthread+0x1d1/0x200 [ 5476.259022] ? set_kthread_struct+0x70/0x70 [ 5476.261617] ret_from_fork+0x1f/0x30 [ 5476.832720] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3546s! [ 5476.838727] Showing busy workqueues and worker pools: [ 5476.842167] workqueue events: flags=0x0 [ 5476.843737] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5476.848477] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 5476.856476] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5476.859611] pending: kfree_rcu_monitor [ 5476.861621] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5476.867046] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 5476.875337] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5476.877757] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5476.882580] workqueue events_long: flags=0x0 [ 5476.885428] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5476.891846] pending: lu_objects_flush [obdclass] [ 5476.896122] workqueue events_unbound: flags=0x2 [ 5476.898503] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5476.905485] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5476.916013] workqueue events_power_efficient: flags=0x80 [ 5476.921012] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5476.927556] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 5476.933568] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5476.943363] pending: neigh_periodic_work [ 5476.946453] workqueue mm_percpu_wq: flags=0x8 [ 5476.952098] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5476.955923] pending: vmstat_update, lru_add_drain_per_cpu [ 5476.958890] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5476.965708] pending: vmstat_update [ 5476.966974] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5476.973680] pending: lru_add_drain_per_cpu [ 5476.975931] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5476.979666] pending: lru_add_drain_per_cpu BAR(48) [ 5476.982344] workqueue writeback: flags=0x4a [ 5476.988673] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5476.997162] pending: wb_workfn [ 5477.000734] workqueue xprtiod: flags=0xa [ 5477.007156] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5477.014748] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5477.023821] workqueue ipv6_addrconf: flags=0x40008 [ 5477.026568] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5477.029292] pending: addrconf_verify_work [ 5477.033182] workqueue knbd0-recv: flags=0x1a [ 5477.035729] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5477.037643] in-flight: 658:recv_work [ 5477.041796] workqueue ptlrpc_pinger: flags=0x2 [ 5477.048235] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5477.051570] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5477.055677] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3546s workers=4 idle: 32984 37 8 [ 5477.060551] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5461s workers=2 idle: 661 [ 5488.056490] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 5488.062333] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5488.093823] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5488.101914] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5488.112138] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5488.121188] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5488.154626] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5488.159554] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 5488.172272] RDX: 000000000000130e RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 5488.184330] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 5488.191029] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 5488.198470] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5488.207037] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 5488.212362] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5488.219618] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5488.227647] Call Trace: [ 5488.229312] [ 5488.234108] ? show_regs.cold.9+0x22/0x2f [ 5488.235860] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5488.243201] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5488.247098] ? watchdog+0x40/0x40 [ 5488.248549] ? hrtimer_interrupt+0x128/0x2c0 [ 5488.253197] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5488.258727] ? apic_timer_interrupt+0xf/0x20 [ 5488.265029] [ 5488.265803] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5488.271165] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5488.274468] do_raw_spin_lock+0x177/0x190 [ 5488.278339] _raw_spin_lock+0x21/0x30 [ 5488.280560] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5488.285046] obd_mod_free+0x46/0x140 [ptlrpc] [ 5488.288142] mdc_commit_open+0x91/0xb0 [mdc] [ 5488.290220] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5488.292775] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5488.297144] after_reply+0xe2c/0x1780 [ptlrpc] [ 5488.299587] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5488.301974] ? __next_timer_interrupt+0x160/0x160 [ 5488.305739] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5488.308059] ? do_wait_intr+0xf0/0xf0 [ 5488.309269] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5488.311521] kthread+0x1d1/0x200 [ 5488.313085] ? set_kthread_struct+0x70/0x70 [ 5488.314989] ret_from_fork+0x1f/0x30 [ 5492.057023] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 5492.060377] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5492.119141] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5492.125254] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5492.137678] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5492.141086] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5492.154670] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5492.157919] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5492.161475] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 5492.165266] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5492.171079] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 5492.184444] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 5492.196181] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 5492.204500] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5492.216371] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 5492.221335] Call Trace: [ 5492.222057] [ 5492.222973] ? show_regs.cold.9+0x22/0x2f [ 5492.224028] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5492.225504] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5492.226961] ? watchdog+0x40/0x40 [ 5492.227832] ? hrtimer_interrupt+0x128/0x2c0 [ 5492.238284] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5492.240553] ? apic_timer_interrupt+0xf/0x20 [ 5492.241879] [ 5492.243948] ? native_safe_halt+0x2e/0x50 [ 5492.245856] kvm_wait+0xcd/0xe0 [ 5492.247309] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5492.249983] do_raw_spin_lock+0x177/0x190 [ 5492.259569] _raw_spin_lock+0x21/0x30 [ 5492.261108] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5492.263704] ? do_raw_spin_unlock+0x75/0x190 [ 5492.265769] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5492.270771] ? do_wait_intr+0xf0/0xf0 [ 5492.279495] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5492.287845] mdc_close+0x53b/0xd70 [mdc] [ 5492.290489] lmv_close+0x18d/0x490 [lmv] [ 5492.291625] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5492.295099] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5492.300199] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5492.303038] ll_md_close+0x274/0x800 [lustre] [ 5492.304408] ll_file_release+0x128/0xc50 [lustre] [ 5492.305977] __fput+0x10d/0x440 [ 5492.307168] ____fput+0x12/0x20 [ 5492.308282] task_work_run+0xc6/0x110 [ 5492.309415] exit_to_usermode_loop+0x1dd/0x1f0 [ 5492.311316] do_syscall_64+0x3d6/0x440 [ 5492.312423] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5492.314486] RIP: 0033:0x7f151bee3cd8 [ 5492.315544] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5492.320243] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5492.323906] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 5492.328302] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 5492.330868] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5492.334480] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5492.338314] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5504.052756] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 5504.056091] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 5504.060123] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) [ 5504.069323] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5504.069459] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5504.069467] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5504.069473] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5504.069923] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5504.070000] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5504.070007] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5504.070012] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5504.070015] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 5504.070018] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 5504.070021] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 5504.070024] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 5504.070027] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 5504.070030] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5504.070033] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5504.070040] Call Trace: [ 5504.070045] [ 5504.070049] ? show_regs.cold.9+0x22/0x2f [ 5504.070058] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5504.070064] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5504.070068] ? watchdog+0x40/0x40 [ 5504.070073] ? hrtimer_interrupt+0x128/0x2c0 [ 5504.070077] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5504.070152] ? apic_timer_interrupt+0xf/0x20 [ 5504.070157] [ 5504.070160] ? native_safe_halt+0x2e/0x50 [ 5504.070164] kvm_wait+0xcd/0xe0 [ 5504.070171] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5504.070179] do_raw_spin_lock+0x177/0x190 [ 5504.070183] _raw_spin_lock+0x21/0x30 [ 5504.070187] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5504.071857] process_one_work+0x2c8/0x700 [ 5504.071879] worker_thread+0x296/0x6e0 [ 5504.071884] ? rescuer_thread+0x570/0x570 [ 5504.071888] kthread+0x1d1/0x200 [ 5504.071893] ? set_kthread_struct+0x70/0x70 [ 5504.071897] ret_from_fork+0x1f/0x30 [ 5504.452010] lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5504.484636] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5504.492895] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5504.499117] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5504.506384] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5504.526164] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5504.535513] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5504.537631] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 5504.552052] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 5504.554912] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 5504.562503] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 5504.570365] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 5504.576925] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5504.584440] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 5504.587289] Call Trace: [ 5504.588377] [ 5504.589286] ? show_regs.cold.9+0x22/0x2f [ 5504.592496] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5504.595293] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5504.598351] ? watchdog+0x40/0x40 [ 5504.601263] ? hrtimer_interrupt+0x128/0x2c0 [ 5504.605119] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5504.606949] ? apic_timer_interrupt+0xf/0x20 [ 5504.608382] [ 5504.613313] ? native_safe_halt+0x2e/0x50 [ 5504.619289] kvm_wait+0xcd/0xe0 [ 5504.622620] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5504.627060] do_raw_spin_lock+0x177/0x190 [ 5504.633815] _raw_spin_lock+0x21/0x30 [ 5504.636851] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5504.639861] ? do_raw_spin_unlock+0x75/0x190 [ 5504.645638] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5504.650650] ? do_wait_intr+0xf0/0xf0 [ 5504.652835] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5504.658172] mdc_reint+0x2d/0x110 [mdc] [ 5504.664476] mdc_unlink+0x658/0x870 [mdc] [ 5504.670910] lmv_unlink+0x34d/0xbd0 [lmv] [ 5504.675887] ll_unlink+0x24d/0x7f0 [lustre] [ 5504.680221] vfs_unlink+0x202/0x330 [ 5504.681605] do_unlinkat+0x2fa/0x460 [ 5504.684517] __x64_sys_unlink+0x37/0x50 [ 5504.685869] do_syscall_64+0xc1/0x440 [ 5504.688145] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5504.691318] RIP: 0033:0x7f151bdee10b [ 5504.693284] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5504.707108] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5504.714307] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 5504.720415] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 5504.726591] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5504.733574] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5504.738567] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5507.552458] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3576s! [ 5507.566746] Showing busy workqueues and worker pools: [ 5507.570931] workqueue events: flags=0x0 [ 5507.574891] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5507.582356] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 5507.605415] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5507.619748] pending: kfree_rcu_monitor [ 5507.630118] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5507.646217] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 5507.664900] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5507.683263] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5507.703239] workqueue events_long: flags=0x0 [ 5507.708377] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5507.719158] pending: lu_objects_flush [obdclass] [ 5507.725333] workqueue events_unbound: flags=0x2 [ 5507.727164] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5507.730968] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5507.754387] workqueue events_power_efficient: flags=0x80 [ 5507.757747] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5507.769524] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 5507.776820] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5507.786754] pending: neigh_periodic_work [ 5507.794712] workqueue mm_percpu_wq: flags=0x8 [ 5507.802062] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5507.806623] pending: vmstat_update, lru_add_drain_per_cpu [ 5507.817496] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5507.826116] pending: vmstat_update [ 5507.829125] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5507.837076] pending: lru_add_drain_per_cpu [ 5507.844440] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5507.850410] pending: lru_add_drain_per_cpu BAR(48) [ 5507.862329] workqueue writeback: flags=0x4a [ 5507.867774] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5507.877540] pending: wb_workfn [ 5507.881490] workqueue xprtiod: flags=0xa [ 5507.886501] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5507.891797] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5507.899184] workqueue ipv6_addrconf: flags=0x40008 [ 5507.900830] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5507.903606] pending: addrconf_verify_work [ 5507.906486] workqueue knbd0-recv: flags=0x1a [ 5507.910565] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5507.918244] in-flight: 658:recv_work [ 5507.922149] workqueue ptlrpc_pinger: flags=0x2 [ 5507.924863] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5507.927975] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5507.937262] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3577s workers=4 idle: 32984 37 8 [ 5507.944584] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5492s workers=2 idle: 661 [ 5516.059825] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 5516.064367] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5516.117520] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5516.125837] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5516.136961] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5516.140624] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5516.164012] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5516.166481] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 5516.177120] RDX: 0000000000002295 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 5516.189117] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 5516.194923] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 5516.198243] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5516.202922] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 5516.210710] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5516.221768] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5516.233426] Call Trace: [ 5516.234910] [ 5516.237525] ? show_regs.cold.9+0x22/0x2f [ 5516.243735] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5516.246218] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5516.253029] ? watchdog+0x40/0x40 [ 5516.254195] ? hrtimer_interrupt+0x128/0x2c0 [ 5516.255620] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5516.267425] ? apic_timer_interrupt+0xf/0x20 [ 5516.270558] [ 5516.272644] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5516.276221] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5516.282126] do_raw_spin_lock+0x177/0x190 [ 5516.283523] _raw_spin_lock+0x21/0x30 [ 5516.288067] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5516.290735] obd_mod_free+0x46/0x140 [ptlrpc] [ 5516.298248] mdc_commit_open+0x91/0xb0 [mdc] [ 5516.302849] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5516.309051] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5516.311254] after_reply+0xe2c/0x1780 [ptlrpc] [ 5516.319321] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5516.331805] ? __next_timer_interrupt+0x160/0x160 [ 5516.338186] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5516.348513] ? do_wait_intr+0xf0/0xf0 [ 5516.352885] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5516.358805] kthread+0x1d1/0x200 [ 5516.364647] ? set_kthread_struct+0x70/0x70 [ 5516.370448] ret_from_fork+0x1f/0x30 [ 5520.057014] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 5520.061890] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5520.095203] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5520.103517] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5520.107217] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5520.110092] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5520.123977] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5520.127271] RAX: 00000000000007a9 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5520.130847] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 5520.136745] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5520.139276] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 5520.144089] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 5520.147115] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 5520.150236] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5520.153437] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 5520.157047] Call Trace: [ 5520.158780] [ 5520.159667] ? show_regs.cold.9+0x22/0x2f [ 5520.161022] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5520.164674] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5520.166562] ? watchdog+0x40/0x40 [ 5520.167946] ? hrtimer_interrupt+0x128/0x2c0 [ 5520.169280] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5520.171713] ? apic_timer_interrupt+0xf/0x20 [ 5520.174493] [ 5520.175224] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5520.177638] do_raw_spin_lock+0x177/0x190 [ 5520.179121] _raw_spin_lock+0x21/0x30 [ 5520.180777] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5520.183088] ? do_raw_spin_unlock+0x75/0x190 [ 5520.184988] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5520.189211] ? do_wait_intr+0xf0/0xf0 [ 5520.192261] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5520.195008] mdc_close+0x53b/0xd70 [mdc] [ 5520.199099] lmv_close+0x18d/0x490 [lmv] [ 5520.203807] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5520.208285] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5520.217031] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5520.218583] ll_md_close+0x274/0x800 [lustre] [ 5520.225891] ll_file_release+0x128/0xc50 [lustre] [ 5520.236950] __fput+0x10d/0x440 [ 5520.239854] ____fput+0x12/0x20 [ 5520.243970] task_work_run+0xc6/0x110 [ 5520.249007] exit_to_usermode_loop+0x1dd/0x1f0 [ 5520.256264] do_syscall_64+0x3d6/0x440 [ 5520.259135] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5520.264074] RIP: 0033:0x7f151bee3cd8 [ 5520.265865] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5520.284983] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5520.287334] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 5520.291029] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 5520.293844] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5520.296218] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5520.298963] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5532.061031] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 5532.061860] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 5532.073990] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5532.074179] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5532.074188] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5532.074204] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5532.074486] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5532.074496] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5532.074501] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5532.074505] RAX: 000000000000788a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5532.074508] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 5532.074510] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 5532.074512] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 5532.074514] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 5532.074516] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 5532.074519] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5532.074521] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5532.074533] Call Trace: [ 5532.074539] [ 5532.074543] ? show_regs.cold.9+0x22/0x2f [ 5532.074552] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5532.074558] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5532.074562] ? watchdog+0x40/0x40 [ 5532.074565] ? hrtimer_interrupt+0x128/0x2c0 [ 5532.074569] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5532.074578] ? apic_timer_interrupt+0xf/0x20 [ 5532.074582] [ 5532.074585] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5532.074589] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5532.074594] do_raw_spin_lock+0x177/0x190 [ 5532.074598] _raw_spin_lock+0x21/0x30 [ 5532.074602] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5532.074829] process_one_work+0x2c8/0x700 [ 5532.074837] worker_thread+0x296/0x6e0 [ 5532.074842] ? rescuer_thread+0x570/0x570 [ 5532.074845] kthread+0x1d1/0x200 [ 5532.074849] ? set_kthread_struct+0x70/0x70 [ 5532.074853] ret_from_fork+0x1f/0x30 [ 5532.525073] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5532.567362] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5532.593450] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5532.608345] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5532.611882] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5532.633056] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5532.641601] RAX: 000000000000748d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5532.652022] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 5532.667357] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 5532.683828] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 5532.694951] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 5532.711196] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 5532.722648] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5532.737276] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 5532.751655] Call Trace: [ 5532.755821] [ 5532.756436] ? show_regs.cold.9+0x22/0x2f [ 5532.768249] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5532.782581] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5532.787758] ? watchdog+0x40/0x40 [ 5532.799315] ? hrtimer_interrupt+0x128/0x2c0 [ 5532.813967] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5532.825747] ? apic_timer_interrupt+0xf/0x20 [ 5532.834773] [ 5532.842909] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5532.850553] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5532.853806] do_raw_spin_lock+0x177/0x190 [ 5532.857661] _raw_spin_lock+0x21/0x30 [ 5532.860342] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5532.867088] ? do_raw_spin_unlock+0x75/0x190 [ 5532.871487] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5532.873462] ? do_wait_intr+0xf0/0xf0 [ 5532.883372] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5532.887598] mdc_reint+0x2d/0x110 [mdc] [ 5532.901066] mdc_unlink+0x658/0x870 [mdc] [ 5532.902532] lmv_unlink+0x34d/0xbd0 [lmv] [ 5532.915805] ll_unlink+0x24d/0x7f0 [lustre] [ 5532.920827] vfs_unlink+0x202/0x330 [ 5532.933080] do_unlinkat+0x2fa/0x460 [ 5532.934326] __x64_sys_unlink+0x37/0x50 [ 5532.951377] do_syscall_64+0xc1/0x440 [ 5532.952727] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5532.954498] RIP: 0033:0x7f151bdee10b [ 5532.974253] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5533.018624] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5533.024551] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 5533.032301] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 5533.045105] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5533.054665] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5533.072580] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5538.272225] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3607s! [ 5538.278279] Showing busy workqueues and worker pools: [ 5538.280800] workqueue events: flags=0x0 [ 5538.283507] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5538.287541] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 5538.293705] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5538.297449] pending: kfree_rcu_monitor [ 5538.299697] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5538.305647] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 5538.316245] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5538.320790] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5538.325496] workqueue events_long: flags=0x0 [ 5538.327788] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5538.331736] pending: lu_objects_flush [obdclass] [ 5538.334706] workqueue events_unbound: flags=0x2 [ 5538.337042] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5538.340292] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5538.346103] workqueue events_power_efficient: flags=0x80 [ 5538.348895] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5538.355729] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 5538.359495] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5538.363671] pending: neigh_periodic_work [ 5538.368098] workqueue mm_percpu_wq: flags=0x8 [ 5538.370491] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5538.373735] pending: vmstat_update, lru_add_drain_per_cpu [ 5538.377445] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5538.381599] pending: vmstat_update [ 5538.384921] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5538.390250] pending: lru_add_drain_per_cpu [ 5538.392860] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5538.396848] pending: lru_add_drain_per_cpu BAR(48) [ 5538.399676] workqueue writeback: flags=0x4a [ 5538.403740] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5538.410814] pending: wb_workfn [ 5538.413016] workqueue xprtiod: flags=0xa [ 5538.419085] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5538.425596] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5538.430092] workqueue ipv6_addrconf: flags=0x40008 [ 5538.432310] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5538.436395] pending: addrconf_verify_work [ 5538.438162] workqueue knbd0-recv: flags=0x1a [ 5538.440037] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5538.443948] in-flight: 658:recv_work [ 5538.446033] workqueue ptlrpc_pinger: flags=0x2 [ 5538.448735] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5538.452244] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5538.458801] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3607s workers=4 idle: 32984 37 8 [ 5538.464105] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5523s workers=2 idle: 661 [ 5544.057253] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 5544.071165] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5544.130024] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5544.143935] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5544.156196] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5544.160195] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5544.180691] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5544.189083] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 5544.194485] RDX: 0000000000000084 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 5544.203654] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 5544.218962] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 5544.227562] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5544.234757] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 5544.239413] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5544.243672] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5544.248383] Call Trace: [ 5544.249586] [ 5544.250959] ? show_regs.cold.9+0x22/0x2f [ 5544.256141] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5544.260256] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5544.263086] ? watchdog+0x40/0x40 [ 5544.268868] ? hrtimer_interrupt+0x128/0x2c0 [ 5544.279546] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5544.291474] ? apic_timer_interrupt+0xf/0x20 [ 5544.296432] [ 5544.298468] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5544.304080] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5544.316700] do_raw_spin_lock+0x177/0x190 [ 5544.319172] _raw_spin_lock+0x21/0x30 [ 5544.323655] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5544.332147] obd_mod_free+0x46/0x140 [ptlrpc] [ 5544.335719] mdc_commit_open+0x91/0xb0 [mdc] [ 5544.338931] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5544.343230] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5544.346924] after_reply+0xe2c/0x1780 [ptlrpc] [ 5544.352805] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5544.359778] ? __next_timer_interrupt+0x160/0x160 [ 5544.363179] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5544.366566] ? do_wait_intr+0xf0/0xf0 [ 5544.368598] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5544.374177] kthread+0x1d1/0x200 [ 5544.376177] ? set_kthread_struct+0x70/0x70 [ 5544.378863] ret_from_fork+0x1f/0x30 [ 5548.057151] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 5548.066968] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5548.101707] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5548.108181] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5548.126727] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5548.132683] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5548.153664] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5548.162077] RAX: 0000000000002208 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5548.166068] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 5548.178170] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5548.185264] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 5548.194425] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 5548.204609] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 5548.210474] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5548.214838] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 5548.219735] Call Trace: [ 5548.220555] [ 5548.221265] ? show_regs.cold.9+0x22/0x2f [ 5548.223521] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5548.225933] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5548.228080] ? watchdog+0x40/0x40 [ 5548.229908] ? hrtimer_interrupt+0x128/0x2c0 [ 5548.231587] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5548.238303] ? apic_timer_interrupt+0xf/0x20 [ 5548.239897] [ 5548.242686] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5548.246824] do_raw_spin_lock+0x177/0x190 [ 5548.250296] _raw_spin_lock+0x21/0x30 [ 5548.251497] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5548.255765] ? do_raw_spin_unlock+0x75/0x190 [ 5548.257006] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5548.260460] ? do_wait_intr+0xf0/0xf0 [ 5548.262008] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5548.266103] mdc_close+0x53b/0xd70 [mdc] [ 5548.270625] lmv_close+0x18d/0x490 [lmv] [ 5548.273589] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5548.278703] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5548.283054] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5548.288488] ll_md_close+0x274/0x800 [lustre] [ 5548.292448] ll_file_release+0x128/0xc50 [lustre] [ 5548.296118] __fput+0x10d/0x440 [ 5548.297194] ____fput+0x12/0x20 [ 5548.300572] task_work_run+0xc6/0x110 [ 5548.302467] exit_to_usermode_loop+0x1dd/0x1f0 [ 5548.304671] do_syscall_64+0x3d6/0x440 [ 5548.306670] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5548.309421] RIP: 0033:0x7f151bee3cd8 [ 5548.311804] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5548.332857] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5548.338230] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 5548.342169] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 5548.348034] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5548.352177] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5548.357936] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5560.058169] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 5560.085098] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5560.131346] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5560.150735] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5560.162817] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5560.169092] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5560.177893] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5560.196600] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5560.202835] RAX: 000000000000700a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5560.211309] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 5560.227944] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 5560.233267] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 5560.240155] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 5560.248816] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 5560.255040] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5560.257039] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5560.264606] Call Trace: [ 5560.265738] [ 5560.266694] ? show_regs.cold.9+0x22/0x2f [ 5560.268617] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5560.270793] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5560.273078] ? watchdog+0x40/0x40 [ 5560.274992] ? hrtimer_interrupt+0x128/0x2c0 [ 5560.276993] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5560.279312] ? apic_timer_interrupt+0xf/0x20 [ 5560.280704] [ 5560.282000] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5560.286000] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5560.291538] do_raw_spin_lock+0x177/0x190 [ 5560.293152] _raw_spin_lock+0x21/0x30 [ 5560.300212] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5560.308129] process_one_work+0x2c8/0x700 [ 5560.309976] worker_thread+0x296/0x6e0 [ 5560.312956] ? rescuer_thread+0x570/0x570 [ 5560.317979] kthread+0x1d1/0x200 [ 5560.319602] ? set_kthread_struct+0x70/0x70 [ 5560.321306] ret_from_fork+0x1f/0x30 [ 5564.060572] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 5564.076155] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5564.152990] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5564.174418] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5564.189415] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5564.202925] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5564.235009] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5564.248201] RAX: 0000000000005c85 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5564.267749] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 5564.280683] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 5564.293528] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 5564.313459] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 5564.336474] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 5564.351654] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5564.360082] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 5564.375645] Call Trace: [ 5564.379198] [ 5564.381054] ? show_regs.cold.9+0x22/0x2f [ 5564.395172] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5564.406452] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5564.422993] ? watchdog+0x40/0x40 [ 5564.424270] ? hrtimer_interrupt+0x128/0x2c0 [ 5564.446490] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5564.448203] ? apic_timer_interrupt+0xf/0x20 [ 5564.465581] [ 5564.469762] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5564.485641] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5564.500576] do_raw_spin_lock+0x177/0x190 [ 5564.505808] _raw_spin_lock+0x21/0x30 [ 5564.517208] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5564.524926] ? do_raw_spin_unlock+0x75/0x190 [ 5564.528202] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5564.532759] ? do_wait_intr+0xf0/0xf0 [ 5564.535963] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5564.539066] mdc_reint+0x2d/0x110 [mdc] [ 5564.541281] mdc_unlink+0x658/0x870 [mdc] [ 5564.543290] lmv_unlink+0x34d/0xbd0 [lmv] [ 5564.545829] ll_unlink+0x24d/0x7f0 [lustre] [ 5564.548419] vfs_unlink+0x202/0x330 [ 5564.550485] do_unlinkat+0x2fa/0x460 [ 5564.552520] __x64_sys_unlink+0x37/0x50 [ 5564.554399] do_syscall_64+0xc1/0x440 [ 5564.560044] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5564.563033] RIP: 0033:0x7f151bdee10b [ 5564.564929] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5564.580451] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5564.585961] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 5564.592197] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 5564.599436] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5564.604364] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5564.611385] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5568.481166] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3637s! [ 5568.484421] Showing busy workqueues and worker pools: [ 5568.486648] workqueue events: flags=0x0 [ 5568.487798] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5568.490185] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 5568.493660] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5568.496575] pending: kfree_rcu_monitor [ 5568.497729] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5568.500546] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 5568.504420] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5568.508254] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5568.512342] workqueue events_long: flags=0x0 [ 5568.514471] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5568.517771] pending: lu_objects_flush [obdclass] [ 5568.520543] workqueue events_unbound: flags=0x2 [ 5568.522989] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5568.526504] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5568.535912] workqueue events_power_efficient: flags=0x80 [ 5568.543739] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5568.547817] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 5568.552649] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5568.560639] pending: neigh_periodic_work [ 5568.562412] workqueue mm_percpu_wq: flags=0x8 [ 5568.563996] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5568.574979] pending: vmstat_update, lru_add_drain_per_cpu [ 5568.579639] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5568.584687] pending: vmstat_update [ 5568.587257] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5568.591490] pending: lru_add_drain_per_cpu [ 5568.592773] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5568.596578] pending: lru_add_drain_per_cpu BAR(48) [ 5568.599993] workqueue writeback: flags=0x4a [ 5568.601762] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5568.604374] pending: wb_workfn [ 5568.605486] workqueue xprtiod: flags=0xa [ 5568.607529] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5568.611728] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5568.615446] workqueue ipv6_addrconf: flags=0x40008 [ 5568.618408] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5568.621879] pending: addrconf_verify_work [ 5568.623681] workqueue knbd0-recv: flags=0x1a [ 5568.625739] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5568.632478] in-flight: 658:recv_work [ 5568.639404] workqueue ptlrpc_pinger: flags=0x2 [ 5568.643738] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5568.649910] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5568.656332] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3638s workers=4 idle: 32984 37 8 [ 5568.661716] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5553s workers=2 idle: 661 [ 5572.056558] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 5572.063635] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5572.096869] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5572.103951] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5572.108860] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5572.111858] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5572.120714] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5572.124356] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 5572.127374] RDX: 00000000000077b5 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 5572.130355] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 5572.133260] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 5572.136708] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5572.145397] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 5572.153164] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5572.158208] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5572.165717] Call Trace: [ 5572.166944] [ 5572.169443] ? show_regs.cold.9+0x22/0x2f [ 5572.177249] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5572.181067] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5572.187769] ? watchdog+0x40/0x40 [ 5572.189046] ? hrtimer_interrupt+0x128/0x2c0 [ 5572.195954] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5572.206283] ? apic_timer_interrupt+0xf/0x20 [ 5572.209118] [ 5572.211890] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5572.218469] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5572.219831] do_raw_spin_lock+0x177/0x190 [ 5572.232683] _raw_spin_lock+0x21/0x30 [ 5572.235873] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5572.240239] obd_mod_free+0x46/0x140 [ptlrpc] [ 5572.243569] mdc_commit_open+0x91/0xb0 [mdc] [ 5572.245508] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5572.248380] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5572.251462] after_reply+0xe2c/0x1780 [ptlrpc] [ 5572.253966] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5572.261314] ? __next_timer_interrupt+0x160/0x160 [ 5572.265224] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5572.267128] ? do_wait_intr+0xf0/0xf0 [ 5572.275977] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5572.279950] kthread+0x1d1/0x200 [ 5572.282343] ? set_kthread_struct+0x70/0x70 [ 5572.283758] ret_from_fork+0x1f/0x30 [ 5576.057005] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 5576.065469] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5576.123552] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5576.141687] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5576.158926] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5576.167323] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5576.187824] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 5576.198924] RAX: 000000000000710a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5576.208188] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 5576.211690] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5576.218362] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 5576.227575] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 5576.235188] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 5576.244201] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5576.247103] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 5576.256542] Call Trace: [ 5576.258504] [ 5576.259922] ? show_regs.cold.9+0x22/0x2f [ 5576.262115] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5576.264087] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5576.266376] ? watchdog+0x40/0x40 [ 5576.268108] ? hrtimer_interrupt+0x128/0x2c0 [ 5576.271018] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5576.273332] ? apic_timer_interrupt+0xf/0x20 [ 5576.275367] [ 5576.276532] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5576.280378] do_raw_spin_lock+0x177/0x190 [ 5576.282724] _raw_spin_lock+0x21/0x30 [ 5576.284378] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5576.286952] ? do_raw_spin_unlock+0x75/0x190 [ 5576.288703] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5576.292249] ? do_wait_intr+0xf0/0xf0 [ 5576.293886] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5576.296972] mdc_close+0x53b/0xd70 [mdc] [ 5576.298645] lmv_close+0x18d/0x490 [lmv] [ 5576.300742] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5576.303420] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5576.305880] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5576.308454] ll_md_close+0x274/0x800 [lustre] [ 5576.310547] ll_file_release+0x128/0xc50 [lustre] [ 5576.312656] __fput+0x10d/0x440 [ 5576.314103] ____fput+0x12/0x20 [ 5576.316417] task_work_run+0xc6/0x110 [ 5576.317917] exit_to_usermode_loop+0x1dd/0x1f0 [ 5576.320423] do_syscall_64+0x3d6/0x440 [ 5576.321946] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5576.324680] RIP: 0033:0x7f151bee3cd8 [ 5576.326430] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5576.335212] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5576.339716] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 5576.343209] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 5576.346909] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5576.350640] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5576.355707] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5584.716871] rcu: INFO: rcu_sched self-detected stall on CPU [ 5584.722464] rcu: 1-...!: (2136568 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 5584.741255] (t=3660069 jiffies g=249657 q=3413) [ 5584.742942] rcu: rcu_sched kthread starved for 3655021 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 5584.757363] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5584.765619] rcu: RCU grace-period kthread stack dump: [ 5584.770120] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5584.779397] Call Trace: [ 5584.781622] __schedule+0x351/0xcb0 [ 5584.786512] schedule+0xc0/0x180 [ 5584.794574] schedule_timeout+0xb4/0x190 [ 5584.799327] ? do_raw_spin_unlock+0x75/0x190 [ 5584.804544] ? __next_timer_interrupt+0x160/0x160 [ 5584.809411] ? prepare_to_swait_event+0xf3/0x140 [ 5584.815550] rcu_gp_kthread+0x8dd/0x15b0 [ 5584.819576] ? force_qs_rnp+0x2c0/0x2c0 [ 5584.824507] kthread+0x1d1/0x200 [ 5584.827220] ? set_kthread_struct+0x70/0x70 [ 5584.832348] ret_from_fork+0x1f/0x30 [ 5584.834176] rcu: Stack dump where RCU GP kthread last ran: [ 5584.839818] Sending NMI from CPU 1 to CPUs 2: [ 5584.844597] NMI backtrace for cpu 2 [ 5584.844603] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5584.844606] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5584.844607] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5584.844610] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5584.844612] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5584.844615] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 [ 5584.844617] RAX: 00000000000043d7 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5584.844619] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 5584.844620] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 5584.844622] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 5584.844623] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 5584.844625] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 5584.844626] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5584.844627] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5584.844629] Call Trace: [ 5584.844629] [ 5584.844630] ? show_regs.cold.9+0x22/0x2f [ 5584.844631] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 5584.844632] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 5584.844633] ? nmi_handle+0xbd/0x210 [ 5584.844634] ? default_do_nmi+0x4b/0x1b0 [ 5584.844635] ? do_nmi+0x280/0x380 [ 5584.844636] ? end_repeat_nmi+0x16/0x1f [ 5584.844637] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5584.844639] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5584.844640] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5584.844641] [ 5584.844642] do_raw_spin_lock+0x177/0x190 [ 5584.844643] _raw_spin_lock+0x21/0x30 [ 5584.844644] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5584.844646] process_one_work+0x2c8/0x700 [ 5584.844646] worker_thread+0x296/0x6e0 [ 5584.844647] ? rescuer_thread+0x570/0x570 [ 5584.844648] kthread+0x1d1/0x200 [ 5584.844650] ? set_kthread_struct+0x70/0x70 [ 5584.844650] ret_from_fork+0x1f/0x30 [ 5584.845189] NMI backtrace for cpu 1 [ 5584.991698] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5584.996431] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5584.999902] Call Trace: [ 5585.001146] [ 5585.002041] ? dump_stack+0xbb/0x10e [ 5585.003419] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5585.009042] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5585.011094] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5585.013038] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5585.014885] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5585.017486] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 5585.019671] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 5585.022281] ? update_process_times+0x6d/0xc0 [ 5585.025034] ? tick_sched_handle.isra.18+0x27/0x90 [ 5585.027767] ? tick_sched_timer+0x53/0xb0 [ 5585.030127] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5585.032260] ? tick_sched_handle.isra.18+0x90/0x90 [ 5585.034870] ? hrtimer_interrupt+0x128/0x2c0 [ 5585.037470] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5585.039769] ? apic_timer_interrupt+0xf/0x20 [ 5585.041682] [ 5585.042715] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5585.045572] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5585.049076] ? do_raw_spin_lock+0x177/0x190 [ 5585.051630] ? _raw_spin_lock+0x21/0x30 [ 5585.054246] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5585.057626] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5585.062190] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5585.064224] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5585.070532] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5585.078055] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5585.081176] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5585.088599] ? __next_timer_interrupt+0x160/0x160 [ 5585.094688] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5585.100168] ? do_wait_intr+0xf0/0xf0 [ 5585.103595] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5585.110250] ? kthread+0x1d1/0x200 [ 5585.114361] ? set_kthread_struct+0x70/0x70 [ 5585.117268] ? ret_from_fork+0x1f/0x30 [ 5596.053721] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 5596.059138] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5596.083261] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5596.093850] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5596.107094] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5596.110483] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5596.126242] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5596.129776] RAX: 0000000000007f91 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5596.134428] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 5596.140267] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 5596.148938] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 5596.166850] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 5596.172415] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 5596.181121] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5596.186338] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 5596.194683] Call Trace: [ 5596.197824] [ 5596.199695] ? show_regs.cold.9+0x22/0x2f [ 5596.204153] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5596.207174] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5596.210565] ? watchdog+0x40/0x40 [ 5596.214240] ? hrtimer_interrupt+0x128/0x2c0 [ 5596.221943] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5596.226211] ? apic_timer_interrupt+0xf/0x20 [ 5596.230450] [ 5596.231945] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5596.239171] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5596.250313] do_raw_spin_lock+0x177/0x190 [ 5596.254683] _raw_spin_lock+0x21/0x30 [ 5596.256681] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5596.261716] ? do_raw_spin_unlock+0x75/0x190 [ 5596.265106] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5596.270202] ? do_wait_intr+0xf0/0xf0 [ 5596.276354] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5596.284782] mdc_reint+0x2d/0x110 [mdc] [ 5596.286913] mdc_unlink+0x658/0x870 [mdc] [ 5596.289230] lmv_unlink+0x34d/0xbd0 [lmv] [ 5596.290774] ll_unlink+0x24d/0x7f0 [lustre] [ 5596.300294] vfs_unlink+0x202/0x330 [ 5596.304476] do_unlinkat+0x2fa/0x460 [ 5596.307730] __x64_sys_unlink+0x37/0x50 [ 5596.310453] do_syscall_64+0xc1/0x440 [ 5596.312721] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5596.315759] RIP: 0033:0x7f151bdee10b [ 5596.317932] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5596.328222] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5596.335078] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 5596.342371] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 5596.346622] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5596.355173] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5596.360929] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5598.693442] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3668s! [ 5598.698421] Showing busy workqueues and worker pools: [ 5598.700502] workqueue events: flags=0x0 [ 5598.704601] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5598.710372] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 5598.714990] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5598.718047] pending: kfree_rcu_monitor [ 5598.719547] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5598.722697] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 5598.726657] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5598.729601] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5598.732893] workqueue events_long: flags=0x0 [ 5598.734484] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5598.737007] pending: lu_objects_flush [obdclass] [ 5598.738871] workqueue events_unbound: flags=0x2 [ 5598.740641] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5598.742933] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5598.746555] workqueue events_power_efficient: flags=0x80 [ 5598.748486] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5598.750929] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 5598.753435] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5598.755705] pending: neigh_periodic_work [ 5598.757402] workqueue mm_percpu_wq: flags=0x8 [ 5598.759045] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5598.761357] pending: vmstat_update, lru_add_drain_per_cpu [ 5598.763488] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5598.765854] pending: vmstat_update [ 5598.767200] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5598.769520] pending: lru_add_drain_per_cpu [ 5598.771320] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5598.773524] pending: lru_add_drain_per_cpu BAR(48) [ 5598.775376] workqueue writeback: flags=0x4a [ 5598.776900] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5598.779098] pending: wb_workfn [ 5598.780637] workqueue xprtiod: flags=0xa [ 5598.782013] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5598.784426] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5598.787202] workqueue ipv6_addrconf: flags=0x40008 [ 5598.788900] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5598.791094] pending: addrconf_verify_work [ 5598.792501] workqueue knbd0-recv: flags=0x1a [ 5598.794440] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5598.796700] in-flight: 658:recv_work [ 5598.798094] workqueue ptlrpc_pinger: flags=0x2 [ 5598.799656] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5598.801714] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5598.804112] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3668s workers=4 idle: 32984 37 8 [ 5598.806679] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5583s workers=2 idle: 661 [ 5604.057870] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 5604.060430] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5604.088434] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5604.103255] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5604.110542] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5604.116602] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5604.134682] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5604.141947] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5604.149094] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 5604.155654] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5604.158919] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 5604.165176] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 5604.171686] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 5604.179923] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5604.183534] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 5604.193243] Call Trace: [ 5604.197889] [ 5604.198601] ? show_regs.cold.9+0x22/0x2f [ 5604.199915] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5604.210466] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5604.213298] ? watchdog+0x40/0x40 [ 5604.217362] ? hrtimer_interrupt+0x128/0x2c0 [ 5604.223337] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5604.225341] ? apic_timer_interrupt+0xf/0x20 [ 5604.229351] [ 5604.230325] ? native_safe_halt+0x2e/0x50 [ 5604.233441] kvm_wait+0xcd/0xe0 [ 5604.238295] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5604.245994] do_raw_spin_lock+0x177/0x190 [ 5604.248482] _raw_spin_lock+0x21/0x30 [ 5604.250461] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5604.255297] ? do_raw_spin_unlock+0x75/0x190 [ 5604.259146] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5604.266447] ? do_wait_intr+0xf0/0xf0 [ 5604.270019] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5604.272672] mdc_close+0x53b/0xd70 [mdc] [ 5604.274962] lmv_close+0x18d/0x490 [lmv] [ 5604.278398] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5604.282471] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5604.286844] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5604.289857] ll_md_close+0x274/0x800 [lustre] [ 5604.293330] ll_file_release+0x128/0xc50 [lustre] [ 5604.299448] __fput+0x10d/0x440 [ 5604.302525] ____fput+0x12/0x20 [ 5604.308960] task_work_run+0xc6/0x110 [ 5604.311915] exit_to_usermode_loop+0x1dd/0x1f0 [ 5604.315694] do_syscall_64+0x3d6/0x440 [ 5604.317482] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5604.322326] RIP: 0033:0x7f151bee3cd8 [ 5604.325449] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5604.341355] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5604.349057] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 5604.356068] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 5604.364022] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5604.367596] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5604.375039] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5612.055277] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 5612.056009] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5612.063363] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5612.063443] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5612.063450] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5612.063453] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5612.063468] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5612.063472] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5612.063476] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 5612.063479] RDX: 0000000000007aff RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 5612.063481] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 5612.063484] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 5612.063486] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5612.063489] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 5612.063492] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5612.063494] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5612.063502] Call Trace: [ 5612.063506] [ 5612.063512] ? show_regs.cold.9+0x22/0x2f [ 5612.063521] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5612.063527] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5612.063531] ? watchdog+0x40/0x40 [ 5612.063536] ? hrtimer_interrupt+0x128/0x2c0 [ 5612.063540] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5612.063547] ? apic_timer_interrupt+0xf/0x20 [ 5612.063551] [ 5612.063555] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5612.063559] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5612.063563] do_raw_spin_lock+0x177/0x190 [ 5612.063567] _raw_spin_lock+0x21/0x30 [ 5612.063573] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5612.074989] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5612.075067] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5612.075074] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5612.075080] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5612.075346] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 5612.075356] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 5612.075360] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5612.075364] RAX: 0000000000001543 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5612.075366] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 5612.075368] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 5612.075370] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 5612.075372] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 5612.075376] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 5612.075378] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5612.075380] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5612.075390] Call Trace: [ 5612.075396] [ 5612.101529] obd_mod_free+0x46/0x140 [ptlrpc] [ 5612.101905] mdc_commit_open+0x91/0xb0 [mdc] [ 5612.101944] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5612.102268] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5612.102469] after_reply+0xe2c/0x1780 [ptlrpc] [ 5612.102724] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5612.103115] ? __next_timer_interrupt+0x160/0x160 [ 5612.103126] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5612.103408] ? do_wait_intr+0xf0/0xf0 [ 5612.103418] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5612.103695] kthread+0x1d1/0x200 [ 5612.103701] ? set_kthread_struct+0x70/0x70 [ 5612.103705] ret_from_fork+0x1f/0x30 [ 5612.455948] ? show_regs.cold.9+0x22/0x2f [ 5612.459454] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5612.463761] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5612.466733] ? watchdog+0x40/0x40 [ 5612.468344] ? hrtimer_interrupt+0x128/0x2c0 [ 5612.470322] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5612.475209] ? apic_timer_interrupt+0xf/0x20 [ 5612.480402] [ 5612.484285] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 5612.486207] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5612.491414] do_raw_spin_lock+0x177/0x190 [ 5612.496387] _raw_spin_lock+0x21/0x30 [ 5612.498353] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5612.501790] process_one_work+0x2c8/0x700 [ 5612.504268] worker_thread+0x296/0x6e0 [ 5612.506248] ? rescuer_thread+0x570/0x570 [ 5612.508427] kthread+0x1d1/0x200 [ 5612.510071] ? set_kthread_struct+0x70/0x70 [ 5612.511509] ret_from_fork+0x1f/0x30 [ 5624.054669] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 5624.066475] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5624.105663] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5624.114968] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5624.121046] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5624.124840] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5624.136836] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5624.141028] RAX: 0000000000001378 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5624.149234] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 5624.154747] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 5624.159902] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 5624.164699] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 5624.167417] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 5624.170322] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5624.172466] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 5624.174899] Call Trace: [ 5624.175834] [ 5624.176460] ? show_regs.cold.9+0x22/0x2f [ 5624.177822] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5624.187593] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5624.189822] ? watchdog+0x40/0x40 [ 5624.191377] ? hrtimer_interrupt+0x128/0x2c0 [ 5624.193413] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5624.196350] ? apic_timer_interrupt+0xf/0x20 [ 5624.198302] [ 5624.199256] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5624.201410] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5624.203644] do_raw_spin_lock+0x177/0x190 [ 5624.205024] _raw_spin_lock+0x21/0x30 [ 5624.206389] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5624.209760] ? do_raw_spin_unlock+0x75/0x190 [ 5624.212407] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5624.215385] ? do_wait_intr+0xf0/0xf0 [ 5624.217187] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5624.218971] mdc_reint+0x2d/0x110 [mdc] [ 5624.220589] mdc_unlink+0x658/0x870 [mdc] [ 5624.221810] lmv_unlink+0x34d/0xbd0 [lmv] [ 5624.223335] ll_unlink+0x24d/0x7f0 [lustre] [ 5624.225025] vfs_unlink+0x202/0x330 [ 5624.226172] do_unlinkat+0x2fa/0x460 [ 5624.227211] __x64_sys_unlink+0x37/0x50 [ 5624.228813] do_syscall_64+0xc1/0x440 [ 5624.230355] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5624.232037] RIP: 0033:0x7f151bdee10b [ 5624.233228] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5624.238665] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5624.241393] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 5624.244009] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 5624.246636] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5624.249244] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5624.251748] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5628.897034] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3698s! [ 5628.901016] Showing busy workqueues and worker pools: [ 5628.904175] workqueue events: flags=0x0 [ 5628.906039] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5628.909814] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 5628.914129] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5628.918180] pending: kfree_rcu_monitor [ 5628.920883] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5628.924097] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 5628.929570] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5628.932960] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5628.937122] workqueue events_long: flags=0x0 [ 5628.939632] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5628.943295] pending: lu_objects_flush [obdclass] [ 5628.946252] workqueue events_unbound: flags=0x2 [ 5628.949067] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5628.952224] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5628.958117] workqueue events_power_efficient: flags=0x80 [ 5628.960835] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5628.964362] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 5628.969020] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5628.975201] pending: neigh_periodic_work [ 5628.977769] workqueue mm_percpu_wq: flags=0x8 [ 5628.979992] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5628.983999] pending: vmstat_update, lru_add_drain_per_cpu [ 5628.987388] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5628.992214] pending: vmstat_update [ 5628.995009] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5628.999152] pending: lru_add_drain_per_cpu [ 5629.001796] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5629.004530] pending: lru_add_drain_per_cpu BAR(48) [ 5629.007025] workqueue writeback: flags=0x4a [ 5629.009149] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5629.011570] pending: wb_workfn [ 5629.012978] workqueue xprtiod: flags=0xa [ 5629.014574] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5629.016817] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5629.020199] workqueue ipv6_addrconf: flags=0x40008 [ 5629.023324] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5629.026217] pending: addrconf_verify_work [ 5629.028132] workqueue knbd0-recv: flags=0x1a [ 5629.030053] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5629.033355] in-flight: 658:recv_work [ 5629.034725] workqueue ptlrpc_pinger: flags=0x2 [ 5629.036719] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5629.040133] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5629.043967] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3698s workers=4 idle: 32984 37 8 [ 5629.047398] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5613s workers=2 idle: 661 [ 5632.059714] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 5632.068343] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5632.110368] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5632.119611] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5632.124957] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5632.127859] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5632.138386] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5632.142408] RAX: 0000000000007c1f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5632.152344] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 5632.162641] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5632.180188] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 5632.192222] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 5632.194481] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 5632.206271] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5632.215730] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 5632.230270] Call Trace: [ 5632.235615] [ 5632.238309] ? show_regs.cold.9+0x22/0x2f [ 5632.242960] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5632.245382] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5632.247876] ? watchdog+0x40/0x40 [ 5632.249198] ? hrtimer_interrupt+0x128/0x2c0 [ 5632.250833] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5632.253541] ? apic_timer_interrupt+0xf/0x20 [ 5632.257118] [ 5632.258932] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5632.260958] do_raw_spin_lock+0x177/0x190 [ 5632.265251] _raw_spin_lock+0x21/0x30 [ 5632.266491] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5632.274124] ? do_raw_spin_unlock+0x75/0x190 [ 5632.276830] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5632.279711] ? do_wait_intr+0xf0/0xf0 [ 5632.286023] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5632.296378] mdc_close+0x53b/0xd70 [mdc] [ 5632.300346] lmv_close+0x18d/0x490 [lmv] [ 5632.309882] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5632.311959] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5632.321467] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5632.326906] ll_md_close+0x274/0x800 [lustre] [ 5632.333410] ll_file_release+0x128/0xc50 [lustre] [ 5632.340917] __fput+0x10d/0x440 [ 5632.346150] ____fput+0x12/0x20 [ 5632.347374] task_work_run+0xc6/0x110 [ 5632.348747] exit_to_usermode_loop+0x1dd/0x1f0 [ 5632.356569] do_syscall_64+0x3d6/0x440 [ 5632.365895] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5632.368089] RIP: 0033:0x7f151bee3cd8 [ 5632.369376] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5632.390731] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5632.399558] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 5632.409940] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 5632.424959] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5632.432203] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5632.439948] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5640.056271] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5640.058596] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 5640.059922] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5640.059994] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5640.060001] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5640.060008] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5640.062015] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5640.062030] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5640.062035] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5640.062039] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5640.062042] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 5640.062044] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 5640.062046] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 5640.062049] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 5640.062053] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 5640.062055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5640.062058] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5640.062066] Call Trace: [ 5640.062070] [ 5640.062075] ? show_regs.cold.9+0x22/0x2f [ 5640.062084] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5640.062089] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5640.062093] ? watchdog+0x40/0x40 [ 5640.062097] ? hrtimer_interrupt+0x128/0x2c0 [ 5640.062101] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5640.062107] ? apic_timer_interrupt+0xf/0x20 [ 5640.062111] [ 5640.062114] ? native_safe_halt+0x2e/0x50 [ 5640.062117] kvm_wait+0xcd/0xe0 [ 5640.062446] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5640.062457] do_raw_spin_lock+0x177/0x190 [ 5640.062461] _raw_spin_lock+0x21/0x30 [ 5640.062466] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5640.069637] Modules linked in: lustre(O) osp(O) ofd(O) [ 5640.099305] process_one_work+0x2c8/0x700 [ 5640.099326] worker_thread+0x296/0x6e0 [ 5640.099332] ? rescuer_thread+0x570/0x570 [ 5640.099338] kthread+0x1d1/0x200 [ 5640.099342] ? set_kthread_struct+0x70/0x70 [ 5640.099346] ret_from_fork+0x1f/0x30 [ 5640.299424] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5640.332983] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5640.351905] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5640.359355] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5640.363218] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5640.383322] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5640.388163] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 5640.395194] RDX: 0000000000007cbc RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 5640.400069] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 5640.404503] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 5640.410111] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5640.418670] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 5640.426122] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5640.428541] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5640.433713] Call Trace: [ 5640.434609] [ 5640.435336] ? show_regs.cold.9+0x22/0x2f [ 5640.437961] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5640.441179] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5640.443615] ? watchdog+0x40/0x40 [ 5640.447315] ? hrtimer_interrupt+0x128/0x2c0 [ 5640.448833] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5640.450450] ? apic_timer_interrupt+0xf/0x20 [ 5640.453721] [ 5640.458632] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5640.467318] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5640.474259] do_raw_spin_lock+0x177/0x190 [ 5640.476539] _raw_spin_lock+0x21/0x30 [ 5640.486608] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5640.488383] obd_mod_free+0x46/0x140 [ptlrpc] [ 5640.495222] mdc_commit_open+0x91/0xb0 [mdc] [ 5640.503153] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5640.509265] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5640.513949] after_reply+0xe2c/0x1780 [ptlrpc] [ 5640.517217] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5640.520255] ? __next_timer_interrupt+0x160/0x160 [ 5640.523529] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5640.528699] ? do_wait_intr+0xf0/0xf0 [ 5640.531579] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5640.535918] kthread+0x1d1/0x200 [ 5640.537648] ? set_kthread_struct+0x70/0x70 [ 5640.539932] ret_from_fork+0x1f/0x30 [ 5656.052080] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 5656.058007] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5656.125593] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5656.143765] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5656.151726] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5656.158202] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5656.187922] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 5656.198172] RAX: 0000000000005f61 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5656.208036] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 5656.222817] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 5656.240808] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 5656.252924] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 5656.261690] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 5656.272572] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5656.284147] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 5656.292285] Call Trace: [ 5656.293139] [ 5656.293810] ? show_regs.cold.9+0x22/0x2f [ 5656.295229] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5656.307126] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5656.313772] ? watchdog+0x40/0x40 [ 5656.314974] ? hrtimer_interrupt+0x128/0x2c0 [ 5656.316563] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5656.327019] ? apic_timer_interrupt+0xf/0x20 [ 5656.334968] [ 5656.339128] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5656.346047] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5656.356083] do_raw_spin_lock+0x177/0x190 [ 5656.361958] _raw_spin_lock+0x21/0x30 [ 5656.365900] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5656.371064] ? do_raw_spin_unlock+0x75/0x190 [ 5656.378910] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5656.387714] ? do_wait_intr+0xf0/0xf0 [ 5656.394273] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5656.400447] mdc_reint+0x2d/0x110 [mdc] [ 5656.405978] mdc_unlink+0x658/0x870 [mdc] [ 5656.410604] lmv_unlink+0x34d/0xbd0 [lmv] [ 5656.413828] ll_unlink+0x24d/0x7f0 [lustre] [ 5656.416297] vfs_unlink+0x202/0x330 [ 5656.421166] do_unlinkat+0x2fa/0x460 [ 5656.424150] __x64_sys_unlink+0x37/0x50 [ 5656.429395] do_syscall_64+0xc1/0x440 [ 5656.436208] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5656.440884] RIP: 0033:0x7f151bdee10b [ 5656.442760] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5656.478966] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5656.483984] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 5656.490074] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 5656.499901] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5656.503454] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5656.508590] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5659.110438] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3728s! [ 5659.133750] Showing busy workqueues and worker pools: [ 5659.148420] workqueue events: flags=0x0 [ 5659.151805] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5659.159582] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 5659.178135] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5659.199488] pending: kfree_rcu_monitor [ 5659.204979] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5659.207288] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 5659.221184] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5659.231367] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5659.234855] workqueue events_long: flags=0x0 [ 5659.246957] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5659.251191] pending: lu_objects_flush [obdclass] [ 5659.262274] workqueue events_unbound: flags=0x2 [ 5659.276017] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5659.290638] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5659.300678] workqueue events_power_efficient: flags=0x80 [ 5659.309050] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5659.317284] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 5659.323446] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5659.330649] pending: neigh_periodic_work [ 5659.338388] workqueue mm_percpu_wq: flags=0x8 [ 5659.350777] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5659.376962] pending: vmstat_update, lru_add_drain_per_cpu [ 5659.378948] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5659.386670] pending: vmstat_update [ 5659.396746] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5659.400394] pending: lru_add_drain_per_cpu [ 5659.403519] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5659.407135] pending: lru_add_drain_per_cpu BAR(48) [ 5659.410070] workqueue writeback: flags=0x4a [ 5659.415564] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5659.421065] pending: wb_workfn [ 5659.422703] workqueue xprtiod: flags=0xa [ 5659.427934] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5659.431574] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5659.436277] workqueue ipv6_addrconf: flags=0x40008 [ 5659.440408] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5659.449100] pending: addrconf_verify_work [ 5659.453236] workqueue knbd0-recv: flags=0x1a [ 5659.456758] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5659.463478] in-flight: 658:recv_work [ 5659.467216] workqueue ptlrpc_pinger: flags=0x2 [ 5659.469542] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5659.472847] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5659.478461] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3728s workers=4 idle: 32984 37 8 [ 5659.484432] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5644s workers=2 idle: 661 [ 5660.059857] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 5660.070411] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5660.111481] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5660.126636] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5660.132425] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5660.134264] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5660.161962] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5660.170633] RAX: 0000000000005b1d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5660.180971] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 5660.193130] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5660.199352] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 5660.203990] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 5660.210562] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 5660.226041] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5660.227984] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 5660.250654] Call Trace: [ 5660.254123] [ 5660.258022] ? show_regs.cold.9+0x22/0x2f [ 5660.272065] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5660.273837] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5660.284942] ? watchdog+0x40/0x40 [ 5660.290157] ? hrtimer_interrupt+0x128/0x2c0 [ 5660.300114] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5660.318705] ? apic_timer_interrupt+0xf/0x20 [ 5660.327561] [ 5660.338308] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5660.341048] do_raw_spin_lock+0x177/0x190 [ 5660.347970] _raw_spin_lock+0x21/0x30 [ 5660.354054] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5660.364579] ? do_raw_spin_unlock+0x75/0x190 [ 5660.366045] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5660.375119] ? do_wait_intr+0xf0/0xf0 [ 5660.378914] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5660.388001] mdc_close+0x53b/0xd70 [mdc] [ 5660.390506] lmv_close+0x18d/0x490 [lmv] [ 5660.398913] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5660.405940] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5660.421165] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5660.428517] ll_md_close+0x274/0x800 [lustre] [ 5660.433486] ll_file_release+0x128/0xc50 [lustre] [ 5660.444449] __fput+0x10d/0x440 [ 5660.446974] ____fput+0x12/0x20 [ 5660.451610] task_work_run+0xc6/0x110 [ 5660.455395] exit_to_usermode_loop+0x1dd/0x1f0 [ 5660.458033] do_syscall_64+0x3d6/0x440 [ 5660.460801] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5660.467013] RIP: 0033:0x7f151bee3cd8 [ 5660.468235] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5660.501349] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5660.513872] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 5660.518955] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 5660.526480] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5660.532023] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5660.538922] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5668.058302] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5668.058850] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 5668.064749] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5668.064828] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5668.064837] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5668.064843] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5668.077766] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5668.077953] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5668.077960] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5668.077963] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5668.077977] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5668.077981] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5668.077985] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 5668.077988] RDX: 00000000000039a3 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 5668.077991] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 5668.077993] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 5668.077995] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5668.077999] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 5668.078002] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5668.078004] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5668.078013] Call Trace: [ 5668.078018] [ 5668.078025] ? show_regs.cold.9+0x22/0x2f [ 5668.078035] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5668.078041] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5668.078046] ? watchdog+0x40/0x40 [ 5668.078051] ? hrtimer_interrupt+0x128/0x2c0 [ 5668.078055] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5668.078064] ? apic_timer_interrupt+0xf/0x20 [ 5668.078069] [ 5668.078292] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5668.078299] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5668.078303] do_raw_spin_lock+0x177/0x190 [ 5668.078308] _raw_spin_lock+0x21/0x30 [ 5668.078314] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5668.138340] [ 5668.138349] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5668.138367] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5668.138371] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5668.138376] RAX: 00000000000077a7 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5668.138378] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 5668.138380] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 5668.138382] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 5668.138385] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 5668.138388] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 5668.138391] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5668.138394] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5668.138402] Call Trace: [ 5668.138407] [ 5668.138413] ? show_regs.cold.9+0x22/0x2f [ 5668.138423] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5668.138429] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5668.138432] ? watchdog+0x40/0x40 [ 5668.138436] ? hrtimer_interrupt+0x128/0x2c0 [ 5668.138440] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5668.138447] ? apic_timer_interrupt+0xf/0x20 [ 5668.138452] [ 5668.138454] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5668.138459] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5668.138466] do_raw_spin_lock+0x177/0x190 [ 5668.138471] _raw_spin_lock+0x21/0x30 [ 5668.138476] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5668.148930] obd_mod_free+0x46/0x140 [ptlrpc] [ 5668.149872] mdc_commit_open+0x91/0xb0 [mdc] [ 5668.149915] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5668.164759] process_one_work+0x2c8/0x700 [ 5668.165328] worker_thread+0x296/0x6e0 [ 5668.165334] ? rescuer_thread+0x570/0x570 [ 5668.165338] kthread+0x1d1/0x200 [ 5668.165343] ? set_kthread_struct+0x70/0x70 [ 5668.165347] ret_from_fork+0x1f/0x30 [ 5668.533708] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5668.539956] after_reply+0xe2c/0x1780 [ptlrpc] [ 5668.545798] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5668.550315] ? __next_timer_interrupt+0x160/0x160 [ 5668.559902] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5668.569398] ? do_wait_intr+0xf0/0xf0 [ 5668.576715] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5668.587886] kthread+0x1d1/0x200 [ 5668.591957] ? set_kthread_struct+0x70/0x70 [ 5668.598748] ret_from_fork+0x1f/0x30 [ 5684.052075] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 5684.056899] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5684.082028] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5684.090240] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5684.092763] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5684.093950] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5684.100234] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5684.106843] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5684.110556] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 5684.113644] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 5684.116875] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 5684.120429] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 5684.121827] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 5684.125292] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5684.126592] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 5684.134184] Call Trace: [ 5684.135579] [ 5684.136976] ? show_regs.cold.9+0x22/0x2f [ 5684.141360] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5684.143531] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5684.145005] ? watchdog+0x40/0x40 [ 5684.146318] ? hrtimer_interrupt+0x128/0x2c0 [ 5684.151263] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5684.152418] ? apic_timer_interrupt+0xf/0x20 [ 5684.153611] [ 5684.154287] ? native_safe_halt+0x2e/0x50 [ 5684.155337] kvm_wait+0xcd/0xe0 [ 5684.156320] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5684.158112] do_raw_spin_lock+0x177/0x190 [ 5684.159540] _raw_spin_lock+0x21/0x30 [ 5684.160876] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5684.163064] ? do_raw_spin_unlock+0x75/0x190 [ 5684.164591] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5684.166801] ? do_wait_intr+0xf0/0xf0 [ 5684.168158] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5684.170232] mdc_reint+0x2d/0x110 [mdc] [ 5684.171899] mdc_unlink+0x658/0x870 [mdc] [ 5684.173790] lmv_unlink+0x34d/0xbd0 [lmv] [ 5684.175418] ll_unlink+0x24d/0x7f0 [lustre] [ 5684.177196] vfs_unlink+0x202/0x330 [ 5684.178577] do_unlinkat+0x2fa/0x460 [ 5684.180069] __x64_sys_unlink+0x37/0x50 [ 5684.181509] do_syscall_64+0xc1/0x440 [ 5684.183205] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5684.185211] RIP: 0033:0x7f151bdee10b [ 5684.186413] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5684.191673] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5684.193737] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 5684.195633] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 5684.198232] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5684.200331] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5684.202194] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5688.057033] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 5688.061513] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5688.102308] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5688.111104] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5688.119857] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5688.127719] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5688.153492] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5688.160591] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5688.171086] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 5688.181479] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5688.186729] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 5688.193999] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 5688.197003] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 5688.207247] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5688.220721] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 5688.232005] Call Trace: [ 5688.233481] [ 5688.234267] ? show_regs.cold.9+0x22/0x2f [ 5688.238786] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5688.241110] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5688.243561] ? watchdog+0x40/0x40 [ 5688.244782] ? hrtimer_interrupt+0x128/0x2c0 [ 5688.253679] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5688.255323] ? apic_timer_interrupt+0xf/0x20 [ 5688.268301] [ 5688.272572] ? native_safe_halt+0x2e/0x50 [ 5688.278701] kvm_wait+0xcd/0xe0 [ 5688.282674] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5688.286099] do_raw_spin_lock+0x177/0x190 [ 5688.290282] _raw_spin_lock+0x21/0x30 [ 5688.292440] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5688.294679] ? do_raw_spin_unlock+0x75/0x190 [ 5688.300502] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5688.306616] ? do_wait_intr+0xf0/0xf0 [ 5688.309500] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5688.313736] mdc_close+0x53b/0xd70 [mdc] [ 5688.316411] lmv_close+0x18d/0x490 [lmv] [ 5688.318047] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5688.320707] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5688.326432] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5688.329718] ll_md_close+0x274/0x800 [lustre] [ 5688.332546] ll_file_release+0x128/0xc50 [lustre] [ 5688.335656] __fput+0x10d/0x440 [ 5688.338773] ____fput+0x12/0x20 [ 5688.343107] task_work_run+0xc6/0x110 [ 5688.345379] exit_to_usermode_loop+0x1dd/0x1f0 [ 5688.348987] do_syscall_64+0x3d6/0x440 [ 5688.352177] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5688.358315] RIP: 0033:0x7f151bee3cd8 [ 5688.359442] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5688.374706] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5688.377827] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 5688.389429] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 5688.393307] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5688.397484] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5688.402365] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5689.826037] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3759s! [ 5689.829829] Showing busy workqueues and worker pools: [ 5689.833411] workqueue events: flags=0x0 [ 5689.834678] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5689.836755] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 5689.841112] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5689.844219] pending: kfree_rcu_monitor [ 5689.845918] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5689.848770] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 5689.853991] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5689.861396] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5689.866219] workqueue events_long: flags=0x0 [ 5689.868120] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5689.871076] pending: lu_objects_flush [obdclass] [ 5689.873453] workqueue events_unbound: flags=0x2 [ 5689.874579] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5689.877169] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5689.882923] workqueue events_power_efficient: flags=0x80 [ 5689.888416] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5689.898524] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 5689.904425] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5689.912277] pending: neigh_periodic_work [ 5689.914650] workqueue mm_percpu_wq: flags=0x8 [ 5689.918116] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5689.925707] pending: vmstat_update, lru_add_drain_per_cpu [ 5689.936768] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5689.946016] pending: vmstat_update [ 5689.948673] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5689.950984] pending: lru_add_drain_per_cpu [ 5689.963043] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5689.968427] pending: lru_add_drain_per_cpu BAR(48) [ 5689.973769] workqueue writeback: flags=0x4a [ 5689.979152] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5689.988195] pending: wb_workfn [ 5689.991833] workqueue xprtiod: flags=0xa [ 5689.994723] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5690.001208] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5690.014561] workqueue ipv6_addrconf: flags=0x40008 [ 5690.016359] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5690.025264] pending: addrconf_verify_work [ 5690.028633] workqueue knbd0-recv: flags=0x1a [ 5690.038607] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5690.045392] in-flight: 658:recv_work [ 5690.046880] workqueue ptlrpc_pinger: flags=0x2 [ 5690.050517] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5690.060483] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5690.069849] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3759s workers=4 idle: 32984 37 8 [ 5690.077447] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5674s workers=2 idle: 661 [ 5696.055577] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 5696.056975] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5696.059605] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5696.059685] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5696.059693] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5696.059696] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5696.059711] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5696.059715] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5696.059720] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 5696.059722] RDX: 0000000000004278 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 5696.059725] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 5696.059727] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 5696.059730] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5696.059733] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 5696.059735] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5696.059737] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5696.059747] Call Trace: [ 5696.059752] [ 5696.059758] ? show_regs.cold.9+0x22/0x2f [ 5696.059768] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5696.059773] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5696.059778] ? watchdog+0x40/0x40 [ 5696.059782] ? hrtimer_interrupt+0x128/0x2c0 [ 5696.059787] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5696.059795] ? apic_timer_interrupt+0xf/0x20 [ 5696.067246] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5696.067320] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5696.067327] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5696.067333] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5696.067613] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5696.067624] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5696.067628] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5696.067634] RAX: 0000000000002d64 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5696.067636] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 5696.067638] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 5696.067640] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 5696.067643] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 5696.067646] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 5696.067649] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5696.067651] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5696.067659] Call Trace: [ 5696.067665] [ 5696.067670] ? show_regs.cold.9+0x22/0x2f [ 5696.067680] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5696.067687] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5696.067691] ? watchdog+0x40/0x40 [ 5696.067696] ? hrtimer_interrupt+0x128/0x2c0 [ 5696.067700] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5696.067708] ? apic_timer_interrupt+0xf/0x20 [ 5696.067713] [ 5696.067716] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5696.067721] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5696.067726] do_raw_spin_lock+0x177/0x190 [ 5696.067731] _raw_spin_lock+0x21/0x30 [ 5696.067736] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5696.067954] process_one_work+0x2c8/0x700 [ 5696.067961] worker_thread+0x296/0x6e0 [ 5696.067966] ? rescuer_thread+0x570/0x570 [ 5696.067970] kthread+0x1d1/0x200 [ 5696.067976] ? set_kthread_struct+0x70/0x70 [ 5696.067979] ret_from_fork+0x1f/0x30 [ 5696.436972] [ 5696.438888] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5696.444329] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5696.447523] do_raw_spin_lock+0x177/0x190 [ 5696.448829] _raw_spin_lock+0x21/0x30 [ 5696.456435] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5696.458400] obd_mod_free+0x46/0x140 [ptlrpc] [ 5696.462787] mdc_commit_open+0x91/0xb0 [mdc] [ 5696.468319] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5696.471337] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5696.474537] after_reply+0xe2c/0x1780 [ptlrpc] [ 5696.477350] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5696.480314] ? __next_timer_interrupt+0x160/0x160 [ 5696.482517] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5696.484620] ? do_wait_intr+0xf0/0xf0 [ 5696.486334] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5696.488705] kthread+0x1d1/0x200 [ 5696.489732] ? set_kthread_struct+0x70/0x70 [ 5696.491029] ret_from_fork+0x1f/0x30 [ 5716.053409] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 5716.057068] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 5716.059041] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5716.059126] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5716.059134] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5716.059137] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5716.059150] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5716.061240] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5716.061247] RAX: 0000000000001661 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5716.061249] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 5716.061252] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 5716.061264] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 5716.061267] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 5716.061270] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 5716.061273] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5716.061275] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 5716.061284] Call Trace: [ 5716.061290] [ 5716.061295] ? show_regs.cold.9+0x22/0x2f [ 5716.061307] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5716.061313] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5716.061316] ? watchdog+0x40/0x40 [ 5716.061321] ? hrtimer_interrupt+0x128/0x2c0 [ 5716.061325] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5716.061333] ? apic_timer_interrupt+0xf/0x20 [ 5716.061337] [ 5716.061340] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5716.061346] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5716.061351] do_raw_spin_lock+0x177/0x190 [ 5716.061356] _raw_spin_lock+0x21/0x30 [ 5716.061362] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5716.061658] ? do_raw_spin_unlock+0x75/0x190 [ 5716.061664] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5716.061912] ? do_wait_intr+0xf0/0xf0 [ 5716.061919] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5716.062122] mdc_reint+0x2d/0x110 [mdc] [ 5716.062159] mdc_unlink+0x658/0x870 [mdc] [ 5716.062185] lmv_unlink+0x34d/0xbd0 [lmv] [ 5716.062217] ll_unlink+0x24d/0x7f0 [lustre] [ 5716.062346] vfs_unlink+0x202/0x330 [ 5716.062354] do_unlinkat+0x2fa/0x460 [ 5716.062359] __x64_sys_unlink+0x37/0x50 [ 5716.062362] do_syscall_64+0xc1/0x440 [ 5716.062369] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5716.062375] RIP: 0033:0x7f151bdee10b [ 5716.062383] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5716.062386] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5716.062390] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 5716.062393] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 5716.062395] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5716.062398] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5716.062400] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5716.493610] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5716.550490] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5716.562822] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5716.572218] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5716.575920] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5716.594085] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5716.601924] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5716.607218] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 5716.616229] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5716.627988] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 5716.632350] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 5716.645509] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 5716.653749] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5716.659310] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 5716.665960] Call Trace: [ 5716.669321] [ 5716.674742] ? show_regs.cold.9+0x22/0x2f [ 5716.681898] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5716.684215] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5716.692147] ? watchdog+0x40/0x40 [ 5716.695273] ? hrtimer_interrupt+0x128/0x2c0 [ 5716.703966] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5716.705550] ? apic_timer_interrupt+0xf/0x20 [ 5716.713500] [ 5716.716763] ? native_safe_halt+0x2e/0x50 [ 5716.727580] kvm_wait+0xcd/0xe0 [ 5716.732160] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5716.735077] do_raw_spin_lock+0x177/0x190 [ 5716.738854] _raw_spin_lock+0x21/0x30 [ 5716.741559] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5716.754513] ? do_raw_spin_unlock+0x75/0x190 [ 5716.758206] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5716.762291] ? do_wait_intr+0xf0/0xf0 [ 5716.764052] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5716.772960] mdc_close+0x53b/0xd70 [mdc] [ 5716.778487] lmv_close+0x18d/0x490 [lmv] [ 5716.782078] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5716.788989] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5716.797257] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5716.802715] ll_md_close+0x274/0x800 [lustre] [ 5716.807424] ll_file_release+0x128/0xc50 [lustre] [ 5716.811549] __fput+0x10d/0x440 [ 5716.813589] ____fput+0x12/0x20 [ 5716.814721] task_work_run+0xc6/0x110 [ 5716.818834] exit_to_usermode_loop+0x1dd/0x1f0 [ 5716.822020] do_syscall_64+0x3d6/0x440 [ 5716.826416] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5716.831431] RIP: 0033:0x7f151bee3cd8 [ 5716.833484] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5716.852037] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5716.854486] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 5716.871523] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 5716.880420] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5716.887524] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5716.895914] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5720.553200] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3789s! [ 5720.562633] Showing busy workqueues and worker pools: [ 5720.567188] workqueue events: flags=0x0 [ 5720.569903] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5720.572404] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 5720.589656] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5720.592152] pending: kfree_rcu_monitor [ 5720.600144] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5720.612201] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 5720.621290] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5720.628438] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5720.635329] workqueue events_long: flags=0x0 [ 5720.645951] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5720.654499] pending: lu_objects_flush [obdclass] [ 5720.663661] workqueue events_unbound: flags=0x2 [ 5720.672372] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5720.676594] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5720.682074] workqueue events_power_efficient: flags=0x80 [ 5720.689301] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5720.691577] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 5720.703010] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5720.708295] pending: neigh_periodic_work [ 5720.710025] workqueue mm_percpu_wq: flags=0x8 [ 5720.713825] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5720.719241] pending: vmstat_update, lru_add_drain_per_cpu [ 5720.721787] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5720.730101] pending: vmstat_update [ 5720.733096] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5720.735568] pending: lru_add_drain_per_cpu [ 5720.739446] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5720.749964] pending: lru_add_drain_per_cpu BAR(48) [ 5720.755875] workqueue writeback: flags=0x4a [ 5720.758031] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5720.761356] pending: wb_workfn [ 5720.762689] workqueue xprtiod: flags=0xa [ 5720.766353] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5720.771703] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5720.782048] workqueue ipv6_addrconf: flags=0x40008 [ 5720.794421] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5720.805932] pending: addrconf_verify_work [ 5720.811223] workqueue knbd0-recv: flags=0x1a [ 5720.814236] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5720.818136] in-flight: 658:recv_work [ 5720.820150] workqueue ptlrpc_pinger: flags=0x2 [ 5720.825168] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5720.830828] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5720.841439] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3790s workers=4 idle: 32984 37 8 [ 5720.845536] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5705s workers=2 idle: 661 [ 5724.056307] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5724.059242] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5724.060707] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 5724.136292] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5724.136308] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5724.136314] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5724.136590] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5724.136599] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5724.136603] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5724.136607] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5724.136609] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 5724.136611] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 5724.136613] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 5724.136616] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 5724.136619] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 5724.136622] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5724.136624] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5724.136632] Call Trace: [ 5724.136638] [ 5724.136643] ? show_regs.cold.9+0x22/0x2f [ 5724.136653] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5724.136659] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5724.136674] ? watchdog+0x40/0x40 [ 5724.136679] ? hrtimer_interrupt+0x128/0x2c0 [ 5724.136682] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5724.136690] ? apic_timer_interrupt+0xf/0x20 [ 5724.136695] [ 5724.136698] ? native_safe_halt+0x2e/0x50 [ 5724.136702] kvm_wait+0xcd/0xe0 [ 5724.136710] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5724.136719] do_raw_spin_lock+0x177/0x190 [ 5724.136722] _raw_spin_lock+0x21/0x30 [ 5724.136727] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5724.136953] process_one_work+0x2c8/0x700 [ 5724.136960] worker_thread+0x296/0x6e0 [ 5724.136965] ? rescuer_thread+0x570/0x570 [ 5724.136969] kthread+0x1d1/0x200 [ 5724.136973] ? set_kthread_struct+0x70/0x70 [ 5724.136977] ret_from_fork+0x1f/0x30 [ 5724.474352] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5724.543093] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5724.557480] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5724.577309] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5724.604023] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5724.646245] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5724.661448] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 5724.676206] RDX: 00000000000008f7 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 5724.685183] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 5724.696188] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 5724.699948] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5724.710563] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 5724.727359] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5724.740512] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5724.759224] Call Trace: [ 5724.761795] [ 5724.767392] ? show_regs.cold.9+0x22/0x2f [ 5724.772108] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5724.778648] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5724.780332] ? watchdog+0x40/0x40 [ 5724.781559] ? hrtimer_interrupt+0x128/0x2c0 [ 5724.801241] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5724.818412] ? apic_timer_interrupt+0xf/0x20 [ 5724.829298] [ 5724.837103] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5724.853308] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5724.862000] do_raw_spin_lock+0x177/0x190 [ 5724.872187] _raw_spin_lock+0x21/0x30 [ 5724.873522] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5724.886291] obd_mod_free+0x46/0x140 [ptlrpc] [ 5724.898048] mdc_commit_open+0x91/0xb0 [mdc] [ 5724.901768] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5724.903650] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5724.926406] after_reply+0xe2c/0x1780 [ptlrpc] [ 5724.934180] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5724.941502] ? __next_timer_interrupt+0x160/0x160 [ 5724.947390] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5724.955302] ? do_wait_intr+0xf0/0xf0 [ 5724.961444] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5724.966579] kthread+0x1d1/0x200 [ 5724.978395] ? set_kthread_struct+0x70/0x70 [ 5724.984149] ret_from_fork+0x1f/0x30 [ 5744.059172] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 5744.068007] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5744.145744] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5744.158394] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5744.165255] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5744.167609] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5744.187961] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5744.200821] RAX: 0000000000007e8c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5744.209926] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 5744.222700] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5744.228055] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 5744.235793] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 5744.240877] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 5744.246130] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5744.249597] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 5744.253578] Call Trace: [ 5744.254961] [ 5744.256038] ? show_regs.cold.9+0x22/0x2f [ 5744.259684] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5744.263826] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5744.266601] ? watchdog+0x40/0x40 [ 5744.268717] ? hrtimer_interrupt+0x128/0x2c0 [ 5744.271154] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5744.273962] ? apic_timer_interrupt+0xf/0x20 [ 5744.276676] [ 5744.278134] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5744.280761] do_raw_spin_lock+0x177/0x190 [ 5744.282977] _raw_spin_lock+0x21/0x30 [ 5744.284800] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5744.287754] ? do_raw_spin_unlock+0x75/0x190 [ 5744.289965] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5744.295375] ? do_wait_intr+0xf0/0xf0 [ 5744.299195] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5744.301228] mdc_close+0x53b/0xd70 [mdc] [ 5744.319661] lmv_close+0x18d/0x490 [lmv] [ 5744.321168] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5744.333487] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5744.341530] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5744.353749] ll_md_close+0x274/0x800 [lustre] [ 5744.360913] ll_file_release+0x128/0xc50 [lustre] [ 5744.371815] __fput+0x10d/0x440 [ 5744.375926] ____fput+0x12/0x20 [ 5744.388060] task_work_run+0xc6/0x110 [ 5744.389490] exit_to_usermode_loop+0x1dd/0x1f0 [ 5744.391058] do_syscall_64+0x3d6/0x440 [ 5744.408190] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5744.409908] RIP: 0033:0x7f151bee3cd8 [ 5744.422637] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5744.452104] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5744.468941] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 5744.476952] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 5744.492996] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5744.501253] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5744.512690] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5748.053098] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 5748.055769] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5748.081467] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5748.087188] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5748.090605] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5748.092930] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5748.109554] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5748.113668] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5748.118637] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 5748.124042] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 5748.127467] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 5748.130773] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 5748.134057] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 5748.137644] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5748.140199] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 5748.143529] Call Trace: [ 5748.144840] [ 5748.145825] ? show_regs.cold.9+0x22/0x2f [ 5748.147604] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5748.149786] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5748.152139] ? watchdog+0x40/0x40 [ 5748.153535] ? hrtimer_interrupt+0x128/0x2c0 [ 5748.155728] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5748.157979] ? apic_timer_interrupt+0xf/0x20 [ 5748.160150] [ 5748.161572] ? native_safe_halt+0x2e/0x50 [ 5748.163590] kvm_wait+0xcd/0xe0 [ 5748.164921] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5748.167800] do_raw_spin_lock+0x177/0x190 [ 5748.169778] _raw_spin_lock+0x21/0x30 [ 5748.171729] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5748.174558] ? do_raw_spin_unlock+0x75/0x190 [ 5748.176267] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5748.180172] ? do_wait_intr+0xf0/0xf0 [ 5748.184137] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5748.191100] mdc_reint+0x2d/0x110 [mdc] [ 5748.196588] mdc_unlink+0x658/0x870 [mdc] [ 5748.200059] lmv_unlink+0x34d/0xbd0 [lmv] [ 5748.202719] ll_unlink+0x24d/0x7f0 [lustre] [ 5748.210899] vfs_unlink+0x202/0x330 [ 5748.214098] do_unlinkat+0x2fa/0x460 [ 5748.217722] __x64_sys_unlink+0x37/0x50 [ 5748.221754] do_syscall_64+0xc1/0x440 [ 5748.224729] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5748.227924] RIP: 0033:0x7f151bdee10b [ 5748.229713] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5748.244650] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5748.249677] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 5748.253780] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 5748.261387] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5748.265408] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5748.271753] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5751.265363] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3820s! [ 5751.271698] Showing busy workqueues and worker pools: [ 5751.274561] workqueue events: flags=0x0 [ 5751.276872] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5751.280042] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 5751.285557] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5751.288107] pending: kfree_rcu_monitor [ 5751.292855] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5751.297941] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 5751.304078] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5751.307952] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5751.312460] workqueue events_long: flags=0x0 [ 5751.315609] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5751.318091] pending: lu_objects_flush [obdclass] [ 5751.320449] workqueue events_unbound: flags=0x2 [ 5751.322040] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5751.324543] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5751.329694] workqueue events_power_efficient: flags=0x80 [ 5751.331830] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5751.335081] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 5751.338069] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5751.341661] pending: neigh_periodic_work [ 5751.343499] workqueue mm_percpu_wq: flags=0x8 [ 5751.345444] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5751.348668] pending: vmstat_update, lru_add_drain_per_cpu [ 5751.351929] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5751.356244] pending: vmstat_update [ 5751.360457] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5751.363844] pending: lru_add_drain_per_cpu [ 5751.367538] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5751.370583] pending: lru_add_drain_per_cpu BAR(48) [ 5751.375946] workqueue writeback: flags=0x4a [ 5751.377496] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5751.387723] pending: wb_workfn [ 5751.390442] workqueue xprtiod: flags=0xa [ 5751.391776] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5751.397993] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5751.408541] workqueue ipv6_addrconf: flags=0x40008 [ 5751.412911] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5751.419322] pending: addrconf_verify_work [ 5751.421471] workqueue knbd0-recv: flags=0x1a [ 5751.435166] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5751.443015] in-flight: 658:recv_work [ 5751.449208] workqueue ptlrpc_pinger: flags=0x2 [ 5751.452458] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5751.462623] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5751.474130] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3820s workers=4 idle: 32984 37 8 [ 5751.480170] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5736s workers=2 idle: 661 [ 5752.055020] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 5752.056018] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 5752.059971] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5752.060174] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5752.060182] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5752.060185] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5752.067595] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5752.067678] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5752.067685] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5752.067691] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5752.094799] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5752.094815] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5752.094821] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 5752.094824] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 5752.094826] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 5752.094828] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 5752.094831] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5752.094833] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 5752.094837] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5752.094839] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5752.094848] Call Trace: [ 5752.094854] [ 5752.094860] ? show_regs.cold.9+0x22/0x2f [ 5752.094872] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5752.094879] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5752.094883] ? watchdog+0x40/0x40 [ 5752.094888] ? hrtimer_interrupt+0x128/0x2c0 [ 5752.094892] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5752.094902] ? apic_timer_interrupt+0xf/0x20 [ 5752.094906] [ 5752.094909] ? native_safe_halt+0x2e/0x50 [ 5752.094913] kvm_wait+0xcd/0xe0 [ 5752.094921] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5752.094927] do_raw_spin_lock+0x177/0x190 [ 5752.094932] _raw_spin_lock+0x21/0x30 [ 5752.094938] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5752.095428] obd_mod_free+0x46/0x140 [ptlrpc] [ 5752.095773] mdc_commit_open+0x91/0xb0 [mdc] [ 5752.095812] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5752.096003] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5752.096464] after_reply+0xe2c/0x1780 [ptlrpc] [ 5752.096803] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5752.096997] ? __next_timer_interrupt+0x160/0x160 [ 5752.097007] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5752.097432] ? do_wait_intr+0xf0/0xf0 [ 5752.097442] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5752.098013] kthread+0x1d1/0x200 [ 5752.098018] ? set_kthread_struct+0x70/0x70 [ 5752.098023] ret_from_fork+0x1f/0x30 [ 5752.270666] [ 5752.271357] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5752.273442] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5752.282476] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5752.286220] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5752.291374] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 5752.298040] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 5752.301822] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 5752.306322] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 5752.309748] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 5752.314132] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5752.316574] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5752.320463] Call Trace: [ 5752.321689] [ 5752.322725] ? show_regs.cold.9+0x22/0x2f [ 5752.324746] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5752.327339] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5752.330260] ? watchdog+0x40/0x40 [ 5752.331782] ? hrtimer_interrupt+0x128/0x2c0 [ 5752.333961] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5752.336796] ? apic_timer_interrupt+0xf/0x20 [ 5752.339123] [ 5752.340217] ? native_safe_halt+0x2e/0x50 [ 5752.342067] kvm_wait+0xcd/0xe0 [ 5752.343623] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5752.346585] do_raw_spin_lock+0x177/0x190 [ 5752.349228] _raw_spin_lock+0x21/0x30 [ 5752.351244] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5752.353764] process_one_work+0x2c8/0x700 [ 5752.355576] worker_thread+0x296/0x6e0 [ 5752.357153] ? rescuer_thread+0x570/0x570 [ 5752.358870] kthread+0x1d1/0x200 [ 5752.360861] ? set_kthread_struct+0x70/0x70 [ 5752.362758] ret_from_fork+0x1f/0x30 [ 5764.719040] rcu: INFO: rcu_sched self-detected stall on CPU [ 5764.721442] rcu: 1-...!: (2225901 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 5764.730057] (t=3840072 jiffies g=249657 q=3477) [ 5764.732120] rcu: rcu_sched kthread starved for 3835024 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 5764.736615] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5764.742780] rcu: RCU grace-period kthread stack dump: [ 5764.746312] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5764.758278] Call Trace: [ 5764.759296] __schedule+0x351/0xcb0 [ 5764.761046] schedule+0xc0/0x180 [ 5764.768944] schedule_timeout+0xb4/0x190 [ 5764.770537] ? do_raw_spin_unlock+0x75/0x190 [ 5764.772117] ? __next_timer_interrupt+0x160/0x160 [ 5764.780939] ? prepare_to_swait_event+0xf3/0x140 [ 5764.789919] rcu_gp_kthread+0x8dd/0x15b0 [ 5764.797738] ? force_qs_rnp+0x2c0/0x2c0 [ 5764.805879] kthread+0x1d1/0x200 [ 5764.807303] ? set_kthread_struct+0x70/0x70 [ 5764.810578] ret_from_fork+0x1f/0x30 [ 5764.812763] rcu: Stack dump where RCU GP kthread last ran: [ 5764.818725] Sending NMI from CPU 1 to CPUs 0: [ 5764.825139] NMI backtrace for cpu 0 [ 5764.827144] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5764.827355] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5764.827357] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5764.827359] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5764.827361] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 [ 5764.827364] RAX: 0000000000003d0b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5764.827365] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 5764.827367] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 5764.827368] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 5764.827370] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 5764.827371] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 5764.827372] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5764.827373] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 5764.827374] Call Trace: [ 5764.827376] [ 5764.827376] ? show_regs.cold.9+0x22/0x2f [ 5764.827377] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 5764.827379] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 5764.827380] ? nmi_handle+0xbd/0x210 [ 5764.827381] ? default_do_nmi+0x4b/0x1b0 [ 5764.827382] ? do_nmi+0x280/0x380 [ 5764.827382] ? end_repeat_nmi+0x16/0x1f [ 5764.827384] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5764.827385] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5764.827386] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5764.827387] [ 5764.827388] do_raw_spin_lock+0x177/0x190 [ 5764.827388] _raw_spin_lock+0x21/0x30 [ 5764.827389] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5764.827390] ? do_raw_spin_unlock+0x75/0x190 [ 5764.827392] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5764.827393] ? do_wait_intr+0xf0/0xf0 [ 5764.827394] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5764.827395] mdc_reint+0x2d/0x110 [mdc] [ 5764.827396] mdc_unlink+0x658/0x870 [mdc] [ 5764.827397] lmv_unlink+0x34d/0xbd0 [lmv] [ 5764.827398] ll_unlink+0x24d/0x7f0 [lustre] [ 5764.827399] vfs_unlink+0x202/0x330 [ 5764.827400] do_unlinkat+0x2fa/0x460 [ 5764.827400] __x64_sys_unlink+0x37/0x50 [ 5764.827401] do_syscall_64+0xc1/0x440 [ 5764.827402] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5764.827403] RIP: 0033:0x7f151bdee10b [ 5764.827405] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5764.827474] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5764.827478] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 5764.827479] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 5764.827480] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5764.827482] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5764.827483] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5764.829857] NMI backtrace for cpu 1 [ 5765.080519] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5765.087114] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5765.098526] Call Trace: [ 5765.099892] [ 5765.100916] ? dump_stack+0xbb/0x10e [ 5765.105743] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5765.115944] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5765.122920] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5765.126483] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5765.130538] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5765.134953] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 5765.138248] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 5765.142484] ? update_process_times+0x6d/0xc0 [ 5765.145779] ? tick_sched_handle.isra.18+0x27/0x90 [ 5765.147733] ? tick_sched_timer+0x53/0xb0 [ 5765.149571] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5765.155046] ? tick_sched_handle.isra.18+0x90/0x90 [ 5765.163537] ? hrtimer_interrupt+0x128/0x2c0 [ 5765.169246] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5765.171585] ? apic_timer_interrupt+0xf/0x20 [ 5765.177843] [ 5765.179250] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5765.182548] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5765.185347] ? do_raw_spin_lock+0x177/0x190 [ 5765.187347] ? _raw_spin_lock+0x21/0x30 [ 5765.189108] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5765.192037] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5765.195565] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5765.198300] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5765.204548] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5765.207107] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5765.209495] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5765.212323] ? __next_timer_interrupt+0x160/0x160 [ 5765.214543] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5765.216610] ? do_wait_intr+0xf0/0xf0 [ 5765.218003] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5765.220409] ? kthread+0x1d1/0x200 [ 5765.221760] ? set_kthread_struct+0x70/0x70 [ 5765.223725] ? ret_from_fork+0x1f/0x30 [ 5772.057028] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 5772.060906] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5772.086394] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5772.098702] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5772.104480] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5772.111415] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5772.133394] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5772.147486] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5772.151771] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 5772.158427] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5772.164001] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 5772.173500] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 5772.176873] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 5772.187520] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5772.190451] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 5772.199020] Call Trace: [ 5772.199892] [ 5772.200639] ? show_regs.cold.9+0x22/0x2f [ 5772.202277] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5772.206081] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5772.210744] ? watchdog+0x40/0x40 [ 5772.216521] ? hrtimer_interrupt+0x128/0x2c0 [ 5772.219904] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5772.222781] ? apic_timer_interrupt+0xf/0x20 [ 5772.224764] [ 5772.225770] ? native_safe_halt+0x2e/0x50 [ 5772.228708] kvm_wait+0xcd/0xe0 [ 5772.230395] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5772.232853] do_raw_spin_lock+0x177/0x190 [ 5772.234773] _raw_spin_lock+0x21/0x30 [ 5772.236363] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5772.239189] ? do_raw_spin_unlock+0x75/0x190 [ 5772.242621] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5772.245486] ? do_wait_intr+0xf0/0xf0 [ 5772.250552] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5772.252418] mdc_close+0x53b/0xd70 [mdc] [ 5772.257643] lmv_close+0x18d/0x490 [lmv] [ 5772.261480] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5772.265367] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5772.269217] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5772.280086] ll_md_close+0x274/0x800 [lustre] [ 5772.281479] ll_file_release+0x128/0xc50 [lustre] [ 5772.282671] __fput+0x10d/0x440 [ 5772.287022] ____fput+0x12/0x20 [ 5772.288495] task_work_run+0xc6/0x110 [ 5772.290286] exit_to_usermode_loop+0x1dd/0x1f0 [ 5772.294336] do_syscall_64+0x3d6/0x440 [ 5772.296039] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5772.297751] RIP: 0033:0x7f151bee3cd8 [ 5772.305721] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5772.325644] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5772.332695] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 5772.337145] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 5772.343102] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5772.346299] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5772.350455] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5780.058907] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 5780.067507] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5780.117519] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5780.127707] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5780.139681] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5780.146806] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5780.150630] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5780.173662] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 5780.181372] RAX: 0000000000006d07 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5780.186556] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 5780.194389] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 5780.208130] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 5780.222339] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 5780.231852] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 5780.238139] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5780.248840] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5780.256186] Call Trace: [ 5780.258575] [ 5780.260801] ? show_regs.cold.9+0x22/0x2f [ 5780.262760] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5780.269167] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5780.274576] ? watchdog+0x40/0x40 [ 5780.276585] ? hrtimer_interrupt+0x128/0x2c0 [ 5780.280832] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5780.287770] ? apic_timer_interrupt+0xf/0x20 [ 5780.291587] [ 5780.292563] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5780.299227] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5780.306664] do_raw_spin_lock+0x177/0x190 [ 5780.307929] _raw_spin_lock+0x21/0x30 [ 5780.309393] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5780.316403] process_one_work+0x2c8/0x700 [ 5780.318589] worker_thread+0x296/0x6e0 [ 5780.322636] ? rescuer_thread+0x570/0x570 [ 5780.325128] kthread+0x1d1/0x200 [ 5780.334330] ? set_kthread_struct+0x70/0x70 [ 5780.338586] ret_from_fork+0x1f/0x30 [ 5781.988236] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3851s! [ 5781.994949] Showing busy workqueues and worker pools: [ 5781.997734] workqueue events: flags=0x0 [ 5781.998829] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5782.001097] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 5782.004268] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5782.006195] pending: kfree_rcu_monitor [ 5782.007326] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5782.009084] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 5782.012186] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5782.014533] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5782.017107] workqueue events_long: flags=0x0 [ 5782.018419] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5782.020520] pending: lu_objects_flush [obdclass] [ 5782.021962] workqueue events_unbound: flags=0x2 [ 5782.023277] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5782.025168] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5782.028844] workqueue events_power_efficient: flags=0x80 [ 5782.031483] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5782.033896] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 5782.043611] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5782.049367] pending: neigh_periodic_work [ 5782.054013] workqueue mm_percpu_wq: flags=0x8 [ 5782.057274] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5782.062497] pending: vmstat_update, lru_add_drain_per_cpu [ 5782.064545] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5782.069775] pending: vmstat_update [ 5782.071836] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5782.079918] pending: lru_add_drain_per_cpu [ 5782.082561] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5782.089349] pending: lru_add_drain_per_cpu BAR(48) [ 5782.092228] workqueue writeback: flags=0x4a [ 5782.094649] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5782.098422] pending: wb_workfn [ 5782.100156] workqueue xprtiod: flags=0xa [ 5782.106144] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5782.110355] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5782.113778] workqueue ipv6_addrconf: flags=0x40008 [ 5782.116397] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5782.118854] pending: addrconf_verify_work [ 5782.120558] workqueue knbd0-recv: flags=0x1a [ 5782.122316] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5782.125073] in-flight: 658:recv_work [ 5782.127055] workqueue ptlrpc_pinger: flags=0x2 [ 5782.128780] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5782.131468] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5782.134335] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3851s workers=4 idle: 32984 37 8 [ 5782.137781] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5766s workers=2 idle: 661 [ 5792.055976] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 5792.065150] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5792.100325] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5792.106293] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5792.111856] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5792.115431] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5792.125945] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5792.129311] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 5792.132636] RDX: 000000000000694e RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 5792.134914] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 5792.136634] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 5792.139164] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5792.141155] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 5792.143400] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5792.145557] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5792.148028] Call Trace: [ 5792.148700] [ 5792.149254] ? show_regs.cold.9+0x22/0x2f [ 5792.151341] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5792.153773] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5792.156011] ? watchdog+0x40/0x40 [ 5792.157396] ? hrtimer_interrupt+0x128/0x2c0 [ 5792.159604] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5792.161741] ? apic_timer_interrupt+0xf/0x20 [ 5792.165265] [ 5792.166240] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5792.170354] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5792.173410] do_raw_spin_lock+0x177/0x190 [ 5792.175609] _raw_spin_lock+0x21/0x30 [ 5792.177469] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5792.180335] obd_mod_free+0x46/0x140 [ptlrpc] [ 5792.181784] mdc_commit_open+0x91/0xb0 [mdc] [ 5792.184264] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5792.187347] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5792.191435] after_reply+0xe2c/0x1780 [ptlrpc] [ 5792.193377] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5792.195726] ? __next_timer_interrupt+0x160/0x160 [ 5792.197501] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5792.200754] ? do_wait_intr+0xf0/0xf0 [ 5792.202589] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5792.207503] kthread+0x1d1/0x200 [ 5792.209658] ? set_kthread_struct+0x70/0x70 [ 5792.211717] ret_from_fork+0x1f/0x30 [ 5800.057084] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 5800.059909] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5800.080011] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5800.085599] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5800.089489] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5800.091781] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5800.099621] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5800.103369] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5800.106301] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 5800.108872] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5800.111444] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 5800.113558] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 5800.116420] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 5800.119436] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5800.121548] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 5800.124259] Call Trace: [ 5800.125259] [ 5800.126088] ? show_regs.cold.9+0x22/0x2f [ 5800.127405] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5800.129314] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5800.131157] ? watchdog+0x40/0x40 [ 5800.132354] ? hrtimer_interrupt+0x128/0x2c0 [ 5800.133954] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5800.135880] ? apic_timer_interrupt+0xf/0x20 [ 5800.137484] [ 5800.138356] ? native_safe_halt+0x2e/0x50 [ 5800.139906] kvm_wait+0xcd/0xe0 [ 5800.141126] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5800.143086] do_raw_spin_lock+0x177/0x190 [ 5800.144302] _raw_spin_lock+0x21/0x30 [ 5800.145564] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5800.147537] ? do_raw_spin_unlock+0x75/0x190 [ 5800.148990] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5800.151053] ? do_wait_intr+0xf0/0xf0 [ 5800.152359] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5800.154316] mdc_close+0x53b/0xd70 [mdc] [ 5800.155738] lmv_close+0x18d/0x490 [lmv] [ 5800.157121] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5800.158973] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5800.160885] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5800.162548] ll_md_close+0x274/0x800 [lustre] [ 5800.163954] ll_file_release+0x128/0xc50 [lustre] [ 5800.165471] __fput+0x10d/0x440 [ 5800.166534] ____fput+0x12/0x20 [ 5800.167509] task_work_run+0xc6/0x110 [ 5800.168594] exit_to_usermode_loop+0x1dd/0x1f0 [ 5800.169891] do_syscall_64+0x3d6/0x440 [ 5800.171143] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5800.172962] RIP: 0033:0x7f151bee3cd8 [ 5800.174360] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5800.181530] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5800.184848] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 5800.187536] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 5800.190249] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5800.192749] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5800.195323] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5808.052125] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 5808.056285] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5808.059729] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 5808.090441] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5808.090457] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5808.090460] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5808.090473] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5808.090477] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5808.090483] RAX: 0000000000007d83 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5808.090485] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 5808.090487] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 5808.090489] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 5808.090491] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 5808.090494] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 5808.090498] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5808.090500] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 5808.090510] Call Trace: [ 5808.090515] [ 5808.090521] ? show_regs.cold.9+0x22/0x2f [ 5808.090532] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5808.090538] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5808.090542] ? watchdog+0x40/0x40 [ 5808.090547] ? hrtimer_interrupt+0x128/0x2c0 [ 5808.090551] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5808.090561] ? apic_timer_interrupt+0xf/0x20 [ 5808.090565] [ 5808.090568] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5808.090574] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5808.090579] do_raw_spin_lock+0x177/0x190 [ 5808.090584] _raw_spin_lock+0x21/0x30 [ 5808.090591] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5808.090872] ? do_raw_spin_unlock+0x75/0x190 [ 5808.090877] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5808.091082] ? do_wait_intr+0xf0/0xf0 [ 5808.091089] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5808.091295] mdc_reint+0x2d/0x110 [mdc] [ 5808.091330] mdc_unlink+0x658/0x870 [mdc] [ 5808.091355] lmv_unlink+0x34d/0xbd0 [lmv] [ 5808.091385] ll_unlink+0x24d/0x7f0 [lustre] [ 5808.091500] vfs_unlink+0x202/0x330 [ 5808.091509] do_unlinkat+0x2fa/0x460 [ 5808.091513] __x64_sys_unlink+0x37/0x50 [ 5808.091516] do_syscall_64+0xc1/0x440 [ 5808.091524] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5808.091529] RIP: 0033:0x7f151bdee10b [ 5808.091535] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5808.091539] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5808.091543] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 5808.091546] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 5808.091548] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5808.091550] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5808.091552] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5808.329425] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5808.363629] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5808.371243] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5808.383974] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5808.387642] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5808.393392] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5808.408434] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5808.414974] RAX: 0000000000007c2e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5808.418084] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 5808.420634] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 5808.423246] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 5808.426630] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 5808.429675] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 5808.431980] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5808.434353] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5808.437101] Call Trace: [ 5808.438128] [ 5808.438677] ? show_regs.cold.9+0x22/0x2f [ 5808.442414] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5808.447279] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5808.452026] ? watchdog+0x40/0x40 [ 5808.453181] ? hrtimer_interrupt+0x128/0x2c0 [ 5808.454705] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5808.456522] ? apic_timer_interrupt+0xf/0x20 [ 5808.457908] [ 5808.458762] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5808.460929] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5808.463135] do_raw_spin_lock+0x177/0x190 [ 5808.464647] _raw_spin_lock+0x21/0x30 [ 5808.466381] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5808.468881] process_one_work+0x2c8/0x700 [ 5808.470628] worker_thread+0x296/0x6e0 [ 5808.472545] ? rescuer_thread+0x570/0x570 [ 5808.474080] kthread+0x1d1/0x200 [ 5808.475561] ? set_kthread_struct+0x70/0x70 [ 5808.485348] ret_from_fork+0x1f/0x30 [ 5812.196592] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3881s! [ 5812.204192] Showing busy workqueues and worker pools: [ 5812.207955] workqueue events: flags=0x0 [ 5812.210119] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5812.215615] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 5812.223582] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5812.226273] pending: kfree_rcu_monitor [ 5812.228296] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5812.231450] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 5812.235840] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5812.238872] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5812.242382] workqueue events_long: flags=0x0 [ 5812.244345] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5812.247322] pending: lu_objects_flush [obdclass] [ 5812.249751] workqueue events_unbound: flags=0x2 [ 5812.251757] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5812.254766] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5812.266656] workqueue events_power_efficient: flags=0x80 [ 5812.271302] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5812.275542] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 5812.281563] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5812.286630] pending: neigh_periodic_work [ 5812.291581] workqueue mm_percpu_wq: flags=0x8 [ 5812.297412] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5812.305827] pending: vmstat_update, lru_add_drain_per_cpu [ 5812.312590] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5812.317683] pending: vmstat_update [ 5812.327362] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5812.337621] pending: lru_add_drain_per_cpu [ 5812.346781] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5812.350054] pending: lru_add_drain_per_cpu BAR(48) [ 5812.355398] workqueue writeback: flags=0x4a [ 5812.358975] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5812.365786] pending: wb_workfn [ 5812.367927] workqueue xprtiod: flags=0xa [ 5812.370888] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5812.376410] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5812.382017] workqueue ipv6_addrconf: flags=0x40008 [ 5812.391923] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5812.397853] pending: addrconf_verify_work [ 5812.407164] workqueue knbd0-recv: flags=0x1a [ 5812.414415] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5812.419598] in-flight: 658:recv_work [ 5812.425596] workqueue ptlrpc_pinger: flags=0x2 [ 5812.432491] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5812.436843] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5812.444384] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3881s workers=4 idle: 32984 37 8 [ 5812.452301] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5797s workers=2 idle: 661 [ 5820.057977] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 5820.064149] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5820.097382] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5820.105537] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5820.110841] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5820.114355] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5820.126894] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5820.131117] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 5820.133887] RDX: 0000000000007ce4 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 5820.136047] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 5820.143301] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 5820.153190] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5820.159257] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 5820.163183] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5820.167223] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5820.171012] Call Trace: [ 5820.172087] [ 5820.173223] ? show_regs.cold.9+0x22/0x2f [ 5820.175657] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5820.179644] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5820.182934] ? watchdog+0x40/0x40 [ 5820.183806] ? hrtimer_interrupt+0x128/0x2c0 [ 5820.184861] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5820.186882] ? apic_timer_interrupt+0xf/0x20 [ 5820.188816] [ 5820.189944] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5820.196095] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5820.200991] do_raw_spin_lock+0x177/0x190 [ 5820.205566] _raw_spin_lock+0x21/0x30 [ 5820.212845] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5820.217495] obd_mod_free+0x46/0x140 [ptlrpc] [ 5820.219622] mdc_commit_open+0x91/0xb0 [mdc] [ 5820.221107] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5820.228864] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5820.235063] after_reply+0xe2c/0x1780 [ptlrpc] [ 5820.243092] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5820.248834] ? __next_timer_interrupt+0x160/0x160 [ 5820.251449] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5820.255984] ? do_wait_intr+0xf0/0xf0 [ 5820.258475] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5820.260527] kthread+0x1d1/0x200 [ 5820.262807] ? set_kthread_struct+0x70/0x70 [ 5820.267112] ret_from_fork+0x1f/0x30 [ 5828.059444] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 5828.063621] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5828.105831] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5828.111394] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5828.117789] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5828.120952] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5828.130426] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5828.133820] RAX: 0000000000004d96 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5828.142166] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 5828.146936] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5828.149855] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 5828.154065] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 5828.157156] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 5828.161232] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5828.165056] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 5828.169382] Call Trace: [ 5828.170256] [ 5828.171100] ? show_regs.cold.9+0x22/0x2f [ 5828.172734] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5828.175545] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5828.177922] ? watchdog+0x40/0x40 [ 5828.180008] ? hrtimer_interrupt+0x128/0x2c0 [ 5828.182123] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5828.184386] ? apic_timer_interrupt+0xf/0x20 [ 5828.186204] [ 5828.187230] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5828.190064] do_raw_spin_lock+0x177/0x190 [ 5828.191698] _raw_spin_lock+0x21/0x30 [ 5828.193091] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5828.196101] ? do_raw_spin_unlock+0x75/0x190 [ 5828.198255] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5828.201087] ? do_wait_intr+0xf0/0xf0 [ 5828.203965] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5828.206759] mdc_close+0x53b/0xd70 [mdc] [ 5828.208925] lmv_close+0x18d/0x490 [lmv] [ 5828.210972] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5828.213928] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5828.217457] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5828.219903] ll_md_close+0x274/0x800 [lustre] [ 5828.222179] ll_file_release+0x128/0xc50 [lustre] [ 5828.224833] __fput+0x10d/0x440 [ 5828.226276] ____fput+0x12/0x20 [ 5828.228047] task_work_run+0xc6/0x110 [ 5828.229974] exit_to_usermode_loop+0x1dd/0x1f0 [ 5828.232109] do_syscall_64+0x3d6/0x440 [ 5828.234003] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5828.236357] RIP: 0033:0x7f151bee3cd8 [ 5828.238161] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5828.247069] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5828.250898] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 5828.254857] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 5828.258449] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5828.261342] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5828.264804] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5836.060333] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 5836.068062] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5836.107144] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5836.117814] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5836.124375] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5836.129315] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5836.134423] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5836.147331] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5836.153310] RAX: 000000000000683e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5836.157683] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 5836.162062] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 5836.167427] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 5836.171867] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 5836.176105] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 5836.181677] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5836.184672] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5836.188738] Call Trace: [ 5836.190210] [ 5836.190815] ? show_regs.cold.9+0x22/0x2f [ 5836.192182] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5836.194749] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5836.196853] ? watchdog+0x40/0x40 [ 5836.199929] ? hrtimer_interrupt+0x128/0x2c0 [ 5836.203415] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5836.205589] ? apic_timer_interrupt+0xf/0x20 [ 5836.208354] [ 5836.209396] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5836.212387] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5836.216890] do_raw_spin_lock+0x177/0x190 [ 5836.218644] _raw_spin_lock+0x21/0x30 [ 5836.221161] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5836.224146] process_one_work+0x2c8/0x700 [ 5836.227162] worker_thread+0x296/0x6e0 [ 5836.229039] ? rescuer_thread+0x570/0x570 [ 5836.231680] kthread+0x1d1/0x200 [ 5836.233061] ? set_kthread_struct+0x70/0x70 [ 5836.237397] ret_from_fork+0x1f/0x30 [ 5840.053052] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 5840.060131] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5840.096118] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5840.112187] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5840.121752] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5840.127929] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5840.149084] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5840.160325] RAX: 0000000000007b18 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5840.173445] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 5840.186968] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 5840.205802] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 5840.222019] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 5840.237240] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 5840.253391] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5840.266832] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 5840.274826] Call Trace: [ 5840.277735] [ 5840.282550] ? show_regs.cold.9+0x22/0x2f [ 5840.285711] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5840.291136] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5840.296280] ? watchdog+0x40/0x40 [ 5840.302851] ? hrtimer_interrupt+0x128/0x2c0 [ 5840.307346] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5840.315553] ? apic_timer_interrupt+0xf/0x20 [ 5840.324725] [ 5840.333930] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5840.345148] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5840.347045] do_raw_spin_lock+0x177/0x190 [ 5840.362560] _raw_spin_lock+0x21/0x30 [ 5840.368126] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5840.373052] ? do_raw_spin_unlock+0x75/0x190 [ 5840.382027] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5840.394019] ? do_wait_intr+0xf0/0xf0 [ 5840.403068] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5840.412225] mdc_reint+0x2d/0x110 [mdc] [ 5840.422099] mdc_unlink+0x658/0x870 [mdc] [ 5840.425268] lmv_unlink+0x34d/0xbd0 [lmv] [ 5840.430949] ll_unlink+0x24d/0x7f0 [lustre] [ 5840.432210] vfs_unlink+0x202/0x330 [ 5840.433014] do_unlinkat+0x2fa/0x460 [ 5840.438693] __x64_sys_unlink+0x37/0x50 [ 5840.440096] do_syscall_64+0xc1/0x440 [ 5840.442837] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5840.445783] RIP: 0033:0x7f151bdee10b [ 5840.449239] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5840.463028] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5840.466847] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 5840.469899] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 5840.473116] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5840.476075] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5840.478963] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5842.914619] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3912s! [ 5842.923401] Showing busy workqueues and worker pools: [ 5842.930131] workqueue events: flags=0x0 [ 5842.934174] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5842.939284] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 5842.945847] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5842.949272] pending: kfree_rcu_monitor [ 5842.952607] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5842.961282] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 5842.967806] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5842.972333] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5842.976266] workqueue events_long: flags=0x0 [ 5842.978711] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5842.982916] pending: lu_objects_flush [obdclass] [ 5842.986742] workqueue events_unbound: flags=0x2 [ 5842.990994] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5842.995014] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5843.003323] workqueue events_power_efficient: flags=0x80 [ 5843.007235] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5843.012808] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 5843.016030] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5843.022909] pending: neigh_periodic_work [ 5843.024329] workqueue mm_percpu_wq: flags=0x8 [ 5843.028165] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5843.033011] pending: vmstat_update, lru_add_drain_per_cpu [ 5843.036040] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5843.040341] pending: vmstat_update [ 5843.043153] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5843.046298] pending: lru_add_drain_per_cpu [ 5843.048258] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5843.051739] pending: lru_add_drain_per_cpu BAR(48) [ 5843.054951] workqueue writeback: flags=0x4a [ 5843.058185] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5843.061915] pending: wb_workfn [ 5843.064867] workqueue xprtiod: flags=0xa [ 5843.067588] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5843.070780] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5843.076275] workqueue ipv6_addrconf: flags=0x40008 [ 5843.079759] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5843.084054] pending: addrconf_verify_work [ 5843.086125] workqueue knbd0-recv: flags=0x1a [ 5843.089271] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5843.093236] in-flight: 658:recv_work [ 5843.095923] workqueue ptlrpc_pinger: flags=0x2 [ 5843.098327] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5843.103199] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5843.108194] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3912s workers=4 idle: 32984 37 8 [ 5843.112352] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5827s workers=2 idle: 661 [ 5848.055049] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 5848.057835] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5848.104314] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5848.112681] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5848.119838] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5848.123945] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5848.143440] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5848.153580] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 5848.160893] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 5848.169553] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 5848.183743] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 5848.195657] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5848.204841] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 5848.212471] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5848.215919] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5848.228208] Call Trace: [ 5848.229587] [ 5848.231010] ? show_regs.cold.9+0x22/0x2f [ 5848.232798] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5848.237432] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5848.239796] ? watchdog+0x40/0x40 [ 5848.243019] ? hrtimer_interrupt+0x128/0x2c0 [ 5848.245152] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5848.248267] ? apic_timer_interrupt+0xf/0x20 [ 5848.250442] [ 5848.251535] ? native_safe_halt+0x2e/0x50 [ 5848.257356] kvm_wait+0xcd/0xe0 [ 5848.258549] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5848.264969] do_raw_spin_lock+0x177/0x190 [ 5848.268923] _raw_spin_lock+0x21/0x30 [ 5848.271491] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5848.275503] obd_mod_free+0x46/0x140 [ptlrpc] [ 5848.278276] mdc_commit_open+0x91/0xb0 [mdc] [ 5848.282836] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5848.287616] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5848.291607] after_reply+0xe2c/0x1780 [ptlrpc] [ 5848.295582] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5848.299308] ? __next_timer_interrupt+0x160/0x160 [ 5848.304208] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5848.307392] ? do_wait_intr+0xf0/0xf0 [ 5848.309923] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5848.314201] kthread+0x1d1/0x200 [ 5848.317048] ? set_kthread_struct+0x70/0x70 [ 5848.323389] ret_from_fork+0x1f/0x30 [ 5856.057009] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 5856.064818] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5856.111627] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5856.128047] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5856.134409] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5856.140522] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5856.155050] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5856.159215] RAX: 0000000000005f91 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5856.163450] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 5856.170959] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5856.176119] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 5856.181065] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 5856.185004] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 5856.190247] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5856.195212] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 5856.200029] Call Trace: [ 5856.201115] [ 5856.202098] ? show_regs.cold.9+0x22/0x2f [ 5856.205895] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5856.208961] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5856.211604] ? watchdog+0x40/0x40 [ 5856.213230] ? hrtimer_interrupt+0x128/0x2c0 [ 5856.215139] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5856.217326] ? apic_timer_interrupt+0xf/0x20 [ 5856.219293] [ 5856.220967] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5856.224025] do_raw_spin_lock+0x177/0x190 [ 5856.226409] _raw_spin_lock+0x21/0x30 [ 5856.228892] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5856.231358] ? do_raw_spin_unlock+0x75/0x190 [ 5856.233705] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5856.238457] ? do_wait_intr+0xf0/0xf0 [ 5856.240625] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5856.247920] mdc_close+0x53b/0xd70 [mdc] [ 5856.250839] lmv_close+0x18d/0x490 [lmv] [ 5856.254289] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5856.262215] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5856.271670] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5856.283674] ll_md_close+0x274/0x800 [lustre] [ 5856.292360] ll_file_release+0x128/0xc50 [lustre] [ 5856.296742] __fput+0x10d/0x440 [ 5856.303731] ____fput+0x12/0x20 [ 5856.305854] task_work_run+0xc6/0x110 [ 5856.308936] exit_to_usermode_loop+0x1dd/0x1f0 [ 5856.314965] do_syscall_64+0x3d6/0x440 [ 5856.317925] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5856.322650] RIP: 0033:0x7f151bee3cd8 [ 5856.325138] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5856.350881] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5856.358185] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 5856.365814] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 5856.370094] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5856.374976] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5856.379047] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5864.061168] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 5864.072469] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5864.091984] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5864.097035] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5864.101223] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5864.114437] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5864.115981] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5864.123753] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5864.126983] RAX: 0000000000007011 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5864.129534] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 5864.132669] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 5864.135988] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 5864.139017] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 5864.141959] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 5864.145361] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5864.147779] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5864.151322] Call Trace: [ 5864.153036] [ 5864.157615] ? show_regs.cold.9+0x22/0x2f [ 5864.159804] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5864.162006] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5864.164050] ? watchdog+0x40/0x40 [ 5864.165707] ? hrtimer_interrupt+0x128/0x2c0 [ 5864.169787] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5864.173833] ? apic_timer_interrupt+0xf/0x20 [ 5864.176504] [ 5864.177744] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5864.180546] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5864.182764] do_raw_spin_lock+0x177/0x190 [ 5864.184772] _raw_spin_lock+0x21/0x30 [ 5864.186387] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5864.189353] process_one_work+0x2c8/0x700 [ 5864.191060] worker_thread+0x296/0x6e0 [ 5864.193159] ? rescuer_thread+0x570/0x570 [ 5864.194657] kthread+0x1d1/0x200 [ 5864.196781] ? set_kthread_struct+0x70/0x70 [ 5864.198664] ret_from_fork+0x1f/0x30 [ 5868.052396] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 5868.055659] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5868.074194] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5868.078970] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5868.082767] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5868.085501] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5868.093003] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5868.096084] RAX: 00000000000067c2 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5868.098987] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 5868.101857] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 5868.104765] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 5868.107730] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 5868.111395] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 5868.114126] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5868.116329] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 5868.119199] Call Trace: [ 5868.120165] [ 5868.120965] ? show_regs.cold.9+0x22/0x2f [ 5868.122661] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5868.124546] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5868.126566] ? watchdog+0x40/0x40 [ 5868.127486] ? hrtimer_interrupt+0x128/0x2c0 [ 5868.128699] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5868.130559] ? apic_timer_interrupt+0xf/0x20 [ 5868.132017] [ 5868.132997] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5868.134904] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5868.137097] do_raw_spin_lock+0x177/0x190 [ 5868.138707] _raw_spin_lock+0x21/0x30 [ 5868.140274] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5868.142786] ? do_raw_spin_unlock+0x75/0x190 [ 5868.144675] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5868.146651] ? do_wait_intr+0xf0/0xf0 [ 5868.148413] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5868.149982] mdc_reint+0x2d/0x110 [mdc] [ 5868.153686] mdc_unlink+0x658/0x870 [mdc] [ 5868.155802] lmv_unlink+0x34d/0xbd0 [lmv] [ 5868.158444] ll_unlink+0x24d/0x7f0 [lustre] [ 5868.160357] vfs_unlink+0x202/0x330 [ 5868.162012] do_unlinkat+0x2fa/0x460 [ 5868.163556] __x64_sys_unlink+0x37/0x50 [ 5868.165115] do_syscall_64+0xc1/0x440 [ 5868.166804] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5868.168817] RIP: 0033:0x7f151bdee10b [ 5868.170477] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5868.191053] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5868.194926] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 5868.204722] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 5868.214090] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5868.220013] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5868.228186] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5873.123407] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3942s! [ 5873.131992] Showing busy workqueues and worker pools: [ 5873.138236] workqueue events: flags=0x0 [ 5873.143573] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5873.152448] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 5873.169363] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5873.177004] pending: kfree_rcu_monitor [ 5873.179623] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5873.184527] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 5873.190036] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5873.194201] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5873.196987] workqueue events_long: flags=0x0 [ 5873.198846] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5873.202277] pending: lu_objects_flush [obdclass] [ 5873.205008] workqueue events_unbound: flags=0x2 [ 5873.207133] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5873.209613] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5873.214782] workqueue events_power_efficient: flags=0x80 [ 5873.217418] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5873.223788] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 5873.233805] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5873.249787] pending: neigh_periodic_work [ 5873.256614] workqueue mm_percpu_wq: flags=0x8 [ 5873.264466] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5873.268426] pending: vmstat_update, lru_add_drain_per_cpu [ 5873.272735] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5873.275336] pending: vmstat_update [ 5873.276959] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5873.279685] pending: lru_add_drain_per_cpu [ 5873.281602] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5873.284555] pending: lru_add_drain_per_cpu BAR(48) [ 5873.287360] workqueue writeback: flags=0x4a [ 5873.288837] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5873.291368] pending: wb_workfn [ 5873.292875] workqueue xprtiod: flags=0xa [ 5873.294520] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5873.296929] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5873.300250] workqueue ipv6_addrconf: flags=0x40008 [ 5873.302377] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5873.305358] pending: addrconf_verify_work [ 5873.307018] workqueue knbd0-recv: flags=0x1a [ 5873.308891] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5873.312068] in-flight: 658:recv_work [ 5873.313873] workqueue ptlrpc_pinger: flags=0x2 [ 5873.316050] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5873.318637] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5873.321642] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3942s workers=4 idle: 32984 37 8 [ 5873.325275] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5858s workers=2 idle: 661 [ 5876.056959] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 5876.060565] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5876.097629] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5876.113372] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5876.121892] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5876.129568] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5876.140367] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5876.144660] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 5876.150802] RDX: 0000000000006c7a RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 5876.162451] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 5876.165847] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 5876.169376] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5876.178328] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 5876.184857] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5876.187975] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5876.191842] Call Trace: [ 5876.193024] [ 5876.193974] ? show_regs.cold.9+0x22/0x2f [ 5876.200463] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5876.202846] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5876.208162] ? watchdog+0x40/0x40 [ 5876.210481] ? hrtimer_interrupt+0x128/0x2c0 [ 5876.217854] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5876.222283] ? apic_timer_interrupt+0xf/0x20 [ 5876.224044] [ 5876.225321] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5876.229363] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5876.232044] do_raw_spin_lock+0x177/0x190 [ 5876.233880] _raw_spin_lock+0x21/0x30 [ 5876.235529] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5876.238175] obd_mod_free+0x46/0x140 [ptlrpc] [ 5876.240288] mdc_commit_open+0x91/0xb0 [mdc] [ 5876.242433] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5876.245039] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5876.251579] after_reply+0xe2c/0x1780 [ptlrpc] [ 5876.254649] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5876.257666] ? __next_timer_interrupt+0x160/0x160 [ 5876.259661] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5876.262267] ? do_wait_intr+0xf0/0xf0 [ 5876.263819] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5876.270001] kthread+0x1d1/0x200 [ 5876.272805] ? set_kthread_struct+0x70/0x70 [ 5876.274689] ret_from_fork+0x1f/0x30 [ 5884.063392] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 5884.069337] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5884.135219] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5884.141817] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5884.159177] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5884.163193] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5884.189706] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5884.198493] RAX: 0000000000007c8a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5884.203032] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 5884.211757] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5884.220648] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 5884.224865] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 5884.230071] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 5884.239572] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5884.248756] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 5884.251326] Call Trace: [ 5884.258494] [ 5884.259226] ? show_regs.cold.9+0x22/0x2f [ 5884.260767] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5884.265298] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5884.267888] ? watchdog+0x40/0x40 [ 5884.269825] ? hrtimer_interrupt+0x128/0x2c0 [ 5884.273089] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5884.279394] ? apic_timer_interrupt+0xf/0x20 [ 5884.280635] [ 5884.281222] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5884.284390] do_raw_spin_lock+0x177/0x190 [ 5884.286179] _raw_spin_lock+0x21/0x30 [ 5884.288863] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5884.300617] ? do_raw_spin_unlock+0x75/0x190 [ 5884.303619] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5884.307469] ? do_wait_intr+0xf0/0xf0 [ 5884.309326] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5884.312079] mdc_close+0x53b/0xd70 [mdc] [ 5884.313466] lmv_close+0x18d/0x490 [lmv] [ 5884.319588] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5884.325619] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5884.327790] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5884.331493] ll_md_close+0x274/0x800 [lustre] [ 5884.333104] ll_file_release+0x128/0xc50 [lustre] [ 5884.344028] __fput+0x10d/0x440 [ 5884.346564] ____fput+0x12/0x20 [ 5884.353163] task_work_run+0xc6/0x110 [ 5884.356159] exit_to_usermode_loop+0x1dd/0x1f0 [ 5884.367315] do_syscall_64+0x3d6/0x440 [ 5884.372538] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5884.379153] RIP: 0033:0x7f151bee3cd8 [ 5884.380492] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5884.419244] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5884.432501] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 5884.440886] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 5884.448725] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5884.456272] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5884.462753] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5892.062250] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5892.067387] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5892.104443] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5892.122419] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5892.131267] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5892.138349] RIP: 0010:kvm_wait+0x1b/0xe0 [ 5892.143023] Code: 00 c3 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 0f 1f 44 00 00 65 8b 05 5c f7 b8 4e a9 00 00 f0 00 75 74 48 83 05 0d 53 e6 02 01 <9c> 58 0f 1f 44 00 00 f6 c4 02 75 11 0f b6 07 48 83 05 fe 52 e6 02 [ 5892.158588] RSP: 0018:ffffb2cb80dabdd0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5892.163009] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5892.172347] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 5892.175249] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 5892.177018] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 5892.182981] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 5892.186521] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 5892.191006] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5892.194085] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5892.197343] Call Trace: [ 5892.198422] [ 5892.199420] ? show_regs.cold.9+0x22/0x2f [ 5892.201507] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5892.205573] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5892.208060] ? watchdog+0x40/0x40 [ 5892.210237] ? hrtimer_interrupt+0x128/0x2c0 [ 5892.212540] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5892.214998] ? apic_timer_interrupt+0xf/0x20 [ 5892.219045] [ 5892.220582] ? kvm_wait+0x1b/0xe0 [ 5892.222340] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5892.227930] do_raw_spin_lock+0x177/0x190 [ 5892.231890] _raw_spin_lock+0x21/0x30 [ 5892.237377] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5892.240976] process_one_work+0x2c8/0x700 [ 5892.246613] worker_thread+0x296/0x6e0 [ 5892.248892] ? rescuer_thread+0x570/0x570 [ 5892.251181] kthread+0x1d1/0x200 [ 5892.252370] ? set_kthread_struct+0x70/0x70 [ 5892.255529] ret_from_fork+0x1f/0x30 [ 5900.055174] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 5900.065649] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5900.103150] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5900.110166] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5900.116217] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5900.119675] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5900.130695] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5900.140719] RAX: 0000000000005385 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5900.146796] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 5900.153719] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 5900.157462] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 5900.161587] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 5900.165162] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 5900.168512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5900.175078] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 5900.183035] Call Trace: [ 5900.184309] [ 5900.185097] ? show_regs.cold.9+0x22/0x2f [ 5900.186592] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5900.188391] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5900.190353] ? watchdog+0x40/0x40 [ 5900.191538] ? hrtimer_interrupt+0x128/0x2c0 [ 5900.193258] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5900.194931] ? apic_timer_interrupt+0xf/0x20 [ 5900.196591] [ 5900.200513] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5900.205268] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5900.212559] do_raw_spin_lock+0x177/0x190 [ 5900.214118] _raw_spin_lock+0x21/0x30 [ 5900.215693] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5900.220731] ? do_raw_spin_unlock+0x75/0x190 [ 5900.222916] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5900.225628] ? do_wait_intr+0xf0/0xf0 [ 5900.228345] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5900.230731] mdc_reint+0x2d/0x110 [mdc] [ 5900.233398] mdc_unlink+0x658/0x870 [mdc] [ 5900.235318] lmv_unlink+0x34d/0xbd0 [lmv] [ 5900.237672] ll_unlink+0x24d/0x7f0 [lustre] [ 5900.244907] vfs_unlink+0x202/0x330 [ 5900.248195] do_unlinkat+0x2fa/0x460 [ 5900.250891] __x64_sys_unlink+0x37/0x50 [ 5900.254949] do_syscall_64+0xc1/0x440 [ 5900.256519] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5900.259631] RIP: 0033:0x7f151bdee10b [ 5900.261785] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5900.270129] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5900.273073] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 5900.279338] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 5900.285297] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5900.288448] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5900.293664] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5903.841034] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3973s! [ 5903.846332] Showing busy workqueues and worker pools: [ 5903.847596] workqueue events: flags=0x0 [ 5903.851859] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5903.856044] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 5903.859193] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5903.861390] pending: kfree_rcu_monitor [ 5903.867498] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5903.870123] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 5903.877700] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5903.880042] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5903.883165] workqueue events_long: flags=0x0 [ 5903.884572] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5903.888231] pending: lu_objects_flush [obdclass] [ 5903.894036] workqueue events_unbound: flags=0x2 [ 5903.897299] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5903.900369] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5903.905058] workqueue events_power_efficient: flags=0x80 [ 5903.909385] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5903.913600] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 5903.918753] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5903.921028] pending: neigh_periodic_work [ 5903.923142] workqueue mm_percpu_wq: flags=0x8 [ 5903.928094] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5903.935317] pending: vmstat_update, lru_add_drain_per_cpu [ 5903.940478] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5903.947413] pending: vmstat_update [ 5903.949211] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5903.954126] pending: lru_add_drain_per_cpu [ 5903.958131] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5903.964420] pending: lru_add_drain_per_cpu BAR(48) [ 5903.966690] workqueue writeback: flags=0x4a [ 5903.973012] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5903.977357] pending: wb_workfn [ 5903.981213] workqueue xprtiod: flags=0xa [ 5903.983134] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5903.985204] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5903.994654] workqueue ipv6_addrconf: flags=0x40008 [ 5903.998334] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5904.002243] pending: addrconf_verify_work [ 5904.005598] workqueue knbd0-recv: flags=0x1a [ 5904.008899] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5904.014770] in-flight: 658:recv_work [ 5904.018183] workqueue ptlrpc_pinger: flags=0x2 [ 5904.020358] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5904.027779] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5904.029882] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3973s workers=4 idle: 32984 37 8 [ 5904.035729] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5888s workers=2 idle: 661 [ 5904.058007] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 5904.061645] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5904.091678] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5904.098257] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5904.113149] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5904.118532] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5904.131614] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5904.144165] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 5904.152115] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 5904.164197] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 5904.171646] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 5904.179226] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5904.187966] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 5904.193963] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5904.208534] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5904.218838] Call Trace: [ 5904.221508] [ 5904.224933] ? show_regs.cold.9+0x22/0x2f [ 5904.229025] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5904.230316] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5904.231810] ? watchdog+0x40/0x40 [ 5904.234331] ? hrtimer_interrupt+0x128/0x2c0 [ 5904.238842] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5904.242940] ? apic_timer_interrupt+0xf/0x20 [ 5904.248449] [ 5904.251157] ? native_safe_halt+0x2e/0x50 [ 5904.256545] kvm_wait+0xcd/0xe0 [ 5904.259991] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5904.265555] do_raw_spin_lock+0x177/0x190 [ 5904.269414] _raw_spin_lock+0x21/0x30 [ 5904.274612] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5904.276972] obd_mod_free+0x46/0x140 [ptlrpc] [ 5904.280545] mdc_commit_open+0x91/0xb0 [mdc] [ 5904.284338] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5904.290258] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5904.294900] after_reply+0xe2c/0x1780 [ptlrpc] [ 5904.299380] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5904.302532] ? __next_timer_interrupt+0x160/0x160 [ 5904.304525] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5904.308624] ? do_wait_intr+0xf0/0xf0 [ 5904.310724] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5904.314060] kthread+0x1d1/0x200 [ 5904.315716] ? set_kthread_struct+0x70/0x70 [ 5904.317951] ret_from_fork+0x1f/0x30 [ 5912.058787] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 5912.070295] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5912.109170] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5912.113583] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5912.116567] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5912.118056] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5912.124165] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5912.126603] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5912.129175] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 5912.131222] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5912.133673] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 5912.136759] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 5912.139218] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 5912.141939] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5912.143646] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 5912.145834] Call Trace: [ 5912.146650] [ 5912.147398] ? show_regs.cold.9+0x22/0x2f [ 5912.148526] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5912.149892] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5912.159249] ? watchdog+0x40/0x40 [ 5912.160319] ? hrtimer_interrupt+0x128/0x2c0 [ 5912.162287] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5912.163810] ? apic_timer_interrupt+0xf/0x20 [ 5912.165854] [ 5912.167015] ? native_safe_halt+0x2e/0x50 [ 5912.168290] kvm_wait+0xcd/0xe0 [ 5912.169442] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5912.171276] do_raw_spin_lock+0x177/0x190 [ 5912.174120] _raw_spin_lock+0x21/0x30 [ 5912.175980] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5912.178480] ? do_raw_spin_unlock+0x75/0x190 [ 5912.180236] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5912.181826] ? do_wait_intr+0xf0/0xf0 [ 5912.182827] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5912.186393] mdc_close+0x53b/0xd70 [mdc] [ 5912.187817] lmv_close+0x18d/0x490 [lmv] [ 5912.189494] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5912.193573] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5912.196080] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5912.197784] ll_md_close+0x274/0x800 [lustre] [ 5912.199539] ll_file_release+0x128/0xc50 [lustre] [ 5912.201243] __fput+0x10d/0x440 [ 5912.202313] ____fput+0x12/0x20 [ 5912.204415] task_work_run+0xc6/0x110 [ 5912.208447] exit_to_usermode_loop+0x1dd/0x1f0 [ 5912.211025] do_syscall_64+0x3d6/0x440 [ 5912.212436] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5912.214104] RIP: 0033:0x7f151bee3cd8 [ 5912.215012] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5912.220188] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5912.222613] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 5912.224646] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 5912.226637] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5912.228780] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5912.230400] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5920.056022] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5920.064614] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5920.094878] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5920.104444] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5920.109185] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5920.118766] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5920.122850] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5920.132989] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5920.138602] RAX: 0000000000005d8e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5920.141318] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 5920.146360] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 5920.153488] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 5920.157090] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 5920.160367] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 5920.164279] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5920.166531] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5920.170855] Call Trace: [ 5920.173452] [ 5920.174881] ? show_regs.cold.9+0x22/0x2f [ 5920.176339] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5920.181529] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5920.182925] ? watchdog+0x40/0x40 [ 5920.185272] ? hrtimer_interrupt+0x128/0x2c0 [ 5920.186482] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5920.188520] ? apic_timer_interrupt+0xf/0x20 [ 5920.196380] [ 5920.197771] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5920.201936] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5920.204606] do_raw_spin_lock+0x177/0x190 [ 5920.206431] _raw_spin_lock+0x21/0x30 [ 5920.208423] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5920.212508] process_one_work+0x2c8/0x700 [ 5920.214565] worker_thread+0x296/0x6e0 [ 5920.216755] ? rescuer_thread+0x570/0x570 [ 5920.220694] kthread+0x1d1/0x200 [ 5920.224302] ? set_kthread_struct+0x70/0x70 [ 5920.226080] ret_from_fork+0x1f/0x30 [ 5928.054575] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 5928.067515] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5928.090507] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5928.100977] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5928.108124] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5928.115030] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5928.136104] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5928.148718] RAX: 0000000000007f85 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5928.154914] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 5928.160730] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 5928.170260] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 5928.178018] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 5928.188128] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 5928.195058] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5928.203131] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 5928.215054] Call Trace: [ 5928.219504] [ 5928.220307] ? show_regs.cold.9+0x22/0x2f [ 5928.230929] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5928.235901] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5928.239021] ? watchdog+0x40/0x40 [ 5928.241042] ? hrtimer_interrupt+0x128/0x2c0 [ 5928.244671] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5928.247450] ? apic_timer_interrupt+0xf/0x20 [ 5928.249026] [ 5928.249987] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5928.255472] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5928.258988] do_raw_spin_lock+0x177/0x190 [ 5928.261712] _raw_spin_lock+0x21/0x30 [ 5928.265868] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5928.271081] ? do_raw_spin_unlock+0x75/0x190 [ 5928.274080] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5928.277885] ? do_wait_intr+0xf0/0xf0 [ 5928.279023] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5928.285100] mdc_reint+0x2d/0x110 [mdc] [ 5928.289343] mdc_unlink+0x658/0x870 [mdc] [ 5928.292974] lmv_unlink+0x34d/0xbd0 [lmv] [ 5928.296018] ll_unlink+0x24d/0x7f0 [lustre] [ 5928.300076] vfs_unlink+0x202/0x330 [ 5928.302307] do_unlinkat+0x2fa/0x460 [ 5928.304701] __x64_sys_unlink+0x37/0x50 [ 5928.307474] do_syscall_64+0xc1/0x440 [ 5928.309709] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5928.311650] RIP: 0033:0x7f151bdee10b [ 5928.325115] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5928.354131] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5928.360064] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 5928.366614] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 5928.374133] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5928.388015] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5928.398697] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5932.055030] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 5932.059621] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5932.092936] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5932.099840] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5932.103902] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5932.105320] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5932.123504] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5932.131715] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 5932.144880] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 5932.153416] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 5932.162907] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 5932.170189] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5932.178723] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 5932.183899] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5932.187752] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5932.191916] Call Trace: [ 5932.194050] [ 5932.194857] ? show_regs.cold.9+0x22/0x2f [ 5932.198261] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5932.204528] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5932.210064] ? watchdog+0x40/0x40 [ 5932.213262] ? hrtimer_interrupt+0x128/0x2c0 [ 5932.216256] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5932.220071] ? apic_timer_interrupt+0xf/0x20 [ 5932.221714] [ 5932.222520] ? native_safe_halt+0x2e/0x50 [ 5932.227386] kvm_wait+0xcd/0xe0 [ 5932.230939] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5932.241916] do_raw_spin_lock+0x177/0x190 [ 5932.251112] _raw_spin_lock+0x21/0x30 [ 5932.257133] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5932.264709] obd_mod_free+0x46/0x140 [ptlrpc] [ 5932.273805] mdc_commit_open+0x91/0xb0 [mdc] [ 5932.275501] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5932.284964] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5932.289330] after_reply+0xe2c/0x1780 [ptlrpc] [ 5932.293876] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5932.304203] ? __next_timer_interrupt+0x160/0x160 [ 5932.308858] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5932.313421] ? do_wait_intr+0xf0/0xf0 [ 5932.314832] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5932.324336] kthread+0x1d1/0x200 [ 5932.327218] ? set_kthread_struct+0x70/0x70 [ 5932.333460] ret_from_fork+0x1f/0x30 [ 5934.049075] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4003s! [ 5934.053475] Showing busy workqueues and worker pools: [ 5934.055610] workqueue events: flags=0x0 [ 5934.058097] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5934.063340] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 5934.068447] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5934.076412] pending: kfree_rcu_monitor [ 5934.084174] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5934.091776] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 5934.097455] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5934.102660] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5934.107264] workqueue events_long: flags=0x0 [ 5934.109341] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5934.112350] pending: lu_objects_flush [obdclass] [ 5934.115223] workqueue events_unbound: flags=0x2 [ 5934.119317] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5934.132327] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5934.146125] workqueue events_power_efficient: flags=0x80 [ 5934.157266] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5934.170739] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 5934.175061] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5934.179504] pending: neigh_periodic_work [ 5934.181896] workqueue mm_percpu_wq: flags=0x8 [ 5934.185132] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5934.189969] pending: vmstat_update, lru_add_drain_per_cpu [ 5934.194417] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5934.202358] pending: vmstat_update [ 5934.208731] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5934.214371] pending: lru_add_drain_per_cpu [ 5934.218140] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5934.224970] pending: lru_add_drain_per_cpu BAR(48) [ 5934.229073] workqueue writeback: flags=0x4a [ 5934.233181] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5934.242332] pending: wb_workfn [ 5934.244434] workqueue xprtiod: flags=0xa [ 5934.248236] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5934.255996] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5934.269053] workqueue ipv6_addrconf: flags=0x40008 [ 5934.273835] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5934.279266] pending: addrconf_verify_work [ 5934.287980] workqueue knbd0-recv: flags=0x1a [ 5934.293245] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5934.297240] in-flight: 658:recv_work [ 5934.300154] workqueue ptlrpc_pinger: flags=0x2 [ 5934.309989] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5934.316063] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5934.321300] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4003s workers=4 idle: 32984 37 8 [ 5934.336128] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5919s workers=2 idle: 661 [ 5940.066715] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 5940.076592] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5940.128507] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5940.135165] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5940.143155] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5940.150044] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5940.165435] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5940.173392] RAX: 000000000000575e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5940.194234] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 5940.210208] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5940.224507] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 5940.235218] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 5940.245263] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 5940.251252] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5940.258910] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 5940.269248] Call Trace: [ 5940.272216] [ 5940.275750] ? show_regs.cold.9+0x22/0x2f [ 5940.278465] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5940.283790] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5940.290266] ? watchdog+0x40/0x40 [ 5940.296100] ? hrtimer_interrupt+0x128/0x2c0 [ 5940.301793] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5940.306233] ? apic_timer_interrupt+0xf/0x20 [ 5940.309587] [ 5940.312769] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5940.319079] do_raw_spin_lock+0x177/0x190 [ 5940.323053] _raw_spin_lock+0x21/0x30 [ 5940.326409] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5940.332978] ? do_raw_spin_unlock+0x75/0x190 [ 5940.334484] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5940.343595] ? do_wait_intr+0xf0/0xf0 [ 5940.345691] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5940.351687] mdc_close+0x53b/0xd70 [mdc] [ 5940.356497] lmv_close+0x18d/0x490 [lmv] [ 5940.361177] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5940.369355] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5940.375164] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5940.378504] ll_md_close+0x274/0x800 [lustre] [ 5940.381249] ll_file_release+0x128/0xc50 [lustre] [ 5940.387216] __fput+0x10d/0x440 [ 5940.388418] ____fput+0x12/0x20 [ 5940.390735] task_work_run+0xc6/0x110 [ 5940.392841] exit_to_usermode_loop+0x1dd/0x1f0 [ 5940.394783] do_syscall_64+0x3d6/0x440 [ 5940.401291] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5940.404306] RIP: 0033:0x7f151bee3cd8 [ 5940.406339] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5940.418434] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5940.428693] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 5940.437362] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 5940.445254] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5940.449713] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5940.459017] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5944.722059] rcu: INFO: rcu_sched self-detected stall on CPU [ 5944.725096] rcu: 1-...!: (2322055 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 5944.730917] (t=4020075 jiffies g=249657 q=3529) [ 5944.733814] rcu: rcu_sched kthread starved for 4015027 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 5944.739631] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5944.744037] rcu: RCU grace-period kthread stack dump: [ 5944.754791] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5944.760498] Call Trace: [ 5944.761836] __schedule+0x351/0xcb0 [ 5944.763323] schedule+0xc0/0x180 [ 5944.765209] schedule_timeout+0xb4/0x190 [ 5944.767776] ? do_raw_spin_unlock+0x75/0x190 [ 5944.770296] ? __next_timer_interrupt+0x160/0x160 [ 5944.775776] ? prepare_to_swait_event+0xf3/0x140 [ 5944.778098] rcu_gp_kthread+0x8dd/0x15b0 [ 5944.780787] ? force_qs_rnp+0x2c0/0x2c0 [ 5944.782170] kthread+0x1d1/0x200 [ 5944.783633] ? set_kthread_struct+0x70/0x70 [ 5944.785591] ret_from_fork+0x1f/0x30 [ 5944.794094] rcu: Stack dump where RCU GP kthread last ran: [ 5944.799303] Sending NMI from CPU 1 to CPUs 0: [ 5944.804134] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [ 5944.806047] NMI backtrace for cpu 1 [ 5944.813103] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5944.819754] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5944.824844] Call Trace: [ 5944.825943] [ 5944.826886] ? dump_stack+0xbb/0x10e [ 5944.828590] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5944.830724] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5944.832957] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5944.835225] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5944.837685] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5944.839989] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 5944.842191] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 5944.844347] ? update_process_times+0x6d/0xc0 [ 5944.846498] ? tick_sched_handle.isra.18+0x27/0x90 [ 5944.848646] ? tick_sched_timer+0x53/0xb0 [ 5944.850607] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5944.853055] ? tick_sched_handle.isra.18+0x90/0x90 [ 5944.858112] ? hrtimer_interrupt+0x128/0x2c0 [ 5944.860080] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5944.863074] ? apic_timer_interrupt+0xf/0x20 [ 5944.866173] [ 5944.866948] ? native_safe_halt+0x2e/0x50 [ 5944.869807] ? kvm_wait+0xcd/0xe0 [ 5944.871974] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5944.877892] ? do_raw_spin_lock+0x177/0x190 [ 5944.883579] ? _raw_spin_lock+0x21/0x30 [ 5944.887027] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5944.889087] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5944.891809] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5944.894595] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5944.897792] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5944.900979] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5944.904566] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5944.907790] ? __next_timer_interrupt+0x160/0x160 [ 5944.910529] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5944.913506] ? do_wait_intr+0xf0/0xf0 [ 5944.915240] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5944.917918] ? kthread+0x1d1/0x200 [ 5944.919758] ? set_kthread_struct+0x70/0x70 [ 5944.921862] ? ret_from_fork+0x1f/0x30 [ 5948.060221] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5948.068103] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5948.110583] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5948.125922] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5948.131683] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5948.137985] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5948.143410] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5948.169620] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5948.182742] RAX: 0000000000007b15 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5948.189733] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 5948.193595] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 5948.197990] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 5948.201820] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 5948.212661] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 5948.219291] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5948.224114] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5948.237718] Call Trace: [ 5948.238552] [ 5948.239262] ? show_regs.cold.9+0x22/0x2f [ 5948.245531] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5948.247343] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5948.249655] ? watchdog+0x40/0x40 [ 5948.251344] ? hrtimer_interrupt+0x128/0x2c0 [ 5948.254732] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5948.259307] ? apic_timer_interrupt+0xf/0x20 [ 5948.262343] [ 5948.263611] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5948.270658] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5948.279765] do_raw_spin_lock+0x177/0x190 [ 5948.283480] _raw_spin_lock+0x21/0x30 [ 5948.284878] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5948.286878] process_one_work+0x2c8/0x700 [ 5948.288020] worker_thread+0x296/0x6e0 [ 5948.289702] ? rescuer_thread+0x570/0x570 [ 5948.291563] kthread+0x1d1/0x200 [ 5948.293279] ? set_kthread_struct+0x70/0x70 [ 5948.296819] ret_from_fork+0x1f/0x30 [ 5964.769045] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4034s! [ 5964.774376] Showing busy workqueues and worker pools: [ 5964.778068] workqueue events: flags=0x0 [ 5964.780971] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5964.785887] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 5964.793498] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5964.799138] pending: kfree_rcu_monitor [ 5964.802145] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5964.807119] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 5964.814026] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5964.818078] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5964.822964] workqueue events_long: flags=0x0 [ 5964.825154] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5964.829201] pending: lu_objects_flush [obdclass] [ 5964.832375] workqueue events_unbound: flags=0x2 [ 5964.835381] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5964.839494] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5964.846669] workqueue events_power_efficient: flags=0x80 [ 5964.850044] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5964.854903] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 5964.859096] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5964.862925] pending: neigh_periodic_work [ 5964.864568] workqueue mm_percpu_wq: flags=0x8 [ 5964.867360] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5964.871793] pending: vmstat_update, lru_add_drain_per_cpu [ 5964.875633] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5964.879363] pending: vmstat_update [ 5964.880687] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5964.884334] pending: lru_add_drain_per_cpu [ 5964.886461] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5964.890904] pending: lru_add_drain_per_cpu BAR(48) [ 5964.894083] workqueue writeback: flags=0x4a [ 5964.896440] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5964.900547] pending: wb_workfn [ 5964.902673] workqueue xprtiod: flags=0xa [ 5964.904969] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5964.908516] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5964.912649] workqueue ipv6_addrconf: flags=0x40008 [ 5964.915305] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5964.919461] pending: addrconf_verify_work [ 5964.922146] workqueue knbd0-recv: flags=0x1a [ 5964.925371] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5964.929318] in-flight: 658:recv_work [ 5964.932160] workqueue ptlrpc_pinger: flags=0x2 [ 5964.935652] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5964.939927] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5964.945459] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4034s workers=4 idle: 32984 37 8 [ 5964.951706] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5949s workers=2 idle: 661 [ 5968.057009] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 5968.063131] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5968.118346] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5968.132047] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5968.137913] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5968.140869] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5968.150144] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5968.153866] RAX: 00000000000066c0 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5968.156720] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 5968.159797] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5968.162247] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 5968.164946] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 5968.176437] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 5968.181625] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5968.190283] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 5968.194598] Call Trace: [ 5968.195438] [ 5968.196210] ? show_regs.cold.9+0x22/0x2f [ 5968.198412] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5968.206098] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5968.211923] ? watchdog+0x40/0x40 [ 5968.213191] ? hrtimer_interrupt+0x128/0x2c0 [ 5968.222655] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5968.224144] ? apic_timer_interrupt+0xf/0x20 [ 5968.229319] [ 5968.230665] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5968.240342] do_raw_spin_lock+0x177/0x190 [ 5968.244361] _raw_spin_lock+0x21/0x30 [ 5968.249959] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5968.257549] ? do_raw_spin_unlock+0x75/0x190 [ 5968.267667] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5968.275158] ? do_wait_intr+0xf0/0xf0 [ 5968.277508] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5968.280148] mdc_close+0x53b/0xd70 [mdc] [ 5968.282351] lmv_close+0x18d/0x490 [lmv] [ 5968.284478] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5968.287803] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5968.290448] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5968.292877] ll_md_close+0x274/0x800 [lustre] [ 5968.295781] ll_file_release+0x128/0xc50 [lustre] [ 5968.302717] __fput+0x10d/0x440 [ 5968.306469] ____fput+0x12/0x20 [ 5968.308273] task_work_run+0xc6/0x110 [ 5968.310309] exit_to_usermode_loop+0x1dd/0x1f0 [ 5968.312393] do_syscall_64+0x3d6/0x440 [ 5968.314762] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5968.317549] RIP: 0033:0x7f151bee3cd8 [ 5968.319755] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5968.329549] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5968.333651] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 5968.337497] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 5968.340762] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5968.344288] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5968.347499] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5972.055023] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 5972.058982] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5972.087090] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5972.090800] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5972.093930] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5972.095979] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5972.105251] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5972.108732] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 5972.112275] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 5972.118549] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 5972.127224] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 5972.133573] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5972.140068] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 5972.143885] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5972.146364] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5972.151324] Call Trace: [ 5972.152092] [ 5972.153217] ? show_regs.cold.9+0x22/0x2f [ 5972.157634] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5972.159954] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5972.162974] ? watchdog+0x40/0x40 [ 5972.166817] ? hrtimer_interrupt+0x128/0x2c0 [ 5972.169573] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5972.175191] ? apic_timer_interrupt+0xf/0x20 [ 5972.176819] [ 5972.179512] ? native_safe_halt+0x2e/0x50 [ 5972.185915] kvm_wait+0xcd/0xe0 [ 5972.188787] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5972.194509] do_raw_spin_lock+0x177/0x190 [ 5972.199182] _raw_spin_lock+0x21/0x30 [ 5972.203069] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5972.209333] obd_mod_free+0x46/0x140 [ptlrpc] [ 5972.213822] mdc_commit_open+0x91/0xb0 [mdc] [ 5972.216741] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5972.223027] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5972.225855] after_reply+0xe2c/0x1780 [ptlrpc] [ 5972.229936] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5972.234293] ? __next_timer_interrupt+0x160/0x160 [ 5972.236779] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5972.242557] ? do_wait_intr+0xf0/0xf0 [ 5972.246090] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5972.248836] kthread+0x1d1/0x200 [ 5972.252768] ? set_kthread_struct+0x70/0x70 [ 5972.258518] ret_from_fork+0x1f/0x30 [ 5976.056025] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 5976.059565] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5976.094273] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5976.110650] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5976.117911] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5976.127049] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5976.135891] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5976.151636] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5976.155844] RAX: 0000000000002dc8 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5976.159781] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 5976.162706] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 5976.166175] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 5976.169535] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 5976.173083] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 5976.176328] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5976.178774] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 5976.182045] Call Trace: [ 5976.183153] [ 5976.184197] ? show_regs.cold.9+0x22/0x2f [ 5976.185667] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5976.187521] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5976.189556] ? watchdog+0x40/0x40 [ 5976.191415] ? hrtimer_interrupt+0x128/0x2c0 [ 5976.192845] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5976.194880] ? apic_timer_interrupt+0xf/0x20 [ 5976.196952] [ 5976.197823] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5976.200744] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5976.203744] do_raw_spin_lock+0x177/0x190 [ 5976.205248] _raw_spin_lock+0x21/0x30 [ 5976.207273] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5976.209788] process_one_work+0x2c8/0x700 [ 5976.211634] worker_thread+0x296/0x6e0 [ 5976.213410] ? rescuer_thread+0x570/0x570 [ 5976.214951] kthread+0x1d1/0x200 [ 5976.216185] ? set_kthread_struct+0x70/0x70 [ 5976.218041] ret_from_fork+0x1f/0x30 [ 5992.055767] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 5992.059297] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5992.081465] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5992.098369] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5992.110913] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5992.118092] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5992.128562] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5992.132585] RAX: 000000000000738c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5992.135889] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 5992.148087] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 5992.151259] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 5992.157135] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 5992.161624] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 5992.169865] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5992.173395] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 5992.175229] Call Trace: [ 5992.175796] [ 5992.181631] ? show_regs.cold.9+0x22/0x2f [ 5992.185152] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5992.189278] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5992.195412] ? watchdog+0x40/0x40 [ 5992.199217] ? hrtimer_interrupt+0x128/0x2c0 [ 5992.202738] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5992.205399] ? apic_timer_interrupt+0xf/0x20 [ 5992.207391] [ 5992.208514] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5992.214332] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5992.217804] do_raw_spin_lock+0x177/0x190 [ 5992.221526] _raw_spin_lock+0x21/0x30 [ 5992.223976] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5992.226015] ? do_raw_spin_unlock+0x75/0x190 [ 5992.230110] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5992.233362] ? do_wait_intr+0xf0/0xf0 [ 5992.241843] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5992.246399] mdc_reint+0x2d/0x110 [mdc] [ 5992.250163] mdc_unlink+0x658/0x870 [mdc] [ 5992.252822] lmv_unlink+0x34d/0xbd0 [lmv] [ 5992.256312] ll_unlink+0x24d/0x7f0 [lustre] [ 5992.258352] vfs_unlink+0x202/0x330 [ 5992.262797] do_unlinkat+0x2fa/0x460 [ 5992.265227] __x64_sys_unlink+0x37/0x50 [ 5992.271605] do_syscall_64+0xc1/0x440 [ 5992.272877] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5992.277797] RIP: 0033:0x7f151bdee10b [ 5992.282246] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5992.318655] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5992.337675] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 5992.346822] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 5992.353446] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5992.362178] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5992.372515] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5994.981053] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4064s! [ 5994.985487] Showing busy workqueues and worker pools: [ 5994.988437] workqueue events: flags=0x0 [ 5994.990526] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5994.993833] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 5994.998718] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5995.002564] pending: kfree_rcu_monitor [ 5995.004856] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5995.008424] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 5995.013693] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5995.017366] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 5995.021019] workqueue events_long: flags=0x0 [ 5995.023185] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5995.026741] pending: lu_objects_flush [obdclass] [ 5995.029392] workqueue events_unbound: flags=0x2 [ 5995.031640] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 5995.034657] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 5995.039703] workqueue events_power_efficient: flags=0x80 [ 5995.042245] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5995.045444] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 5995.049524] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5995.052700] pending: neigh_periodic_work [ 5995.054763] workqueue mm_percpu_wq: flags=0x8 [ 5995.056706] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5995.060435] pending: vmstat_update, lru_add_drain_per_cpu [ 5995.063640] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5995.066915] pending: vmstat_update [ 5995.068677] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5995.072119] pending: lru_add_drain_per_cpu [ 5995.074235] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5995.076663] pending: lru_add_drain_per_cpu BAR(48) [ 5995.079607] workqueue writeback: flags=0x4a [ 5995.081720] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 5995.084661] pending: wb_workfn [ 5995.086371] workqueue xprtiod: flags=0xa [ 5995.087833] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 5995.090401] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 5995.093446] workqueue ipv6_addrconf: flags=0x40008 [ 5995.095363] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5995.097581] pending: addrconf_verify_work [ 5995.099470] workqueue knbd0-recv: flags=0x1a [ 5995.101085] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5995.103378] in-flight: 658:recv_work [ 5995.105246] workqueue ptlrpc_pinger: flags=0x2 [ 5995.107344] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5995.109934] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 5995.113053] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4064s workers=4 idle: 32984 37 8 [ 5995.116020] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5979s workers=2 idle: 661 [ 5996.058122] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 5996.066365] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5996.106716] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5996.122417] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5996.135481] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5996.143484] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5996.160576] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5996.167100] RAX: 0000000000005564 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 5996.172155] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 5996.178472] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5996.184075] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 5996.187213] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 5996.191072] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 5996.197371] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5996.199669] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 5996.204297] Call Trace: [ 5996.209662] [ 5996.210638] ? show_regs.cold.9+0x22/0x2f [ 5996.215976] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5996.220304] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5996.221787] ? watchdog+0x40/0x40 [ 5996.222770] ? hrtimer_interrupt+0x128/0x2c0 [ 5996.227412] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5996.233231] ? apic_timer_interrupt+0xf/0x20 [ 5996.239313] [ 5996.243160] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5996.245457] do_raw_spin_lock+0x177/0x190 [ 5996.249087] _raw_spin_lock+0x21/0x30 [ 5996.254337] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5996.259083] ? do_raw_spin_unlock+0x75/0x190 [ 5996.264618] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5996.268864] ? do_wait_intr+0xf0/0xf0 [ 5996.271073] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5996.273118] mdc_close+0x53b/0xd70 [mdc] [ 5996.277638] lmv_close+0x18d/0x490 [lmv] [ 5996.279477] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 5996.285170] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 5996.290486] ll_md_real_close+0xa5/0x2a0 [lustre] [ 5996.294591] ll_md_close+0x274/0x800 [lustre] [ 5996.297661] ll_file_release+0x128/0xc50 [lustre] [ 5996.300369] __fput+0x10d/0x440 [ 5996.302123] ____fput+0x12/0x20 [ 5996.304197] task_work_run+0xc6/0x110 [ 5996.306176] exit_to_usermode_loop+0x1dd/0x1f0 [ 5996.309721] do_syscall_64+0x3d6/0x440 [ 5996.311491] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5996.318622] RIP: 0033:0x7f151bee3cd8 [ 5996.322010] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 5996.341187] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 5996.347352] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 5996.349563] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 5996.361372] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 5996.365630] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 5996.379679] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6000.057621] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 6000.060041] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6000.123884] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6000.154049] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6000.165702] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6000.170214] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6000.191475] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6000.204549] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 6000.207187] RDX: 0000000000007f0e RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 6000.213167] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 6000.215451] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 6000.229221] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6000.232619] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 6000.237517] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6000.240608] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6000.245178] Call Trace: [ 6000.246400] [ 6000.247451] ? show_regs.cold.9+0x22/0x2f [ 6000.249364] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6000.253978] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6000.263310] ? watchdog+0x40/0x40 [ 6000.264132] ? hrtimer_interrupt+0x128/0x2c0 [ 6000.265658] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6000.267637] ? apic_timer_interrupt+0xf/0x20 [ 6000.269685] [ 6000.271068] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6000.276049] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6000.280618] do_raw_spin_lock+0x177/0x190 [ 6000.282302] _raw_spin_lock+0x21/0x30 [ 6000.283777] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6000.286023] obd_mod_free+0x46/0x140 [ptlrpc] [ 6000.288218] mdc_commit_open+0x91/0xb0 [mdc] [ 6000.291225] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6000.297417] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6000.300382] after_reply+0xe2c/0x1780 [ptlrpc] [ 6000.306620] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6000.308444] ? __next_timer_interrupt+0x160/0x160 [ 6000.312558] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6000.317340] ? do_wait_intr+0xf0/0xf0 [ 6000.325413] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6000.328218] kthread+0x1d1/0x200 [ 6000.329421] ? set_kthread_struct+0x70/0x70 [ 6000.334524] ret_from_fork+0x1f/0x30 [ 6004.056149] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6004.061710] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6004.085754] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6004.094676] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6004.099346] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6004.102628] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6004.105475] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6004.114282] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 6004.118211] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6004.121491] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 6004.125216] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 6004.128752] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 6004.131678] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 6004.135752] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 6004.144200] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6004.151397] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6004.155274] Call Trace: [ 6004.161041] [ 6004.161785] ? show_regs.cold.9+0x22/0x2f [ 6004.165395] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6004.171321] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6004.174039] ? watchdog+0x40/0x40 [ 6004.176081] ? hrtimer_interrupt+0x128/0x2c0 [ 6004.178103] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6004.183415] ? apic_timer_interrupt+0xf/0x20 [ 6004.185864] [ 6004.186763] ? native_safe_halt+0x2e/0x50 [ 6004.189009] kvm_wait+0xcd/0xe0 [ 6004.190603] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6004.193081] do_raw_spin_lock+0x177/0x190 [ 6004.196793] _raw_spin_lock+0x21/0x30 [ 6004.198326] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6004.201403] process_one_work+0x2c8/0x700 [ 6004.205176] worker_thread+0x296/0x6e0 [ 6004.207378] ? rescuer_thread+0x570/0x570 [ 6004.209616] kthread+0x1d1/0x200 [ 6004.213188] ? set_kthread_struct+0x70/0x70 [ 6004.218133] ret_from_fork+0x1f/0x30 [ 6020.053498] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 6020.056496] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6020.115022] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6020.126249] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6020.143317] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6020.146996] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6020.173867] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6020.183089] RAX: 0000000000005b4e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6020.192476] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 6020.196555] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 6020.207867] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 6020.215814] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 6020.221449] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 6020.230475] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6020.233119] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 6020.238962] Call Trace: [ 6020.245657] [ 6020.247499] ? show_regs.cold.9+0x22/0x2f [ 6020.252741] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6020.258489] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6020.264405] ? watchdog+0x40/0x40 [ 6020.267575] ? hrtimer_interrupt+0x128/0x2c0 [ 6020.270706] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6020.276224] ? apic_timer_interrupt+0xf/0x20 [ 6020.278615] [ 6020.284645] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6020.287512] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6020.296098] do_raw_spin_lock+0x177/0x190 [ 6020.303892] _raw_spin_lock+0x21/0x30 [ 6020.305307] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6020.311811] ? do_raw_spin_unlock+0x75/0x190 [ 6020.317344] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6020.330836] ? do_wait_intr+0xf0/0xf0 [ 6020.332440] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6020.338687] mdc_reint+0x2d/0x110 [mdc] [ 6020.340484] mdc_unlink+0x658/0x870 [mdc] [ 6020.346357] lmv_unlink+0x34d/0xbd0 [lmv] [ 6020.349668] ll_unlink+0x24d/0x7f0 [lustre] [ 6020.354015] vfs_unlink+0x202/0x330 [ 6020.358740] do_unlinkat+0x2fa/0x460 [ 6020.363451] __x64_sys_unlink+0x37/0x50 [ 6020.372536] do_syscall_64+0xc1/0x440 [ 6020.379860] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6020.389887] RIP: 0033:0x7f151bdee10b [ 6020.395010] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6020.416313] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6020.424021] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 6020.435397] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 6020.443381] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6020.446708] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6020.451526] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6024.061949] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 6024.079383] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6024.135110] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6024.145317] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6024.148402] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6024.156185] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6024.170271] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6024.178575] RAX: 0000000000003b65 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6024.185470] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 6024.194514] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6024.206939] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 6024.214390] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 6024.221016] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 6024.231799] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6024.236442] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 6024.246451] Call Trace: [ 6024.249032] [ 6024.251086] ? show_regs.cold.9+0x22/0x2f [ 6024.257052] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6024.262579] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6024.276396] ? watchdog+0x40/0x40 [ 6024.278708] ? hrtimer_interrupt+0x128/0x2c0 [ 6024.284078] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6024.285682] ? apic_timer_interrupt+0xf/0x20 [ 6024.293690] [ 6024.296599] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6024.307475] do_raw_spin_lock+0x177/0x190 [ 6024.313066] _raw_spin_lock+0x21/0x30 [ 6024.314421] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6024.328781] ? do_raw_spin_unlock+0x75/0x190 [ 6024.339804] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6024.341646] ? do_wait_intr+0xf0/0xf0 [ 6024.355806] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6024.357713] mdc_close+0x53b/0xd70 [mdc] [ 6024.369538] lmv_close+0x18d/0x490 [lmv] [ 6024.374144] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6024.382173] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6024.386166] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6024.389343] ll_md_close+0x274/0x800 [lustre] [ 6024.395319] ll_file_release+0x128/0xc50 [lustre] [ 6024.398300] __fput+0x10d/0x440 [ 6024.403355] ____fput+0x12/0x20 [ 6024.407867] task_work_run+0xc6/0x110 [ 6024.409060] exit_to_usermode_loop+0x1dd/0x1f0 [ 6024.411292] do_syscall_64+0x3d6/0x440 [ 6024.412492] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6024.416889] RIP: 0033:0x7f151bee3cd8 [ 6024.418346] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6024.427202] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6024.433367] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 6024.436433] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 6024.439368] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6024.443716] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6024.446924] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6025.185177] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4094s! [ 6025.187585] Showing busy workqueues and worker pools: [ 6025.189201] workqueue events: flags=0x0 [ 6025.190476] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6025.193086] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 6025.196307] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6025.198413] pending: kfree_rcu_monitor [ 6025.200091] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6025.202483] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 6025.205592] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6025.207703] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 6025.210444] workqueue events_long: flags=0x0 [ 6025.212173] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6025.214818] pending: lu_objects_flush [obdclass] [ 6025.216586] workqueue events_unbound: flags=0x2 [ 6025.218107] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6025.220091] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6025.223372] workqueue events_power_efficient: flags=0x80 [ 6025.225095] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6025.227107] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 6025.229798] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6025.232580] pending: neigh_periodic_work [ 6025.234044] workqueue mm_percpu_wq: flags=0x8 [ 6025.235392] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6025.237470] pending: vmstat_update, lru_add_drain_per_cpu [ 6025.239266] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6025.242049] pending: vmstat_update [ 6025.243218] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6025.245301] pending: lru_add_drain_per_cpu [ 6025.246714] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6025.248864] pending: lru_add_drain_per_cpu BAR(48) [ 6025.250467] workqueue writeback: flags=0x4a [ 6025.251823] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 6025.253627] pending: wb_workfn [ 6025.254566] workqueue xprtiod: flags=0xa [ 6025.256076] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6025.258149] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6025.260927] workqueue ipv6_addrconf: flags=0x40008 [ 6025.262403] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6025.264360] pending: addrconf_verify_work [ 6025.265761] workqueue knbd0-recv: flags=0x1a [ 6025.267874] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6025.270395] in-flight: 658:recv_work [ 6025.272348] workqueue ptlrpc_pinger: flags=0x2 [ 6025.274131] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6025.276361] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6025.278601] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4094s workers=4 idle: 32984 37 8 [ 6025.281557] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6010s workers=2 idle: 661 [ 6028.055024] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 6028.058546] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6028.084021] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6028.093906] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6028.100350] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6028.104564] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6028.118088] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6028.122680] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 6028.127974] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 6028.130911] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 6028.134371] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 6028.140430] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6028.150613] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 6028.160740] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6028.169081] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6028.176963] Call Trace: [ 6028.178904] [ 6028.182071] ? show_regs.cold.9+0x22/0x2f [ 6028.184691] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6028.186238] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6028.189020] ? watchdog+0x40/0x40 [ 6028.191465] ? hrtimer_interrupt+0x128/0x2c0 [ 6028.192895] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6028.195184] ? apic_timer_interrupt+0xf/0x20 [ 6028.197470] [ 6028.199499] ? native_safe_halt+0x2e/0x50 [ 6028.201314] kvm_wait+0xcd/0xe0 [ 6028.202379] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6028.206825] do_raw_spin_lock+0x177/0x190 [ 6028.208985] _raw_spin_lock+0x21/0x30 [ 6028.210966] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6028.215701] obd_mod_free+0x46/0x140 [ptlrpc] [ 6028.218322] mdc_commit_open+0x91/0xb0 [mdc] [ 6028.222041] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6028.227159] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6028.230838] after_reply+0xe2c/0x1780 [ptlrpc] [ 6028.233456] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6028.236405] ? __next_timer_interrupt+0x160/0x160 [ 6028.238897] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6028.242560] ? do_wait_intr+0xf0/0xf0 [ 6028.252926] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6028.260992] kthread+0x1d1/0x200 [ 6028.268383] ? set_kthread_struct+0x70/0x70 [ 6028.273275] ret_from_fork+0x1f/0x30 [ 6032.058434] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6032.070924] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6032.138691] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6032.152789] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6032.162521] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6032.168778] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6032.173017] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6032.197100] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6032.201409] RAX: 0000000000007d47 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6032.207981] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 6032.211127] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 6032.219791] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 6032.224061] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 6032.230464] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 6032.235510] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6032.237538] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6032.244272] Call Trace: [ 6032.246154] [ 6032.247370] ? show_regs.cold.9+0x22/0x2f [ 6032.251367] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6032.255426] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6032.259968] ? watchdog+0x40/0x40 [ 6032.261849] ? hrtimer_interrupt+0x128/0x2c0 [ 6032.263868] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6032.266242] ? apic_timer_interrupt+0xf/0x20 [ 6032.267895] [ 6032.271242] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6032.275979] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6032.278987] do_raw_spin_lock+0x177/0x190 [ 6032.281308] _raw_spin_lock+0x21/0x30 [ 6032.285597] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6032.290459] process_one_work+0x2c8/0x700 [ 6032.293987] worker_thread+0x296/0x6e0 [ 6032.296232] ? rescuer_thread+0x570/0x570 [ 6032.298128] kthread+0x1d1/0x200 [ 6032.299744] ? set_kthread_struct+0x70/0x70 [ 6032.304918] ret_from_fork+0x1f/0x30 [ 6052.060201] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 6052.060933] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 6052.086736] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6052.086812] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6052.086820] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6052.086822] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6052.086835] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6052.086840] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6052.086845] RAX: 0000000000004585 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6052.086847] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 6052.086849] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 6052.086852] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 6052.086854] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 6052.086857] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 6052.086860] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6052.086863] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 6052.086871] Call Trace: [ 6052.086877] [ 6052.086883] ? show_regs.cold.9+0x22/0x2f [ 6052.086891] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6052.086897] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6052.086901] ? watchdog+0x40/0x40 [ 6052.086904] ? hrtimer_interrupt+0x128/0x2c0 [ 6052.086908] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6052.086916] ? apic_timer_interrupt+0xf/0x20 [ 6052.086921] [ 6052.086924] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6052.086929] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6052.086934] do_raw_spin_lock+0x177/0x190 [ 6052.086939] _raw_spin_lock+0x21/0x30 [ 6052.086945] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6052.087228] ? do_raw_spin_unlock+0x75/0x190 [ 6052.087234] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6052.087453] ? do_wait_intr+0xf0/0xf0 [ 6052.087461] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6052.087683] mdc_reint+0x2d/0x110 [mdc] [ 6052.087728] mdc_unlink+0x658/0x870 [mdc] [ 6052.087758] lmv_unlink+0x34d/0xbd0 [lmv] [ 6052.087788] ll_unlink+0x24d/0x7f0 [lustre] [ 6052.087912] vfs_unlink+0x202/0x330 [ 6052.087920] do_unlinkat+0x2fa/0x460 [ 6052.087926] __x64_sys_unlink+0x37/0x50 [ 6052.087930] do_syscall_64+0xc1/0x440 [ 6052.087937] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6052.087944] RIP: 0033:0x7f151bdee10b [ 6052.087950] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6052.087954] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6052.087959] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 6052.087961] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 6052.087963] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6052.087966] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6052.087968] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6052.506900] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6052.530959] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6052.538165] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6052.542622] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6052.547104] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6052.556018] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6052.560161] RAX: 0000000000006c88 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6052.565687] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 6052.571074] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6052.578937] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 6052.582918] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 6052.587093] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 6052.591023] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6052.596076] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 6052.604182] Call Trace: [ 6052.606501] [ 6052.607697] ? show_regs.cold.9+0x22/0x2f [ 6052.609559] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6052.612257] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6052.615626] ? watchdog+0x40/0x40 [ 6052.623281] ? hrtimer_interrupt+0x128/0x2c0 [ 6052.630172] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6052.631816] ? apic_timer_interrupt+0xf/0x20 [ 6052.633045] [ 6052.641636] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6052.649738] do_raw_spin_lock+0x177/0x190 [ 6052.653696] _raw_spin_lock+0x21/0x30 [ 6052.655637] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6052.657757] ? do_raw_spin_unlock+0x75/0x190 [ 6052.659229] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6052.664740] ? do_wait_intr+0xf0/0xf0 [ 6052.665903] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6052.669594] mdc_close+0x53b/0xd70 [mdc] [ 6052.671755] lmv_close+0x18d/0x490 [lmv] [ 6052.673662] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6052.680113] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6052.683741] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6052.686073] ll_md_close+0x274/0x800 [lustre] [ 6052.688711] ll_file_release+0x128/0xc50 [lustre] [ 6052.691623] __fput+0x10d/0x440 [ 6052.694204] ____fput+0x12/0x20 [ 6052.695509] task_work_run+0xc6/0x110 [ 6052.697027] exit_to_usermode_loop+0x1dd/0x1f0 [ 6052.698936] do_syscall_64+0x3d6/0x440 [ 6052.700765] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6052.703604] RIP: 0033:0x7f151bee3cd8 [ 6052.705132] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6052.721285] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6052.726953] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 6052.729214] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 6052.733076] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6052.736300] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6052.739580] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6055.393946] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4124s! [ 6055.415052] Showing busy workqueues and worker pools: [ 6055.416921] workqueue events: flags=0x0 [ 6055.427330] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6055.435278] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 6055.446487] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6055.460777] pending: kfree_rcu_monitor [ 6055.474293] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6055.493254] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 6055.515068] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6055.533278] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 6055.551459] workqueue events_long: flags=0x0 [ 6055.556199] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6055.562258] pending: lu_objects_flush [obdclass] [ 6055.565623] workqueue events_unbound: flags=0x2 [ 6055.568135] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6055.575473] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6055.589889] workqueue events_power_efficient: flags=0x80 [ 6055.595279] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6055.599141] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 6055.603693] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6055.609440] pending: neigh_periodic_work [ 6055.612081] workqueue mm_percpu_wq: flags=0x8 [ 6055.616540] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6055.621207] pending: vmstat_update, lru_add_drain_per_cpu [ 6055.628111] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6055.632087] pending: vmstat_update [ 6055.638071] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6055.645252] pending: lru_add_drain_per_cpu [ 6055.646687] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6055.654050] pending: lru_add_drain_per_cpu BAR(48) [ 6055.664029] workqueue writeback: flags=0x4a [ 6055.667115] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 6055.681199] pending: wb_workfn [ 6055.685895] workqueue xprtiod: flags=0xa [ 6055.687842] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6055.698455] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6055.714724] workqueue ipv6_addrconf: flags=0x40008 [ 6055.721636] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6055.734253] pending: addrconf_verify_work [ 6055.739993] workqueue knbd0-recv: flags=0x1a [ 6055.745421] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6055.752438] in-flight: 658:recv_work [ 6055.756349] workqueue ptlrpc_pinger: flags=0x2 [ 6055.757577] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6055.759203] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6055.768671] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4125s workers=4 idle: 32984 37 8 [ 6055.788246] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6040s workers=2 idle: 661 [ 6056.055126] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 6056.059469] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6056.108286] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6056.124191] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6056.131015] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6056.136655] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6056.167724] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6056.178516] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 6056.185130] RDX: 0000000000003775 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 6056.192239] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 6056.199931] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 6056.208037] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6056.214144] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 6056.225044] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6056.228870] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6056.236369] Call Trace: [ 6056.237064] [ 6056.238005] ? show_regs.cold.9+0x22/0x2f [ 6056.243432] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6056.246732] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6056.256057] ? watchdog+0x40/0x40 [ 6056.258240] ? hrtimer_interrupt+0x128/0x2c0 [ 6056.261497] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6056.264889] ? apic_timer_interrupt+0xf/0x20 [ 6056.273488] [ 6056.282636] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6056.286566] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6056.290334] do_raw_spin_lock+0x177/0x190 [ 6056.291961] _raw_spin_lock+0x21/0x30 [ 6056.296011] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6056.303274] obd_mod_free+0x46/0x140 [ptlrpc] [ 6056.306195] mdc_commit_open+0x91/0xb0 [mdc] [ 6056.308230] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6056.314431] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6056.316930] after_reply+0xe2c/0x1780 [ptlrpc] [ 6056.319629] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6056.322502] ? __next_timer_interrupt+0x160/0x160 [ 6056.325009] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6056.328737] ? do_wait_intr+0xf0/0xf0 [ 6056.329856] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6056.334111] kthread+0x1d1/0x200 [ 6056.336434] ? set_kthread_struct+0x70/0x70 [ 6056.338474] ret_from_fork+0x1f/0x30 [ 6060.060222] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6060.065489] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6060.132269] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6060.141833] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6060.159850] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6060.165129] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6060.167558] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6060.201701] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6060.207375] RAX: 0000000000007e05 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6060.209779] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 6060.227426] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 6060.240241] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 6060.254207] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 6060.265617] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 6060.272290] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6060.284065] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6060.305670] Call Trace: [ 6060.311022] [ 6060.313147] ? show_regs.cold.9+0x22/0x2f [ 6060.327356] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6060.329078] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6060.345253] ? watchdog+0x40/0x40 [ 6060.353542] ? hrtimer_interrupt+0x128/0x2c0 [ 6060.363770] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6060.379070] ? apic_timer_interrupt+0xf/0x20 [ 6060.387836] [ 6060.396596] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6060.411638] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6060.415511] do_raw_spin_lock+0x177/0x190 [ 6060.421275] _raw_spin_lock+0x21/0x30 [ 6060.431300] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6060.434908] process_one_work+0x2c8/0x700 [ 6060.436793] worker_thread+0x296/0x6e0 [ 6060.447966] ? rescuer_thread+0x570/0x570 [ 6060.451146] kthread+0x1d1/0x200 [ 6060.464816] ? set_kthread_struct+0x70/0x70 [ 6060.472444] ret_from_fork+0x1f/0x30 [ 6080.053436] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 6080.060350] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6080.061215] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 6080.117183] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6080.117200] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6080.117202] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6080.117227] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6080.117231] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6080.117237] RAX: 000000000000680b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6080.117240] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 6080.117242] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 6080.117244] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 6080.117246] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 6080.117250] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 6080.117252] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6080.117255] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 6080.117263] Call Trace: [ 6080.117269] [ 6080.117274] ? show_regs.cold.9+0x22/0x2f [ 6080.117283] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6080.117290] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6080.117294] ? watchdog+0x40/0x40 [ 6080.117298] ? hrtimer_interrupt+0x128/0x2c0 [ 6080.117302] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6080.117310] ? apic_timer_interrupt+0xf/0x20 [ 6080.117315] [ 6080.117318] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6080.117323] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6080.117328] do_raw_spin_lock+0x177/0x190 [ 6080.117333] _raw_spin_lock+0x21/0x30 [ 6080.117341] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6080.117645] ? do_raw_spin_unlock+0x75/0x190 [ 6080.117651] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6080.117869] ? do_wait_intr+0xf0/0xf0 [ 6080.117877] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6080.118079] mdc_reint+0x2d/0x110 [mdc] [ 6080.118117] mdc_unlink+0x658/0x870 [mdc] [ 6080.118143] lmv_unlink+0x34d/0xbd0 [lmv] [ 6080.118174] ll_unlink+0x24d/0x7f0 [lustre] [ 6080.118306] vfs_unlink+0x202/0x330 [ 6080.118315] do_unlinkat+0x2fa/0x460 [ 6080.118321] __x64_sys_unlink+0x37/0x50 [ 6080.118325] do_syscall_64+0xc1/0x440 [ 6080.118332] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6080.118338] RIP: 0033:0x7f151bdee10b [ 6080.118344] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6080.118349] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6080.118354] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 6080.118357] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 6080.118359] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6080.118361] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6080.118363] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6080.374502] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6080.403771] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6080.410680] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6080.419291] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6080.424096] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6080.452407] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6080.456632] RAX: 0000000000004083 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6080.465754] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 6080.471654] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6080.482023] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 6080.492865] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 6080.500278] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 6080.504537] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6080.508491] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 6080.514374] Call Trace: [ 6080.515794] [ 6080.516883] ? show_regs.cold.9+0x22/0x2f [ 6080.518990] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6080.523241] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6080.527861] ? watchdog+0x40/0x40 [ 6080.530405] ? hrtimer_interrupt+0x128/0x2c0 [ 6080.534463] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6080.536420] ? apic_timer_interrupt+0xf/0x20 [ 6080.537495] [ 6080.538009] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6080.540294] do_raw_spin_lock+0x177/0x190 [ 6080.541765] _raw_spin_lock+0x21/0x30 [ 6080.547373] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6080.550879] ? do_raw_spin_unlock+0x75/0x190 [ 6080.554418] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6080.558965] ? do_wait_intr+0xf0/0xf0 [ 6080.564237] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6080.570150] mdc_close+0x53b/0xd70 [mdc] [ 6080.573693] lmv_close+0x18d/0x490 [lmv] [ 6080.577610] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6080.584649] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6080.589561] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6080.591294] ll_md_close+0x274/0x800 [lustre] [ 6080.592536] ll_file_release+0x128/0xc50 [lustre] [ 6080.593997] __fput+0x10d/0x440 [ 6080.595558] ____fput+0x12/0x20 [ 6080.596769] task_work_run+0xc6/0x110 [ 6080.598394] exit_to_usermode_loop+0x1dd/0x1f0 [ 6080.601002] do_syscall_64+0x3d6/0x440 [ 6080.605506] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6080.608470] RIP: 0033:0x7f151bee3cd8 [ 6080.609755] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6080.620844] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6080.627941] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 6080.630944] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 6080.633439] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6080.637555] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6080.640937] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6084.055113] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 6084.065963] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6084.111284] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6084.126382] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6084.141455] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6084.150803] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6084.178272] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6084.186485] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 6084.194306] RDX: 000000000000670c RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 6084.202665] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 6084.207293] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 6084.210644] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6084.214161] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 6084.217807] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6084.221306] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6084.224892] Call Trace: [ 6084.225960] [ 6084.227297] ? show_regs.cold.9+0x22/0x2f [ 6084.228948] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6084.231737] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6084.235082] ? watchdog+0x40/0x40 [ 6084.236786] ? hrtimer_interrupt+0x128/0x2c0 [ 6084.239308] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6084.242118] ? apic_timer_interrupt+0xf/0x20 [ 6084.243752] [ 6084.245429] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6084.250321] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6084.254217] do_raw_spin_lock+0x177/0x190 [ 6084.257244] _raw_spin_lock+0x21/0x30 [ 6084.259649] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6084.265612] obd_mod_free+0x46/0x140 [ptlrpc] [ 6084.270573] mdc_commit_open+0x91/0xb0 [mdc] [ 6084.273939] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6084.276612] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6084.279883] after_reply+0xe2c/0x1780 [ptlrpc] [ 6084.282711] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6084.285511] ? __next_timer_interrupt+0x160/0x160 [ 6084.287481] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6084.289581] ? do_wait_intr+0xf0/0xf0 [ 6084.290942] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6084.292948] kthread+0x1d1/0x200 [ 6084.294016] ? set_kthread_struct+0x70/0x70 [ 6084.299507] ret_from_fork+0x1f/0x30 [ 6086.117157] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4155s! [ 6086.127335] Showing busy workqueues and worker pools: [ 6086.134621] workqueue events: flags=0x0 [ 6086.142093] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6086.151718] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 6086.164337] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6086.170642] pending: kfree_rcu_monitor [ 6086.176936] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6086.180918] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 6086.195217] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6086.203835] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 6086.213725] workqueue events_long: flags=0x0 [ 6086.218597] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6086.226311] pending: lu_objects_flush [obdclass] [ 6086.229266] workqueue events_unbound: flags=0x2 [ 6086.231888] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6086.238583] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6086.243489] workqueue events_power_efficient: flags=0x80 [ 6086.247886] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6086.254352] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 6086.263959] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6086.266631] pending: neigh_periodic_work [ 6086.278705] workqueue mm_percpu_wq: flags=0x8 [ 6086.285244] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6086.287570] pending: vmstat_update, lru_add_drain_per_cpu [ 6086.292168] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6086.303378] pending: vmstat_update [ 6086.312370] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6086.318003] pending: lru_add_drain_per_cpu [ 6086.319902] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6086.323843] pending: lru_add_drain_per_cpu BAR(48) [ 6086.327617] workqueue writeback: flags=0x4a [ 6086.329667] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 6086.335627] pending: wb_workfn [ 6086.337362] workqueue xprtiod: flags=0xa [ 6086.339902] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6086.344363] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6086.353126] workqueue ipv6_addrconf: flags=0x40008 [ 6086.361321] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6086.365639] pending: addrconf_verify_work [ 6086.370945] workqueue knbd0-recv: flags=0x1a [ 6086.375878] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6086.379076] in-flight: 658:recv_work [ 6086.382637] workqueue ptlrpc_pinger: flags=0x2 [ 6086.387434] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6086.390113] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6086.393104] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4155s workers=4 idle: 32984 37 8 [ 6086.396811] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6071s workers=2 idle: 661 [ 6088.057751] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6088.063589] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6088.117671] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6088.124756] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6088.132808] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6088.146929] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6088.152950] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6088.163988] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6088.167883] RAX: 0000000000004d8f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6088.175795] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 6088.180277] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 6088.183811] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 6088.187519] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 6088.190814] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 6088.194753] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6088.197721] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6088.201330] Call Trace: [ 6088.202604] [ 6088.203603] ? show_regs.cold.9+0x22/0x2f [ 6088.205531] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6088.207779] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6088.210170] ? watchdog+0x40/0x40 [ 6088.211769] ? hrtimer_interrupt+0x128/0x2c0 [ 6088.214304] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6088.216704] ? apic_timer_interrupt+0xf/0x20 [ 6088.221880] [ 6088.222717] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6088.229319] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6088.235904] do_raw_spin_lock+0x177/0x190 [ 6088.240095] _raw_spin_lock+0x21/0x30 [ 6088.241828] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6088.247323] process_one_work+0x2c8/0x700 [ 6088.251339] worker_thread+0x296/0x6e0 [ 6088.252762] ? rescuer_thread+0x570/0x570 [ 6088.254073] kthread+0x1d1/0x200 [ 6088.255181] ? set_kthread_struct+0x70/0x70 [ 6088.261334] ret_from_fork+0x1f/0x30 [ 6108.057472] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 6108.061372] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6108.098954] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6108.109637] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6108.112708] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6108.115669] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6108.124270] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6108.130148] RAX: 0000000000006789 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6108.133311] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 6108.136366] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6108.139504] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 6108.143433] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 6108.149132] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 6108.152924] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6108.158616] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 6108.161380] Call Trace: [ 6108.162419] [ 6108.163459] ? show_regs.cold.9+0x22/0x2f [ 6108.166529] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6108.168058] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6108.169815] ? watchdog+0x40/0x40 [ 6108.171029] ? hrtimer_interrupt+0x128/0x2c0 [ 6108.173023] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6108.175683] ? apic_timer_interrupt+0xf/0x20 [ 6108.179776] [ 6108.181714] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6108.186353] do_raw_spin_lock+0x177/0x190 [ 6108.188578] _raw_spin_lock+0x21/0x30 [ 6108.190319] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6108.194022] ? do_raw_spin_unlock+0x75/0x190 [ 6108.196036] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6108.198534] ? do_wait_intr+0xf0/0xf0 [ 6108.201760] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6108.207365] mdc_close+0x53b/0xd70 [mdc] [ 6108.210537] lmv_close+0x18d/0x490 [lmv] [ 6108.211967] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6108.213755] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6108.217973] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6108.220111] ll_md_close+0x274/0x800 [lustre] [ 6108.222264] ll_file_release+0x128/0xc50 [lustre] [ 6108.224544] __fput+0x10d/0x440 [ 6108.226029] ____fput+0x12/0x20 [ 6108.228787] task_work_run+0xc6/0x110 [ 6108.230342] exit_to_usermode_loop+0x1dd/0x1f0 [ 6108.232137] do_syscall_64+0x3d6/0x440 [ 6108.233431] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6108.236033] RIP: 0033:0x7f151bee3cd8 [ 6108.237521] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6108.247519] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6108.253391] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 6108.257112] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 6108.260322] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6108.267489] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6108.274927] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6112.055077] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 6112.058270] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 6112.065127] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6112.065202] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6112.065209] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6112.065212] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6112.065226] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6112.065231] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6112.065235] RAX: 0000000000007f14 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6112.065237] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 6112.065239] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 6112.065241] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 6112.065243] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 6112.065246] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 6112.065249] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6112.065251] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 6112.065259] Call Trace: [ 6112.065264] [ 6112.065269] ? show_regs.cold.9+0x22/0x2f [ 6112.065279] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6112.065286] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6112.065290] ? watchdog+0x40/0x40 [ 6112.065295] ? hrtimer_interrupt+0x128/0x2c0 [ 6112.065299] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6112.065318] ? apic_timer_interrupt+0xf/0x20 [ 6112.065323] [ 6112.065327] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6112.065332] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6112.065336] do_raw_spin_lock+0x177/0x190 [ 6112.065341] _raw_spin_lock+0x21/0x30 [ 6112.065346] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6112.065664] ? do_raw_spin_unlock+0x75/0x190 [ 6112.065669] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6112.065874] ? do_wait_intr+0xf0/0xf0 [ 6112.065882] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6112.066101] mdc_reint+0x2d/0x110 [mdc] [ 6112.066143] mdc_unlink+0x658/0x870 [mdc] [ 6112.066169] lmv_unlink+0x34d/0xbd0 [lmv] [ 6112.066201] ll_unlink+0x24d/0x7f0 [lustre] [ 6112.066314] vfs_unlink+0x202/0x330 [ 6112.066322] do_unlinkat+0x2fa/0x460 [ 6112.066327] __x64_sys_unlink+0x37/0x50 [ 6112.066330] do_syscall_64+0xc1/0x440 [ 6112.066336] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6112.066344] RIP: 0033:0x7f151bdee10b [ 6112.066352] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6112.066355] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6112.066360] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 6112.066363] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 6112.066365] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6112.066367] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6112.066370] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6112.261698] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6112.284058] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6112.288609] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6112.291568] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6112.294126] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6112.302405] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6112.306034] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 6112.309670] RDX: 0000000000005c31 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 6112.312473] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 6112.315879] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 6112.322156] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6112.327718] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 6112.331730] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6112.335324] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6112.339729] Call Trace: [ 6112.341232] [ 6112.342178] ? show_regs.cold.9+0x22/0x2f [ 6112.344251] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6112.346929] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6112.351189] ? watchdog+0x40/0x40 [ 6112.352530] ? hrtimer_interrupt+0x128/0x2c0 [ 6112.354284] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6112.359241] ? apic_timer_interrupt+0xf/0x20 [ 6112.362188] [ 6112.364906] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6112.367907] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6112.371235] do_raw_spin_lock+0x177/0x190 [ 6112.375502] _raw_spin_lock+0x21/0x30 [ 6112.376680] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6112.378535] obd_mod_free+0x46/0x140 [ptlrpc] [ 6112.383238] mdc_commit_open+0x91/0xb0 [mdc] [ 6112.385644] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6112.391137] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6112.394104] after_reply+0xe2c/0x1780 [ptlrpc] [ 6112.396318] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6112.398694] ? __next_timer_interrupt+0x160/0x160 [ 6112.400957] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6112.403026] ? do_wait_intr+0xf0/0xf0 [ 6112.404731] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6112.408346] kthread+0x1d1/0x200 [ 6112.409911] ? set_kthread_struct+0x70/0x70 [ 6112.412158] ret_from_fork+0x1f/0x30 [ 6116.059011] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 6116.062638] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6116.097066] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6116.106352] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6116.112834] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6116.117069] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6116.121490] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6116.130351] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6116.133253] RAX: 0000000000007107 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6116.135917] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 6116.138620] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 6116.141367] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 6116.144850] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 6116.150051] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 6116.153415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6116.155512] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6116.161962] Call Trace: [ 6116.163460] [ 6116.164225] ? show_regs.cold.9+0x22/0x2f [ 6116.165886] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6116.167823] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6116.169649] ? watchdog+0x40/0x40 [ 6116.171252] ? hrtimer_interrupt+0x128/0x2c0 [ 6116.172842] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6116.174512] ? apic_timer_interrupt+0xf/0x20 [ 6116.177441] [ 6116.178431] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6116.180091] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6116.182098] do_raw_spin_lock+0x177/0x190 [ 6116.184157] _raw_spin_lock+0x21/0x30 [ 6116.186155] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6116.191637] process_one_work+0x2c8/0x700 [ 6116.192905] worker_thread+0x296/0x6e0 [ 6116.196225] ? rescuer_thread+0x570/0x570 [ 6116.198586] kthread+0x1d1/0x200 [ 6116.199929] ? set_kthread_struct+0x70/0x70 [ 6116.201596] ret_from_fork+0x1f/0x30 [ 6116.832169] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4186s! [ 6116.837221] Showing busy workqueues and worker pools: [ 6116.840373] workqueue events: flags=0x0 [ 6116.842574] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6116.844668] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 6116.849414] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6116.852325] pending: kfree_rcu_monitor [ 6116.853838] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6116.856388] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 6116.859701] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6116.863318] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 6116.873241] workqueue events_long: flags=0x0 [ 6116.877955] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6116.882141] pending: lu_objects_flush [obdclass] [ 6116.884965] workqueue events_unbound: flags=0x2 [ 6116.887387] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6116.890372] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6116.895922] workqueue events_power_efficient: flags=0x80 [ 6116.898166] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6116.901181] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 6116.904618] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6116.907840] pending: neigh_periodic_work [ 6116.909877] workqueue mm_percpu_wq: flags=0x8 [ 6116.911972] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6116.914673] pending: vmstat_update, lru_add_drain_per_cpu [ 6116.917234] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6116.920504] pending: vmstat_update [ 6116.922342] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6116.925708] pending: lru_add_drain_per_cpu [ 6116.928219] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6116.931548] pending: lru_add_drain_per_cpu BAR(48) [ 6116.934030] workqueue writeback: flags=0x4a [ 6116.936103] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 6116.938961] pending: wb_workfn [ 6116.940502] workqueue xprtiod: flags=0xa [ 6116.942227] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6116.944896] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6116.949180] workqueue ipv6_addrconf: flags=0x40008 [ 6116.952342] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6116.958905] pending: addrconf_verify_work [ 6116.962014] workqueue knbd0-recv: flags=0x1a [ 6116.964650] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6116.966387] in-flight: 658:recv_work [ 6116.967611] workqueue ptlrpc_pinger: flags=0x2 [ 6116.969776] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6116.972582] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6116.975552] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4186s workers=4 idle: 32984 37 8 [ 6116.978842] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6101s workers=2 idle: 661 [ 6124.725153] rcu: INFO: rcu_sched self-detected stall on CPU [ 6124.728350] rcu: 1-...!: (2426344 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 6124.735371] (t=4200078 jiffies g=249657 q=3560) [ 6124.739449] rcu: rcu_sched kthread starved for 4195030 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 6124.748244] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 6124.753567] rcu: RCU grace-period kthread stack dump: [ 6124.759192] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 6124.767286] Call Trace: [ 6124.771240] __schedule+0x351/0xcb0 [ 6124.773495] schedule+0xc0/0x180 [ 6124.776536] schedule_timeout+0xb4/0x190 [ 6124.779967] ? do_raw_spin_unlock+0x75/0x190 [ 6124.784555] ? __next_timer_interrupt+0x160/0x160 [ 6124.788250] ? prepare_to_swait_event+0xf3/0x140 [ 6124.794004] rcu_gp_kthread+0x8dd/0x15b0 [ 6124.795472] ? force_qs_rnp+0x2c0/0x2c0 [ 6124.799608] kthread+0x1d1/0x200 [ 6124.801950] ? set_kthread_struct+0x70/0x70 [ 6124.806906] ret_from_fork+0x1f/0x30 [ 6124.809377] rcu: Stack dump where RCU GP kthread last ran: [ 6124.814989] Sending NMI from CPU 1 to CPUs 0: [ 6124.818912] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [ 6124.820786] NMI backtrace for cpu 1 [ 6124.824378] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6124.833389] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6124.841466] Call Trace: [ 6124.843594] [ 6124.845460] ? dump_stack+0xbb/0x10e [ 6124.846685] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6124.852538] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6124.858124] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6124.861189] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6124.866232] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6124.869320] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 6124.874981] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 6124.877577] ? update_process_times+0x6d/0xc0 [ 6124.883354] ? tick_sched_handle.isra.18+0x27/0x90 [ 6124.888613] ? tick_sched_timer+0x53/0xb0 [ 6124.890158] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6124.895612] ? tick_sched_handle.isra.18+0x90/0x90 [ 6124.899940] ? hrtimer_interrupt+0x128/0x2c0 [ 6124.904408] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6124.908881] ? apic_timer_interrupt+0xf/0x20 [ 6124.911368] [ 6124.912334] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6124.917647] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6124.922324] ? do_raw_spin_lock+0x177/0x190 [ 6124.925495] ? _raw_spin_lock+0x21/0x30 [ 6124.929231] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6124.931153] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 6124.939042] ? mdc_commit_open+0x91/0xb0 [mdc] [ 6124.944271] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6124.946876] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6124.952866] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 6124.959062] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6124.961824] ? __next_timer_interrupt+0x160/0x160 [ 6124.965549] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6124.969902] ? do_wait_intr+0xf0/0xf0 [ 6124.971972] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6124.974146] ? kthread+0x1d1/0x200 [ 6124.975383] ? set_kthread_struct+0x70/0x70 [ 6124.978211] ? ret_from_fork+0x1f/0x30 [ 6136.057232] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 6136.066743] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6136.108058] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6136.113677] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6136.117147] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 6136.121533] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 6136.134793] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6136.138139] RAX: 0000000000000308 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6136.143367] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 6136.147508] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6136.151382] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 6136.159963] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 6136.165063] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 6136.171430] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6136.176022] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 6136.179520] Call Trace: [ 6136.182115] [ 6136.184275] ? show_regs.cold.9+0x22/0x2f [ 6136.186849] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6136.192318] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6136.195764] ? watchdog+0x40/0x40 [ 6136.197496] ? hrtimer_interrupt+0x128/0x2c0 [ 6136.201896] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6136.206071] ? apic_timer_interrupt+0xf/0x20 [ 6136.208250] [ 6136.210084] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 6136.212217] do_raw_spin_lock+0x177/0x190 [ 6136.214433] _raw_spin_lock+0x21/0x30 [ 6136.215790] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6136.218350] ? do_raw_spin_unlock+0x75/0x190 [ 6136.220284] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6136.224732] ? do_wait_intr+0xf0/0xf0 [ 6136.226646] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6136.229341] mdc_close+0x53b/0xd70 [mdc] [ 6136.233804] lmv_close+0x18d/0x490 [lmv] [ 6136.236812] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6136.242436] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6136.248846] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6136.253309] ll_md_close+0x274/0x800 [lustre] [ 6136.260459] ll_file_release+0x128/0xc50 [lustre] [ 6136.266796] __fput+0x10d/0x440 [ 6136.271314] ____fput+0x12/0x20 [ 6136.274302] task_work_run+0xc6/0x110 [ 6136.279250] exit_to_usermode_loop+0x1dd/0x1f0 [ 6136.283455] do_syscall_64+0x3d6/0x440 [ 6136.289181] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6136.293858] RIP: 0033:0x7f151bee3cd8 [ 6136.295802] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6136.324105] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6136.330128] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 6136.335705] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 6136.343827] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6136.348993] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6136.354867] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6144.056147] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 6144.064134] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6144.087405] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6144.091897] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6144.095775] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6144.098714] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6144.101075] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6144.108966] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6144.112105] RAX: 000000000000340a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6144.115182] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 6144.118387] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 6144.121320] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 6144.124744] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 6144.127624] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 6144.130721] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6144.133064] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6144.135301] Call Trace: [ 6144.136548] [ 6144.137318] ? show_regs.cold.9+0x22/0x2f [ 6144.139060] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6144.141075] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6144.143004] ? watchdog+0x40/0x40 [ 6144.144076] ? hrtimer_interrupt+0x128/0x2c0 [ 6144.145955] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6144.147837] ? apic_timer_interrupt+0xf/0x20 [ 6144.149679] [ 6144.150637] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6144.152748] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6144.154895] do_raw_spin_lock+0x177/0x190 [ 6144.156678] _raw_spin_lock+0x21/0x30 [ 6144.158056] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6144.160612] process_one_work+0x2c8/0x700 [ 6144.162287] worker_thread+0x296/0x6e0 [ 6144.163783] ? rescuer_thread+0x570/0x570 [ 6144.165556] kthread+0x1d1/0x200 [ 6144.166879] ? set_kthread_struct+0x70/0x70 [ 6144.168739] ret_from_fork+0x1f/0x30 [ 6147.040616] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4216s! [ 6147.046992] Showing busy workqueues and worker pools: [ 6147.049693] workqueue events: flags=0x0 [ 6147.051880] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6147.054932] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 6147.059406] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6147.062784] pending: kfree_rcu_monitor [ 6147.064643] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6147.067622] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 6147.071906] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6147.074658] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 6147.078011] workqueue events_long: flags=0x0 [ 6147.080013] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6147.083073] pending: lu_objects_flush [obdclass] [ 6147.085811] workqueue events_unbound: flags=0x2 [ 6147.088054] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6147.090982] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6147.095579] workqueue events_power_efficient: flags=0x80 [ 6147.097774] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6147.101085] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 6147.104462] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6147.107009] pending: neigh_periodic_work [ 6147.108747] workqueue mm_percpu_wq: flags=0x8 [ 6147.110391] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6147.113307] pending: vmstat_update, lru_add_drain_per_cpu [ 6147.115980] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6147.118597] pending: vmstat_update [ 6147.120368] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6147.123328] pending: lru_add_drain_per_cpu [ 6147.125389] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6147.128096] pending: lru_add_drain_per_cpu BAR(48) [ 6147.129913] workqueue writeback: flags=0x4a [ 6147.132011] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 6147.134929] pending: wb_workfn [ 6147.136371] workqueue xprtiod: flags=0xa [ 6147.138695] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6147.141532] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6147.144979] workqueue ipv6_addrconf: flags=0x40008 [ 6147.147101] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6147.149974] pending: addrconf_verify_work [ 6147.152052] workqueue knbd0-recv: flags=0x1a [ 6147.154013] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6147.156800] in-flight: 658:recv_work [ 6147.158709] workqueue ptlrpc_pinger: flags=0x2 [ 6147.160376] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6147.163248] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6147.166090] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4216s workers=4 idle: 32984 37 8 [ 6147.170067] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6131s workers=2 idle: 661 [ 6152.056233] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 6152.068376] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6152.096909] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6152.104825] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6152.111515] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6152.115077] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6152.144740] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6152.151990] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 6152.164902] RDX: 0000000000007a7c RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 6152.172431] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 6152.176649] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 6152.179881] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6152.182684] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 6152.186502] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6152.188124] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6152.190683] Call Trace: [ 6152.192855] [ 6152.193945] ? show_regs.cold.9+0x22/0x2f [ 6152.196056] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6152.198830] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6152.201519] ? watchdog+0x40/0x40 [ 6152.204002] ? hrtimer_interrupt+0x128/0x2c0 [ 6152.206654] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6152.211678] ? apic_timer_interrupt+0xf/0x20 [ 6152.214478] [ 6152.215672] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6152.218784] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6152.223675] do_raw_spin_lock+0x177/0x190 [ 6152.228010] _raw_spin_lock+0x21/0x30 [ 6152.230666] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6152.237095] obd_mod_free+0x46/0x140 [ptlrpc] [ 6152.242853] mdc_commit_open+0x91/0xb0 [mdc] [ 6152.245615] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6152.249663] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6152.254122] after_reply+0xe2c/0x1780 [ptlrpc] [ 6152.257596] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6152.261544] ? __next_timer_interrupt+0x160/0x160 [ 6152.267369] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6152.271450] ? do_wait_intr+0xf0/0xf0 [ 6152.274909] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6152.277956] kthread+0x1d1/0x200 [ 6152.280052] ? set_kthread_struct+0x70/0x70 [ 6152.282565] ret_from_fork+0x1f/0x30 [ 6164.058294] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 6164.070599] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6164.118725] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6164.123896] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6164.127813] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6164.130803] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6164.143349] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6164.148600] RAX: 0000000000005a8a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6164.151923] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 6164.154492] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6164.156817] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 6164.159115] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 6164.161080] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 6164.164092] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6164.170073] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 6164.174961] Call Trace: [ 6164.176985] [ 6164.178767] ? show_regs.cold.9+0x22/0x2f [ 6164.180332] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6164.182684] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6164.184761] ? watchdog+0x40/0x40 [ 6164.186218] ? hrtimer_interrupt+0x128/0x2c0 [ 6164.188319] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6164.190617] ? apic_timer_interrupt+0xf/0x20 [ 6164.192618] [ 6164.193667] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6164.196360] do_raw_spin_lock+0x177/0x190 [ 6164.198652] _raw_spin_lock+0x21/0x30 [ 6164.200633] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6164.203593] ? do_raw_spin_unlock+0x75/0x190 [ 6164.205748] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6164.208517] ? do_wait_intr+0xf0/0xf0 [ 6164.210399] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6164.213141] mdc_close+0x53b/0xd70 [mdc] [ 6164.215007] lmv_close+0x18d/0x490 [lmv] [ 6164.217650] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6164.221340] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6164.224207] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6164.226647] ll_md_close+0x274/0x800 [lustre] [ 6164.233031] ll_file_release+0x128/0xc50 [lustre] [ 6164.240028] __fput+0x10d/0x440 [ 6164.241956] ____fput+0x12/0x20 [ 6164.243186] task_work_run+0xc6/0x110 [ 6164.244996] exit_to_usermode_loop+0x1dd/0x1f0 [ 6164.247237] do_syscall_64+0x3d6/0x440 [ 6164.249429] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6164.254674] RIP: 0033:0x7f151bee3cd8 [ 6164.257956] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6164.270744] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6164.274418] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 6164.277819] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 6164.281082] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6164.283801] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6164.289798] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6172.052012] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 6172.056791] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6172.058408] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 6172.080400] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6172.080415] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6172.080418] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6172.080432] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6172.080436] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6172.080442] RAX: 0000000000002e66 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6172.080444] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 6172.080447] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 6172.080449] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 6172.080451] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 6172.080454] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 6172.080457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6172.080460] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 6172.080469] Call Trace: [ 6172.080474] [ 6172.080479] ? show_regs.cold.9+0x22/0x2f [ 6172.080489] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6172.080496] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6172.080500] ? watchdog+0x40/0x40 [ 6172.080504] ? hrtimer_interrupt+0x128/0x2c0 [ 6172.080509] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6172.080518] ? apic_timer_interrupt+0xf/0x20 [ 6172.080522] [ 6172.080526] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6172.080530] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6172.080535] do_raw_spin_lock+0x177/0x190 [ 6172.080540] _raw_spin_lock+0x21/0x30 [ 6172.080548] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6172.084985] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6172.085060] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6172.085069] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6172.085075] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6172.085349] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6172.085360] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6172.085364] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6172.085368] RAX: 0000000000007488 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6172.085371] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 6172.085373] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 6172.085375] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 6172.085378] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 6172.085381] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 6172.085384] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6172.085387] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6172.085395] Call Trace: [ 6172.085401] [ 6172.085406] ? show_regs.cold.9+0x22/0x2f [ 6172.085414] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6172.085421] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6172.085426] ? watchdog+0x40/0x40 [ 6172.085430] ? hrtimer_interrupt+0x128/0x2c0 [ 6172.085434] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6172.085442] ? apic_timer_interrupt+0xf/0x20 [ 6172.085447] [ 6172.085449] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6172.085454] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6172.085459] do_raw_spin_lock+0x177/0x190 [ 6172.085465] _raw_spin_lock+0x21/0x30 [ 6172.085469] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6172.085682] process_one_work+0x2c8/0x700 [ 6172.085689] worker_thread+0x296/0x6e0 [ 6172.085693] ? rescuer_thread+0x570/0x570 [ 6172.085697] kthread+0x1d1/0x200 [ 6172.085701] ? set_kthread_struct+0x70/0x70 [ 6172.085705] ret_from_fork+0x1f/0x30 [ 6172.421530] ? do_raw_spin_unlock+0x75/0x190 [ 6172.424914] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6172.434872] ? do_wait_intr+0xf0/0xf0 [ 6172.439909] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6172.450679] mdc_reint+0x2d/0x110 [mdc] [ 6172.453205] mdc_unlink+0x658/0x870 [mdc] [ 6172.454942] lmv_unlink+0x34d/0xbd0 [lmv] [ 6172.458546] ll_unlink+0x24d/0x7f0 [lustre] [ 6172.465436] vfs_unlink+0x202/0x330 [ 6172.467075] do_unlinkat+0x2fa/0x460 [ 6172.471108] __x64_sys_unlink+0x37/0x50 [ 6172.474957] do_syscall_64+0xc1/0x440 [ 6172.480060] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6172.482336] RIP: 0033:0x7f151bdee10b [ 6172.487041] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6172.502582] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6172.510034] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 6172.518163] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 6172.528123] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6172.535504] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6172.539663] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6177.250812] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4246s! [ 6177.275613] Showing busy workqueues and worker pools: [ 6177.282225] workqueue events: flags=0x0 [ 6177.286608] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6177.288982] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 6177.300699] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6177.311603] pending: kfree_rcu_monitor [ 6177.317703] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6177.328640] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 6177.343717] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6177.348655] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 6177.357581] workqueue events_long: flags=0x0 [ 6177.361575] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6177.366627] pending: lu_objects_flush [obdclass] [ 6177.373199] workqueue events_unbound: flags=0x2 [ 6177.374948] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6177.386719] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6177.400589] workqueue events_power_efficient: flags=0x80 [ 6177.406354] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6177.418576] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 6177.431118] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6177.446354] pending: neigh_periodic_work [ 6177.448736] workqueue mm_percpu_wq: flags=0x8 [ 6177.450381] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6177.455700] pending: vmstat_update, lru_add_drain_per_cpu [ 6177.467526] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6177.476863] pending: vmstat_update [ 6177.479548] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6177.489478] pending: lru_add_drain_per_cpu [ 6177.495243] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6177.504370] pending: lru_add_drain_per_cpu BAR(48) [ 6177.508460] workqueue writeback: flags=0x4a [ 6177.511671] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 6177.517497] pending: wb_workfn [ 6177.519951] workqueue xprtiod: flags=0xa [ 6177.522245] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6177.524085] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6177.532894] workqueue ipv6_addrconf: flags=0x40008 [ 6177.541339] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6177.545983] pending: addrconf_verify_work [ 6177.550551] workqueue knbd0-recv: flags=0x1a [ 6177.556017] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6177.560704] in-flight: 658:recv_work [ 6177.567310] workqueue ptlrpc_pinger: flags=0x2 [ 6177.573303] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6177.579646] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6177.585516] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4246s workers=4 idle: 32984 37 8 [ 6177.596512] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6162s workers=2 idle: 661 [ 6180.055688] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 6180.065576] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6180.104680] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6180.115752] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6180.121844] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6180.124039] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6180.139759] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6180.144858] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 6180.148852] RDX: 000000000000399b RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 6180.151279] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 6180.166340] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 6180.174608] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6180.178756] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 6180.184518] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6180.190925] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6180.195736] Call Trace: [ 6180.197348] [ 6180.197960] ? show_regs.cold.9+0x22/0x2f [ 6180.200565] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6180.203508] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6180.205260] ? watchdog+0x40/0x40 [ 6180.206666] ? hrtimer_interrupt+0x128/0x2c0 [ 6180.208350] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6180.210640] ? apic_timer_interrupt+0xf/0x20 [ 6180.212747] [ 6180.213539] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6180.215960] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6180.218341] do_raw_spin_lock+0x177/0x190 [ 6180.219914] _raw_spin_lock+0x21/0x30 [ 6180.221300] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6180.223606] obd_mod_free+0x46/0x140 [ptlrpc] [ 6180.228298] mdc_commit_open+0x91/0xb0 [mdc] [ 6180.229914] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6180.233764] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6180.236859] after_reply+0xe2c/0x1780 [ptlrpc] [ 6180.240343] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6180.244540] ? __next_timer_interrupt+0x160/0x160 [ 6180.246224] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6180.257432] ? do_wait_intr+0xf0/0xf0 [ 6180.260080] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6180.263949] kthread+0x1d1/0x200 [ 6180.270340] ? set_kthread_struct+0x70/0x70 [ 6180.277347] ret_from_fork+0x1f/0x30 [ 6192.057071] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 6192.061196] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6192.089334] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6192.102280] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6192.115590] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6192.117775] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6192.135660] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6192.139502] RAX: 000000000000735c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6192.143739] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 6192.148392] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6192.151696] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 6192.157683] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 6192.168298] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 6192.177380] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6192.187868] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 6192.192963] Call Trace: [ 6192.193901] [ 6192.195301] ? show_regs.cold.9+0x22/0x2f [ 6192.197678] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6192.201455] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6192.203594] ? watchdog+0x40/0x40 [ 6192.205332] ? hrtimer_interrupt+0x128/0x2c0 [ 6192.207287] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6192.209457] ? apic_timer_interrupt+0xf/0x20 [ 6192.211630] [ 6192.212590] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6192.215366] do_raw_spin_lock+0x177/0x190 [ 6192.217284] _raw_spin_lock+0x21/0x30 [ 6192.218719] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6192.220689] ? do_raw_spin_unlock+0x75/0x190 [ 6192.222341] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6192.226905] ? do_wait_intr+0xf0/0xf0 [ 6192.228944] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6192.232690] mdc_close+0x53b/0xd70 [mdc] [ 6192.234942] lmv_close+0x18d/0x490 [lmv] [ 6192.238201] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6192.246613] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6192.251359] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6192.253680] ll_md_close+0x274/0x800 [lustre] [ 6192.256423] ll_file_release+0x128/0xc50 [lustre] [ 6192.260779] __fput+0x10d/0x440 [ 6192.262415] ____fput+0x12/0x20 [ 6192.266557] task_work_run+0xc6/0x110 [ 6192.268807] exit_to_usermode_loop+0x1dd/0x1f0 [ 6192.272290] do_syscall_64+0x3d6/0x440 [ 6192.273709] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6192.277278] RIP: 0033:0x7f151bee3cd8 [ 6192.279978] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6192.290552] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6192.294357] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 6192.297833] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 6192.301798] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6192.306015] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6192.313805] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6200.056238] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 6200.059286] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6200.093288] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6200.098924] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6200.104012] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6200.111798] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6200.114389] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6200.128971] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6200.135734] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6200.144703] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 6200.152640] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 6200.159122] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 6200.161685] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 6200.174447] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 6200.181335] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6200.188520] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6200.195513] Call Trace: [ 6200.197636] [ 6200.200087] ? show_regs.cold.9+0x22/0x2f [ 6200.207841] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6200.212763] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6200.219248] ? watchdog+0x40/0x40 [ 6200.222879] ? hrtimer_interrupt+0x128/0x2c0 [ 6200.225477] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6200.227764] ? apic_timer_interrupt+0xf/0x20 [ 6200.233758] [ 6200.234746] ? native_safe_halt+0x2e/0x50 [ 6200.241255] kvm_wait+0xcd/0xe0 [ 6200.243925] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6200.247763] do_raw_spin_lock+0x177/0x190 [ 6200.250569] _raw_spin_lock+0x21/0x30 [ 6200.253505] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6200.262321] process_one_work+0x2c8/0x700 [ 6200.265732] worker_thread+0x296/0x6e0 [ 6200.269894] ? rescuer_thread+0x570/0x570 [ 6200.271897] kthread+0x1d1/0x200 [ 6200.273459] ? set_kthread_struct+0x70/0x70 [ 6200.274860] ret_from_fork+0x1f/0x30 [ 6204.052027] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 6204.056199] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6204.106595] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6204.119416] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6204.127683] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6204.130521] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6204.142304] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6204.145221] RAX: 0000000000004589 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6204.148631] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 6204.154902] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 6204.157248] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 6204.160485] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 6204.163570] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 6204.168421] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6204.171759] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 6204.177201] Call Trace: [ 6204.178301] [ 6204.179081] ? show_regs.cold.9+0x22/0x2f [ 6204.184483] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6204.186919] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6204.192790] ? watchdog+0x40/0x40 [ 6204.194468] ? hrtimer_interrupt+0x128/0x2c0 [ 6204.197892] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6204.200735] ? apic_timer_interrupt+0xf/0x20 [ 6204.206698] [ 6204.207699] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6204.213655] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6204.216010] do_raw_spin_lock+0x177/0x190 [ 6204.218596] _raw_spin_lock+0x21/0x30 [ 6204.220191] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6204.223538] ? do_raw_spin_unlock+0x75/0x190 [ 6204.225260] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6204.228310] ? do_wait_intr+0xf0/0xf0 [ 6204.230571] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6204.232807] mdc_reint+0x2d/0x110 [mdc] [ 6204.234650] mdc_unlink+0x658/0x870 [mdc] [ 6204.236555] lmv_unlink+0x34d/0xbd0 [lmv] [ 6204.238376] ll_unlink+0x24d/0x7f0 [lustre] [ 6204.240777] vfs_unlink+0x202/0x330 [ 6204.242387] do_unlinkat+0x2fa/0x460 [ 6204.244493] __x64_sys_unlink+0x37/0x50 [ 6204.246310] do_syscall_64+0xc1/0x440 [ 6204.248164] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6204.250734] RIP: 0033:0x7f151bdee10b [ 6204.256709] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6204.279669] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6204.285453] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 6204.288483] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 6204.291134] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6204.294143] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6204.299037] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6207.970221] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4277s! [ 6207.972728] Showing busy workqueues and worker pools: [ 6207.975602] workqueue events: flags=0x0 [ 6207.977087] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6207.979938] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 6207.983825] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6207.988714] pending: kfree_rcu_monitor [ 6207.990079] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6207.992317] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 6207.996097] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6207.998288] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 6208.001395] workqueue events_long: flags=0x0 [ 6208.003667] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6208.005863] pending: lu_objects_flush [obdclass] [ 6208.008128] workqueue events_unbound: flags=0x2 [ 6208.010765] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6208.013483] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6208.020673] workqueue events_power_efficient: flags=0x80 [ 6208.023018] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6208.027814] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 6208.032298] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6208.035680] pending: neigh_periodic_work [ 6208.037806] workqueue mm_percpu_wq: flags=0x8 [ 6208.039409] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6208.043369] pending: vmstat_update, lru_add_drain_per_cpu [ 6208.048425] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6208.053374] pending: vmstat_update [ 6208.055592] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6208.057021] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 6208.059628] pending: lru_add_drain_per_cpu [ 6208.059671] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6208.059685] pending: lru_add_drain_per_cpu BAR(48) [ 6208.059698] workqueue writeback: flags=0x4a [ 6208.059701] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 6208.059709] pending: wb_workfn [ 6208.059722] workqueue xprtiod: flags=0xa [ 6208.059724] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6208.059730] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6208.059749] workqueue ipv6_addrconf: flags=0x40008 [ 6208.059751] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6208.059760] pending: addrconf_verify_work [ 6208.059771] workqueue knbd0-recv: flags=0x1a [ 6208.059773] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6208.059792] in-flight: 658:recv_work [ 6208.059802] workqueue ptlrpc_pinger: flags=0x2 [ 6208.059804] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6208.059810] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6208.066714] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6208.066789] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6208.066797] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6208.066800] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6208.066813] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6208.066818] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6208.066822] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 6208.066878] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 6208.066880] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 6208.066883] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 6208.066885] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6208.066888] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 6208.066890] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6208.066894] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6208.066901] Call Trace: [ 6208.066906] [ 6208.066912] ? show_regs.cold.9+0x22/0x2f [ 6208.066923] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6208.066929] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6208.066932] ? watchdog+0x40/0x40 [ 6208.066936] ? hrtimer_interrupt+0x128/0x2c0 [ 6208.066940] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6208.066945] ? apic_timer_interrupt+0xf/0x20 [ 6208.066949] [ 6208.066952] ? native_safe_halt+0x2e/0x50 [ 6208.066955] kvm_wait+0xcd/0xe0 [ 6208.066972] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6208.066981] do_raw_spin_lock+0x177/0x190 [ 6208.066986] _raw_spin_lock+0x21/0x30 [ 6208.066991] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6208.068904] [ 6208.068934] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4277s workers=4 idle: 32984 37 8 [ 6208.068953] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6192s workers=2 idle: 661 [ 6208.296028] obd_mod_free+0x46/0x140 [ptlrpc] [ 6208.297758] mdc_commit_open+0x91/0xb0 [mdc] [ 6208.300426] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6208.304775] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6208.310330] after_reply+0xe2c/0x1780 [ptlrpc] [ 6208.312240] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6208.314634] ? __next_timer_interrupt+0x160/0x160 [ 6208.316504] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6208.318971] ? do_wait_intr+0xf0/0xf0 [ 6208.320098] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6208.321869] kthread+0x1d1/0x200 [ 6208.323291] ? set_kthread_struct+0x70/0x70 [ 6208.327667] ret_from_fork+0x1f/0x30 [ 6220.057112] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 6220.062289] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6220.103528] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6220.108717] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6220.111623] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6220.118006] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6220.134536] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6220.139556] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6220.143363] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 6220.148964] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6220.156823] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 6220.166478] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 6220.173823] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 6220.177445] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6220.188902] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 6220.194158] Call Trace: [ 6220.196042] [ 6220.196975] ? show_regs.cold.9+0x22/0x2f [ 6220.200122] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6220.203380] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6220.205421] ? watchdog+0x40/0x40 [ 6220.207109] ? hrtimer_interrupt+0x128/0x2c0 [ 6220.211634] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6220.214410] ? apic_timer_interrupt+0xf/0x20 [ 6220.217153] [ 6220.218076] ? native_safe_halt+0x2e/0x50 [ 6220.221167] kvm_wait+0xcd/0xe0 [ 6220.229184] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6220.233136] do_raw_spin_lock+0x177/0x190 [ 6220.238570] _raw_spin_lock+0x21/0x30 [ 6220.241162] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6220.253092] ? do_raw_spin_unlock+0x75/0x190 [ 6220.256953] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6220.264472] ? do_wait_intr+0xf0/0xf0 [ 6220.267557] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6220.274052] mdc_close+0x53b/0xd70 [mdc] [ 6220.276016] lmv_close+0x18d/0x490 [lmv] [ 6220.282564] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6220.291405] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6220.301153] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6220.304650] ll_md_close+0x274/0x800 [lustre] [ 6220.307479] ll_file_release+0x128/0xc50 [lustre] [ 6220.311116] __fput+0x10d/0x440 [ 6220.320187] ____fput+0x12/0x20 [ 6220.325530] task_work_run+0xc6/0x110 [ 6220.327842] exit_to_usermode_loop+0x1dd/0x1f0 [ 6220.331305] do_syscall_64+0x3d6/0x440 [ 6220.332659] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6220.336803] RIP: 0033:0x7f151bee3cd8 [ 6220.338709] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6220.351657] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6220.358403] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 6220.371687] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 6220.376308] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6220.381074] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6220.387655] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6228.057223] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 6228.064324] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6228.098303] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6228.109544] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6228.119496] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6228.124265] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6228.137514] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6228.163434] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6228.176247] RAX: 0000000000007f0c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6228.184249] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 6228.191032] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 6228.206945] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 6228.217112] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 6228.226387] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 6228.237212] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6228.248164] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6228.251097] Call Trace: [ 6228.253158] [ 6228.255450] ? show_regs.cold.9+0x22/0x2f [ 6228.258445] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6228.262329] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6228.267279] ? watchdog+0x40/0x40 [ 6228.269828] ? hrtimer_interrupt+0x128/0x2c0 [ 6228.271868] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6228.274044] ? apic_timer_interrupt+0xf/0x20 [ 6228.275841] [ 6228.276713] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6228.279070] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6228.280777] do_raw_spin_lock+0x177/0x190 [ 6228.281866] _raw_spin_lock+0x21/0x30 [ 6228.282889] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6228.285143] process_one_work+0x2c8/0x700 [ 6228.286882] worker_thread+0x296/0x6e0 [ 6228.288375] ? rescuer_thread+0x570/0x570 [ 6228.289979] kthread+0x1d1/0x200 [ 6228.291371] ? set_kthread_struct+0x70/0x70 [ 6228.293164] ret_from_fork+0x1f/0x30 [ 6236.054164] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 6236.056828] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 6236.064726] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6236.064813] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6236.064821] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6236.064823] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6236.064839] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6236.064844] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 6236.064848] RAX: 000000000000662b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6236.064851] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 6236.064854] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 6236.064856] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 6236.064858] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 6236.064862] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 6236.064865] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6236.064867] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 6236.064877] Call Trace: [ 6236.064883] [ 6236.064889] ? show_regs.cold.9+0x22/0x2f [ 6236.064899] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6236.064906] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6236.064910] ? watchdog+0x40/0x40 [ 6236.064915] ? hrtimer_interrupt+0x128/0x2c0 [ 6236.064919] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6236.064927] ? apic_timer_interrupt+0xf/0x20 [ 6236.064932] [ 6236.064935] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6236.064939] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6236.064944] do_raw_spin_lock+0x177/0x190 [ 6236.064949] _raw_spin_lock+0x21/0x30 [ 6236.064954] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6236.065259] ? do_raw_spin_unlock+0x75/0x190 [ 6236.065264] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6236.080137] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6236.080281] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6236.080293] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6236.080296] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6236.080311] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6236.080315] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6236.080320] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 6236.080322] RDX: 00000000000040ff RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 6236.080325] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 6236.080327] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 6236.080329] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6236.080332] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 6236.080335] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6236.080338] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6236.080498] Call Trace: [ 6236.080506] [ 6236.080512] ? show_regs.cold.9+0x22/0x2f [ 6236.080718] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6236.080726] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6236.080731] ? watchdog+0x40/0x40 [ 6236.080735] ? hrtimer_interrupt+0x128/0x2c0 [ 6236.080739] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6236.081144] ? apic_timer_interrupt+0xf/0x20 [ 6236.081149] [ 6236.081821] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6236.081833] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6236.081838] do_raw_spin_lock+0x177/0x190 [ 6236.081844] _raw_spin_lock+0x21/0x30 [ 6236.081928] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6236.085174] obd_mod_free+0x46/0x140 [ptlrpc] [ 6236.086546] mdc_commit_open+0x91/0xb0 [mdc] [ 6236.086585] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6236.087542] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6236.088470] after_reply+0xe2c/0x1780 [ptlrpc] [ 6236.089773] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6236.090078] ? __next_timer_interrupt+0x160/0x160 [ 6236.090090] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6236.142177] ? do_wait_intr+0xf0/0xf0 [ 6236.148355] ? do_wait_intr+0xf0/0xf0 [ 6236.148381] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6236.149315] kthread+0x1d1/0x200 [ 6236.149323] ? set_kthread_struct+0x70/0x70 [ 6236.149327] ret_from_fork+0x1f/0x30 [ 6236.781267] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6236.783382] mdc_reint+0x2d/0x110 [mdc] [ 6236.794545] mdc_unlink+0x658/0x870 [mdc] [ 6236.805256] lmv_unlink+0x34d/0xbd0 [lmv] [ 6236.806676] ll_unlink+0x24d/0x7f0 [lustre] [ 6236.808276] vfs_unlink+0x202/0x330 [ 6236.821502] do_unlinkat+0x2fa/0x460 [ 6236.829769] __x64_sys_unlink+0x37/0x50 [ 6236.838114] do_syscall_64+0xc1/0x440 [ 6236.839512] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6236.851914] RIP: 0033:0x7f151bdee10b [ 6236.853406] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6236.865986] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6236.871781] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 6236.876530] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 6236.881504] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6236.885495] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6236.890907] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6238.176628] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4307s! [ 6238.186809] Showing busy workqueues and worker pools: [ 6238.191644] workqueue events: flags=0x0 [ 6238.195880] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6238.200087] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 6238.205716] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6238.212975] pending: kfree_rcu_monitor [ 6238.216140] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6238.220602] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 6238.225778] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6238.229243] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 6238.233265] workqueue events_long: flags=0x0 [ 6238.235042] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6238.238714] pending: lu_objects_flush [obdclass] [ 6238.242568] workqueue events_unbound: flags=0x2 [ 6238.247387] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6238.250285] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6238.257792] workqueue events_power_efficient: flags=0x80 [ 6238.259987] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6238.263465] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 6238.266348] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6238.269085] pending: neigh_periodic_work [ 6238.270943] workqueue mm_percpu_wq: flags=0x8 [ 6238.272714] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6238.275220] pending: vmstat_update, lru_add_drain_per_cpu [ 6238.278010] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6238.280913] pending: vmstat_update [ 6238.282342] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6238.284995] pending: lru_add_drain_per_cpu [ 6238.286635] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6238.289128] pending: lru_add_drain_per_cpu BAR(48) [ 6238.291028] workqueue writeback: flags=0x4a [ 6238.294913] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 6238.297080] pending: wb_workfn [ 6238.298258] workqueue xprtiod: flags=0xa [ 6238.299943] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6238.302151] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6238.305227] workqueue ipv6_addrconf: flags=0x40008 [ 6238.309413] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6238.317686] pending: addrconf_verify_work [ 6238.322879] workqueue knbd0-recv: flags=0x1a [ 6238.324760] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6238.327247] in-flight: 658:recv_work [ 6238.330640] workqueue ptlrpc_pinger: flags=0x2 [ 6238.331824] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6238.333353] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6238.336025] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4307s workers=4 idle: 32984 37 8 [ 6238.340242] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6223s workers=2 idle: 661 [ 6248.057320] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 6248.061921] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6248.115107] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6248.125894] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6248.152743] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6248.154825] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6248.183533] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6248.191051] RAX: 0000000000004662 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6248.207315] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 6248.216607] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6248.224456] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 6248.232455] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 6248.235048] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 6248.258790] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6248.262714] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 6248.266517] Call Trace: [ 6248.267700] [ 6248.269310] ? show_regs.cold.9+0x22/0x2f [ 6248.274494] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6248.284424] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6248.286677] ? watchdog+0x40/0x40 [ 6248.295810] ? hrtimer_interrupt+0x128/0x2c0 [ 6248.300760] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6248.304683] ? apic_timer_interrupt+0xf/0x20 [ 6248.306798] [ 6248.308228] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6248.314149] do_raw_spin_lock+0x177/0x190 [ 6248.316302] _raw_spin_lock+0x21/0x30 [ 6248.318328] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6248.324405] ? do_raw_spin_unlock+0x75/0x190 [ 6248.332979] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6248.337300] ? do_wait_intr+0xf0/0xf0 [ 6248.341662] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6248.344908] mdc_close+0x53b/0xd70 [mdc] [ 6248.346699] lmv_close+0x18d/0x490 [lmv] [ 6248.348048] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6248.356409] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6248.362044] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6248.367075] ll_md_close+0x274/0x800 [lustre] [ 6248.368953] ll_file_release+0x128/0xc50 [lustre] [ 6248.370215] __fput+0x10d/0x440 [ 6248.371398] ____fput+0x12/0x20 [ 6248.375139] task_work_run+0xc6/0x110 [ 6248.377857] exit_to_usermode_loop+0x1dd/0x1f0 [ 6248.379826] do_syscall_64+0x3d6/0x440 [ 6248.382174] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6248.385800] RIP: 0033:0x7f151bee3cd8 [ 6248.388787] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6248.404637] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6248.410174] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 6248.417080] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 6248.421672] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6248.423531] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6248.426547] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6256.062411] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6256.067285] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6256.120412] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6256.129736] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6256.137895] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6256.146577] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6256.151834] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6256.173288] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6256.180714] RAX: 0000000000000ab7 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6256.191576] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 6256.200591] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 6256.209891] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 6256.218488] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 6256.223715] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 6256.239243] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6256.245376] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6256.254361] Call Trace: [ 6256.256715] [ 6256.259285] ? show_regs.cold.9+0x22/0x2f [ 6256.267931] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6256.276656] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6256.278350] ? watchdog+0x40/0x40 [ 6256.283175] ? hrtimer_interrupt+0x128/0x2c0 [ 6256.296950] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6256.303907] ? apic_timer_interrupt+0xf/0x20 [ 6256.309779] [ 6256.316245] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6256.321834] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6256.326031] do_raw_spin_lock+0x177/0x190 [ 6256.330787] _raw_spin_lock+0x21/0x30 [ 6256.335664] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6256.340719] process_one_work+0x2c8/0x700 [ 6256.342233] worker_thread+0x296/0x6e0 [ 6256.353195] ? rescuer_thread+0x570/0x570 [ 6256.355954] kthread+0x1d1/0x200 [ 6256.365536] ? set_kthread_struct+0x70/0x70 [ 6256.376760] ret_from_fork+0x1f/0x30 [ 6264.052123] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 6264.056729] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 6264.070861] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6264.070934] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6264.070942] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6264.070945] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6264.070959] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6264.070964] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6264.070969] RAX: 0000000000006d86 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6264.070971] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 6264.070974] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 6264.070976] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 6264.070978] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 6264.070981] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 6264.070984] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6264.071084] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 6264.071093] Call Trace: [ 6264.071099] [ 6264.071105] ? show_regs.cold.9+0x22/0x2f [ 6264.071115] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6264.071122] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6264.071126] ? watchdog+0x40/0x40 [ 6264.071131] ? hrtimer_interrupt+0x128/0x2c0 [ 6264.071135] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6264.071143] ? apic_timer_interrupt+0xf/0x20 [ 6264.071148] [ 6264.071151] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6264.071156] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6264.071161] do_raw_spin_lock+0x177/0x190 [ 6264.071166] _raw_spin_lock+0x21/0x30 [ 6264.071172] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6264.071742] ? do_raw_spin_unlock+0x75/0x190 [ 6264.071748] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6264.071937] ? do_wait_intr+0xf0/0xf0 [ 6264.071944] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6264.087832] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6264.087988] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6264.087995] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6264.087998] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6264.088013] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6264.088017] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6264.088022] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 6264.088025] RDX: 00000000000036ce RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 6264.088027] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 6264.088030] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 6264.088032] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6264.088036] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 6264.088039] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6264.088041] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6264.088050] Call Trace: [ 6264.088055] [ 6264.088061] ? show_regs.cold.9+0x22/0x2f [ 6264.088069] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6264.088074] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6264.088078] ? watchdog+0x40/0x40 [ 6264.088083] ? hrtimer_interrupt+0x128/0x2c0 [ 6264.088087] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6264.088097] ? apic_timer_interrupt+0xf/0x20 [ 6264.088102] [ 6264.088105] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6264.088110] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6264.088114] do_raw_spin_lock+0x177/0x190 [ 6264.088119] _raw_spin_lock+0x21/0x30 [ 6264.088125] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6264.089201] obd_mod_free+0x46/0x140 [ptlrpc] [ 6264.089417] mdc_commit_open+0x91/0xb0 [mdc] [ 6264.089469] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6264.132879] mdc_reint+0x2d/0x110 [mdc] [ 6264.132935] mdc_unlink+0x658/0x870 [mdc] [ 6264.132963] lmv_unlink+0x34d/0xbd0 [lmv] [ 6264.137898] ll_unlink+0x24d/0x7f0 [lustre] [ 6264.138014] vfs_unlink+0x202/0x330 [ 6264.138023] do_unlinkat+0x2fa/0x460 [ 6264.138031] __x64_sys_unlink+0x37/0x50 [ 6264.138034] do_syscall_64+0xc1/0x440 [ 6264.138040] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6264.138050] RIP: 0033:0x7f151bdee10b [ 6264.138059] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6264.138063] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6264.138068] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 6264.138070] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 6264.138073] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6264.138075] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6264.138077] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6264.696090] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6264.710587] after_reply+0xe2c/0x1780 [ptlrpc] [ 6264.718965] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6264.733604] ? __next_timer_interrupt+0x160/0x160 [ 6264.734964] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6264.749224] ? do_wait_intr+0xf0/0xf0 [ 6264.754787] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6264.772033] kthread+0x1d1/0x200 [ 6264.776076] ? set_kthread_struct+0x70/0x70 [ 6264.780244] ret_from_fork+0x1f/0x30 [ 6268.396676] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4337s! [ 6268.402811] Showing busy workqueues and worker pools: [ 6268.406463] workqueue events: flags=0x0 [ 6268.408593] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6268.427798] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 6268.442096] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6268.448022] pending: kfree_rcu_monitor [ 6268.452735] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6268.466668] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 6268.479654] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6268.486478] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 6268.498899] workqueue events_long: flags=0x0 [ 6268.506158] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6268.514812] pending: lu_objects_flush [obdclass] [ 6268.524983] workqueue events_unbound: flags=0x2 [ 6268.530076] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6268.540673] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6268.551551] workqueue events_power_efficient: flags=0x80 [ 6268.558180] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6268.566033] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 6268.577613] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6268.587989] pending: neigh_periodic_work [ 6268.594653] workqueue mm_percpu_wq: flags=0x8 [ 6268.598803] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6268.603712] pending: vmstat_update, lru_add_drain_per_cpu [ 6268.607774] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6268.612051] pending: vmstat_update [ 6268.621293] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6268.630394] pending: lru_add_drain_per_cpu [ 6268.632040] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6268.636882] pending: lru_add_drain_per_cpu BAR(48) [ 6268.651589] workqueue writeback: flags=0x4a [ 6268.657878] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 6268.661102] pending: wb_workfn [ 6268.667441] workqueue xprtiod: flags=0xa [ 6268.674021] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6268.685533] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6268.699110] workqueue ipv6_addrconf: flags=0x40008 [ 6268.703443] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6268.713674] pending: addrconf_verify_work [ 6268.718490] workqueue knbd0-recv: flags=0x1a [ 6268.727743] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6268.738574] in-flight: 658:recv_work [ 6268.750543] workqueue ptlrpc_pinger: flags=0x2 [ 6268.753631] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6268.764771] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6268.774565] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4338s workers=4 idle: 32984 37 8 [ 6268.782036] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6253s workers=2 idle: 661 [ 6276.063830] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 6276.074108] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6276.142070] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6276.158880] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6276.177794] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6276.188983] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6276.211317] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6276.227363] RAX: 0000000000007a86 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6276.237112] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 6276.247669] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6276.264484] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 6276.293116] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 6276.316047] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 6276.327101] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6276.329264] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 6276.332278] Call Trace: [ 6276.333338] [ 6276.334516] ? show_regs.cold.9+0x22/0x2f [ 6276.335716] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6276.336980] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6276.338505] ? watchdog+0x40/0x40 [ 6276.339389] ? hrtimer_interrupt+0x128/0x2c0 [ 6276.340458] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6276.342273] ? apic_timer_interrupt+0xf/0x20 [ 6276.343903] [ 6276.344642] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6276.348449] do_raw_spin_lock+0x177/0x190 [ 6276.350427] _raw_spin_lock+0x21/0x30 [ 6276.352002] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6276.354455] ? do_raw_spin_unlock+0x75/0x190 [ 6276.356362] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6276.358416] ? do_wait_intr+0xf0/0xf0 [ 6276.359725] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6276.361807] mdc_close+0x53b/0xd70 [mdc] [ 6276.363263] lmv_close+0x18d/0x490 [lmv] [ 6276.365095] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6276.371496] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6276.373853] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6276.376120] ll_md_close+0x274/0x800 [lustre] [ 6276.378039] ll_file_release+0x128/0xc50 [lustre] [ 6276.379954] __fput+0x10d/0x440 [ 6276.381164] ____fput+0x12/0x20 [ 6276.382530] task_work_run+0xc6/0x110 [ 6276.383916] exit_to_usermode_loop+0x1dd/0x1f0 [ 6276.391149] do_syscall_64+0x3d6/0x440 [ 6276.392547] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6276.394522] RIP: 0033:0x7f151bee3cd8 [ 6276.395873] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6276.402930] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6276.405788] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 6276.408630] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 6276.411414] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6276.414061] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6276.416948] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6284.056333] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6284.067032] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6284.123706] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6284.139557] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6284.150920] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6284.158827] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6284.165389] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6284.178922] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6284.193996] RAX: 0000000000002291 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6284.202649] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 6284.208483] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 6284.217449] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 6284.222632] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 6284.231442] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 6284.236242] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6284.240437] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6284.244321] Call Trace: [ 6284.245493] [ 6284.246811] ? show_regs.cold.9+0x22/0x2f [ 6284.251093] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6284.254882] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6284.262444] ? watchdog+0x40/0x40 [ 6284.267073] ? hrtimer_interrupt+0x128/0x2c0 [ 6284.271007] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6284.276488] ? apic_timer_interrupt+0xf/0x20 [ 6284.278833] [ 6284.279688] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6284.282417] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6284.285145] do_raw_spin_lock+0x177/0x190 [ 6284.287136] _raw_spin_lock+0x21/0x30 [ 6284.289297] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6284.292422] process_one_work+0x2c8/0x700 [ 6284.301967] worker_thread+0x296/0x6e0 [ 6284.306792] ? rescuer_thread+0x570/0x570 [ 6284.311397] kthread+0x1d1/0x200 [ 6284.312737] ? set_kthread_struct+0x70/0x70 [ 6284.317405] ret_from_fork+0x1f/0x30 [ 6292.055164] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 6292.059759] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6292.097571] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6292.104236] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6292.110761] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6292.114579] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6292.134028] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6292.143415] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 6292.150303] RDX: 0000000000001be0 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 6292.160578] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 6292.164541] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 6292.167269] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6292.174895] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 6292.181523] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6292.185243] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6292.189266] Call Trace: [ 6292.190691] [ 6292.192188] ? show_regs.cold.9+0x22/0x2f [ 6292.194893] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6292.197484] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6292.200408] ? watchdog+0x40/0x40 [ 6292.202337] ? hrtimer_interrupt+0x128/0x2c0 [ 6292.206547] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6292.209249] ? apic_timer_interrupt+0xf/0x20 [ 6292.211301] [ 6292.211911] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6292.214183] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6292.216375] do_raw_spin_lock+0x177/0x190 [ 6292.221348] _raw_spin_lock+0x21/0x30 [ 6292.222988] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6292.227337] obd_mod_free+0x46/0x140 [ptlrpc] [ 6292.232893] mdc_commit_open+0x91/0xb0 [mdc] [ 6292.240432] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6292.243695] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6292.247815] after_reply+0xe2c/0x1780 [ptlrpc] [ 6292.249861] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6292.252696] ? __next_timer_interrupt+0x160/0x160 [ 6292.254603] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6292.258176] ? do_wait_intr+0xf0/0xf0 [ 6292.260335] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6292.263607] kthread+0x1d1/0x200 [ 6292.267560] ? set_kthread_struct+0x70/0x70 [ 6292.271394] ret_from_fork+0x1f/0x30 [ 6296.054143] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 6296.066431] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6296.151116] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6296.162799] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6296.170565] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6296.174512] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6296.183442] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6296.186742] RAX: 0000000000007e08 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6296.194161] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 6296.196788] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 6296.199791] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 6296.207291] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 6296.216639] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 6296.219787] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6296.225269] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 6296.228215] Call Trace: [ 6296.228817] [ 6296.229471] ? show_regs.cold.9+0x22/0x2f [ 6296.231432] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6296.233256] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6296.234826] ? watchdog+0x40/0x40 [ 6296.235685] ? hrtimer_interrupt+0x128/0x2c0 [ 6296.237712] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6296.239735] ? apic_timer_interrupt+0xf/0x20 [ 6296.241650] [ 6296.242340] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6296.244567] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6296.250348] do_raw_spin_lock+0x177/0x190 [ 6296.251885] _raw_spin_lock+0x21/0x30 [ 6296.255948] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6296.259068] ? do_raw_spin_unlock+0x75/0x190 [ 6296.263818] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6296.267639] ? do_wait_intr+0xf0/0xf0 [ 6296.271412] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6296.273436] mdc_reint+0x2d/0x110 [mdc] [ 6296.275587] mdc_unlink+0x658/0x870 [mdc] [ 6296.277695] lmv_unlink+0x34d/0xbd0 [lmv] [ 6296.279640] ll_unlink+0x24d/0x7f0 [lustre] [ 6296.282117] vfs_unlink+0x202/0x330 [ 6296.284761] do_unlinkat+0x2fa/0x460 [ 6296.287228] __x64_sys_unlink+0x37/0x50 [ 6296.290464] do_syscall_64+0xc1/0x440 [ 6296.294810] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6296.297624] RIP: 0033:0x7f151bdee10b [ 6296.298946] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6296.309773] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6296.314660] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 6296.321120] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 6296.325681] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6296.329165] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6296.332227] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6299.106575] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4368s! [ 6299.111467] Showing busy workqueues and worker pools: [ 6299.115887] workqueue events: flags=0x0 [ 6299.125914] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6299.140429] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 6299.149931] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6299.155485] pending: kfree_rcu_monitor [ 6299.160136] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6299.167626] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 6299.174710] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6299.176829] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 6299.183411] workqueue events_long: flags=0x0 [ 6299.185800] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6299.193269] pending: lu_objects_flush [obdclass] [ 6299.202905] workqueue events_unbound: flags=0x2 [ 6299.208779] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6299.218528] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6299.229673] workqueue events_power_efficient: flags=0x80 [ 6299.238906] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6299.248408] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 6299.259630] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6299.272459] pending: neigh_periodic_work [ 6299.279408] workqueue mm_percpu_wq: flags=0x8 [ 6299.285326] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6299.296564] pending: vmstat_update, lru_add_drain_per_cpu [ 6299.303868] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6299.315396] pending: vmstat_update [ 6299.316610] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6299.329771] pending: lru_add_drain_per_cpu [ 6299.337434] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6299.349386] pending: lru_add_drain_per_cpu BAR(48) [ 6299.355515] workqueue writeback: flags=0x4a [ 6299.357214] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 6299.359717] pending: wb_workfn [ 6299.361097] workqueue xprtiod: flags=0xa [ 6299.362778] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6299.365052] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6299.368636] workqueue ipv6_addrconf: flags=0x40008 [ 6299.370276] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6299.375676] pending: addrconf_verify_work [ 6299.380700] workqueue knbd0-recv: flags=0x1a [ 6299.389255] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6299.399519] in-flight: 658:recv_work [ 6299.405887] workqueue ptlrpc_pinger: flags=0x2 [ 6299.409107] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6299.415867] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6299.420866] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4368s workers=4 idle: 32984 37 8 [ 6299.428065] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6284s workers=2 idle: 661 [ 6304.058273] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 6304.061046] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6304.100160] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6304.109291] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6304.116604] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6304.122056] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6304.137236] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6304.145743] RAX: 0000000000003d84 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6304.152874] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 6304.155410] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6304.161788] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 6304.168471] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 6304.170960] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 6304.180557] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6304.186979] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 6304.192277] Call Trace: [ 6304.196644] [ 6304.197566] ? show_regs.cold.9+0x22/0x2f [ 6304.203365] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6304.205563] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6304.211463] ? watchdog+0x40/0x40 [ 6304.213752] ? hrtimer_interrupt+0x128/0x2c0 [ 6304.218561] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6304.221610] ? apic_timer_interrupt+0xf/0x20 [ 6304.226208] [ 6304.229289] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6304.233678] do_raw_spin_lock+0x177/0x190 [ 6304.239201] _raw_spin_lock+0x21/0x30 [ 6304.241310] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6304.251347] ? do_raw_spin_unlock+0x75/0x190 [ 6304.255956] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6304.261514] ? do_wait_intr+0xf0/0xf0 [ 6304.269268] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6304.274777] mdc_close+0x53b/0xd70 [mdc] [ 6304.277323] lmv_close+0x18d/0x490 [lmv] [ 6304.282845] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6304.286833] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6304.292601] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6304.298951] ll_md_close+0x274/0x800 [lustre] [ 6304.301573] ll_file_release+0x128/0xc50 [lustre] [ 6304.304521] __fput+0x10d/0x440 [ 6304.312302] ____fput+0x12/0x20 [ 6304.314331] task_work_run+0xc6/0x110 [ 6304.315654] exit_to_usermode_loop+0x1dd/0x1f0 [ 6304.320857] do_syscall_64+0x3d6/0x440 [ 6304.322101] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6304.328028] RIP: 0033:0x7f151bee3cd8 [ 6304.329403] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6304.348774] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6304.374549] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 6304.382708] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 6304.385591] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6304.391177] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6304.402652] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6304.734373] rcu: INFO: rcu_sched self-detected stall on CPU [ 6304.743168] rcu: 1-...!: (2517588 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 6304.759529] (t=4380087 jiffies g=249657 q=3596) [ 6304.765636] rcu: rcu_sched kthread starved for 4375039 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 6304.776208] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 6304.797621] rcu: RCU grace-period kthread stack dump: [ 6304.812375] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 6304.824715] Call Trace: [ 6304.825600] __schedule+0x351/0xcb0 [ 6304.826747] schedule+0xc0/0x180 [ 6304.838537] schedule_timeout+0xb4/0x190 [ 6304.841686] ? do_raw_spin_unlock+0x75/0x190 [ 6304.846774] ? __next_timer_interrupt+0x160/0x160 [ 6304.856439] ? prepare_to_swait_event+0xf3/0x140 [ 6304.859656] rcu_gp_kthread+0x8dd/0x15b0 [ 6304.866087] ? force_qs_rnp+0x2c0/0x2c0 [ 6304.869388] kthread+0x1d1/0x200 [ 6304.878723] ? set_kthread_struct+0x70/0x70 [ 6304.884578] ret_from_fork+0x1f/0x30 [ 6304.892091] rcu: Stack dump where RCU GP kthread last ran: [ 6304.897552] Sending NMI from CPU 1 to CPUs 2: [ 6304.908557] NMI backtrace for cpu 2 [ 6304.908564] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6304.908566] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6304.908568] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6304.908571] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6304.908573] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6304.908575] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 [ 6304.908577] RAX: 0000000000005b0a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6304.908579] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 6304.908580] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 6304.908581] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 6304.908582] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 6304.908583] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 6304.908585] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6304.908586] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6304.908592] Call Trace: [ 6304.908593] [ 6304.908594] ? show_regs.cold.9+0x22/0x2f [ 6304.908595] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 6304.908597] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 6304.908598] ? nmi_handle+0xbd/0x210 [ 6304.908599] ? default_do_nmi+0x4b/0x1b0 [ 6304.908600] ? do_nmi+0x280/0x380 [ 6304.908602] ? end_repeat_nmi+0x16/0x1f [ 6304.908603] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6304.908604] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6304.908606] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6304.908607] [ 6304.908608] do_raw_spin_lock+0x177/0x190 [ 6304.908609] _raw_spin_lock+0x21/0x30 [ 6304.908610] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6304.908612] process_one_work+0x2c8/0x700 [ 6304.908613] worker_thread+0x296/0x6e0 [ 6304.908614] ? rescuer_thread+0x570/0x570 [ 6304.908615] kthread+0x1d1/0x200 [ 6304.908616] ? set_kthread_struct+0x70/0x70 [ 6304.908617] ret_from_fork+0x1f/0x30 [ 6304.916148] NMI backtrace for cpu 1 [ 6305.124477] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6305.143096] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6305.162540] Call Trace: [ 6305.163426] [ 6305.164241] ? dump_stack+0xbb/0x10e [ 6305.173818] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6305.179121] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6305.191249] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6305.198725] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6305.213260] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6305.221378] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 6305.222772] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 6305.231713] ? update_process_times+0x6d/0xc0 [ 6305.249368] ? tick_sched_handle.isra.18+0x27/0x90 [ 6305.250617] ? tick_sched_timer+0x53/0xb0 [ 6305.251704] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6305.258540] ? tick_sched_handle.isra.18+0x90/0x90 [ 6305.266281] ? hrtimer_interrupt+0x128/0x2c0 [ 6305.271808] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6305.277572] ? apic_timer_interrupt+0xf/0x20 [ 6305.282277] [ 6305.283110] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6305.288320] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6305.292406] ? do_raw_spin_lock+0x177/0x190 [ 6305.300298] ? _raw_spin_lock+0x21/0x30 [ 6305.303684] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6305.308469] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 6305.316143] ? mdc_commit_open+0x91/0xb0 [mdc] [ 6305.321898] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6305.326272] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6305.345713] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 6305.347753] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6305.358840] ? __next_timer_interrupt+0x160/0x160 [ 6305.360464] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6305.369524] ? do_wait_intr+0xf0/0xf0 [ 6305.376594] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6305.380289] ? kthread+0x1d1/0x200 [ 6305.385330] ? set_kthread_struct+0x70/0x70 [ 6305.393948] ? ret_from_fork+0x1f/0x30 [ 6324.054620] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 6324.061947] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6324.122610] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6324.133508] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6324.138972] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6324.143537] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6324.169078] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6324.180325] RAX: 0000000000000c8a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6324.188336] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 6324.192109] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 6324.197234] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 6324.203657] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 6324.213243] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 6324.225635] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6324.233547] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 6324.239154] Call Trace: [ 6324.249541] [ 6324.251272] ? show_regs.cold.9+0x22/0x2f [ 6324.255193] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6324.260693] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6324.269962] ? watchdog+0x40/0x40 [ 6324.273529] ? hrtimer_interrupt+0x128/0x2c0 [ 6324.279168] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6324.284124] ? apic_timer_interrupt+0xf/0x20 [ 6324.288417] [ 6324.291172] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6324.293635] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6324.298000] do_raw_spin_lock+0x177/0x190 [ 6324.303759] _raw_spin_lock+0x21/0x30 [ 6324.313453] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6324.321735] ? do_raw_spin_unlock+0x75/0x190 [ 6324.324875] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6324.328236] ? do_wait_intr+0xf0/0xf0 [ 6324.331288] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6324.336299] mdc_reint+0x2d/0x110 [mdc] [ 6324.339486] mdc_unlink+0x658/0x870 [mdc] [ 6324.346927] lmv_unlink+0x34d/0xbd0 [lmv] [ 6324.348566] ll_unlink+0x24d/0x7f0 [lustre] [ 6324.352441] vfs_unlink+0x202/0x330 [ 6324.355475] do_unlinkat+0x2fa/0x460 [ 6324.358866] __x64_sys_unlink+0x37/0x50 [ 6324.364406] do_syscall_64+0xc1/0x440 [ 6324.372627] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6324.380089] RIP: 0033:0x7f151bdee10b [ 6324.384043] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6324.416353] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6324.430078] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 6324.435912] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 6324.443486] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6324.450320] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6324.454780] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6329.829919] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4399s! [ 6329.844624] Showing busy workqueues and worker pools: [ 6329.848577] workqueue events: flags=0x0 [ 6329.855105] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6329.869898] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 6329.889855] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6329.898849] pending: kfree_rcu_monitor [ 6329.901540] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6329.905571] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 6329.911129] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6329.914801] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 6329.919227] workqueue events_long: flags=0x0 [ 6329.921435] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6329.924799] pending: lu_objects_flush [obdclass] [ 6329.928140] workqueue events_unbound: flags=0x2 [ 6329.931156] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6329.934468] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6329.942618] workqueue events_power_efficient: flags=0x80 [ 6329.945830] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6329.950021] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 6329.954465] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6329.957815] pending: neigh_periodic_work [ 6329.964119] workqueue mm_percpu_wq: flags=0x8 [ 6329.966534] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6329.969807] pending: vmstat_update, lru_add_drain_per_cpu [ 6329.972083] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6329.976475] pending: vmstat_update [ 6329.979491] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6329.982592] pending: lru_add_drain_per_cpu [ 6329.988228] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6329.993299] pending: lru_add_drain_per_cpu BAR(48) [ 6329.997507] workqueue writeback: flags=0x4a [ 6329.999591] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 6330.003755] pending: wb_workfn [ 6330.005591] workqueue xprtiod: flags=0xa [ 6330.010744] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6330.015322] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6330.026578] workqueue ipv6_addrconf: flags=0x40008 [ 6330.028788] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6330.032215] pending: addrconf_verify_work [ 6330.034119] workqueue knbd0-recv: flags=0x1a [ 6330.035641] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6330.038581] in-flight: 658:recv_work [ 6330.040189] workqueue ptlrpc_pinger: flags=0x2 [ 6330.041695] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6330.044375] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6330.047381] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4399s workers=4 idle: 32984 37 8 [ 6330.050689] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6314s workers=2 idle: 661 [ 6332.056064] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 6332.059306] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 6332.059299] Modules linked in: [ 6332.060034] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 6332.060041] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6332.060123] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6332.060129] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6332.060132] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6332.060145] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6332.060150] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6332.060154] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 6332.060157] RDX: 0000000000006d2b RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 6332.060160] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 6332.060163] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 6332.060166] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6332.060169] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 6332.060172] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6332.060174] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6332.060183] Call Trace: [ 6332.060188] [ 6332.060193] ? show_regs.cold.9+0x22/0x2f [ 6332.060204] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6332.060211] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6332.060216] ? watchdog+0x40/0x40 [ 6332.060220] ? hrtimer_interrupt+0x128/0x2c0 [ 6332.060225] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6332.060234] ? apic_timer_interrupt+0xf/0x20 [ 6332.060239] [ 6332.060242] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6332.060247] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6332.060252] do_raw_spin_lock+0x177/0x190 [ 6332.060257] _raw_spin_lock+0x21/0x30 [ 6332.060263] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6332.060562] obd_mod_free+0x46/0x140 [ptlrpc] [ 6332.060758] mdc_commit_open+0x91/0xb0 [mdc] [ 6332.060816] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6332.061009] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6332.061217] after_reply+0xe2c/0x1780 [ptlrpc] [ 6332.061414] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6332.061610] ? __next_timer_interrupt+0x160/0x160 [ 6332.061620] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6332.061839] ? do_wait_intr+0xf0/0xf0 [ 6332.061849] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6332.062064] kthread+0x1d1/0x200 [ 6332.062070] ? set_kthread_struct+0x70/0x70 [ 6332.062074] ret_from_fork+0x1f/0x30 [ 6332.067065] Modules linked in: [ 6332.068472] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6332.068550] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6332.068557] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6332.068564] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6332.069090] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6332.069104] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6332.069108] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6332.069112] RAX: 0000000000003e0c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6332.069114] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 6332.069116] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 6332.069118] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 6332.069121] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 6332.069124] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 6332.069127] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6332.069129] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6332.069138] Call Trace: [ 6332.069144] [ 6332.069149] ? show_regs.cold.9+0x22/0x2f [ 6332.069158] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6332.069165] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6332.069170] ? watchdog+0x40/0x40 [ 6332.069175] ? hrtimer_interrupt+0x128/0x2c0 [ 6332.069179] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6332.069188] ? apic_timer_interrupt+0xf/0x20 [ 6332.069194] [ 6332.069197] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6332.069201] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6332.069207] do_raw_spin_lock+0x177/0x190 [ 6332.069211] _raw_spin_lock+0x21/0x30 [ 6332.069217] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6332.069709] process_one_work+0x2c8/0x700 [ 6332.069717] worker_thread+0x296/0x6e0 [ 6332.069722] ? rescuer_thread+0x570/0x570 [ 6332.069726] kthread+0x1d1/0x200 [ 6332.069730] ? set_kthread_struct+0x70/0x70 [ 6332.069734] ret_from_fork+0x1f/0x30 [ 6332.302224] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6332.321982] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6332.327036] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6332.330672] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 6332.334533] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 6332.341553] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6332.343981] RAX: 00000000000040bc RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6332.346604] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 6332.352797] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6332.356015] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 6332.359723] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 6332.363031] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 6332.366049] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6332.369198] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 6332.372743] Call Trace: [ 6332.373674] [ 6332.374470] ? show_regs.cold.9+0x22/0x2f [ 6332.376476] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6332.379405] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6332.381534] ? watchdog+0x40/0x40 [ 6332.382949] ? hrtimer_interrupt+0x128/0x2c0 [ 6332.385248] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6332.387975] ? apic_timer_interrupt+0xf/0x20 [ 6332.389706] [ 6332.390672] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 6332.393204] do_raw_spin_lock+0x177/0x190 [ 6332.395068] _raw_spin_lock+0x21/0x30 [ 6332.396737] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6332.399177] ? do_raw_spin_unlock+0x75/0x190 [ 6332.401606] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6332.403992] ? do_wait_intr+0xf0/0xf0 [ 6332.405548] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6332.409873] mdc_close+0x53b/0xd70 [mdc] [ 6332.411807] lmv_close+0x18d/0x490 [lmv] [ 6332.413521] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6332.415913] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6332.419512] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6332.421542] ll_md_close+0x274/0x800 [lustre] [ 6332.423432] ll_file_release+0x128/0xc50 [lustre] [ 6332.425595] __fput+0x10d/0x440 [ 6332.426858] ____fput+0x12/0x20 [ 6332.428248] task_work_run+0xc6/0x110 [ 6332.430097] exit_to_usermode_loop+0x1dd/0x1f0 [ 6332.432381] do_syscall_64+0x3d6/0x440 [ 6332.434532] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6332.436508] RIP: 0033:0x7f151bee3cd8 [ 6332.438025] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6332.446844] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6332.450578] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 6332.454221] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 6332.457266] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6332.460233] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6332.463549] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6356.052019] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 6356.058383] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6356.091844] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6356.098022] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6356.105144] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6356.106989] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6356.118036] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6356.121837] RAX: 0000000000006565 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6356.128474] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 6356.132087] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 6356.135248] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 6356.138610] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 6356.142482] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 6356.146327] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6356.148689] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 6356.152015] Call Trace: [ 6356.153046] [ 6356.153933] ? show_regs.cold.9+0x22/0x2f [ 6356.155232] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6356.156770] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6356.158553] ? watchdog+0x40/0x40 [ 6356.160607] ? hrtimer_interrupt+0x128/0x2c0 [ 6356.161866] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6356.163386] ? apic_timer_interrupt+0xf/0x20 [ 6356.164900] [ 6356.165583] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6356.167005] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6356.168714] do_raw_spin_lock+0x177/0x190 [ 6356.170460] _raw_spin_lock+0x21/0x30 [ 6356.171861] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6356.174268] ? do_raw_spin_unlock+0x75/0x190 [ 6356.177005] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6356.179779] ? do_wait_intr+0xf0/0xf0 [ 6356.181036] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6356.187230] mdc_reint+0x2d/0x110 [mdc] [ 6356.189515] mdc_unlink+0x658/0x870 [mdc] [ 6356.192575] lmv_unlink+0x34d/0xbd0 [lmv] [ 6356.194800] ll_unlink+0x24d/0x7f0 [lustre] [ 6356.200814] vfs_unlink+0x202/0x330 [ 6356.204311] do_unlinkat+0x2fa/0x460 [ 6356.206860] __x64_sys_unlink+0x37/0x50 [ 6356.211307] do_syscall_64+0xc1/0x440 [ 6356.214712] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6356.219792] RIP: 0033:0x7f151bdee10b [ 6356.221356] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6356.234596] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6356.238612] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 6356.243656] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 6356.248866] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6356.258258] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6356.263825] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6360.060000] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 6360.060447] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 6360.063596] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 6360.063609] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6360.063681] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6360.063687] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6360.063693] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6360.063976] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6360.063987] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6360.063992] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6360.063996] RAX: 0000000000007d93 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6360.063999] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 6360.064001] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 6360.064004] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 6360.064006] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 6360.064010] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 6360.064013] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6360.064016] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6360.064026] Call Trace: [ 6360.064033] [ 6360.064039] ? show_regs.cold.9+0x22/0x2f [ 6360.064049] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6360.064056] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6360.064061] ? watchdog+0x40/0x40 [ 6360.064066] ? hrtimer_interrupt+0x128/0x2c0 [ 6360.064070] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6360.064080] ? apic_timer_interrupt+0xf/0x20 [ 6360.064084] [ 6360.064087] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6360.064093] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6360.064099] do_raw_spin_lock+0x177/0x190 [ 6360.064117] _raw_spin_lock+0x21/0x30 [ 6360.064124] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6360.064334] process_one_work+0x2c8/0x700 [ 6360.064342] worker_thread+0x296/0x6e0 [ 6360.064346] ? rescuer_thread+0x570/0x570 [ 6360.064350] kthread+0x1d1/0x200 [ 6360.064354] ? set_kthread_struct+0x70/0x70 [ 6360.064357] ret_from_fork+0x1f/0x30 [ 6360.065946] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6360.066019] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6360.066026] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6360.066029] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6360.066041] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6360.066046] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6360.066050] RAX: 0000000000007d83 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6360.066052] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 6360.066054] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6360.066056] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 6360.066058] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 6360.066061] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 6360.066064] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6360.066066] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 6360.066073] Call Trace: [ 6360.066078] [ 6360.066082] ? show_regs.cold.9+0x22/0x2f [ 6360.066090] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6360.066097] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6360.066101] ? watchdog+0x40/0x40 [ 6360.066105] ? hrtimer_interrupt+0x128/0x2c0 [ 6360.066109] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6360.066116] ? apic_timer_interrupt+0xf/0x20 [ 6360.066120] [ 6360.066123] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6360.066129] do_raw_spin_lock+0x177/0x190 [ 6360.066133] _raw_spin_lock+0x21/0x30 [ 6360.066138] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6360.066398] ? do_raw_spin_unlock+0x75/0x190 [ 6360.066404] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6360.067111] ? do_wait_intr+0xf0/0xf0 [ 6360.067119] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6360.067320] mdc_close+0x53b/0xd70 [mdc] [ 6360.068513] lmv_close+0x18d/0x490 [lmv] [ 6360.068549] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6360.068720] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6360.068831] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6360.068911] ll_md_close+0x274/0x800 [lustre] [ 6360.069576] ll_file_release+0x128/0xc50 [lustre] [ 6360.069840] __fput+0x10d/0x440 [ 6360.069850] ____fput+0x12/0x20 [ 6360.069855] task_work_run+0xc6/0x110 [ 6360.069862] exit_to_usermode_loop+0x1dd/0x1f0 [ 6360.069870] do_syscall_64+0x3d6/0x440 [ 6360.069875] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6360.069882] RIP: 0033:0x7f151bee3cd8 [ 6360.069890] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6360.069894] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6360.069898] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 6360.069901] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 6360.069903] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6360.069905] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6360.069907] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6360.718663] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6360.747410] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6360.752998] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6360.758842] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6360.761023] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6360.770630] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6360.773450] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 6360.776936] RDX: 0000000000005f78 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 6360.780218] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 6360.783345] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 6360.786428] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6360.789559] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 6360.792532] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6360.794548] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6360.798780] Call Trace: [ 6360.799851] [ 6360.800955] ? show_regs.cold.9+0x22/0x2f [ 6360.802610] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6360.804002] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6360.805586] ? watchdog+0x40/0x40 [ 6360.806827] ? hrtimer_interrupt+0x128/0x2c0 [ 6360.808253] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6360.810128] ? apic_timer_interrupt+0xf/0x20 [ 6360.811855] [ 6360.812715] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6360.815003] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6360.817143] do_raw_spin_lock+0x177/0x190 [ 6360.818711] _raw_spin_lock+0x21/0x30 [ 6360.820022] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6360.822800] obd_mod_free+0x46/0x140 [ptlrpc] [ 6360.824787] mdc_commit_open+0x91/0xb0 [mdc] [ 6360.826299] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6360.829151] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6360.832626] after_reply+0xe2c/0x1780 [ptlrpc] [ 6360.834844] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6360.837472] ? __next_timer_interrupt+0x160/0x160 [ 6360.839896] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6360.842305] ? do_wait_intr+0xf0/0xf0 [ 6360.844048] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6360.849122] kthread+0x1d1/0x200 [ 6360.851084] ? set_kthread_struct+0x70/0x70 [ 6360.854851] ret_from_fork+0x1f/0x30 [ 6360.858996] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4430s! [ 6360.872021] Showing busy workqueues and worker pools: [ 6360.879622] workqueue events: flags=0x0 [ 6360.881102] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6360.894107] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 6360.902812] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6360.921028] pending: kfree_rcu_monitor [ 6360.922290] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6360.932264] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 6360.937722] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6360.940841] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 6360.946880] workqueue events_long: flags=0x0 [ 6360.948783] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6360.967880] pending: lu_objects_flush [obdclass] [ 6360.973101] workqueue events_unbound: flags=0x2 [ 6360.975780] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6360.985217] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6361.000632] workqueue events_power_efficient: flags=0x80 [ 6361.010229] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6361.021953] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 6361.033468] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6361.035780] pending: neigh_periodic_work [ 6361.048068] workqueue mm_percpu_wq: flags=0x8 [ 6361.053905] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6361.062107] pending: vmstat_update, lru_add_drain_per_cpu [ 6361.075198] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6361.077374] pending: vmstat_update [ 6361.090245] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6361.103263] pending: lru_add_drain_per_cpu [ 6361.106282] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6361.117017] pending: lru_add_drain_per_cpu BAR(48) [ 6361.123610] workqueue writeback: flags=0x4a [ 6361.132171] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 6361.136489] pending: wb_workfn [ 6361.138110] workqueue xprtiod: flags=0xa [ 6361.140197] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6361.143865] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6361.157300] workqueue ipv6_addrconf: flags=0x40008 [ 6361.161114] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6361.165326] pending: addrconf_verify_work [ 6361.171796] workqueue knbd0-recv: flags=0x1a [ 6361.182838] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6361.191661] in-flight: 658:recv_work [ 6361.195768] workqueue ptlrpc_pinger: flags=0x2 [ 6361.205208] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6361.217932] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6361.224691] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4430s workers=4 idle: 32984 37 8 [ 6361.229195] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6345s workers=2 idle: 661 [ 6388.054430] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 6388.055915] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 6388.060313] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 6388.060327] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6388.060400] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6388.060406] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6388.060411] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6388.060741] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6388.060753] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6388.060757] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6388.060761] RAX: 000000000000348a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6388.060764] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 6388.060766] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 6388.060768] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 6388.060771] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 6388.060774] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 6388.060777] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6388.060779] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6388.060788] Call Trace: [ 6388.060793] [ 6388.060798] ? show_regs.cold.9+0x22/0x2f [ 6388.060807] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6388.060814] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6388.060818] ? watchdog+0x40/0x40 [ 6388.060822] ? hrtimer_interrupt+0x128/0x2c0 [ 6388.060826] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6388.060835] ? apic_timer_interrupt+0xf/0x20 [ 6388.060840] [ 6388.060842] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6388.060848] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6388.060853] do_raw_spin_lock+0x177/0x190 [ 6388.060858] _raw_spin_lock+0x21/0x30 [ 6388.060863] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6388.061021] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6388.061082] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6388.061087] process_one_work+0x2c8/0x700 [ 6388.061091] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6388.061101] worker_thread+0x296/0x6e0 [ 6388.061106] ? rescuer_thread+0x570/0x570 [ 6388.061110] kthread+0x1d1/0x200 [ 6388.061114] ? set_kthread_struct+0x70/0x70 [ 6388.061117] ret_from_fork+0x1f/0x30 [ 6388.061159] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6388.061167] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6388.061169] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6388.061173] RAX: 0000000000002f63 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6388.061175] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 6388.061176] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 6388.061178] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 6388.061180] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 6388.063077] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 6388.063089] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6388.063159] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6388.063164] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6388.063167] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6388.063178] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6388.063182] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6388.063186] RAX: 0000000000004634 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6388.063188] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 6388.063191] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6388.063193] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 6388.063195] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 6388.063199] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 6388.063202] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6388.063205] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 6388.063213] Call Trace: [ 6388.063227] [ 6388.063233] ? show_regs.cold.9+0x22/0x2f [ 6388.063242] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6388.063248] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6388.063253] ? watchdog+0x40/0x40 [ 6388.063257] ? hrtimer_interrupt+0x128/0x2c0 [ 6388.063262] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6388.063270] ? apic_timer_interrupt+0xf/0x20 [ 6388.063274] [ 6388.063277] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6388.063284] do_raw_spin_lock+0x177/0x190 [ 6388.063289] _raw_spin_lock+0x21/0x30 [ 6388.063293] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6388.065061] ? do_raw_spin_unlock+0x75/0x190 [ 6388.065072] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6388.065285] ? do_wait_intr+0xf0/0xf0 [ 6388.065294] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6388.065765] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6388.065833] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6388.069598] mdc_close+0x53b/0xd70 [mdc] [ 6388.069638] lmv_close+0x18d/0x490 [lmv] [ 6388.069667] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6388.069827] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6388.069937] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6388.070023] ll_md_close+0x274/0x800 [lustre] [ 6388.070112] ll_file_release+0x128/0xc50 [lustre] [ 6388.070199] __fput+0x10d/0x440 [ 6388.070209] ____fput+0x12/0x20 [ 6388.070214] task_work_run+0xc6/0x110 [ 6388.070234] exit_to_usermode_loop+0x1dd/0x1f0 [ 6388.070240] do_syscall_64+0x3d6/0x440 [ 6388.070246] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6388.070254] RIP: 0033:0x7f151bee3cd8 [ 6388.070263] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6388.070267] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6388.070272] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 6388.070274] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 6388.070276] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6388.070278] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6388.070280] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6388.070911] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 6388.070917] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6388.070920] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 6388.070929] Call Trace: [ 6388.070934] [ 6388.070937] ? show_regs.cold.9+0x22/0x2f [ 6388.070949] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6388.070955] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6388.070959] ? watchdog+0x40/0x40 [ 6388.070964] ? hrtimer_interrupt+0x128/0x2c0 [ 6388.070968] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6388.070974] ? apic_timer_interrupt+0xf/0x20 [ 6388.070979] [ 6388.070982] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6388.070990] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6388.070995] do_raw_spin_lock+0x177/0x190 [ 6388.070999] _raw_spin_lock+0x21/0x30 [ 6388.071005] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6388.071422] ? do_raw_spin_unlock+0x75/0x190 [ 6388.071428] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6388.107059] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6388.111826] ? do_wait_intr+0xf0/0xf0 [ 6388.111849] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6388.121463] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6388.121485] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6388.121489] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6388.121495] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 6388.121498] RDX: 0000000000004d49 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 6388.121501] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 6388.121503] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 6388.121505] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6388.121509] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 6388.121512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6388.121514] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6388.121522] Call Trace: [ 6388.121527] [ 6388.121532] ? show_regs.cold.9+0x22/0x2f [ 6388.121543] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6388.121549] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6388.121554] ? watchdog+0x40/0x40 [ 6388.121558] ? hrtimer_interrupt+0x128/0x2c0 [ 6388.121562] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6388.121570] ? apic_timer_interrupt+0xf/0x20 [ 6388.121575] [ 6388.121577] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6388.121582] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6388.121587] do_raw_spin_lock+0x177/0x190 [ 6388.121591] _raw_spin_lock+0x21/0x30 [ 6388.121599] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6388.121874] obd_mod_free+0x46/0x140 [ptlrpc] [ 6388.122083] mdc_commit_open+0x91/0xb0 [mdc] [ 6388.122118] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6388.122342] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6388.122555] after_reply+0xe2c/0x1780 [ptlrpc] [ 6388.122766] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6388.122977] ? __next_timer_interrupt+0x160/0x160 [ 6388.122986] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6388.123203] ? do_wait_intr+0xf0/0xf0 [ 6388.123212] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6388.123578] kthread+0x1d1/0x200 [ 6388.123584] ? set_kthread_struct+0x70/0x70 [ 6388.123588] ret_from_fork+0x1f/0x30 [ 6388.953700] mdc_reint+0x2d/0x110 [mdc] [ 6388.957934] mdc_unlink+0x658/0x870 [mdc] [ 6388.959842] lmv_unlink+0x34d/0xbd0 [lmv] [ 6388.962957] ll_unlink+0x24d/0x7f0 [lustre] [ 6388.967411] vfs_unlink+0x202/0x330 [ 6388.969520] do_unlinkat+0x2fa/0x460 [ 6388.972129] __x64_sys_unlink+0x37/0x50 [ 6388.980916] do_syscall_64+0xc1/0x440 [ 6388.983555] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6388.991182] RIP: 0033:0x7f151bdee10b [ 6388.992490] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6389.011279] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6389.018608] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 6389.026487] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 6389.028628] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6389.035200] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6389.046035] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6391.265053] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4460s! [ 6391.271607] Showing busy workqueues and worker pools: [ 6391.274212] workqueue events: flags=0x0 [ 6391.275725] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6391.286472] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 6391.299903] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6391.302701] pending: kfree_rcu_monitor [ 6391.304546] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6391.309474] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 6391.315273] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6391.318094] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 6391.334564] workqueue events_long: flags=0x0 [ 6391.349431] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6391.362661] pending: lu_objects_flush [obdclass] [ 6391.371718] workqueue events_unbound: flags=0x2 [ 6391.376572] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6391.387217] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6391.408936] workqueue events_power_efficient: flags=0x80 [ 6391.423768] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6391.434501] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 6391.443904] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6391.451627] pending: neigh_periodic_work [ 6391.457331] workqueue mm_percpu_wq: flags=0x8 [ 6391.459032] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6391.469491] pending: vmstat_update, lru_add_drain_per_cpu [ 6391.476270] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6391.479303] pending: vmstat_update [ 6391.482438] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6391.493765] pending: lru_add_drain_per_cpu [ 6391.499266] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6391.507467] pending: lru_add_drain_per_cpu BAR(48) [ 6391.512675] workqueue writeback: flags=0x4a [ 6391.517527] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 6391.527899] pending: wb_workfn [ 6391.534801] workqueue xprtiod: flags=0xa [ 6391.546379] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6391.560440] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6391.567399] workqueue ipv6_addrconf: flags=0x40008 [ 6391.569430] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6391.577624] pending: addrconf_verify_work [ 6391.580979] workqueue knbd0-recv: flags=0x1a [ 6391.583411] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6391.589362] in-flight: 658:recv_work [ 6391.592291] workqueue ptlrpc_pinger: flags=0x2 [ 6391.595543] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6391.597596] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6391.600077] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4461s workers=4 idle: 32984 37 8 [ 6391.606009] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6376s workers=2 idle: 661 [ 6416.055281] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 6416.056084] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6416.057698] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 6416.057708] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6416.057782] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6416.057788] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6416.057791] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6416.057807] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6416.057811] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6416.057815] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 6416.057818] RDX: 000000000000248a RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 6416.057821] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 6416.057823] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 6416.057825] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6416.057829] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 6416.057831] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6416.057834] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6416.057844] Call Trace: [ 6416.057850] [ 6416.057856] ? show_regs.cold.9+0x22/0x2f [ 6416.057867] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6416.057874] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6416.057879] ? watchdog+0x40/0x40 [ 6416.057883] ? hrtimer_interrupt+0x128/0x2c0 [ 6416.057887] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6416.057897] ? apic_timer_interrupt+0xf/0x20 [ 6416.057902] [ 6416.057905] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6416.057910] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6416.057915] do_raw_spin_lock+0x177/0x190 [ 6416.057921] _raw_spin_lock+0x21/0x30 [ 6416.058454] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) [ 6416.057927] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6416.059127] ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6416.059153] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6416.059157] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6416.059159] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6416.059169] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6416.059171] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 6416.059174] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6416.059182] Modules linked in: [ 6416.059182] RAX: 000000000000588c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6416.059188] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 6416.059188] lustre(O) [ 6416.059191] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 6416.059195] osp(O) [ 6416.059195] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 6416.059199] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 6416.059201] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 6416.059202] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6416.059204] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 6416.059209] Call Trace: [ 6416.059212] [ 6416.059213] ofd(O) lod(O) mdt(O) mdd(O) [ 6416.059217] ? show_regs.cold.9+0x22/0x2f [ 6416.059224] mgs(O) lquota(O) lfsck(O) [ 6416.059230] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6416.059236] obdecho(O) mgc(O) [ 6416.059241] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6416.059245] mdc(O) lov(O) [ 6416.059250] ? watchdog+0x40/0x40 [ 6416.059254] osc(O) lmv(O) [ 6416.059261] ? hrtimer_interrupt+0x128/0x2c0 [ 6416.059270] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6416.059278] ? apic_timer_interrupt+0xf/0x20 [ 6416.059282] [ 6416.059284] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6416.059288] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6416.059290] do_raw_spin_lock+0x177/0x190 [ 6416.059294] _raw_spin_lock+0x21/0x30 [ 6416.059297] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6416.059559] ? do_raw_spin_unlock+0x75/0x190 [ 6416.059565] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6416.059755] ? do_wait_intr+0xf0/0xf0 [ 6416.059763] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6416.059975] mdc_reint+0x2d/0x110 [mdc] [ 6416.060013] mdc_unlink+0x658/0x870 [mdc] [ 6416.060041] lmv_unlink+0x34d/0xbd0 [lmv] [ 6416.059264] ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6416.061087] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6416.061094] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6416.061096] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6416.061106] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6416.061109] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6416.061113] RAX: 0000000000005e0e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6416.061116] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 6416.061118] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6416.061120] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 6416.061123] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 6416.061126] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 6416.061129] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6416.061135] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 6416.061144] Call Trace: [ 6416.061148] [ 6416.061151] ? show_regs.cold.9+0x22/0x2f [ 6416.061161] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6416.061174] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6416.061179] ? watchdog+0x40/0x40 [ 6416.061183] ? hrtimer_interrupt+0x128/0x2c0 [ 6416.061187] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6416.061195] ? apic_timer_interrupt+0xf/0x20 [ 6416.061199] [ 6416.061202] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6416.061141] ll_unlink+0x24d/0x7f0 [lustre] [ 6416.061210] do_raw_spin_lock+0x177/0x190 [ 6416.061221] _raw_spin_lock+0x21/0x30 [ 6416.061395] vfs_unlink+0x202/0x330 [ 6416.061405] do_unlinkat+0x2fa/0x460 [ 6416.061410] __x64_sys_unlink+0x37/0x50 [ 6416.061414] do_syscall_64+0xc1/0x440 [ 6416.061421] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6416.061426] RIP: 0033:0x7f151bdee10b [ 6416.061432] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6416.061435] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6416.061438] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 6416.061440] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 6416.061442] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6416.061444] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6416.061446] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6416.061226] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6416.061553] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul [ 6416.061609] ? do_raw_spin_unlock+0x75/0x190 [ 6416.061614] crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6416.061635] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6416.061643] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6416.061622] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6416.061647] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6416.062333] ? do_wait_intr+0xf0/0xf0 [ 6416.062341] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6416.062552] mdc_close+0x53b/0xd70 [mdc] [ 6416.062593] lmv_close+0x18d/0x490 [lmv] [ 6416.062628] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6416.062802] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6416.062907] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6416.062995] ll_md_close+0x274/0x800 [lustre] [ 6416.063077] ll_file_release+0x128/0xc50 [lustre] [ 6416.063158] __fput+0x10d/0x440 [ 6416.063168] ____fput+0x12/0x20 [ 6416.063172] task_work_run+0xc6/0x110 [ 6416.063179] exit_to_usermode_loop+0x1dd/0x1f0 [ 6416.063186] do_syscall_64+0x3d6/0x440 [ 6416.063190] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6416.063196] RIP: 0033:0x7f151bee3cd8 [ 6416.063202] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6416.063205] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6416.063209] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 6416.063211] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 6416.063214] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6416.063216] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6416.063218] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6416.064498] obd_mod_free+0x46/0x140 [ptlrpc] [ 6416.064766] mdc_commit_open+0x91/0xb0 [mdc] [ 6416.064806] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6416.065016] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6416.065228] after_reply+0xe2c/0x1780 [ptlrpc] [ 6416.065441] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6416.065653] ? __next_timer_interrupt+0x160/0x160 [ 6416.065664] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6416.065867] ? do_wait_intr+0xf0/0xf0 [ 6416.065877] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6416.092881] [ 6416.099734] kthread+0x1d1/0x200 [ 6416.099754] ? set_kthread_struct+0x70/0x70 [ 6416.099759] ret_from_fork+0x1f/0x30 [ 6416.787211] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6416.789885] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6416.802236] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6416.809007] RAX: 0000000000003012 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6416.816472] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 6416.821085] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 6416.824338] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 6416.828345] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 6416.834114] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 6416.842364] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6416.852031] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6416.861846] Call Trace: [ 6416.862662] [ 6416.863372] ? show_regs.cold.9+0x22/0x2f [ 6416.864715] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6416.867457] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6416.870008] ? watchdog+0x40/0x40 [ 6416.871347] ? hrtimer_interrupt+0x128/0x2c0 [ 6416.873220] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6416.875115] ? apic_timer_interrupt+0xf/0x20 [ 6416.879133] [ 6416.881125] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6416.884338] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6416.887193] do_raw_spin_lock+0x177/0x190 [ 6416.888767] _raw_spin_lock+0x21/0x30 [ 6416.891782] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6416.898065] process_one_work+0x2c8/0x700 [ 6416.901551] worker_thread+0x296/0x6e0 [ 6416.906083] ? rescuer_thread+0x570/0x570 [ 6416.912343] kthread+0x1d1/0x200 [ 6416.917371] ? set_kthread_struct+0x70/0x70 [ 6416.921196] ret_from_fork+0x1f/0x30 [ 6421.996946] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4491s! [ 6422.001670] Showing busy workqueues and worker pools: [ 6422.006491] workqueue events: flags=0x0 [ 6422.008571] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6422.017358] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 6422.030536] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6422.041258] pending: kfree_rcu_monitor [ 6422.042953] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6422.052143] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 6422.056373] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6422.059294] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 6422.063034] workqueue events_long: flags=0x0 [ 6422.064940] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6422.068269] pending: lu_objects_flush [obdclass] [ 6422.073725] workqueue events_unbound: flags=0x2 [ 6422.080045] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6422.088141] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6422.096541] workqueue events_power_efficient: flags=0x80 [ 6422.103098] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6422.106560] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 6422.113861] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6422.123213] pending: neigh_periodic_work [ 6422.124770] workqueue mm_percpu_wq: flags=0x8 [ 6422.133632] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6422.143581] pending: vmstat_update, lru_add_drain_per_cpu [ 6422.150596] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6422.156951] pending: vmstat_update [ 6422.160524] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6422.167397] pending: lru_add_drain_per_cpu [ 6422.172232] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6422.177145] pending: lru_add_drain_per_cpu BAR(48) [ 6422.185468] workqueue writeback: flags=0x4a [ 6422.192156] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 6422.200021] pending: wb_workfn [ 6422.204386] workqueue xprtiod: flags=0xa [ 6422.210027] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6422.213453] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6422.217240] workqueue ipv6_addrconf: flags=0x40008 [ 6422.219410] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6422.222419] pending: addrconf_verify_work [ 6422.224563] workqueue knbd0-recv: flags=0x1a [ 6422.226481] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6422.229490] in-flight: 658:recv_work [ 6422.231176] workqueue ptlrpc_pinger: flags=0x2 [ 6422.233105] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6422.236142] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6422.238635] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4491s workers=4 idle: 32984 37 8 [ 6422.242216] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6406s workers=2 idle: 661 [ 6444.055095] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 6444.059810] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6444.060549] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 6444.060557] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6444.060627] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6444.060633] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6444.060636] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6444.060649] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6444.060653] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6444.060658] RAX: 000000000000790e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6444.060660] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 6444.060663] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6444.060665] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 6444.060668] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 6444.060672] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 6444.060707] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6444.060710] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 6444.060719] Call Trace: [ 6444.060726] [ 6444.060731] ? show_regs.cold.9+0x22/0x2f [ 6444.060742] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6444.060749] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6444.060754] ? watchdog+0x40/0x40 [ 6444.060758] ? hrtimer_interrupt+0x128/0x2c0 [ 6444.060763] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6444.060772] ? apic_timer_interrupt+0xf/0x20 [ 6444.060776] [ 6444.060779] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6444.060787] do_raw_spin_lock+0x177/0x190 [ 6444.060792] _raw_spin_lock+0x21/0x30 [ 6444.060796] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6444.061095] ? do_raw_spin_unlock+0x75/0x190 [ 6444.061102] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6444.061336] ? do_wait_intr+0xf0/0xf0 [ 6444.061344] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6444.061538] mdc_close+0x53b/0xd70 [mdc] [ 6444.061576] lmv_close+0x18d/0x490 [lmv] [ 6444.061604] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6444.061796] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6444.061918] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6444.062005] ll_md_close+0x274/0x800 [lustre] [ 6444.062096] ll_file_release+0x128/0xc50 [lustre] [ 6444.062193] __fput+0x10d/0x440 [ 6444.062202] ____fput+0x12/0x20 [ 6444.062207] task_work_run+0xc6/0x110 [ 6444.062215] exit_to_usermode_loop+0x1dd/0x1f0 [ 6444.062222] do_syscall_64+0x3d6/0x440 [ 6444.062227] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6444.062235] RIP: 0033:0x7f151bee3cd8 [ 6444.062242] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6444.062245] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6444.062251] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 6444.062253] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 6444.062255] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6444.062258] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6444.062261] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6444.066513] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6444.066704] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6444.066711] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6444.066714] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6444.066729] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6444.066733] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6444.066738] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 6444.066741] RDX: 0000000000006cae RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 6444.066743] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 6444.066745] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 6444.066747] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6444.066750] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 6444.066753] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6444.066755] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6444.066763] Call Trace: [ 6444.066769] [ 6444.066774] ? show_regs.cold.9+0x22/0x2f [ 6444.066784] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6444.066790] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6444.066794] ? watchdog+0x40/0x40 [ 6444.066798] ? hrtimer_interrupt+0x128/0x2c0 [ 6444.066802] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6444.066809] ? apic_timer_interrupt+0xf/0x20 [ 6444.066814] [ 6444.066816] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6444.066820] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6444.066825] do_raw_spin_lock+0x177/0x190 [ 6444.066829] _raw_spin_lock+0x21/0x30 [ 6444.066835] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6444.067850] obd_mod_free+0x46/0x140 [ptlrpc] [ 6444.068111] mdc_commit_open+0x91/0xb0 [mdc] [ 6444.068146] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6444.068335] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6444.068554] after_reply+0xe2c/0x1780 [ptlrpc] [ 6444.069122] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6444.069335] ? __next_timer_interrupt+0x160/0x160 [ 6444.069357] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6444.069575] ? do_wait_intr+0xf0/0xf0 [ 6444.069584] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6444.070448] kthread+0x1d1/0x200 [ 6444.070455] ? set_kthread_struct+0x70/0x70 [ 6444.070460] ret_from_fork+0x1f/0x30 [ 6444.682346] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6444.713988] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6444.721562] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6444.728376] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6444.738947] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6444.741986] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6444.763343] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6444.769908] RAX: 0000000000007c8d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6444.778857] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 6444.784846] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 6444.788725] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 6444.798293] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 6444.815873] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 6444.822024] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6444.826042] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6444.837898] Call Trace: [ 6444.843125] [ 6444.844734] ? show_regs.cold.9+0x22/0x2f [ 6444.851190] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6444.854342] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6444.857296] ? watchdog+0x40/0x40 [ 6444.859341] ? hrtimer_interrupt+0x128/0x2c0 [ 6444.862796] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6444.870650] ? apic_timer_interrupt+0xf/0x20 [ 6444.875783] [ 6444.877015] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6444.880451] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6444.881924] do_raw_spin_lock+0x177/0x190 [ 6444.884267] _raw_spin_lock+0x21/0x30 [ 6444.887328] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6444.894153] process_one_work+0x2c8/0x700 [ 6444.899796] worker_thread+0x296/0x6e0 [ 6444.902929] ? rescuer_thread+0x570/0x570 [ 6444.909429] kthread+0x1d1/0x200 [ 6444.916072] ? set_kthread_struct+0x70/0x70 [ 6444.917613] ret_from_fork+0x1f/0x30 [ 6448.053401] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 6448.065693] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6448.139017] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6448.153503] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6448.159904] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6448.167843] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6448.207775] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6448.224249] RAX: 0000000000001b56 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6448.237114] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 6448.250078] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 6448.259844] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 6448.262213] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 6448.280692] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 6448.295036] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6448.308086] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 6448.320245] Call Trace: [ 6448.322371] [ 6448.332752] ? show_regs.cold.9+0x22/0x2f [ 6448.336322] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6448.342154] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6448.357022] ? watchdog+0x40/0x40 [ 6448.358478] ? hrtimer_interrupt+0x128/0x2c0 [ 6448.364922] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6448.374069] ? apic_timer_interrupt+0xf/0x20 [ 6448.381289] [ 6448.384273] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6448.393844] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6448.402228] do_raw_spin_lock+0x177/0x190 [ 6448.411512] _raw_spin_lock+0x21/0x30 [ 6448.412755] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6448.430104] ? do_raw_spin_unlock+0x75/0x190 [ 6448.432447] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6448.442760] ? do_wait_intr+0xf0/0xf0 [ 6448.445486] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6448.458032] mdc_reint+0x2d/0x110 [mdc] [ 6448.464740] mdc_unlink+0x658/0x870 [mdc] [ 6448.469740] lmv_unlink+0x34d/0xbd0 [lmv] [ 6448.472249] ll_unlink+0x24d/0x7f0 [lustre] [ 6448.477083] vfs_unlink+0x202/0x330 [ 6448.478995] do_unlinkat+0x2fa/0x460 [ 6448.481037] __x64_sys_unlink+0x37/0x50 [ 6448.487296] do_syscall_64+0xc1/0x440 [ 6448.489713] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6448.492433] RIP: 0033:0x7f151bdee10b [ 6448.494375] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6448.514210] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6448.530030] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 6448.537266] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 6448.548427] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6448.571457] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6448.578514] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6452.721214] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4522s! [ 6452.733065] Showing busy workqueues and worker pools: [ 6452.736481] workqueue events: flags=0x0 [ 6452.752406] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6452.762823] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 6452.775064] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6452.782758] pending: kfree_rcu_monitor [ 6452.785160] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6452.792436] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 6452.801748] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6452.821034] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 6452.836252] workqueue events_long: flags=0x0 [ 6452.852272] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6452.862823] pending: lu_objects_flush [obdclass] [ 6452.872440] workqueue events_unbound: flags=0x2 [ 6452.880525] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6452.891907] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6452.901720] workqueue events_power_efficient: flags=0x80 [ 6452.905859] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6452.910652] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 6452.915493] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6452.920468] pending: neigh_periodic_work [ 6452.927610] workqueue mm_percpu_wq: flags=0x8 [ 6452.936533] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6452.949836] pending: vmstat_update, lru_add_drain_per_cpu [ 6452.965754] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6452.976859] pending: vmstat_update [ 6452.981657] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6452.989017] pending: lru_add_drain_per_cpu [ 6452.994690] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6453.002004] pending: lru_add_drain_per_cpu BAR(48) [ 6453.009646] workqueue writeback: flags=0x4a [ 6453.014393] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 6453.025680] pending: wb_workfn [ 6453.035110] workqueue xprtiod: flags=0xa [ 6453.040133] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6453.052195] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6453.060474] workqueue ipv6_addrconf: flags=0x40008 [ 6453.070187] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6453.080734] pending: addrconf_verify_work [ 6453.086381] workqueue knbd0-recv: flags=0x1a [ 6453.088714] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6453.095438] in-flight: 658:recv_work [ 6453.098026] workqueue ptlrpc_pinger: flags=0x2 [ 6453.100304] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6453.103734] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6453.107205] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4522s workers=4 idle: 32984 37 8 [ 6453.111958] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6437s workers=2 idle: 661 [ 6472.055027] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 6472.056264] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6472.060730] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6472.060817] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6472.060825] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6472.060828] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6472.060906] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6472.060910] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6472.060915] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 6472.060918] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 6472.060920] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 6472.060922] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 6472.060924] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6472.060928] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 6472.060930] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6472.060933] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6472.060941] Call Trace: [ 6472.060946] [ 6472.060952] ? show_regs.cold.9+0x22/0x2f [ 6472.060961] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6472.060968] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6472.060972] ? watchdog+0x40/0x40 [ 6472.060977] ? hrtimer_interrupt+0x128/0x2c0 [ 6472.060981] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6472.060989] ? apic_timer_interrupt+0xf/0x20 [ 6472.060994] [ 6472.060997] ? native_safe_halt+0x2e/0x50 [ 6472.061001] kvm_wait+0xcd/0xe0 [ 6472.061009] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6472.061017] do_raw_spin_lock+0x177/0x190 [ 6472.061022] _raw_spin_lock+0x21/0x30 [ 6472.061028] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6472.061799] obd_mod_free+0x46/0x140 [ptlrpc] [ 6472.062100] mdc_commit_open+0x91/0xb0 [mdc] [ 6472.062140] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6472.062752] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6472.063042] after_reply+0xe2c/0x1780 [ptlrpc] [ 6472.063235] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 6472.063242] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6472.063304] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6472.063309] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6472.063312] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6472.063323] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6472.063327] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6472.063331] RAX: 0000000000007f86 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6472.063335] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 6472.063337] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6472.063339] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 6472.063342] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 6472.063345] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 6472.063347] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6472.063350] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 6472.063358] Call Trace: [ 6472.063362] [ 6472.063365] ? show_regs.cold.9+0x22/0x2f [ 6472.063374] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6472.063380] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6472.063384] ? watchdog+0x40/0x40 [ 6472.063387] ? hrtimer_interrupt+0x128/0x2c0 [ 6472.063391] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6472.063399] ? apic_timer_interrupt+0xf/0x20 [ 6472.063413] [ 6472.063417] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6472.063423] do_raw_spin_lock+0x177/0x190 [ 6472.063435] _raw_spin_lock+0x21/0x30 [ 6472.063441] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6472.063941] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6472.064134] ? __next_timer_interrupt+0x160/0x160 [ 6472.064143] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6472.064761] ? do_wait_intr+0xf0/0xf0 [ 6472.064772] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6472.064984] kthread+0x1d1/0x200 [ 6472.064990] ? set_kthread_struct+0x70/0x70 [ 6472.064994] ret_from_fork+0x1f/0x30 [ 6472.071189] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) [ 6472.122137] ? do_raw_spin_unlock+0x75/0x190 [ 6472.122159] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6472.123905] ? do_wait_intr+0xf0/0xf0 [ 6472.123916] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6472.124117] mdc_close+0x53b/0xd70 [mdc] [ 6472.124155] lmv_close+0x18d/0x490 [lmv] [ 6472.124189] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6472.125668] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6472.125798] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6472.125876] ll_md_close+0x274/0x800 [lustre] [ 6472.125968] ll_file_release+0x128/0xc50 [lustre] [ 6472.126075] __fput+0x10d/0x440 [ 6472.126085] ____fput+0x12/0x20 [ 6472.126090] task_work_run+0xc6/0x110 [ 6472.126098] exit_to_usermode_loop+0x1dd/0x1f0 [ 6472.126105] do_syscall_64+0x3d6/0x440 [ 6472.126109] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6472.126119] RIP: 0033:0x7f151bee3cd8 [ 6472.126127] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6472.126131] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6472.126136] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 6472.126139] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 6472.126141] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6472.126143] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6472.126145] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6472.791013] mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6472.886302] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6472.919433] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6472.940067] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6472.949905] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6472.951952] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6472.975451] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6472.981557] RAX: 00000000000077b7 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6472.984821] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 6472.987847] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 6472.990602] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 6472.993351] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 6472.995778] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 6472.998729] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6473.000762] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6473.003427] Call Trace: [ 6473.004321] [ 6473.005042] ? show_regs.cold.9+0x22/0x2f [ 6473.006701] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6473.008602] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6473.010101] ? watchdog+0x40/0x40 [ 6473.010986] ? hrtimer_interrupt+0x128/0x2c0 [ 6473.012114] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6473.013444] ? apic_timer_interrupt+0xf/0x20 [ 6473.014905] [ 6473.015732] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6473.017610] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6473.019607] do_raw_spin_lock+0x177/0x190 [ 6473.020957] _raw_spin_lock+0x21/0x30 [ 6473.022423] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6473.024671] process_one_work+0x2c8/0x700 [ 6473.026365] worker_thread+0x296/0x6e0 [ 6473.027796] ? rescuer_thread+0x570/0x570 [ 6473.029043] kthread+0x1d1/0x200 [ 6473.030163] ? set_kthread_struct+0x70/0x70 [ 6473.031632] ret_from_fork+0x1f/0x30 [ 6480.055294] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 6480.062543] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6480.113444] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6480.126924] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6480.135662] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6480.142828] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6480.163073] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6480.169631] RAX: 0000000000005587 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6480.173418] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 6480.180610] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 6480.185575] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 6480.196584] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 6480.201546] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 6480.216202] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6480.218198] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 6480.221787] Call Trace: [ 6480.225982] [ 6480.226930] ? show_regs.cold.9+0x22/0x2f [ 6480.229294] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6480.234836] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6480.237794] ? watchdog+0x40/0x40 [ 6480.239700] ? hrtimer_interrupt+0x128/0x2c0 [ 6480.244937] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6480.253581] ? apic_timer_interrupt+0xf/0x20 [ 6480.255232] [ 6480.256216] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6480.257620] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6480.260170] do_raw_spin_lock+0x177/0x190 [ 6480.262942] _raw_spin_lock+0x21/0x30 [ 6480.265539] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6480.272926] ? do_raw_spin_unlock+0x75/0x190 [ 6480.274778] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6480.276994] ? do_wait_intr+0xf0/0xf0 [ 6480.278980] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6480.283059] mdc_reint+0x2d/0x110 [mdc] [ 6480.284533] mdc_unlink+0x658/0x870 [mdc] [ 6480.285885] lmv_unlink+0x34d/0xbd0 [lmv] [ 6480.288602] ll_unlink+0x24d/0x7f0 [lustre] [ 6480.290825] vfs_unlink+0x202/0x330 [ 6480.292449] do_unlinkat+0x2fa/0x460 [ 6480.294629] __x64_sys_unlink+0x37/0x50 [ 6480.296835] do_syscall_64+0xc1/0x440 [ 6480.299900] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6480.305474] RIP: 0033:0x7f151bdee10b [ 6480.307500] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6480.317759] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6480.324718] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 6480.333753] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 6480.337646] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6480.341044] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6480.344675] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6483.427939] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4552s! [ 6483.435734] Showing busy workqueues and worker pools: [ 6483.438382] workqueue events: flags=0x0 [ 6483.439784] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6483.449788] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 6483.459921] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6483.466029] pending: kfree_rcu_monitor [ 6483.471548] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6483.478048] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 6483.488791] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6483.491921] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 6483.497593] workqueue events_long: flags=0x0 [ 6483.501414] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6483.504153] pending: lu_objects_flush [obdclass] [ 6483.506625] workqueue events_unbound: flags=0x2 [ 6483.508377] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6483.520587] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6483.532467] workqueue events_power_efficient: flags=0x80 [ 6483.546208] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6483.552121] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 6483.564699] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6483.567144] pending: neigh_periodic_work [ 6483.583568] workqueue mm_percpu_wq: flags=0x8 [ 6483.594159] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6483.611503] pending: vmstat_update, lru_add_drain_per_cpu [ 6483.618263] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6483.626020] pending: vmstat_update [ 6483.628375] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6483.632134] pending: lru_add_drain_per_cpu [ 6483.635391] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6483.640012] pending: lru_add_drain_per_cpu BAR(48) [ 6483.647653] workqueue writeback: flags=0x4a [ 6483.650477] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 6483.656288] pending: wb_workfn [ 6483.657884] workqueue xprtiod: flags=0xa [ 6483.659186] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6483.660893] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6483.663574] workqueue ipv6_addrconf: flags=0x40008 [ 6483.668945] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6483.680276] pending: addrconf_verify_work [ 6483.682525] workqueue knbd0-recv: flags=0x1a [ 6483.689047] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6483.693390] in-flight: 658:recv_work [ 6483.695713] workqueue ptlrpc_pinger: flags=0x2 [ 6483.698871] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6483.701568] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6483.707787] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4553s workers=4 idle: 32984 37 8 [ 6483.716629] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6468s workers=2 idle: 661 [ 6484.737093] rcu: INFO: rcu_sched self-detected stall on CPU [ 6484.740410] rcu: 1-...!: (2591177 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 6484.745512] (t=4560090 jiffies g=249657 q=3671) [ 6484.747637] rcu: rcu_sched kthread starved for 4555042 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 6484.752920] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 6484.756807] rcu: RCU grace-period kthread stack dump: [ 6484.758750] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 6484.763259] Call Trace: [ 6484.764372] __schedule+0x351/0xcb0 [ 6484.765988] schedule+0xc0/0x180 [ 6484.767745] schedule_timeout+0xb4/0x190 [ 6484.769594] ? do_raw_spin_unlock+0x75/0x190 [ 6484.771592] ? __next_timer_interrupt+0x160/0x160 [ 6484.774068] ? prepare_to_swait_event+0xf3/0x140 [ 6484.776634] rcu_gp_kthread+0x8dd/0x15b0 [ 6484.778698] ? force_qs_rnp+0x2c0/0x2c0 [ 6484.780591] kthread+0x1d1/0x200 [ 6484.782208] ? set_kthread_struct+0x70/0x70 [ 6484.784117] ret_from_fork+0x1f/0x30 [ 6484.785805] rcu: Stack dump where RCU GP kthread last ran: [ 6484.787975] NMI backtrace for cpu 1 [ 6484.789833] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6484.794622] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6484.798946] Call Trace: [ 6484.800379] [ 6484.801326] ? dump_stack+0xbb/0x10e [ 6484.803462] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6484.805124] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6484.806714] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6484.809106] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6484.811647] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6484.814127] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 6484.816739] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 6484.819875] ? update_process_times+0x6d/0xc0 [ 6484.821721] ? tick_sched_handle.isra.18+0x27/0x90 [ 6484.824606] ? tick_sched_timer+0x53/0xb0 [ 6484.826456] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6484.828737] ? tick_sched_handle.isra.18+0x90/0x90 [ 6484.830809] ? hrtimer_interrupt+0x128/0x2c0 [ 6484.832420] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6484.834400] ? apic_timer_interrupt+0xf/0x20 [ 6484.835836] [ 6484.836714] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6484.839542] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6484.841833] ? do_raw_spin_lock+0x177/0x190 [ 6484.843642] ? _raw_spin_lock+0x21/0x30 [ 6484.845439] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6484.848096] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 6484.850524] ? mdc_commit_open+0x91/0xb0 [mdc] [ 6484.853338] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6484.857953] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6484.862011] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 6484.864776] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6484.867687] ? __next_timer_interrupt+0x160/0x160 [ 6484.870101] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6484.872939] ? do_wait_intr+0xf0/0xf0 [ 6484.874601] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6484.878915] ? kthread+0x1d1/0x200 [ 6484.880637] ? set_kthread_struct+0x70/0x70 [ 6484.882662] ? ret_from_fork+0x1f/0x30 [ 6484.884345] NMI backtrace for cpu 1 [ 6484.885712] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6484.890314] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6484.894003] Call Trace: [ 6484.895206] [ 6484.895997] ? dump_stack+0xbb/0x10e [ 6484.897703] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6484.899828] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6484.901905] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6484.904628] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6484.906874] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6484.909492] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 6484.911471] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 6484.913607] ? update_process_times+0x6d/0xc0 [ 6484.915132] ? tick_sched_handle.isra.18+0x27/0x90 [ 6484.917140] ? tick_sched_timer+0x53/0xb0 [ 6484.918644] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6484.920276] ? tick_sched_handle.isra.18+0x90/0x90 [ 6484.922930] ? hrtimer_interrupt+0x128/0x2c0 [ 6484.924081] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6484.925938] ? apic_timer_interrupt+0xf/0x20 [ 6484.928487] [ 6484.929846] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6484.932354] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6484.934638] ? do_raw_spin_lock+0x177/0x190 [ 6484.936488] ? _raw_spin_lock+0x21/0x30 [ 6484.938481] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6484.941007] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 6484.943802] ? mdc_commit_open+0x91/0xb0 [mdc] [ 6484.946200] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6484.949013] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6484.952769] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 6484.954953] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6484.957506] ? __next_timer_interrupt+0x160/0x160 [ 6484.959369] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6484.961324] ? do_wait_intr+0xf0/0xf0 [ 6484.962778] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6484.965850] ? kthread+0x1d1/0x200 [ 6484.967578] ? set_kthread_struct+0x70/0x70 [ 6484.969806] ? ret_from_fork+0x1f/0x30 [ 6500.056108] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6500.058355] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 6500.060434] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6500.060742] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6500.060751] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6500.060760] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6500.061398] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6500.061409] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6500.061414] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6500.061418] RAX: 0000000000007e0c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6500.061420] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 6500.061422] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 6500.061425] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 6500.061427] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 6500.061431] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 6500.061434] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6500.061437] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6500.061446] Call Trace: [ 6500.061450] [ 6500.061455] ? show_regs.cold.9+0x22/0x2f [ 6500.061464] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6500.061471] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6500.061475] ? watchdog+0x40/0x40 [ 6500.061480] ? hrtimer_interrupt+0x128/0x2c0 [ 6500.061484] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6500.061493] ? apic_timer_interrupt+0xf/0x20 [ 6500.061497] [ 6500.061500] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6500.061505] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6500.061510] do_raw_spin_lock+0x177/0x190 [ 6500.061515] _raw_spin_lock+0x21/0x30 [ 6500.061519] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6500.070126] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6500.070216] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6500.070224] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6500.070226] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6500.070241] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6500.070246] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6500.070251] RAX: 0000000000007f96 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6500.070254] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 6500.070256] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6500.070259] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 6500.070261] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 6500.070265] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 6500.070268] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6500.070271] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 6500.070278] Call Trace: [ 6500.070283] [ 6500.070289] ? show_regs.cold.9+0x22/0x2f [ 6500.070297] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6500.070304] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6500.070309] ? watchdog+0x40/0x40 [ 6500.070314] ? hrtimer_interrupt+0x128/0x2c0 [ 6500.070318] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6500.070325] ? apic_timer_interrupt+0xf/0x20 [ 6500.070330] [ 6500.070333] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6500.070339] do_raw_spin_lock+0x177/0x190 [ 6500.070344] _raw_spin_lock+0x21/0x30 [ 6500.070348] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6500.104666] process_one_work+0x2c8/0x700 [ 6500.104685] worker_thread+0x296/0x6e0 [ 6500.104691] ? rescuer_thread+0x570/0x570 [ 6500.104695] kthread+0x1d1/0x200 [ 6500.104699] ? set_kthread_struct+0x70/0x70 [ 6500.104704] ret_from_fork+0x1f/0x30 [ 6500.538435] ? do_raw_spin_unlock+0x75/0x190 [ 6500.547721] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6500.549687] ? do_wait_intr+0xf0/0xf0 [ 6500.551012] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6500.564222] mdc_close+0x53b/0xd70 [mdc] [ 6500.568536] lmv_close+0x18d/0x490 [lmv] [ 6500.571912] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6500.578856] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6500.582471] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6500.588144] ll_md_close+0x274/0x800 [lustre] [ 6500.592146] ll_file_release+0x128/0xc50 [lustre] [ 6500.596952] __fput+0x10d/0x440 [ 6500.602590] ____fput+0x12/0x20 [ 6500.605663] task_work_run+0xc6/0x110 [ 6500.606979] exit_to_usermode_loop+0x1dd/0x1f0 [ 6500.611084] do_syscall_64+0x3d6/0x440 [ 6500.622309] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6500.631284] RIP: 0033:0x7f151bee3cd8 [ 6500.639180] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6500.661592] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6500.671937] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 6500.682454] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 6500.689765] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6500.697781] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6500.705266] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6508.052037] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 6508.055188] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6508.083617] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6508.090520] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6508.093625] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6508.095332] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6508.102608] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6508.105628] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6508.108344] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 6508.111343] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 6508.116017] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 6508.120335] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 6508.123533] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 6508.129491] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6508.133111] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 6508.142677] Call Trace: [ 6508.146140] [ 6508.146744] ? show_regs.cold.9+0x22/0x2f [ 6508.149185] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6508.152189] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6508.154173] ? watchdog+0x40/0x40 [ 6508.156295] ? hrtimer_interrupt+0x128/0x2c0 [ 6508.158522] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6508.163384] ? apic_timer_interrupt+0xf/0x20 [ 6508.165176] [ 6508.165740] ? native_safe_halt+0x2e/0x50 [ 6508.168664] kvm_wait+0xcd/0xe0 [ 6508.171510] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6508.173359] do_raw_spin_lock+0x177/0x190 [ 6508.174544] _raw_spin_lock+0x21/0x30 [ 6508.176737] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6508.178417] ? do_raw_spin_unlock+0x75/0x190 [ 6508.180369] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6508.182521] ? do_wait_intr+0xf0/0xf0 [ 6508.188303] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6508.189814] mdc_reint+0x2d/0x110 [mdc] [ 6508.190849] mdc_unlink+0x658/0x870 [mdc] [ 6508.195685] lmv_unlink+0x34d/0xbd0 [lmv] [ 6508.197342] ll_unlink+0x24d/0x7f0 [lustre] [ 6508.198678] vfs_unlink+0x202/0x330 [ 6508.200074] do_unlinkat+0x2fa/0x460 [ 6508.201123] __x64_sys_unlink+0x37/0x50 [ 6508.202994] do_syscall_64+0xc1/0x440 [ 6508.204008] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6508.205776] RIP: 0033:0x7f151bdee10b [ 6508.207541] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6508.215547] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6508.220108] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 6508.223713] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 6508.226711] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6508.229312] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6508.231697] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6512.058217] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 6512.066818] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6512.099478] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6512.105135] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6512.111256] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6512.113426] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6512.138111] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6512.143277] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 6512.148504] RDX: 0000000000001fe3 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 6512.155383] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 6512.160283] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 6512.171414] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6512.179437] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 6512.187401] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6512.194776] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6512.199316] Call Trace: [ 6512.202591] [ 6512.206566] ? show_regs.cold.9+0x22/0x2f [ 6512.208025] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6512.209709] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6512.221385] ? watchdog+0x40/0x40 [ 6512.225642] ? hrtimer_interrupt+0x128/0x2c0 [ 6512.232383] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6512.236563] ? apic_timer_interrupt+0xf/0x20 [ 6512.238749] [ 6512.247337] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6512.249543] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6512.252631] do_raw_spin_lock+0x177/0x190 [ 6512.259126] _raw_spin_lock+0x21/0x30 [ 6512.260493] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6512.269429] obd_mod_free+0x46/0x140 [ptlrpc] [ 6512.274336] mdc_commit_open+0x91/0xb0 [mdc] [ 6512.276351] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6512.284751] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6512.289471] after_reply+0xe2c/0x1780 [ptlrpc] [ 6512.295477] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6512.298511] ? __next_timer_interrupt+0x160/0x160 [ 6512.304454] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6512.310045] ? do_wait_intr+0xf0/0xf0 [ 6512.311580] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6512.315421] kthread+0x1d1/0x200 [ 6512.317145] ? set_kthread_struct+0x70/0x70 [ 6512.319266] ret_from_fork+0x1f/0x30 [ 6514.153941] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4583s! [ 6514.162658] Showing busy workqueues and worker pools: [ 6514.167085] workqueue events: flags=0x0 [ 6514.171044] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6514.175626] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 6514.181560] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6514.185654] pending: kfree_rcu_monitor [ 6514.187986] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6514.191391] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 6514.195599] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6514.198748] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 6514.202757] workqueue events_long: flags=0x0 [ 6514.204406] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6514.206989] pending: lu_objects_flush [obdclass] [ 6514.212083] workqueue events_unbound: flags=0x2 [ 6514.214916] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6514.216822] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6514.221628] workqueue events_power_efficient: flags=0x80 [ 6514.225768] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6514.228356] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 6514.236937] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6514.241851] pending: neigh_periodic_work [ 6514.244329] workqueue mm_percpu_wq: flags=0x8 [ 6514.249280] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6514.254275] pending: vmstat_update, lru_add_drain_per_cpu [ 6514.258647] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6514.268595] pending: vmstat_update [ 6514.269891] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6514.277336] pending: lru_add_drain_per_cpu [ 6514.280020] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6514.285204] pending: lru_add_drain_per_cpu BAR(48) [ 6514.291547] workqueue writeback: flags=0x4a [ 6514.293542] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 6514.299034] pending: wb_workfn [ 6514.303905] workqueue xprtiod: flags=0xa [ 6514.307372] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6514.314177] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6514.326279] workqueue ipv6_addrconf: flags=0x40008 [ 6514.334585] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6514.339707] pending: addrconf_verify_work [ 6514.342370] workqueue knbd0-recv: flags=0x1a [ 6514.346571] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6514.354191] in-flight: 658:recv_work [ 6514.358956] workqueue ptlrpc_pinger: flags=0x2 [ 6514.364286] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6514.373382] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6514.376125] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4583s workers=4 idle: 32984 37 8 [ 6514.386807] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6499s workers=2 idle: 661 [ 6528.056030] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6528.057025] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 6528.057853] Modules linked in: [ 6528.065457] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6528.065532] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6528.065540] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6528.065543] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6528.065557] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6528.065561] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6528.065566] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6528.065568] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 6528.065570] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6528.065573] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 6528.065575] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 6528.065578] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 6528.065581] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6528.065583] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 6528.065592] Call Trace: [ 6528.065596] [ 6528.065601] ? show_regs.cold.9+0x22/0x2f [ 6528.065611] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6528.065617] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6528.065621] ? watchdog+0x40/0x40 [ 6528.065625] ? hrtimer_interrupt+0x128/0x2c0 [ 6528.065628] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6528.065634] ? apic_timer_interrupt+0xf/0x20 [ 6528.065639] [ 6528.065642] ? native_safe_halt+0x2e/0x50 [ 6528.065646] kvm_wait+0xcd/0xe0 [ 6528.065653] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6528.065661] do_raw_spin_lock+0x177/0x190 [ 6528.065665] _raw_spin_lock+0x21/0x30 [ 6528.065669] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6528.066596] ? do_raw_spin_unlock+0x75/0x190 [ 6528.066604] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6528.106509] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 [ 6528.124841] ? do_wait_intr+0xf0/0xf0 [ 6528.124874] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6528.125405] mdc_close+0x53b/0xd70 [mdc] [ 6528.125448] lmv_close+0x18d/0x490 [lmv] [ 6528.125480] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6528.125641] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6528.125886] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6528.125967] ll_md_close+0x274/0x800 [lustre] [ 6528.126051] ll_file_release+0x128/0xc50 [lustre] [ 6528.126301] __fput+0x10d/0x440 [ 6528.126311] ____fput+0x12/0x20 [ 6528.126315] task_work_run+0xc6/0x110 [ 6528.126322] exit_to_usermode_loop+0x1dd/0x1f0 [ 6528.126329] do_syscall_64+0x3d6/0x440 [ 6528.126333] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6528.126341] RIP: 0033:0x7f151bee3cd8 [ 6528.126349] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6528.126353] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6528.126358] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 6528.126361] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 6528.126363] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6528.126364] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6528.126367] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6528.372984] sha512_generic [ 6528.373971] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6528.378099] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6528.381309] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6528.383694] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6528.385503] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6528.392333] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6528.395696] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6528.398578] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 6528.401125] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 6528.405017] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 6528.406886] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 6528.409208] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 6528.412705] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6528.415000] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6528.417675] Call Trace: [ 6528.418619] [ 6528.419394] ? show_regs.cold.9+0x22/0x2f [ 6528.421107] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6528.422785] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6528.424542] ? watchdog+0x40/0x40 [ 6528.425785] ? hrtimer_interrupt+0x128/0x2c0 [ 6528.427353] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6528.429477] ? apic_timer_interrupt+0xf/0x20 [ 6528.431132] [ 6528.431995] ? native_safe_halt+0x2e/0x50 [ 6528.433675] kvm_wait+0xcd/0xe0 [ 6528.435126] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6528.437244] do_raw_spin_lock+0x177/0x190 [ 6528.439052] _raw_spin_lock+0x21/0x30 [ 6528.440681] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6528.443140] process_one_work+0x2c8/0x700 [ 6528.444955] worker_thread+0x296/0x6e0 [ 6528.446400] ? rescuer_thread+0x570/0x570 [ 6528.448081] kthread+0x1d1/0x200 [ 6528.449354] ? set_kthread_struct+0x70/0x70 [ 6528.451013] ret_from_fork+0x1f/0x30 [ 6540.055042] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 6540.056705] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 6540.062194] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6540.062345] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6540.062353] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6540.062356] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6540.062377] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6540.062381] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6540.062386] RAX: 0000000000007e14 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6540.062388] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 6540.062390] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 6540.062392] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 6540.062394] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 6540.062397] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 6540.062400] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6540.062402] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 6540.062410] Call Trace: [ 6540.062416] [ 6540.062422] ? show_regs.cold.9+0x22/0x2f [ 6540.062442] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6540.062448] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6540.062452] ? watchdog+0x40/0x40 [ 6540.062456] ? hrtimer_interrupt+0x128/0x2c0 [ 6540.062461] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6540.062469] ? apic_timer_interrupt+0xf/0x20 [ 6540.062474] [ 6540.062477] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6540.062482] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6540.063050] do_raw_spin_lock+0x177/0x190 [ 6540.063055] _raw_spin_lock+0x21/0x30 [ 6540.063062] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6540.063420] ? do_raw_spin_unlock+0x75/0x190 [ 6540.063473] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6540.064395] ? do_wait_intr+0xf0/0xf0 [ 6540.064403] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6540.065083] mdc_reint+0x2d/0x110 [mdc] [ 6540.065123] mdc_unlink+0x658/0x870 [mdc] [ 6540.065151] lmv_unlink+0x34d/0xbd0 [lmv] [ 6540.065183] ll_unlink+0x24d/0x7f0 [lustre] [ 6540.075252] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) [ 6540.142838] vfs_unlink+0x202/0x330 [ 6540.142859] do_unlinkat+0x2fa/0x460 [ 6540.142865] __x64_sys_unlink+0x37/0x50 [ 6540.142869] do_syscall_64+0xc1/0x440 [ 6540.158361] obdecho(O) [ 6540.168981] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6540.169006] RIP: 0033:0x7f151bdee10b [ 6540.172172] mgc(O) [ 6540.193163] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6540.193176] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6540.193183] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 6540.193186] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 6540.193189] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6540.193192] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6540.193194] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6540.483127] mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6540.547378] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6540.575788] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6540.590846] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6540.600073] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6540.614327] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6540.619728] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 6540.628341] RDX: 00000000000050ec RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 6540.638497] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 6540.644850] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 6540.647114] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6540.666905] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 6540.679214] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6540.683856] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6540.694228] Call Trace: [ 6540.699583] [ 6540.700362] ? show_regs.cold.9+0x22/0x2f [ 6540.706236] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6540.709368] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6540.714818] ? watchdog+0x40/0x40 [ 6540.720122] ? hrtimer_interrupt+0x128/0x2c0 [ 6540.727866] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6540.731795] ? apic_timer_interrupt+0xf/0x20 [ 6540.734104] [ 6540.735566] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6540.739859] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6540.746552] do_raw_spin_lock+0x177/0x190 [ 6540.749270] _raw_spin_lock+0x21/0x30 [ 6540.751852] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6540.754902] obd_mod_free+0x46/0x140 [ptlrpc] [ 6540.757691] mdc_commit_open+0x91/0xb0 [mdc] [ 6540.759913] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6540.763568] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6540.767320] after_reply+0xe2c/0x1780 [ptlrpc] [ 6540.770042] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6540.772829] ? __next_timer_interrupt+0x160/0x160 [ 6540.775144] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6540.777247] ? do_wait_intr+0xf0/0xf0 [ 6540.778613] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6540.781964] kthread+0x1d1/0x200 [ 6540.783689] ? set_kthread_struct+0x70/0x70 [ 6540.785848] ret_from_fork+0x1f/0x30 [ 6544.866101] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4614s! [ 6544.869932] Showing busy workqueues and worker pools: [ 6544.872513] workqueue events: flags=0x0 [ 6544.874540] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6544.878192] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 6544.883643] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6544.887264] pending: kfree_rcu_monitor [ 6544.889304] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6544.892803] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 6544.897904] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6544.901306] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 6544.905255] workqueue events_long: flags=0x0 [ 6544.907692] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6544.911417] pending: lu_objects_flush [obdclass] [ 6544.914194] workqueue events_unbound: flags=0x2 [ 6544.916378] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6544.919510] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6544.925013] workqueue events_power_efficient: flags=0x80 [ 6544.927871] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6544.931155] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 6544.935116] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6544.938538] pending: neigh_periodic_work [ 6544.940881] workqueue mm_percpu_wq: flags=0x8 [ 6544.943113] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6544.947037] pending: vmstat_update, lru_add_drain_per_cpu [ 6544.952548] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6544.956327] pending: vmstat_update [ 6544.959109] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6544.962543] pending: lru_add_drain_per_cpu [ 6544.964899] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6544.968039] pending: lru_add_drain_per_cpu BAR(48) [ 6544.970779] workqueue writeback: flags=0x4a [ 6544.972953] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 6544.976325] pending: wb_workfn [ 6544.978151] workqueue xprtiod: flags=0xa [ 6544.980045] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6544.983271] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6544.988118] workqueue ipv6_addrconf: flags=0x40008 [ 6544.990748] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6544.994422] pending: addrconf_verify_work [ 6544.996743] workqueue knbd0-recv: flags=0x1a [ 6544.999377] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6545.003333] in-flight: 658:recv_work [ 6545.005258] workqueue ptlrpc_pinger: flags=0x2 [ 6545.007736] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6545.011269] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6545.014274] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4614s workers=4 idle: 32984 37 8 [ 6545.018200] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6529s workers=2 idle: 661 [ 6556.056030] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6556.057036] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 6556.067780] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6556.068619] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6556.068626] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6556.068632] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6556.069712] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6556.069725] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6556.069729] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6556.069734] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6556.069737] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 6556.069739] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 6556.069741] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 6556.069744] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 6556.069748] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 6556.069751] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6556.069753] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6556.069762] Call Trace: [ 6556.069767] [ 6556.069772] ? show_regs.cold.9+0x22/0x2f [ 6556.069782] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6556.069804] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6556.069808] ? watchdog+0x40/0x40 [ 6556.069812] ? hrtimer_interrupt+0x128/0x2c0 [ 6556.069816] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6556.069822] ? apic_timer_interrupt+0xf/0x20 [ 6556.069827] [ 6556.069829] ? native_safe_halt+0x2e/0x50 [ 6556.069833] kvm_wait+0xcd/0xe0 [ 6556.069840] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6556.069848] do_raw_spin_lock+0x177/0x190 [ 6556.072773] _raw_spin_lock+0x21/0x30 [ 6556.088389] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6556.088529] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6556.088537] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6556.088540] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6556.088553] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6556.088557] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6556.088562] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6556.088565] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 6556.088567] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6556.088569] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 6556.088571] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 6556.088574] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 6556.088577] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6556.088579] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 6556.088597] Call Trace: [ 6556.088603] [ 6556.088608] ? show_regs.cold.9+0x22/0x2f [ 6556.088619] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6556.088625] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6556.088629] ? watchdog+0x40/0x40 [ 6556.088634] ? hrtimer_interrupt+0x128/0x2c0 [ 6556.088638] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6556.088645] ? apic_timer_interrupt+0xf/0x20 [ 6556.088650] [ 6556.088653] ? native_safe_halt+0x2e/0x50 [ 6556.088657] kvm_wait+0xcd/0xe0 [ 6556.088665] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6556.088673] do_raw_spin_lock+0x177/0x190 [ 6556.088678] _raw_spin_lock+0x21/0x30 [ 6556.088683] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6556.088950] ? do_raw_spin_unlock+0x75/0x190 [ 6556.088956] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6556.154557] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6556.158207] process_one_work+0x2c8/0x700 [ 6556.158218] worker_thread+0x296/0x6e0 [ 6556.158223] ? rescuer_thread+0x570/0x570 [ 6556.158227] kthread+0x1d1/0x200 [ 6556.158232] ? set_kthread_struct+0x70/0x70 [ 6556.158235] ret_from_fork+0x1f/0x30 [ 6556.766071] ? do_wait_intr+0xf0/0xf0 [ 6556.774424] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6556.779903] mdc_close+0x53b/0xd70 [mdc] [ 6556.781269] lmv_close+0x18d/0x490 [lmv] [ 6556.788182] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6556.793272] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6556.800818] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6556.805893] ll_md_close+0x274/0x800 [lustre] [ 6556.811030] ll_file_release+0x128/0xc50 [lustre] [ 6556.818093] __fput+0x10d/0x440 [ 6556.820265] ____fput+0x12/0x20 [ 6556.825117] task_work_run+0xc6/0x110 [ 6556.826476] exit_to_usermode_loop+0x1dd/0x1f0 [ 6556.832724] do_syscall_64+0x3d6/0x440 [ 6556.838651] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6556.845767] RIP: 0033:0x7f151bee3cd8 [ 6556.855727] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6556.873603] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6556.878756] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 6556.882420] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 6556.892110] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6556.900678] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6556.908690] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6568.057123] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 6568.065486] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6568.100831] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6568.114113] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6568.124385] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6568.126355] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6568.157671] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6568.165869] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 6568.172246] RDX: 00000000000010fa RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 6568.182685] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 6568.187081] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 6568.193897] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6568.201861] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 6568.206860] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6568.212613] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6568.217746] Call Trace: [ 6568.219715] [ 6568.221685] ? show_regs.cold.9+0x22/0x2f [ 6568.225160] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6568.229612] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6568.233657] ? watchdog+0x40/0x40 [ 6568.234838] ? hrtimer_interrupt+0x128/0x2c0 [ 6568.236173] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6568.237906] ? apic_timer_interrupt+0xf/0x20 [ 6568.241909] [ 6568.242871] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6568.246893] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6568.255902] do_raw_spin_lock+0x177/0x190 [ 6568.263762] _raw_spin_lock+0x21/0x30 [ 6568.266827] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6568.274914] obd_mod_free+0x46/0x140 [ptlrpc] [ 6568.278632] mdc_commit_open+0x91/0xb0 [mdc] [ 6568.281505] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6568.285308] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6568.290206] after_reply+0xe2c/0x1780 [ptlrpc] [ 6568.295750] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6568.300804] ? __next_timer_interrupt+0x160/0x160 [ 6568.303306] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6568.310712] ? do_wait_intr+0xf0/0xf0 [ 6568.313946] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6568.318864] kthread+0x1d1/0x200 [ 6568.321048] ? set_kthread_struct+0x70/0x70 [ 6568.323598] ret_from_fork+0x1f/0x30 [ 6572.053604] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 6572.062723] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6572.082266] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6572.086545] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6572.089473] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6572.091322] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6572.097793] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6572.100695] RAX: 0000000000000b44 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6572.103140] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 6572.105007] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 6572.107141] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 6572.109551] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 6572.111915] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 6572.114216] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6572.119753] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 6572.124651] Call Trace: [ 6572.126209] [ 6572.129504] ? show_regs.cold.9+0x22/0x2f [ 6572.132044] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6572.135900] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6572.140996] ? watchdog+0x40/0x40 [ 6572.142256] ? hrtimer_interrupt+0x128/0x2c0 [ 6572.146424] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6572.150702] ? apic_timer_interrupt+0xf/0x20 [ 6572.154801] [ 6572.159898] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6572.162835] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6572.166066] do_raw_spin_lock+0x177/0x190 [ 6572.170080] _raw_spin_lock+0x21/0x30 [ 6572.173949] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6572.185830] ? do_raw_spin_unlock+0x75/0x190 [ 6572.190793] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6572.197556] ? do_wait_intr+0xf0/0xf0 [ 6572.202221] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6572.204129] mdc_reint+0x2d/0x110 [mdc] [ 6572.212491] mdc_unlink+0x658/0x870 [mdc] [ 6572.215885] lmv_unlink+0x34d/0xbd0 [lmv] [ 6572.217796] ll_unlink+0x24d/0x7f0 [lustre] [ 6572.220304] vfs_unlink+0x202/0x330 [ 6572.222241] do_unlinkat+0x2fa/0x460 [ 6572.225512] __x64_sys_unlink+0x37/0x50 [ 6572.227673] do_syscall_64+0xc1/0x440 [ 6572.234422] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6572.240655] RIP: 0033:0x7f151bdee10b [ 6572.244216] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6572.256961] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6572.264483] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 6572.273834] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 6572.280647] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6572.288211] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6572.295860] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6575.076370] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4644s! [ 6575.084293] Showing busy workqueues and worker pools: [ 6575.090834] workqueue events: flags=0x0 [ 6575.092494] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6575.101611] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 6575.111607] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6575.121241] pending: kfree_rcu_monitor [ 6575.126911] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6575.129819] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 6575.141907] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6575.147560] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 6575.156674] workqueue events_long: flags=0x0 [ 6575.164817] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6575.174194] pending: lu_objects_flush [obdclass] [ 6575.177531] workqueue events_unbound: flags=0x2 [ 6575.179310] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6575.190176] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6575.198134] workqueue events_power_efficient: flags=0x80 [ 6575.203585] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6575.207141] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 6575.213379] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6575.221157] pending: neigh_periodic_work [ 6575.222737] workqueue mm_percpu_wq: flags=0x8 [ 6575.227648] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6575.234962] pending: vmstat_update, lru_add_drain_per_cpu [ 6575.239797] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6575.243438] pending: vmstat_update [ 6575.248414] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6575.254552] pending: lru_add_drain_per_cpu [ 6575.262281] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6575.266969] pending: lru_add_drain_per_cpu BAR(48) [ 6575.269813] workqueue writeback: flags=0x4a [ 6575.274819] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 6575.281680] pending: wb_workfn [ 6575.284041] workqueue xprtiod: flags=0xa [ 6575.286266] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6575.289776] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6575.294231] workqueue ipv6_addrconf: flags=0x40008 [ 6575.296852] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6575.299790] pending: addrconf_verify_work [ 6575.301606] workqueue knbd0-recv: flags=0x1a [ 6575.303666] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6575.306504] in-flight: 658:recv_work [ 6575.312476] workqueue ptlrpc_pinger: flags=0x2 [ 6575.314991] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6575.323701] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6575.328762] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4644s workers=4 idle: 32984 37 8 [ 6575.335825] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6560s workers=2 idle: 661 [ 6584.056109] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6584.064095] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6584.065608] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 6584.095970] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6584.095985] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6584.095992] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6584.096273] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6584.096284] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6584.096288] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6584.096292] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6584.096295] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 6584.096297] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 6584.096299] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 6584.096301] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 6584.096305] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 6584.096308] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6584.096310] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6584.096329] Call Trace: [ 6584.096335] [ 6584.096340] ? show_regs.cold.9+0x22/0x2f [ 6584.096350] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6584.096356] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6584.096360] ? watchdog+0x40/0x40 [ 6584.096364] ? hrtimer_interrupt+0x128/0x2c0 [ 6584.096369] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6584.096376] ? apic_timer_interrupt+0xf/0x20 [ 6584.096382] [ 6584.096385] ? native_safe_halt+0x2e/0x50 [ 6584.096389] kvm_wait+0xcd/0xe0 [ 6584.096397] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6584.096404] do_raw_spin_lock+0x177/0x190 [ 6584.096408] _raw_spin_lock+0x21/0x30 [ 6584.096412] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6584.105698] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6584.105783] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6584.105790] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6584.105793] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6584.105806] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6584.105810] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6584.105814] RAX: 0000000000007f87 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6584.105817] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 6584.105819] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6584.105821] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 6584.105823] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 6584.105827] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 6584.105830] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6584.105832] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 6584.105839] Call Trace: [ 6584.105843] [ 6584.105848] ? show_regs.cold.9+0x22/0x2f [ 6584.105856] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6584.105862] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6584.105867] ? watchdog+0x40/0x40 [ 6584.105871] ? hrtimer_interrupt+0x128/0x2c0 [ 6584.105874] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6584.105882] ? apic_timer_interrupt+0xf/0x20 [ 6584.105887] [ 6584.105891] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6584.105897] do_raw_spin_lock+0x177/0x190 [ 6584.105901] _raw_spin_lock+0x21/0x30 [ 6584.105906] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6584.106174] ? do_raw_spin_unlock+0x75/0x190 [ 6584.106586] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6584.106790] ? do_wait_intr+0xf0/0xf0 [ 6584.106804] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6584.106999] mdc_close+0x53b/0xd70 [mdc] [ 6584.107034] lmv_close+0x18d/0x490 [lmv] [ 6584.107062] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6584.107210] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6584.119219] process_one_work+0x2c8/0x700 [ 6584.119240] worker_thread+0x296/0x6e0 [ 6584.119246] ? rescuer_thread+0x570/0x570 [ 6584.119250] kthread+0x1d1/0x200 [ 6584.119254] ? set_kthread_struct+0x70/0x70 [ 6584.119258] ret_from_fork+0x1f/0x30 [ 6584.456148] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6584.459976] ll_md_close+0x274/0x800 [lustre] [ 6584.462678] ll_file_release+0x128/0xc50 [lustre] [ 6584.464427] __fput+0x10d/0x440 [ 6584.465707] ____fput+0x12/0x20 [ 6584.466783] task_work_run+0xc6/0x110 [ 6584.468614] exit_to_usermode_loop+0x1dd/0x1f0 [ 6584.471396] do_syscall_64+0x3d6/0x440 [ 6584.473588] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6584.475697] RIP: 0033:0x7f151bee3cd8 [ 6584.477881] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6584.487656] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6584.491608] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 6584.495573] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 6584.499750] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6584.502989] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6584.506032] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6596.059569] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 6596.066636] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6596.106407] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6596.109456] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6596.113131] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6596.115707] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6596.123590] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6596.127590] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 6596.132381] RDX: 0000000000007af3 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 6596.134922] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 6596.138342] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 6596.141308] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6596.144235] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 6596.148304] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6596.150965] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6596.156468] Call Trace: [ 6596.157400] [ 6596.158932] ? show_regs.cold.9+0x22/0x2f [ 6596.160331] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6596.162408] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6596.163538] ? watchdog+0x40/0x40 [ 6596.165998] ? hrtimer_interrupt+0x128/0x2c0 [ 6596.167290] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6596.168626] ? apic_timer_interrupt+0xf/0x20 [ 6596.170166] [ 6596.170928] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6596.172936] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6596.175160] do_raw_spin_lock+0x177/0x190 [ 6596.177263] _raw_spin_lock+0x21/0x30 [ 6596.180386] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6596.183343] obd_mod_free+0x46/0x140 [ptlrpc] [ 6596.186093] mdc_commit_open+0x91/0xb0 [mdc] [ 6596.189384] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6596.191841] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6596.197243] after_reply+0xe2c/0x1780 [ptlrpc] [ 6596.200579] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6596.202767] ? __next_timer_interrupt+0x160/0x160 [ 6596.204760] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6596.206764] ? do_wait_intr+0xf0/0xf0 [ 6596.208233] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6596.210541] kthread+0x1d1/0x200 [ 6596.211858] ? set_kthread_struct+0x70/0x70 [ 6596.214601] ret_from_fork+0x1f/0x30 [ 6600.054336] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 6600.060488] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6600.085512] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6600.089880] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6600.094002] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6600.097719] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6600.106894] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6600.110100] RAX: 0000000000004575 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6600.113083] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 6600.117673] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 6600.128455] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 6600.138204] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 6600.144017] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 6600.152976] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6600.156707] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 6600.160071] Call Trace: [ 6600.161811] [ 6600.162699] ? show_regs.cold.9+0x22/0x2f [ 6600.166379] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6600.168793] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6600.171800] ? watchdog+0x40/0x40 [ 6600.176246] ? hrtimer_interrupt+0x128/0x2c0 [ 6600.178245] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6600.180674] ? apic_timer_interrupt+0xf/0x20 [ 6600.182657] [ 6600.183606] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6600.186141] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6600.189014] do_raw_spin_lock+0x177/0x190 [ 6600.190877] _raw_spin_lock+0x21/0x30 [ 6600.192703] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6600.195466] ? do_raw_spin_unlock+0x75/0x190 [ 6600.197649] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6600.200475] ? do_wait_intr+0xf0/0xf0 [ 6600.202311] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6600.207938] mdc_reint+0x2d/0x110 [mdc] [ 6600.210729] mdc_unlink+0x658/0x870 [mdc] [ 6600.214894] lmv_unlink+0x34d/0xbd0 [lmv] [ 6600.216370] ll_unlink+0x24d/0x7f0 [lustre] [ 6600.218362] vfs_unlink+0x202/0x330 [ 6600.219853] do_unlinkat+0x2fa/0x460 [ 6600.221027] __x64_sys_unlink+0x37/0x50 [ 6600.222139] do_syscall_64+0xc1/0x440 [ 6600.226039] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6600.227452] RIP: 0033:0x7f151bdee10b [ 6600.231477] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6600.245518] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6600.248635] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 6600.250796] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 6600.255781] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6600.260456] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6600.264649] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6605.795132] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4675s! [ 6605.798761] Showing busy workqueues and worker pools: [ 6605.800923] workqueue events: flags=0x0 [ 6605.802746] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6605.805949] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 6605.810169] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6605.813135] pending: kfree_rcu_monitor [ 6605.814740] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6605.817548] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 6605.821391] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6605.824160] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 6605.827425] workqueue events_long: flags=0x0 [ 6605.829245] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6605.831966] pending: lu_objects_flush [obdclass] [ 6605.834182] workqueue events_unbound: flags=0x2 [ 6605.835870] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6605.838393] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6605.842507] workqueue events_power_efficient: flags=0x80 [ 6605.844594] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6605.847490] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 6605.850403] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6605.853115] pending: neigh_periodic_work [ 6605.854661] workqueue mm_percpu_wq: flags=0x8 [ 6605.856568] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6605.859437] pending: vmstat_update, lru_add_drain_per_cpu [ 6605.862008] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6605.864664] pending: vmstat_update [ 6605.866115] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6605.868764] pending: lru_add_drain_per_cpu [ 6605.870315] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6605.872807] pending: lru_add_drain_per_cpu BAR(48) [ 6605.875476] workqueue writeback: flags=0x4a [ 6605.877196] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 6605.879858] pending: wb_workfn [ 6605.881287] workqueue xprtiod: flags=0xa [ 6605.882827] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6605.885312] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6605.888497] workqueue ipv6_addrconf: flags=0x40008 [ 6605.890556] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6605.893144] pending: addrconf_verify_work [ 6605.895075] workqueue knbd0-recv: flags=0x1a [ 6605.896636] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6605.899213] in-flight: 658:recv_work [ 6605.901404] workqueue ptlrpc_pinger: flags=0x2 [ 6605.903344] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6605.906562] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6605.909117] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4675s workers=4 idle: 32984 37 8 [ 6605.912351] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6590s workers=2 idle: 661 [ 6612.056029] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6612.057026] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 6612.060002] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6612.060098] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6612.060105] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6612.060111] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6612.060545] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6612.060556] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6612.060559] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6612.060563] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6612.060566] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 6612.060568] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 6612.060570] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 6612.060572] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 6612.060575] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 6612.060577] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6612.060580] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6612.060588] Call Trace: [ 6612.060841] [ 6612.060851] ? show_regs.cold.9+0x22/0x2f [ 6612.060861] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6612.060869] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6612.060873] ? watchdog+0x40/0x40 [ 6612.060878] ? hrtimer_interrupt+0x128/0x2c0 [ 6612.060882] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6612.060889] ? apic_timer_interrupt+0xf/0x20 [ 6612.060893] [ 6612.060896] ? native_safe_halt+0x2e/0x50 [ 6612.060900] kvm_wait+0xcd/0xe0 [ 6612.060976] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6612.060987] do_raw_spin_lock+0x177/0x190 [ 6612.060991] _raw_spin_lock+0x21/0x30 [ 6612.060996] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6612.062599] process_one_work+0x2c8/0x700 [ 6612.062609] worker_thread+0x296/0x6e0 [ 6612.062614] ? rescuer_thread+0x570/0x570 [ 6612.062617] kthread+0x1d1/0x200 [ 6612.062622] ? set_kthread_struct+0x70/0x70 [ 6612.062626] ret_from_fork+0x1f/0x30 [ 6612.314445] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6612.354541] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6612.359692] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6612.363403] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6612.368531] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6612.384255] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6612.387265] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6612.390329] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 6612.394006] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6612.398140] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 6612.401422] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 6612.405569] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 6612.409192] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6612.412744] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 6612.415618] Call Trace: [ 6612.417099] [ 6612.417789] ? show_regs.cold.9+0x22/0x2f [ 6612.419501] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6612.421624] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6612.423829] ? watchdog+0x40/0x40 [ 6612.425169] ? hrtimer_interrupt+0x128/0x2c0 [ 6612.427368] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6612.430664] ? apic_timer_interrupt+0xf/0x20 [ 6612.434729] [ 6612.437405] ? native_safe_halt+0x2e/0x50 [ 6612.438770] kvm_wait+0xcd/0xe0 [ 6612.441432] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6612.443490] do_raw_spin_lock+0x177/0x190 [ 6612.445559] _raw_spin_lock+0x21/0x30 [ 6612.448861] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6612.456254] ? do_raw_spin_unlock+0x75/0x190 [ 6612.458333] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6612.462363] ? do_wait_intr+0xf0/0xf0 [ 6612.465565] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6612.468444] mdc_close+0x53b/0xd70 [mdc] [ 6612.470325] lmv_close+0x18d/0x490 [lmv] [ 6612.471968] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6612.474575] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6612.477376] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6612.480065] ll_md_close+0x274/0x800 [lustre] [ 6612.489858] ll_file_release+0x128/0xc50 [lustre] [ 6612.496201] __fput+0x10d/0x440 [ 6612.497486] ____fput+0x12/0x20 [ 6612.501447] task_work_run+0xc6/0x110 [ 6612.503336] exit_to_usermode_loop+0x1dd/0x1f0 [ 6612.505483] do_syscall_64+0x3d6/0x440 [ 6612.507282] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6612.509305] RIP: 0033:0x7f151bee3cd8 [ 6612.510878] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6612.523564] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6612.531337] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 6612.538404] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 6612.546229] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6612.550386] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6612.554204] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6624.055908] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 6624.068055] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6624.120514] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6624.133948] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6624.148115] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6624.158386] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6624.175287] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6624.183499] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 6624.197707] RDX: 0000000000002d14 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 6624.209370] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 6624.211515] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 6624.221210] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6624.231346] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 6624.244287] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6624.254885] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6624.266369] Call Trace: [ 6624.267388] [ 6624.271545] ? show_regs.cold.9+0x22/0x2f [ 6624.278508] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6624.283640] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6624.288391] ? watchdog+0x40/0x40 [ 6624.295675] ? hrtimer_interrupt+0x128/0x2c0 [ 6624.301230] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6624.310413] ? apic_timer_interrupt+0xf/0x20 [ 6624.316443] [ 6624.318531] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6624.326627] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6624.328529] do_raw_spin_lock+0x177/0x190 [ 6624.339630] _raw_spin_lock+0x21/0x30 [ 6624.348189] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6624.355553] obd_mod_free+0x46/0x140 [ptlrpc] [ 6624.363966] mdc_commit_open+0x91/0xb0 [mdc] [ 6624.365330] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6624.371529] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6624.380363] after_reply+0xe2c/0x1780 [ptlrpc] [ 6624.384504] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6624.393228] ? __next_timer_interrupt+0x160/0x160 [ 6624.399283] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6624.407037] ? do_wait_intr+0xf0/0xf0 [ 6624.408246] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6624.418387] kthread+0x1d1/0x200 [ 6624.426281] ? set_kthread_struct+0x70/0x70 [ 6624.433134] ret_from_fork+0x1f/0x30 [ 6632.053360] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 6632.062653] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6632.102193] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6632.114582] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6632.125444] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6632.128897] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6632.143664] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 6632.148659] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6632.153898] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 6632.161608] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 6632.166655] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 6632.170607] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 6632.174207] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 6632.178192] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6632.180256] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 6632.184449] Call Trace: [ 6632.186066] [ 6632.187025] ? show_regs.cold.9+0x22/0x2f [ 6632.188678] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6632.191063] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6632.193102] ? watchdog+0x40/0x40 [ 6632.194570] ? hrtimer_interrupt+0x128/0x2c0 [ 6632.198922] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6632.204150] ? apic_timer_interrupt+0xf/0x20 [ 6632.211624] [ 6632.220517] ? native_safe_halt+0x2e/0x50 [ 6632.229511] kvm_wait+0xcd/0xe0 [ 6632.243653] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6632.245586] do_raw_spin_lock+0x177/0x190 [ 6632.258241] _raw_spin_lock+0x21/0x30 [ 6632.260523] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6632.273058] ? do_raw_spin_unlock+0x75/0x190 [ 6632.287050] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6632.295668] ? do_wait_intr+0xf0/0xf0 [ 6632.308677] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6632.324787] mdc_reint+0x2d/0x110 [mdc] [ 6632.331810] mdc_unlink+0x658/0x870 [mdc] [ 6632.339726] lmv_unlink+0x34d/0xbd0 [lmv] [ 6632.346626] ll_unlink+0x24d/0x7f0 [lustre] [ 6632.355688] vfs_unlink+0x202/0x330 [ 6632.357572] do_unlinkat+0x2fa/0x460 [ 6632.360949] __x64_sys_unlink+0x37/0x50 [ 6632.362124] do_syscall_64+0xc1/0x440 [ 6632.371589] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6632.374317] RIP: 0033:0x7f151bdee10b [ 6632.378331] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6632.399705] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6632.403375] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 6632.406236] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 6632.410615] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6632.413093] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6632.417146] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6636.011163] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4705s! [ 6636.029063] Showing busy workqueues and worker pools: [ 6636.030934] workqueue events: flags=0x0 [ 6636.032554] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6636.035517] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 6636.039322] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6636.042030] pending: kfree_rcu_monitor [ 6636.043852] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6636.046605] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 6636.050336] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6636.053234] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 6636.056342] workqueue events_long: flags=0x0 [ 6636.058159] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6636.061011] pending: lu_objects_flush [obdclass] [ 6636.063293] workqueue events_unbound: flags=0x2 [ 6636.065089] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6636.067792] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6636.072136] workqueue events_power_efficient: flags=0x80 [ 6636.074477] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6636.077213] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 6636.080382] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6636.083352] pending: neigh_periodic_work [ 6636.085299] workqueue mm_percpu_wq: flags=0x8 [ 6636.087172] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6636.090144] pending: vmstat_update, lru_add_drain_per_cpu [ 6636.093568] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6636.097021] pending: vmstat_update [ 6636.102189] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6636.107931] pending: lru_add_drain_per_cpu [ 6636.109567] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6636.111309] pending: lru_add_drain_per_cpu BAR(48) [ 6636.115069] workqueue writeback: flags=0x4a [ 6636.118388] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 6636.130999] pending: wb_workfn [ 6636.132329] workqueue xprtiod: flags=0xa [ 6636.137446] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6636.146601] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6636.152869] workqueue ipv6_addrconf: flags=0x40008 [ 6636.157524] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6636.161985] pending: addrconf_verify_work [ 6636.164367] workqueue knbd0-recv: flags=0x1a [ 6636.167342] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6636.173094] in-flight: 658:recv_work [ 6636.176966] workqueue ptlrpc_pinger: flags=0x2 [ 6636.183257] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6636.188626] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6636.193970] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4705s workers=4 idle: 32984 37 8 [ 6636.201561] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6620s workers=2 idle: 661 [ 6640.056028] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 6640.057031] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 6640.058352] Modules linked in: [ 6640.064803] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6640.064888] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6640.064896] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6640.064898] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6640.064912] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6640.064916] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6640.064921] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6640.064924] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 6640.064926] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6640.064928] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 6640.064931] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 6640.064934] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 6640.064937] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6640.064939] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 6640.066594] Call Trace: [ 6640.071273] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6640.071359] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6640.071368] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6640.071376] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6640.074065] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6640.074079] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6640.074084] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6640.074089] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6640.074092] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 6640.074094] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 6640.074096] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 6640.074098] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 6640.074102] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 6640.074105] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6640.074107] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6640.074116] Call Trace: [ 6640.074123] [ 6640.074128] ? show_regs.cold.9+0x22/0x2f [ 6640.074139] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6640.074146] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6640.074151] ? watchdog+0x40/0x40 [ 6640.074156] ? hrtimer_interrupt+0x128/0x2c0 [ 6640.074160] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6640.074169] ? apic_timer_interrupt+0xf/0x20 [ 6640.074174] [ 6640.074177] ? native_safe_halt+0x2e/0x50 [ 6640.074182] kvm_wait+0xcd/0xe0 [ 6640.074189] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6640.074197] do_raw_spin_lock+0x177/0x190 [ 6640.074202] _raw_spin_lock+0x21/0x30 [ 6640.074206] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6640.074439] process_one_work+0x2c8/0x700 [ 6640.074447] worker_thread+0x296/0x6e0 [ 6640.074452] ? rescuer_thread+0x570/0x570 [ 6640.074456] kthread+0x1d1/0x200 [ 6640.074461] ? set_kthread_struct+0x70/0x70 [ 6640.112368] [ 6640.112381] ? show_regs.cold.9+0x22/0x2f [ 6640.112395] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6640.112401] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6640.112406] ? watchdog+0x40/0x40 [ 6640.112411] ? hrtimer_interrupt+0x128/0x2c0 [ 6640.112414] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6640.112422] ? apic_timer_interrupt+0xf/0x20 [ 6640.112427] [ 6640.112430] ? native_safe_halt+0x2e/0x50 [ 6640.112434] kvm_wait+0xcd/0xe0 [ 6640.112442] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6640.112453] do_raw_spin_lock+0x177/0x190 [ 6640.112458] _raw_spin_lock+0x21/0x30 [ 6640.112462] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6640.112739] ? do_raw_spin_unlock+0x75/0x190 [ 6640.112745] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6640.112950] ? do_wait_intr+0xf0/0xf0 [ 6640.112959] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6640.113155] mdc_close+0x53b/0xd70 [mdc] [ 6640.113204] lmv_close+0x18d/0x490 [lmv] [ 6640.113238] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6640.113414] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6640.113527] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6640.113608] ll_md_close+0x274/0x800 [lustre] [ 6640.113690] ll_file_release+0x128/0xc50 [lustre] [ 6640.114429] __fput+0x10d/0x440 [ 6640.114443] ____fput+0x12/0x20 [ 6640.114447] task_work_run+0xc6/0x110 [ 6640.114456] exit_to_usermode_loop+0x1dd/0x1f0 [ 6640.114464] do_syscall_64+0x3d6/0x440 [ 6640.114468] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6640.114477] RIP: 0033:0x7f151bee3cd8 [ 6640.114486] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6640.114491] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6640.114495] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 6640.114497] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 6640.114500] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6640.114502] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6640.114504] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6640.694591] ret_from_fork+0x1f/0x30 [ 6652.055010] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 6652.060245] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6652.086092] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6652.090385] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6652.093225] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6652.095044] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6652.105491] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6652.108223] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 6652.109999] RDX: 0000000000002f57 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 6652.111647] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 6652.113277] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 6652.115054] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6652.116860] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 6652.118818] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6652.120461] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6652.122415] Call Trace: [ 6652.123333] [ 6652.124178] ? show_regs.cold.9+0x22/0x2f [ 6652.125833] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6652.127507] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6652.129277] ? watchdog+0x40/0x40 [ 6652.130355] ? hrtimer_interrupt+0x128/0x2c0 [ 6652.131696] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6652.132937] ? apic_timer_interrupt+0xf/0x20 [ 6652.135364] [ 6652.136327] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6652.138654] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6652.141403] do_raw_spin_lock+0x177/0x190 [ 6652.144070] _raw_spin_lock+0x21/0x30 [ 6652.145826] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6652.148417] obd_mod_free+0x46/0x140 [ptlrpc] [ 6652.150637] mdc_commit_open+0x91/0xb0 [mdc] [ 6652.152346] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6652.154587] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6652.157078] after_reply+0xe2c/0x1780 [ptlrpc] [ 6652.159240] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6652.161834] ? __next_timer_interrupt+0x160/0x160 [ 6652.163470] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6652.165414] ? do_wait_intr+0xf0/0xf0 [ 6652.167766] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6652.172831] kthread+0x1d1/0x200 [ 6652.174806] ? set_kthread_struct+0x70/0x70 [ 6652.176755] ret_from_fork+0x1f/0x30 [ 6664.052054] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 6664.057895] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6664.116995] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6664.129106] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6664.133862] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6664.137763] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6664.149272] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6664.154759] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6664.160377] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 6664.164447] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 6664.170177] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 6664.174242] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 6664.187734] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 6664.199097] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6664.201221] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 6664.213728] Call Trace: [ 6664.218110] [ 6664.219547] ? show_regs.cold.9+0x22/0x2f [ 6664.226379] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6664.229557] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6664.233280] ? watchdog+0x40/0x40 [ 6664.235733] ? hrtimer_interrupt+0x128/0x2c0 [ 6664.240168] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6664.243573] ? apic_timer_interrupt+0xf/0x20 [ 6664.248129] [ 6664.250872] ? native_safe_halt+0x2e/0x50 [ 6664.253013] kvm_wait+0xcd/0xe0 [ 6664.254379] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6664.258116] do_raw_spin_lock+0x177/0x190 [ 6664.261096] _raw_spin_lock+0x21/0x30 [ 6664.263714] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6664.266948] ? do_raw_spin_unlock+0x75/0x190 [ 6664.273252] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6664.282333] ? do_wait_intr+0xf0/0xf0 [ 6664.285788] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6664.298058] mdc_reint+0x2d/0x110 [mdc] [ 6664.305983] mdc_unlink+0x658/0x870 [mdc] [ 6664.321089] lmv_unlink+0x34d/0xbd0 [lmv] [ 6664.327261] ll_unlink+0x24d/0x7f0 [lustre] [ 6664.336859] vfs_unlink+0x202/0x330 [ 6664.342570] do_unlinkat+0x2fa/0x460 [ 6664.349414] __x64_sys_unlink+0x37/0x50 [ 6664.356338] do_syscall_64+0xc1/0x440 [ 6664.361115] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6664.366886] RIP: 0033:0x7f151bdee10b [ 6664.370104] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6664.388285] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6664.396355] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 6664.408012] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 6664.415986] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6664.427889] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6664.450412] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6664.742116] rcu: INFO: rcu_sched self-detected stall on CPU [ 6664.745756] rcu: 1-...!: (2688566 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 6664.763251] (t=4740095 jiffies g=249657 q=3707) [ 6664.765053] rcu: rcu_sched kthread starved for 4735047 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 6664.786820] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 6664.802910] rcu: RCU grace-period kthread stack dump: [ 6664.806998] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 6664.818262] Call Trace: [ 6664.820849] __schedule+0x351/0xcb0 [ 6664.826315] schedule+0xc0/0x180 [ 6664.828574] schedule_timeout+0xb4/0x190 [ 6664.831096] ? do_raw_spin_unlock+0x75/0x190 [ 6664.837021] ? __next_timer_interrupt+0x160/0x160 [ 6664.848769] ? prepare_to_swait_event+0xf3/0x140 [ 6664.851261] rcu_gp_kthread+0x8dd/0x15b0 [ 6664.860081] ? force_qs_rnp+0x2c0/0x2c0 [ 6664.871160] kthread+0x1d1/0x200 [ 6664.873783] ? set_kthread_struct+0x70/0x70 [ 6664.887354] ret_from_fork+0x1f/0x30 [ 6664.888680] rcu: Stack dump where RCU GP kthread last ran: [ 6664.912302] Sending NMI from CPU 1 to CPUs 3: [ 6664.923890] INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 8.618 msecs [ 6664.924025] NMI backtrace for cpu 3 [ 6664.924029] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6664.924031] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6664.924032] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6664.924034] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6664.924036] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000212 [ 6664.924038] RAX: 0000000000005c8a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6664.924040] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 6664.924041] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6664.924042] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 6664.924043] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 6664.924045] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 6664.924046] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6664.924047] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 6664.924048] Call Trace: [ 6664.924049] [ 6664.924050] ? show_regs.cold.9+0x22/0x2f [ 6664.924051] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 6664.924052] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 6664.924053] ? nmi_handle+0xbd/0x210 [ 6664.924054] ? default_do_nmi+0x4b/0x1b0 [ 6664.924055] ? do_nmi+0x280/0x380 [ 6664.924056] ? end_repeat_nmi+0x16/0x1f [ 6664.924057] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6664.924058] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6664.924059] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6664.924060] [ 6664.924061] do_raw_spin_lock+0x177/0x190 [ 6664.924062] _raw_spin_lock+0x21/0x30 [ 6664.924063] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6664.924064] ? do_raw_spin_unlock+0x75/0x190 [ 6664.924066] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6664.924067] ? do_wait_intr+0xf0/0xf0 [ 6664.924068] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6664.924069] mdc_close+0x53b/0xd70 [mdc] [ 6664.924070] lmv_close+0x18d/0x490 [lmv] [ 6664.924071] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6664.924073] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6664.924074] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6664.924075] ll_md_close+0x274/0x800 [lustre] [ 6664.924076] ll_file_release+0x128/0xc50 [lustre] [ 6664.924077] __fput+0x10d/0x440 [ 6664.924078] ____fput+0x12/0x20 [ 6664.924079] task_work_run+0xc6/0x110 [ 6664.924080] exit_to_usermode_loop+0x1dd/0x1f0 [ 6664.924081] do_syscall_64+0x3d6/0x440 [ 6664.924083] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6664.924084] RIP: 0033:0x7f151bee3cd8 [ 6664.924085] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6664.924088] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6664.924091] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 6664.924093] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 6664.924095] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6664.924096] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6664.924098] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6664.924103] NMI backtrace for cpu 1 [ 6665.440041] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6665.456613] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6665.463179] Call Trace: [ 6665.466305] [ 6665.468018] ? dump_stack+0xbb/0x10e [ 6665.470236] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6665.474181] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6665.480596] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6665.486811] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6665.490800] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6665.492357] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 6665.507597] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 6665.509383] ? update_process_times+0x6d/0xc0 [ 6665.512518] ? tick_sched_handle.isra.18+0x27/0x90 [ 6665.514456] ? tick_sched_timer+0x53/0xb0 [ 6665.516208] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6665.518042] ? tick_sched_handle.isra.18+0x90/0x90 [ 6665.526133] ? hrtimer_interrupt+0x128/0x2c0 [ 6665.527598] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6665.533697] ? apic_timer_interrupt+0xf/0x20 [ 6665.535991] [ 6665.539713] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6665.543328] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6665.546969] ? do_raw_spin_lock+0x177/0x190 [ 6665.548552] ? _raw_spin_lock+0x21/0x30 [ 6665.550334] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6665.558551] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 6665.560479] ? mdc_commit_open+0x91/0xb0 [mdc] [ 6665.564061] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6665.566664] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6665.569730] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 6665.572200] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6665.575290] ? __next_timer_interrupt+0x160/0x160 [ 6665.577814] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6665.580414] ? do_wait_intr+0xf0/0xf0 [ 6665.582759] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6665.585821] ? kthread+0x1d1/0x200 [ 6665.588589] ? set_kthread_struct+0x70/0x70 [ 6665.594046] ? ret_from_fork+0x1f/0x30 [ 6666.213654] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4735s! [ 6666.218397] Showing busy workqueues and worker pools: [ 6666.221796] workqueue events: flags=0x0 [ 6666.223571] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6666.232226] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 6666.244025] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6666.247194] pending: kfree_rcu_monitor [ 6666.248459] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6666.252687] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 6666.265689] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6666.276632] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 6666.288070] workqueue events_long: flags=0x0 [ 6666.299359] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6666.310193] pending: lu_objects_flush [obdclass] [ 6666.320847] workqueue events_unbound: flags=0x2 [ 6666.325354] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6666.328611] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6666.348779] workqueue events_power_efficient: flags=0x80 [ 6666.361849] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6666.374624] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 6666.382877] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6666.390960] pending: neigh_periodic_work [ 6666.394438] workqueue mm_percpu_wq: flags=0x8 [ 6666.396139] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6666.404002] pending: vmstat_update, lru_add_drain_per_cpu [ 6666.409173] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6666.425000] pending: vmstat_update [ 6666.427859] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6666.434756] pending: lru_add_drain_per_cpu [ 6666.448761] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6666.454354] pending: lru_add_drain_per_cpu BAR(48) [ 6666.458984] workqueue writeback: flags=0x4a [ 6666.465358] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 6666.484128] pending: wb_workfn [ 6666.488138] workqueue xprtiod: flags=0xa [ 6666.491916] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6666.498020] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6666.508958] workqueue ipv6_addrconf: flags=0x40008 [ 6666.513412] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6666.525765] pending: addrconf_verify_work [ 6666.528872] workqueue knbd0-recv: flags=0x1a [ 6666.542898] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6666.551331] in-flight: 658:recv_work [ 6666.556995] workqueue ptlrpc_pinger: flags=0x2 [ 6666.561608] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6666.569817] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6666.591582] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4735s workers=4 idle: 32984 37 8 [ 6666.615118] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6651s workers=2 idle: 661 [ 6668.058305] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 6668.066795] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6668.141749] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6668.156603] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6668.168086] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6668.178656] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6668.187645] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6668.208632] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 6668.216334] RAX: 0000000000001205 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6668.229199] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 6668.238745] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 6668.250681] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 6668.257966] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 6668.272762] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 6668.288535] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6668.297344] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6668.308146] Call Trace: [ 6668.312684] [ 6668.316510] ? show_regs.cold.9+0x22/0x2f [ 6668.319736] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6668.324130] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6668.328579] ? watchdog+0x40/0x40 [ 6668.331689] ? hrtimer_interrupt+0x128/0x2c0 [ 6668.334039] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6668.340190] ? apic_timer_interrupt+0xf/0x20 [ 6668.347820] [ 6668.351171] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6668.356836] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6668.362857] do_raw_spin_lock+0x177/0x190 [ 6668.366199] _raw_spin_lock+0x21/0x30 [ 6668.371347] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6668.375625] process_one_work+0x2c8/0x700 [ 6668.381644] worker_thread+0x296/0x6e0 [ 6668.386045] ? rescuer_thread+0x570/0x570 [ 6668.397922] kthread+0x1d1/0x200 [ 6668.408080] ? set_kthread_struct+0x70/0x70 [ 6668.409639] ret_from_fork+0x1f/0x30 [ 6692.052068] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 6692.055026] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 6692.056441] Modules linked in: [ 6692.057514] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 6692.057522] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6692.058262] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6692.058279] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6692.058282] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6692.058296] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6692.058300] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6692.058305] RAX: 0000000000004d70 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6692.058307] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 6692.058309] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6692.058312] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 6692.058314] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 6692.058317] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 6692.058321] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6692.058323] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 6692.058331] Call Trace: [ 6692.058337] [ 6692.058342] ? show_regs.cold.9+0x22/0x2f [ 6692.058351] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6692.058358] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6692.058363] ? watchdog+0x40/0x40 [ 6692.058367] ? hrtimer_interrupt+0x128/0x2c0 [ 6692.058371] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6692.058380] ? apic_timer_interrupt+0xf/0x20 [ 6692.058385] [ 6692.058388] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6692.058394] do_raw_spin_lock+0x177/0x190 [ 6692.058399] _raw_spin_lock+0x21/0x30 [ 6692.058403] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6692.059010] ? do_raw_spin_unlock+0x75/0x190 [ 6692.059017] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6692.059215] ? do_wait_intr+0xf0/0xf0 [ 6692.059223] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6692.059438] mdc_close+0x53b/0xd70 [mdc] [ 6692.059474] lmv_close+0x18d/0x490 [lmv] [ 6692.059517] Modules linked in: lustre(O) [ 6692.059503] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6692.060970] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6692.061086] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6692.061174] ll_md_close+0x274/0x800 [lustre] [ 6692.061257] ll_file_release+0x128/0xc50 [lustre] [ 6692.061350] __fput+0x10d/0x440 [ 6692.061359] ____fput+0x12/0x20 [ 6692.061364] task_work_run+0xc6/0x110 [ 6692.061373] exit_to_usermode_loop+0x1dd/0x1f0 [ 6692.061380] do_syscall_64+0x3d6/0x440 [ 6692.061385] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6692.061394] RIP: 0033:0x7f151bee3cd8 [ 6692.061403] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6692.061407] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6692.061412] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 6692.061414] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 6692.061417] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6692.061419] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6692.061421] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6692.064776] lustre(O) [ 6692.067933] osp(O) [ 6692.091310] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6692.091386] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6692.091393] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6692.091397] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6692.091410] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6692.091415] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6692.091420] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6692.091422] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 6692.091425] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 6692.091427] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 6692.091430] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 6692.091433] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 6692.091436] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6692.091438] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 6692.091446] Call Trace: [ 6692.091452] [ 6692.091457] ? show_regs.cold.9+0x22/0x2f [ 6692.091467] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6692.091473] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6692.091477] ? watchdog+0x40/0x40 [ 6692.091482] ? hrtimer_interrupt+0x128/0x2c0 [ 6692.091486] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6692.091495] ? apic_timer_interrupt+0xf/0x20 [ 6692.091500] [ 6692.091502] ? native_safe_halt+0x2e/0x50 [ 6692.091506] kvm_wait+0xcd/0xe0 [ 6692.091513] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6692.091521] do_raw_spin_lock+0x177/0x190 [ 6692.091526] _raw_spin_lock+0x21/0x30 [ 6692.091531] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6692.091941] ? do_raw_spin_unlock+0x75/0x190 [ 6692.091948] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6692.099196] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6692.099276] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6692.099284] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6692.099287] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6692.099304] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6692.099308] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6692.099312] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 6692.099315] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 6692.099317] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 6692.099319] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 6692.099321] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6692.099324] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 6692.099327] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6692.099330] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6692.099338] Call Trace: [ 6692.099343] [ 6692.099349] ? show_regs.cold.9+0x22/0x2f [ 6692.099358] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6692.099364] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6692.099368] ? watchdog+0x40/0x40 [ 6692.099373] ? hrtimer_interrupt+0x128/0x2c0 [ 6692.099378] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6692.099385] ? apic_timer_interrupt+0xf/0x20 [ 6692.099389] [ 6692.099392] ? native_safe_halt+0x2e/0x50 [ 6692.099396] kvm_wait+0xcd/0xe0 [ 6692.099403] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6692.099411] do_raw_spin_lock+0x177/0x190 [ 6692.099415] _raw_spin_lock+0x21/0x30 [ 6692.099420] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6692.106086] ? do_wait_intr+0xf0/0xf0 [ 6692.106108] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6692.106515] mdc_reint+0x2d/0x110 [mdc] [ 6692.106552] mdc_unlink+0x658/0x870 [mdc] [ 6692.106578] lmv_unlink+0x34d/0xbd0 [lmv] [ 6692.106611] ll_unlink+0x24d/0x7f0 [lustre] [ 6692.106740] vfs_unlink+0x202/0x330 [ 6692.106748] do_unlinkat+0x2fa/0x460 [ 6692.106753] __x64_sys_unlink+0x37/0x50 [ 6692.106756] do_syscall_64+0xc1/0x440 [ 6692.106763] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6692.106772] RIP: 0033:0x7f151bdee10b [ 6692.106780] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6692.106785] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6692.106791] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 6692.106793] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 6692.106796] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6692.106798] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6692.106800] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6692.733467] obd_mod_free+0x46/0x140 [ptlrpc] [ 6692.740948] mdc_commit_open+0x91/0xb0 [mdc] [ 6692.743193] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6692.746042] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6692.751481] after_reply+0xe2c/0x1780 [ptlrpc] [ 6692.755902] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6692.759897] ? __next_timer_interrupt+0x160/0x160 [ 6692.762171] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6692.764579] ? do_wait_intr+0xf0/0xf0 [ 6692.766886] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6692.769613] kthread+0x1d1/0x200 [ 6692.771122] ? set_kthread_struct+0x70/0x70 [ 6692.773553] ret_from_fork+0x1f/0x30 [ 6696.056019] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 6696.058282] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6696.070416] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6696.086820] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6696.099553] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6696.112396] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6696.117740] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6696.147918] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6696.162860] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6696.170613] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 6696.194534] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 6696.212667] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 6696.230402] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 6696.240547] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 6696.262712] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6696.277925] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6696.289514] Call Trace: [ 6696.291701] [ 6696.293746] ? show_regs.cold.9+0x22/0x2f [ 6696.309891] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6696.319962] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6696.323443] ? watchdog+0x40/0x40 [ 6696.327780] ? hrtimer_interrupt+0x128/0x2c0 [ 6696.333768] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6696.336612] ? apic_timer_interrupt+0xf/0x20 [ 6696.340282] [ 6696.341017] ? native_safe_halt+0x2e/0x50 [ 6696.344083] kvm_wait+0xcd/0xe0 [ 6696.346767] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6696.351673] do_raw_spin_lock+0x177/0x190 [ 6696.354082] _raw_spin_lock+0x21/0x30 [ 6696.356844] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6696.362384] process_one_work+0x2c8/0x700 [ 6696.373461] worker_thread+0x296/0x6e0 [ 6696.374832] ? rescuer_thread+0x570/0x570 [ 6696.387437] kthread+0x1d1/0x200 [ 6696.393492] ? set_kthread_struct+0x70/0x70 [ 6696.408888] ret_from_fork+0x1f/0x30 [ 6696.928090] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4766s! [ 6696.932580] Showing busy workqueues and worker pools: [ 6696.935348] workqueue events: flags=0x0 [ 6696.937410] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6696.941487] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 6696.945946] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6696.950933] pending: kfree_rcu_monitor [ 6696.953448] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6696.957395] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 6696.963263] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6696.967273] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 6696.971603] workqueue events_long: flags=0x0 [ 6696.974477] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6696.978440] pending: lu_objects_flush [obdclass] [ 6696.981797] workqueue events_unbound: flags=0x2 [ 6696.983930] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6696.987150] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6696.992252] workqueue events_power_efficient: flags=0x80 [ 6696.994887] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6696.998537] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 6697.001571] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6697.004788] pending: neigh_periodic_work [ 6697.006842] workqueue mm_percpu_wq: flags=0x8 [ 6697.008602] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6697.011905] pending: vmstat_update, lru_add_drain_per_cpu [ 6697.014835] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6697.017964] pending: vmstat_update [ 6697.020084] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6697.023257] pending: lru_add_drain_per_cpu [ 6697.025293] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6697.028258] pending: lru_add_drain_per_cpu BAR(48) [ 6697.031268] workqueue writeback: flags=0x4a [ 6697.033078] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 6697.036467] pending: wb_workfn [ 6697.037968] workqueue xprtiod: flags=0xa [ 6697.039687] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6697.042700] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6697.046721] workqueue ipv6_addrconf: flags=0x40008 [ 6697.049605] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6697.052500] pending: addrconf_verify_work [ 6697.054490] workqueue knbd0-recv: flags=0x1a [ 6697.056580] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6697.059761] in-flight: 658:recv_work [ 6697.061617] workqueue ptlrpc_pinger: flags=0x2 [ 6697.063613] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6697.065735] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6697.068772] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4766s workers=4 idle: 32984 37 8 [ 6697.072485] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6681s workers=2 idle: 661 [ 6720.055131] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 6720.057055] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 6720.061465] Modules linked in: [ 6720.064932] Modules linked in: [ 6720.064955] lustre(O) [ 6720.066642] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6720.066794] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6720.066802] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6720.066804] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6720.066833] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6720.066838] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6720.066843] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6720.066845] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 6720.066847] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6720.066850] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 6720.066853] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 6720.066856] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 6720.066859] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6720.066862] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 6720.066874] Call Trace: [ 6720.066891] [ 6720.066898] ? show_regs.cold.9+0x22/0x2f [ 6720.066908] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6720.066914] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6720.066919] ? watchdog+0x40/0x40 [ 6720.066923] ? hrtimer_interrupt+0x128/0x2c0 [ 6720.066928] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6720.066936] ? apic_timer_interrupt+0xf/0x20 [ 6720.066941] [ 6720.066945] ? native_safe_halt+0x2e/0x50 [ 6720.066949] kvm_wait+0xcd/0xe0 [ 6720.066958] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6720.067051] do_raw_spin_lock+0x177/0x190 [ 6720.067056] _raw_spin_lock+0x21/0x30 [ 6720.067061] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6720.067554] ? do_raw_spin_unlock+0x75/0x190 [ 6720.067561] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6720.067802] ? do_wait_intr+0xf0/0xf0 [ 6720.067811] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6720.068394] mdc_close+0x53b/0xd70 [mdc] [ 6720.073396] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6720.073470] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6720.073477] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6720.073479] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6720.073494] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6720.073498] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6720.073503] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 6720.073505] RDX: 00000000000046d7 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 6720.073508] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 6720.073510] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 6720.073512] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6720.073516] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 6720.073518] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6720.073521] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6720.073530] Call Trace: [ 6720.073535] [ 6720.073541] ? show_regs.cold.9+0x22/0x2f [ 6720.073551] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6720.073558] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6720.073562] ? watchdog+0x40/0x40 [ 6720.073566] ? hrtimer_interrupt+0x128/0x2c0 [ 6720.073571] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6720.073579] ? apic_timer_interrupt+0xf/0x20 [ 6720.073584] [ 6720.073587] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6720.073592] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6720.073597] do_raw_spin_lock+0x177/0x190 [ 6720.073602] _raw_spin_lock+0x21/0x30 [ 6720.073609] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6720.073957] obd_mod_free+0x46/0x140 [ptlrpc] [ 6720.074163] mdc_commit_open+0x91/0xb0 [mdc] [ 6720.074205] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6720.074400] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6720.074597] after_reply+0xe2c/0x1780 [ptlrpc] [ 6720.079920] lmv_close+0x18d/0x490 [lmv] [ 6720.079953] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6720.080127] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6720.080243] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6720.136304] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6720.151313] ll_md_close+0x274/0x800 [lustre] [ 6720.151447] ll_file_release+0x128/0xc50 [lustre] [ 6720.151527] __fput+0x10d/0x440 [ 6720.151538] ____fput+0x12/0x20 [ 6720.151542] task_work_run+0xc6/0x110 [ 6720.151550] exit_to_usermode_loop+0x1dd/0x1f0 [ 6720.151557] do_syscall_64+0x3d6/0x440 [ 6720.151562] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6720.151570] RIP: 0033:0x7f151bee3cd8 [ 6720.151634] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6720.151640] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6720.151645] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 6720.151647] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 6720.151649] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6720.151651] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6720.168623] ? __next_timer_interrupt+0x160/0x160 [ 6720.168656] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6720.169303] ? do_wait_intr+0xf0/0xf0 [ 6720.169314] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6720.169528] kthread+0x1d1/0x200 [ 6720.169535] ? set_kthread_struct+0x70/0x70 [ 6720.169554] ret_from_fork+0x1f/0x30 [ 6720.776413] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6724.052073] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 6724.056227] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 6724.057281] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6724.057432] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6724.057439] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6724.057442] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6724.057455] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6724.057460] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6724.057465] RAX: 0000000000007786 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6724.057467] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 6724.057470] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 6724.057472] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 6724.057475] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 6724.057479] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 6724.057481] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6724.057483] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 6724.057491] Call Trace: [ 6724.057496] [ 6724.057501] ? show_regs.cold.9+0x22/0x2f [ 6724.057512] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6724.057517] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6724.057522] ? watchdog+0x40/0x40 [ 6724.057526] ? hrtimer_interrupt+0x128/0x2c0 [ 6724.057530] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6724.057540] ? apic_timer_interrupt+0xf/0x20 [ 6724.057544] [ 6724.057548] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6724.057553] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6724.057558] do_raw_spin_lock+0x177/0x190 [ 6724.059555] _raw_spin_lock+0x21/0x30 [ 6724.062870] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6724.062952] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6724.062973] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6724.063016] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6724.063287] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6724.063297] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6724.063301] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6724.063306] RAX: 000000000000758a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6724.063308] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 6724.063310] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 6724.063311] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 6724.063314] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 6724.063317] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 6724.063319] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6724.063321] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6724.063330] Call Trace: [ 6724.063335] [ 6724.064021] ? show_regs.cold.9+0x22/0x2f [ 6724.064033] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6724.064039] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6724.064044] ? watchdog+0x40/0x40 [ 6724.064049] ? hrtimer_interrupt+0x128/0x2c0 [ 6724.064054] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6724.064062] ? apic_timer_interrupt+0xf/0x20 [ 6724.064067] [ 6724.064071] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6724.064076] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6724.064083] do_raw_spin_lock+0x177/0x190 [ 6724.064088] _raw_spin_lock+0x21/0x30 [ 6724.064093] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6724.064349] process_one_work+0x2c8/0x700 [ 6724.064357] worker_thread+0x296/0x6e0 [ 6724.064361] ? rescuer_thread+0x570/0x570 [ 6724.064365] kthread+0x1d1/0x200 [ 6724.064370] ? set_kthread_struct+0x70/0x70 [ 6724.064373] ret_from_fork+0x1f/0x30 [ 6724.513442] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6724.520085] ? do_raw_spin_unlock+0x75/0x190 [ 6724.524655] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6724.528209] ? do_wait_intr+0xf0/0xf0 [ 6724.532550] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6724.536983] mdc_reint+0x2d/0x110 [mdc] [ 6724.541964] mdc_unlink+0x658/0x870 [mdc] [ 6724.545453] lmv_unlink+0x34d/0xbd0 [lmv] [ 6724.547370] ll_unlink+0x24d/0x7f0 [lustre] [ 6724.552244] vfs_unlink+0x202/0x330 [ 6724.554928] do_unlinkat+0x2fa/0x460 [ 6724.559291] __x64_sys_unlink+0x37/0x50 [ 6724.562447] do_syscall_64+0xc1/0x440 [ 6724.566434] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6724.570944] RIP: 0033:0x7f151bdee10b [ 6724.575364] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6724.597306] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6724.601117] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 6724.607081] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 6724.617255] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6724.627694] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6724.634188] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6727.137127] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4796s! [ 6727.147756] Showing busy workqueues and worker pools: [ 6727.152642] workqueue events: flags=0x0 [ 6727.156308] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6727.163592] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 6727.172197] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6727.178183] pending: kfree_rcu_monitor [ 6727.180475] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6727.188383] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 6727.195996] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6727.199123] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 6727.207077] workqueue events_long: flags=0x0 [ 6727.210803] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6727.220442] pending: lu_objects_flush [obdclass] [ 6727.224092] workqueue events_unbound: flags=0x2 [ 6727.226218] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6727.229345] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6727.236687] workqueue events_power_efficient: flags=0x80 [ 6727.241635] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6727.247404] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 6727.258188] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6727.264885] pending: neigh_periodic_work [ 6727.271410] workqueue mm_percpu_wq: flags=0x8 [ 6727.279086] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6727.284648] pending: vmstat_update, lru_add_drain_per_cpu [ 6727.293522] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6727.297289] pending: vmstat_update [ 6727.302497] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6727.305547] pending: lru_add_drain_per_cpu [ 6727.307677] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6727.313754] pending: lru_add_drain_per_cpu BAR(48) [ 6727.320405] workqueue writeback: flags=0x4a [ 6727.321914] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 6727.336907] pending: wb_workfn [ 6727.340231] workqueue xprtiod: flags=0xa [ 6727.344383] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6727.351697] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6727.358303] workqueue ipv6_addrconf: flags=0x40008 [ 6727.361840] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6727.368471] pending: addrconf_verify_work [ 6727.373290] workqueue knbd0-recv: flags=0x1a [ 6727.379330] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6727.388326] in-flight: 658:recv_work [ 6727.391286] workqueue ptlrpc_pinger: flags=0x2 [ 6727.392993] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6727.399208] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6727.402778] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4796s workers=4 idle: 32984 37 8 [ 6727.406685] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6712s workers=2 idle: 661 [ 6748.058885] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 6748.059771] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 6748.068088] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6748.068165] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6748.068173] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6748.068176] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6748.068191] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6748.068195] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6748.068199] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 6748.068201] RDX: 0000000000007c37 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 6748.068204] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 6748.068206] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 6748.068208] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6748.068211] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 6748.068214] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6748.068216] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6748.068225] Call Trace: [ 6748.068232] [ 6748.068238] ? show_regs.cold.9+0x22/0x2f [ 6748.068250] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6748.068256] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6748.068261] ? watchdog+0x40/0x40 [ 6748.068265] ? hrtimer_interrupt+0x128/0x2c0 [ 6748.068270] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6748.068277] ? apic_timer_interrupt+0xf/0x20 [ 6748.068281] [ 6748.068284] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6748.068289] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6748.068293] do_raw_spin_lock+0x177/0x190 [ 6748.068298] _raw_spin_lock+0x21/0x30 [ 6748.068304] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6748.074564] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6748.074642] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6748.074651] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6748.074654] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6748.074669] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6748.074674] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6748.074679] RAX: 0000000000006e11 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6748.074681] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 6748.074683] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6748.074686] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 6748.074688] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 6748.074691] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 6748.074694] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6748.074696] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 6748.074706] Call Trace: [ 6748.074711] [ 6748.074718] ? show_regs.cold.9+0x22/0x2f [ 6748.074845] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6748.074851] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6748.074856] ? watchdog+0x40/0x40 [ 6748.074860] ? hrtimer_interrupt+0x128/0x2c0 [ 6748.074864] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6748.074880] ? apic_timer_interrupt+0xf/0x20 [ 6748.074885] [ 6748.074888] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6748.074895] do_raw_spin_lock+0x177/0x190 [ 6748.074899] _raw_spin_lock+0x21/0x30 [ 6748.074905] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6748.075282] ? do_raw_spin_unlock+0x75/0x190 [ 6748.075288] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6748.075749] ? do_wait_intr+0xf0/0xf0 [ 6748.075758] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6748.075967] mdc_close+0x53b/0xd70 [mdc] [ 6748.076012] lmv_close+0x18d/0x490 [lmv] [ 6748.076047] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6748.109296] obd_mod_free+0x46/0x140 [ptlrpc] [ 6748.109608] mdc_commit_open+0x91/0xb0 [mdc] [ 6748.109645] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6748.111042] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6748.111272] after_reply+0xe2c/0x1780 [ptlrpc] [ 6748.111474] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6748.121228] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6748.121365] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6748.127516] ? __next_timer_interrupt+0x160/0x160 [ 6748.127541] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6748.127897] ? do_wait_intr+0xf0/0xf0 [ 6748.127908] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6748.128114] kthread+0x1d1/0x200 [ 6748.128121] ? set_kthread_struct+0x70/0x70 [ 6748.128125] ret_from_fork+0x1f/0x30 [ 6748.433978] ll_md_close+0x274/0x800 [lustre] [ 6748.438061] ll_file_release+0x128/0xc50 [lustre] [ 6748.444084] __fput+0x10d/0x440 [ 6748.446816] ____fput+0x12/0x20 [ 6748.448884] task_work_run+0xc6/0x110 [ 6748.451106] exit_to_usermode_loop+0x1dd/0x1f0 [ 6748.454098] do_syscall_64+0x3d6/0x440 [ 6748.455987] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6748.458191] RIP: 0033:0x7f151bee3cd8 [ 6748.460469] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6748.475845] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6748.479879] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 6748.496203] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 6748.500319] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6748.504827] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6748.508131] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6752.052662] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 6752.056910] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata [ 6752.057034] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 6752.057705] dm_mirror [ 6752.089284] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6752.089363] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6752.089370] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6752.089377] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6752.092288] dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6752.092305] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6752.092312] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6752.092315] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6752.092330] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6752.092334] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6752.092338] RAX: 0000000000002f25 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6752.092340] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 6752.092342] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 6752.092344] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 6752.092346] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 6752.092350] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 6752.092353] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6752.092355] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 6752.092364] Call Trace: [ 6752.092370] [ 6752.092375] ? show_regs.cold.9+0x22/0x2f [ 6752.092385] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6752.092392] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6752.092396] ? watchdog+0x40/0x40 [ 6752.092400] ? hrtimer_interrupt+0x128/0x2c0 [ 6752.092404] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6752.092412] ? apic_timer_interrupt+0xf/0x20 [ 6752.092417] [ 6752.092419] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6752.092424] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6752.092428] do_raw_spin_lock+0x177/0x190 [ 6752.092432] _raw_spin_lock+0x21/0x30 [ 6752.092439] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6752.093369] ? do_raw_spin_unlock+0x75/0x190 [ 6752.093377] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6752.093595] ? do_wait_intr+0xf0/0xf0 [ 6752.093603] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6752.099803] [ 6752.120408] mdc_reint+0x2d/0x110 [mdc] [ 6752.120520] mdc_unlink+0x658/0x870 [mdc] [ 6752.120551] lmv_unlink+0x34d/0xbd0 [lmv] [ 6752.120587] ll_unlink+0x24d/0x7f0 [lustre] [ 6752.120724] vfs_unlink+0x202/0x330 [ 6752.120740] do_unlinkat+0x2fa/0x460 [ 6752.120745] __x64_sys_unlink+0x37/0x50 [ 6752.120749] do_syscall_64+0xc1/0x440 [ 6752.120757] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6752.120765] RIP: 0033:0x7f151bdee10b [ 6752.120778] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6752.120782] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6752.120788] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 6752.120791] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 6752.120793] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6752.120800] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6752.120802] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6752.295977] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6752.298103] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6752.305048] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6752.307839] RAX: 0000000000003292 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6752.310430] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 6752.313008] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 6752.315613] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 6752.318142] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 6752.321636] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 6752.325290] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6752.328268] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6752.331132] Call Trace: [ 6752.332273] [ 6752.333104] ? show_regs.cold.9+0x22/0x2f [ 6752.335461] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6752.337658] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6752.339731] ? watchdog+0x40/0x40 [ 6752.341420] ? hrtimer_interrupt+0x128/0x2c0 [ 6752.343435] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6752.345698] ? apic_timer_interrupt+0xf/0x20 [ 6752.347696] [ 6752.349009] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6752.351392] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6752.354480] do_raw_spin_lock+0x177/0x190 [ 6752.356131] _raw_spin_lock+0x21/0x30 [ 6752.358058] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6752.361324] process_one_work+0x2c8/0x700 [ 6752.363224] worker_thread+0x296/0x6e0 [ 6752.364954] ? rescuer_thread+0x570/0x570 [ 6752.366867] kthread+0x1d1/0x200 [ 6752.368196] ? set_kthread_struct+0x70/0x70 [ 6752.370023] ret_from_fork+0x1f/0x30 [ 6757.857054] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4827s! [ 6757.867935] Showing busy workqueues and worker pools: [ 6757.871078] workqueue events: flags=0x0 [ 6757.873919] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6757.877276] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 6757.881890] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6757.885154] pending: kfree_rcu_monitor [ 6757.886849] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6757.892216] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 6757.896156] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6757.898339] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 6757.902035] workqueue events_long: flags=0x0 [ 6757.904355] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6757.907424] pending: lu_objects_flush [obdclass] [ 6757.909839] workqueue events_unbound: flags=0x2 [ 6757.911759] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6757.914436] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6757.919300] workqueue events_power_efficient: flags=0x80 [ 6757.921561] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6757.924833] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 6757.928057] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6757.930913] pending: neigh_periodic_work [ 6757.932412] workqueue mm_percpu_wq: flags=0x8 [ 6757.934371] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6757.936908] pending: vmstat_update, lru_add_drain_per_cpu [ 6757.939557] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6757.942034] pending: vmstat_update [ 6757.943189] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6757.945963] pending: lru_add_drain_per_cpu [ 6757.947913] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6757.951160] pending: lru_add_drain_per_cpu BAR(48) [ 6757.953120] workqueue writeback: flags=0x4a [ 6757.954485] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 6757.956470] pending: wb_workfn [ 6757.957500] workqueue xprtiod: flags=0xa [ 6757.959217] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6757.961468] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6757.964743] workqueue ipv6_addrconf: flags=0x40008 [ 6757.966740] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6757.969566] pending: addrconf_verify_work [ 6757.971827] workqueue knbd0-recv: flags=0x1a [ 6757.974169] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6757.976953] in-flight: 658:recv_work [ 6757.978439] workqueue ptlrpc_pinger: flags=0x2 [ 6757.980325] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6757.983101] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6757.985779] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4827s workers=4 idle: 32984 37 8 [ 6757.989168] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6742s workers=2 idle: 661 [ 6776.055021] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 6776.057023] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 6776.057259] Modules linked in: [ 6776.059679] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6776.059759] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6776.059767] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6776.059770] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6776.059784] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6776.059788] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6776.059793] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6776.059796] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 6776.059798] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6776.059801] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 6776.059803] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 6776.059807] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 6776.059810] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6776.059812] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 6776.059820] Call Trace: [ 6776.059827] [ 6776.059833] ? show_regs.cold.9+0x22/0x2f [ 6776.059842] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6776.059848] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6776.059853] ? watchdog+0x40/0x40 [ 6776.059857] ? hrtimer_interrupt+0x128/0x2c0 [ 6776.059862] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6776.059868] ? apic_timer_interrupt+0xf/0x20 [ 6776.059874] [ 6776.059877] ? native_safe_halt+0x2e/0x50 [ 6776.059882] kvm_wait+0xcd/0xe0 [ 6776.059889] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6776.059898] do_raw_spin_lock+0x177/0x190 [ 6776.059903] _raw_spin_lock+0x21/0x30 [ 6776.059907] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6776.060168] ? do_raw_spin_unlock+0x75/0x190 [ 6776.060173] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6776.060360] ? do_wait_intr+0xf0/0xf0 [ 6776.060368] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6776.060655] mdc_close+0x53b/0xd70 [mdc] [ 6776.060693] lmv_close+0x18d/0x490 [lmv] [ 6776.060722] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6776.060876] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6776.060987] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6776.061070] ll_md_close+0x274/0x800 [lustre] [ 6776.061155] ll_file_release+0x128/0xc50 [lustre] [ 6776.061239] __fput+0x10d/0x440 [ 6776.061247] ____fput+0x12/0x20 [ 6776.061251] task_work_run+0xc6/0x110 [ 6776.061258] exit_to_usermode_loop+0x1dd/0x1f0 [ 6776.061265] do_syscall_64+0x3d6/0x440 [ 6776.061270] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6776.061276] RIP: 0033:0x7f151bee3cd8 [ 6776.061282] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6776.061286] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6776.061291] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 6776.061293] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 6776.061296] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6776.061297] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6776.061299] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6776.192611] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6776.209429] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6776.213706] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6776.216857] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6776.218639] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6776.227588] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6776.232782] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 6776.235836] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 6776.237890] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 6776.240894] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 6776.243446] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6776.245828] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 6776.248574] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6776.250646] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6776.252948] Call Trace: [ 6776.253922] [ 6776.254774] ? show_regs.cold.9+0x22/0x2f [ 6776.256447] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6776.258419] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6776.259944] ? watchdog+0x40/0x40 [ 6776.261229] ? hrtimer_interrupt+0x128/0x2c0 [ 6776.262987] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6776.264844] ? apic_timer_interrupt+0xf/0x20 [ 6776.266531] [ 6776.267315] ? native_safe_halt+0x2e/0x50 [ 6776.268773] kvm_wait+0xcd/0xe0 [ 6776.269867] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6776.271740] do_raw_spin_lock+0x177/0x190 [ 6776.273175] _raw_spin_lock+0x21/0x30 [ 6776.275375] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6776.277620] obd_mod_free+0x46/0x140 [ptlrpc] [ 6776.279790] mdc_commit_open+0x91/0xb0 [mdc] [ 6776.281591] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6776.284821] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6776.287717] after_reply+0xe2c/0x1780 [ptlrpc] [ 6776.289976] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6776.293894] ? __next_timer_interrupt+0x160/0x160 [ 6776.297609] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6776.300917] ? do_wait_intr+0xf0/0xf0 [ 6776.302564] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6776.305880] kthread+0x1d1/0x200 [ 6776.308614] ? set_kthread_struct+0x70/0x70 [ 6776.313604] ret_from_fork+0x1f/0x30 [ 6780.060800] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6780.066573] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6780.110981] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6780.128954] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6780.143238] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6780.149258] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6780.151478] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6780.162985] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6780.171758] RAX: 0000000000005283 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6780.175594] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 6780.181881] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 6780.189012] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 6780.194800] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 6780.199394] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 6780.203059] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6780.208556] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6780.217907] Call Trace: [ 6780.219483] [ 6780.224411] ? show_regs.cold.9+0x22/0x2f [ 6780.229463] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6780.233140] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6780.238779] ? watchdog+0x40/0x40 [ 6780.244395] ? hrtimer_interrupt+0x128/0x2c0 [ 6780.246656] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6780.253704] ? apic_timer_interrupt+0xf/0x20 [ 6780.260079] [ 6780.262752] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6780.267572] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6780.272249] do_raw_spin_lock+0x177/0x190 [ 6780.274585] _raw_spin_lock+0x21/0x30 [ 6780.276434] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6780.279233] process_one_work+0x2c8/0x700 [ 6780.281110] worker_thread+0x296/0x6e0 [ 6780.282907] ? rescuer_thread+0x570/0x570 [ 6780.290931] kthread+0x1d1/0x200 [ 6780.292710] ? set_kthread_struct+0x70/0x70 [ 6780.295133] ret_from_fork+0x1f/0x30 [ 6784.053305] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 6784.064951] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6784.129886] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6784.143713] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6784.155413] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6784.164864] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6784.185398] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6784.194864] RAX: 0000000000007c8e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6784.200250] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 6784.210852] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 6784.221561] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 6784.229853] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 6784.236610] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 6784.246835] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6784.256536] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 6784.259716] Call Trace: [ 6784.260665] [ 6784.261714] ? show_regs.cold.9+0x22/0x2f [ 6784.265852] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6784.268591] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6784.272573] ? watchdog+0x40/0x40 [ 6784.274338] ? hrtimer_interrupt+0x128/0x2c0 [ 6784.275516] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6784.277437] ? apic_timer_interrupt+0xf/0x20 [ 6784.278936] [ 6784.279581] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6784.281475] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6784.283485] do_raw_spin_lock+0x177/0x190 [ 6784.285110] _raw_spin_lock+0x21/0x30 [ 6784.286903] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6784.291360] ? do_raw_spin_unlock+0x75/0x190 [ 6784.296223] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6784.303302] ? do_wait_intr+0xf0/0xf0 [ 6784.304825] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6784.309580] mdc_reint+0x2d/0x110 [mdc] [ 6784.312919] mdc_unlink+0x658/0x870 [mdc] [ 6784.320078] lmv_unlink+0x34d/0xbd0 [lmv] [ 6784.322084] ll_unlink+0x24d/0x7f0 [lustre] [ 6784.324304] vfs_unlink+0x202/0x330 [ 6784.326208] do_unlinkat+0x2fa/0x460 [ 6784.328833] __x64_sys_unlink+0x37/0x50 [ 6784.332516] do_syscall_64+0xc1/0x440 [ 6784.338087] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6784.346686] RIP: 0033:0x7f151bdee10b [ 6784.352751] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6784.370550] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6784.374394] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 6784.377444] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 6784.380622] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6784.384075] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6784.387748] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6788.071537] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4857s! [ 6788.079425] Showing busy workqueues and worker pools: [ 6788.082113] workqueue events: flags=0x0 [ 6788.083987] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6788.087878] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 6788.092910] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6788.096512] pending: kfree_rcu_monitor [ 6788.098520] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6788.101498] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 6788.105594] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6788.108546] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 6788.112541] workqueue events_long: flags=0x0 [ 6788.114460] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6788.123049] pending: lu_objects_flush [obdclass] [ 6788.126071] workqueue events_unbound: flags=0x2 [ 6788.137663] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6788.142927] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6788.153552] workqueue events_power_efficient: flags=0x80 [ 6788.158065] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6788.171049] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 6788.179115] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6788.195052] pending: neigh_periodic_work [ 6788.198735] workqueue mm_percpu_wq: flags=0x8 [ 6788.211077] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6788.221258] pending: vmstat_update, lru_add_drain_per_cpu [ 6788.225094] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6788.247706] pending: vmstat_update [ 6788.249121] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6788.268017] pending: lru_add_drain_per_cpu [ 6788.276693] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6788.285012] pending: lru_add_drain_per_cpu BAR(48) [ 6788.296412] workqueue writeback: flags=0x4a [ 6788.300536] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 6788.306127] pending: wb_workfn [ 6788.310077] workqueue xprtiod: flags=0xa [ 6788.313099] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6788.317357] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6788.321973] workqueue ipv6_addrconf: flags=0x40008 [ 6788.330193] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6788.338183] pending: addrconf_verify_work [ 6788.349772] workqueue knbd0-recv: flags=0x1a [ 6788.355074] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6788.363005] in-flight: 658:recv_work [ 6788.366833] workqueue ptlrpc_pinger: flags=0x2 [ 6788.368613] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6788.376355] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6788.382115] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4857s workers=4 idle: 32984 37 8 [ 6788.388966] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6773s workers=2 idle: 661 [ 6804.055010] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 6804.061052] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 6804.061441] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6804.061629] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6804.061636] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6804.061706] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6804.061722] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6804.061726] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6804.061731] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 6804.061733] RDX: 000000000000293b RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 6804.061736] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 6804.061738] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 6804.061740] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6804.061743] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 6804.061746] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6804.061749] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6804.061757] Call Trace: [ 6804.061763] [ 6804.061834] ? show_regs.cold.9+0x22/0x2f [ 6804.061845] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6804.061852] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6804.061857] ? watchdog+0x40/0x40 [ 6804.061861] ? hrtimer_interrupt+0x128/0x2c0 [ 6804.061866] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6804.061935] ? apic_timer_interrupt+0xf/0x20 [ 6804.061940] [ 6804.061943] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6804.061949] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6804.061954] do_raw_spin_lock+0x177/0x190 [ 6804.061960] _raw_spin_lock+0x21/0x30 [ 6804.061965] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6804.062373] obd_mod_free+0x46/0x140 [ptlrpc] [ 6804.062660] mdc_commit_open+0x91/0xb0 [mdc] [ 6804.062697] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6804.063257] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6804.063646] after_reply+0xe2c/0x1780 [ptlrpc] [ 6804.063882] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6804.064104] ? __next_timer_interrupt+0x160/0x160 [ 6804.064113] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6804.064334] ? do_wait_intr+0xf0/0xf0 [ 6804.064344] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6804.064777] kthread+0x1d1/0x200 [ 6804.064784] ? set_kthread_struct+0x70/0x70 [ 6804.064791] ret_from_fork+0x1f/0x30 [ 6804.260400] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6804.286961] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6804.292948] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6804.297266] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6804.300645] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6804.311017] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6804.315247] RAX: 0000000000007f8b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6804.317739] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 6804.321569] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6804.324398] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 6804.326994] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 6804.330407] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 6804.333280] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6804.335461] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 6804.339536] Call Trace: [ 6804.340738] [ 6804.342276] ? show_regs.cold.9+0x22/0x2f [ 6804.345926] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6804.348259] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6804.349747] ? watchdog+0x40/0x40 [ 6804.351388] ? hrtimer_interrupt+0x128/0x2c0 [ 6804.354414] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6804.357950] ? apic_timer_interrupt+0xf/0x20 [ 6804.360772] [ 6804.361927] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6804.365149] do_raw_spin_lock+0x177/0x190 [ 6804.367345] _raw_spin_lock+0x21/0x30 [ 6804.369314] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6804.373085] ? do_raw_spin_unlock+0x75/0x190 [ 6804.375240] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6804.378159] ? do_wait_intr+0xf0/0xf0 [ 6804.379383] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6804.382674] mdc_close+0x53b/0xd70 [mdc] [ 6804.384952] lmv_close+0x18d/0x490 [lmv] [ 6804.387342] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6804.390507] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6804.394263] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6804.397356] ll_md_close+0x274/0x800 [lustre] [ 6804.400189] ll_file_release+0x128/0xc50 [lustre] [ 6804.404535] __fput+0x10d/0x440 [ 6804.406274] ____fput+0x12/0x20 [ 6804.408082] task_work_run+0xc6/0x110 [ 6804.410139] exit_to_usermode_loop+0x1dd/0x1f0 [ 6804.413029] do_syscall_64+0x3d6/0x440 [ 6804.415372] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6804.418789] RIP: 0033:0x7f151bee3cd8 [ 6804.420297] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6804.431226] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6804.435570] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 6804.439790] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 6804.444116] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6804.448496] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6804.452629] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6808.056533] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6808.060241] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6808.096825] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6808.107796] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6808.122199] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6808.125676] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6808.132345] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6808.141900] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6808.148199] RAX: 0000000000000124 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6808.153735] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 6808.160458] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 6808.164744] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 6808.169811] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 6808.179712] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 6808.191312] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6808.201299] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6808.203798] Call Trace: [ 6808.215750] [ 6808.218789] ? show_regs.cold.9+0x22/0x2f [ 6808.220931] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6808.223993] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6808.227403] ? watchdog+0x40/0x40 [ 6808.228526] ? hrtimer_interrupt+0x128/0x2c0 [ 6808.233520] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6808.240370] ? apic_timer_interrupt+0xf/0x20 [ 6808.243633] [ 6808.245579] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6808.252684] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6808.258534] do_raw_spin_lock+0x177/0x190 [ 6808.261182] _raw_spin_lock+0x21/0x30 [ 6808.264293] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6808.267653] process_one_work+0x2c8/0x700 [ 6808.271663] worker_thread+0x296/0x6e0 [ 6808.274692] ? rescuer_thread+0x570/0x570 [ 6808.280262] kthread+0x1d1/0x200 [ 6808.283247] ? set_kthread_struct+0x70/0x70 [ 6808.286968] ret_from_fork+0x1f/0x30 [ 6816.052153] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 6816.057342] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6816.088146] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6816.095576] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6816.102516] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6816.105497] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6816.116601] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6816.131246] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6816.136760] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 6816.141316] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 6816.147215] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 6816.153466] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 6816.158440] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 6816.163307] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6816.166535] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 6816.171573] Call Trace: [ 6816.173457] [ 6816.174699] ? show_regs.cold.9+0x22/0x2f [ 6816.176692] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6816.178895] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6816.182482] ? watchdog+0x40/0x40 [ 6816.184649] ? hrtimer_interrupt+0x128/0x2c0 [ 6816.187708] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6816.190941] ? apic_timer_interrupt+0xf/0x20 [ 6816.193452] [ 6816.194228] ? native_safe_halt+0x2e/0x50 [ 6816.196653] kvm_wait+0xcd/0xe0 [ 6816.198653] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6816.206469] do_raw_spin_lock+0x177/0x190 [ 6816.211181] _raw_spin_lock+0x21/0x30 [ 6816.213908] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6816.222941] ? do_raw_spin_unlock+0x75/0x190 [ 6816.225380] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6816.228723] ? do_wait_intr+0xf0/0xf0 [ 6816.231133] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6816.234808] mdc_reint+0x2d/0x110 [mdc] [ 6816.236824] mdc_unlink+0x658/0x870 [mdc] [ 6816.237891] lmv_unlink+0x34d/0xbd0 [lmv] [ 6816.242690] ll_unlink+0x24d/0x7f0 [lustre] [ 6816.245229] vfs_unlink+0x202/0x330 [ 6816.247325] do_unlinkat+0x2fa/0x460 [ 6816.250310] __x64_sys_unlink+0x37/0x50 [ 6816.252531] do_syscall_64+0xc1/0x440 [ 6816.256668] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6816.260798] RIP: 0033:0x7f151bdee10b [ 6816.263760] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6816.278370] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6816.280210] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 6816.284778] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 6816.288671] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6816.293850] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6816.300323] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6818.793433] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4888s! [ 6818.797183] Showing busy workqueues and worker pools: [ 6818.799120] workqueue events: flags=0x0 [ 6818.800931] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6818.808508] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 6818.817458] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6818.826454] pending: kfree_rcu_monitor [ 6818.828606] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6818.835460] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 6818.845520] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6818.849591] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 6818.852827] workqueue events_long: flags=0x0 [ 6818.858124] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6818.861060] pending: lu_objects_flush [obdclass] [ 6818.863729] workqueue events_unbound: flags=0x2 [ 6818.865779] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6818.873886] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6818.882589] workqueue events_power_efficient: flags=0x80 [ 6818.885488] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6818.888836] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 6818.895770] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6818.899104] pending: neigh_periodic_work [ 6818.900500] workqueue mm_percpu_wq: flags=0x8 [ 6818.903866] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6818.913229] pending: vmstat_update, lru_add_drain_per_cpu [ 6818.917235] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6818.920982] pending: vmstat_update [ 6818.926169] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6818.929581] pending: lru_add_drain_per_cpu [ 6818.935494] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6818.938379] pending: lru_add_drain_per_cpu BAR(48) [ 6818.940529] workqueue writeback: flags=0x4a [ 6818.942626] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 6818.948161] pending: wb_workfn [ 6818.949730] workqueue xprtiod: flags=0xa [ 6818.952074] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6818.955479] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6818.963468] workqueue ipv6_addrconf: flags=0x40008 [ 6818.966445] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6818.969758] pending: addrconf_verify_work [ 6818.975357] workqueue knbd0-recv: flags=0x1a [ 6818.977557] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6818.981237] in-flight: 658:recv_work [ 6818.989676] workqueue ptlrpc_pinger: flags=0x2 [ 6818.991851] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6818.996065] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6819.002666] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4888s workers=4 idle: 32984 37 8 [ 6819.010449] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6803s workers=2 idle: 661 [ 6832.055342] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 6832.057227] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 6832.058894] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) [ 6832.065252] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6832.065334] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6832.065343] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6832.065346] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6832.065358] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6832.065362] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6832.065367] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6832.065369] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 6832.065372] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6832.065374] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 6832.065377] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 6832.065380] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 6832.065383] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6832.065386] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 6832.065394] Call Trace: [ 6832.065400] [ 6832.065405] ? show_regs.cold.9+0x22/0x2f [ 6832.065415] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6832.065423] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6832.065428] ? watchdog+0x40/0x40 [ 6832.065432] ? hrtimer_interrupt+0x128/0x2c0 [ 6832.065436] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6832.065443] ? apic_timer_interrupt+0xf/0x20 [ 6832.065449] [ 6832.065452] ? native_safe_halt+0x2e/0x50 [ 6832.065456] kvm_wait+0xcd/0xe0 [ 6832.065464] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6832.065472] do_raw_spin_lock+0x177/0x190 [ 6832.065476] _raw_spin_lock+0x21/0x30 [ 6832.065480] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6832.067069] ? do_raw_spin_unlock+0x75/0x190 [ 6832.067079] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6832.067420] ? do_wait_intr+0xf0/0xf0 [ 6832.067428] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6832.067623] mdc_close+0x53b/0xd70 [mdc] [ 6832.067667] lmv_close+0x18d/0x490 [lmv] [ 6832.074170] lquota(O) [ 6832.096160] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6832.096609] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6832.096723] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6832.096813] ll_md_close+0x274/0x800 [lustre] [ 6832.096896] ll_file_release+0x128/0xc50 [lustre] [ 6832.096992] __fput+0x10d/0x440 [ 6832.097002] ____fput+0x12/0x20 [ 6832.097008] task_work_run+0xc6/0x110 [ 6832.097016] exit_to_usermode_loop+0x1dd/0x1f0 [ 6832.097024] do_syscall_64+0x3d6/0x440 [ 6832.116218] lfsck(O) [ 6832.120485] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6832.120516] RIP: 0033:0x7f151bee3cd8 [ 6832.120529] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6832.120534] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6832.120540] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 6832.120542] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 6832.120548] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6832.120551] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6832.120623] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6832.524400] obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6832.564608] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6832.571703] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6832.575674] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6832.578091] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6832.598208] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6832.608593] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 6832.611375] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 6832.614483] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 6832.617654] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 6832.620461] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6832.627418] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 6832.636494] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6832.642964] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6832.652269] Call Trace: [ 6832.653257] [ 6832.654090] ? show_regs.cold.9+0x22/0x2f [ 6832.664143] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6832.667612] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6832.672073] ? watchdog+0x40/0x40 [ 6832.673249] ? hrtimer_interrupt+0x128/0x2c0 [ 6832.681555] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6832.683882] ? apic_timer_interrupt+0xf/0x20 [ 6832.687596] [ 6832.688711] ? native_safe_halt+0x2e/0x50 [ 6832.691520] kvm_wait+0xcd/0xe0 [ 6832.693579] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6832.703212] do_raw_spin_lock+0x177/0x190 [ 6832.705826] _raw_spin_lock+0x21/0x30 [ 6832.708224] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6832.714524] obd_mod_free+0x46/0x140 [ptlrpc] [ 6832.720687] mdc_commit_open+0x91/0xb0 [mdc] [ 6832.726870] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6832.733091] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6832.742516] after_reply+0xe2c/0x1780 [ptlrpc] [ 6832.747535] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6832.754143] ? __next_timer_interrupt+0x160/0x160 [ 6832.758569] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6832.766268] ? do_wait_intr+0xf0/0xf0 [ 6832.771391] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6832.774102] kthread+0x1d1/0x200 [ 6832.775003] ? set_kthread_struct+0x70/0x70 [ 6832.779436] ret_from_fork+0x1f/0x30 [ 6836.058850] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6836.062647] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6836.109121] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6836.115357] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6836.125631] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6836.130695] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6836.140617] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6836.166731] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6836.170694] RAX: 0000000000007d86 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6836.183194] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 6836.186718] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 6836.191804] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 6836.195690] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 6836.199266] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 6836.207130] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6836.212579] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6836.218523] Call Trace: [ 6836.221698] [ 6836.222467] ? show_regs.cold.9+0x22/0x2f [ 6836.231638] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6836.234012] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6836.246661] ? watchdog+0x40/0x40 [ 6836.252611] ? hrtimer_interrupt+0x128/0x2c0 [ 6836.263147] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6836.264884] ? apic_timer_interrupt+0xf/0x20 [ 6836.273019] [ 6836.274612] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6836.282282] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6836.288708] do_raw_spin_lock+0x177/0x190 [ 6836.295084] _raw_spin_lock+0x21/0x30 [ 6836.298665] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6836.304727] process_one_work+0x2c8/0x700 [ 6836.307428] worker_thread+0x296/0x6e0 [ 6836.311311] ? rescuer_thread+0x570/0x570 [ 6836.315994] kthread+0x1d1/0x200 [ 6836.319527] ? set_kthread_struct+0x70/0x70 [ 6836.329451] ret_from_fork+0x1f/0x30 [ 6844.054412] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 6844.061444] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6844.107881] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6844.117945] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6844.123504] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6844.128852] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6844.141245] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6844.147806] RAX: 0000000000007e08 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6844.151055] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 6844.159566] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 6844.165917] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 6844.169893] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 6844.173536] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 6844.180746] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6844.183429] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 6844.195239] Call Trace: [ 6844.196454] [ 6844.198316] ? show_regs.cold.9+0x22/0x2f [ 6844.200786] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6844.202920] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6844.207507] ? watchdog+0x40/0x40 [ 6844.213284] ? hrtimer_interrupt+0x128/0x2c0 [ 6844.217365] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6844.220905] ? apic_timer_interrupt+0xf/0x20 [ 6844.225722] [ 6844.226484] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6844.235090] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6844.241247] do_raw_spin_lock+0x177/0x190 [ 6844.248929] _raw_spin_lock+0x21/0x30 [ 6844.251174] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6844.256204] ? do_raw_spin_unlock+0x75/0x190 [ 6844.270097] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6844.272123] ? do_wait_intr+0xf0/0xf0 [ 6844.282161] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6844.292174] mdc_reint+0x2d/0x110 [mdc] [ 6844.299165] mdc_unlink+0x658/0x870 [mdc] [ 6844.305631] lmv_unlink+0x34d/0xbd0 [lmv] [ 6844.308890] ll_unlink+0x24d/0x7f0 [lustre] [ 6844.312241] vfs_unlink+0x202/0x330 [ 6844.315027] do_unlinkat+0x2fa/0x460 [ 6844.323923] __x64_sys_unlink+0x37/0x50 [ 6844.326428] do_syscall_64+0xc1/0x440 [ 6844.328002] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6844.331213] RIP: 0033:0x7f151bdee10b [ 6844.335360] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6844.353311] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6844.366564] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 6844.370448] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 6844.376916] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6844.383082] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6844.388589] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6844.748818] rcu: INFO: rcu_sched self-detected stall on CPU [ 6844.757256] rcu: 1-...!: (2787280 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 6844.763348] (t=4920101 jiffies g=249657 q=3766) [ 6844.766187] rcu: rcu_sched kthread starved for 4915053 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 6844.773092] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 6844.778931] rcu: RCU grace-period kthread stack dump: [ 6844.780448] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 6844.783677] Call Trace: [ 6844.784306] __schedule+0x351/0xcb0 [ 6844.785574] schedule+0xc0/0x180 [ 6844.786553] schedule_timeout+0xb4/0x190 [ 6844.787597] ? do_raw_spin_unlock+0x75/0x190 [ 6844.788874] ? __next_timer_interrupt+0x160/0x160 [ 6844.790271] ? prepare_to_swait_event+0xf3/0x140 [ 6844.791955] rcu_gp_kthread+0x8dd/0x15b0 [ 6844.793543] ? force_qs_rnp+0x2c0/0x2c0 [ 6844.795098] kthread+0x1d1/0x200 [ 6844.796140] ? set_kthread_struct+0x70/0x70 [ 6844.797576] ret_from_fork+0x1f/0x30 [ 6844.798955] rcu: Stack dump where RCU GP kthread last ran: [ 6844.801140] NMI backtrace for cpu 1 [ 6844.802349] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6844.806134] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6844.809045] Call Trace: [ 6844.809958] [ 6844.810915] ? dump_stack+0xbb/0x10e [ 6844.812140] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6844.813763] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6844.815241] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6844.817263] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6844.821753] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6844.831711] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 6844.840003] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 6844.844629] ? update_process_times+0x6d/0xc0 [ 6844.848188] ? tick_sched_handle.isra.18+0x27/0x90 [ 6844.852380] ? tick_sched_timer+0x53/0xb0 [ 6844.855367] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6844.859466] ? tick_sched_handle.isra.18+0x90/0x90 [ 6844.863010] ? hrtimer_interrupt+0x128/0x2c0 [ 6844.866240] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6844.871575] ? apic_timer_interrupt+0xf/0x20 [ 6844.873965] [ 6844.875359] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6844.878014] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6844.882015] ? do_raw_spin_lock+0x177/0x190 [ 6844.884309] ? _raw_spin_lock+0x21/0x30 [ 6844.887294] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6844.891586] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 6844.898521] ? mdc_commit_open+0x91/0xb0 [mdc] [ 6844.900956] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6844.904223] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6844.909386] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 6844.914281] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6844.920026] ? __next_timer_interrupt+0x160/0x160 [ 6844.922913] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6844.929707] ? do_wait_intr+0xf0/0xf0 [ 6844.932029] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6844.934455] ? kthread+0x1d1/0x200 [ 6844.935611] ? set_kthread_struct+0x70/0x70 [ 6844.937937] ? ret_from_fork+0x1f/0x30 [ 6844.941526] NMI backtrace for cpu 1 [ 6844.946570] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6844.952397] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6844.958268] Call Trace: [ 6844.960251] [ 6844.962174] ? dump_stack+0xbb/0x10e [ 6844.968291] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6844.971103] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6844.973386] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6844.976382] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6844.980146] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6844.984136] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 6844.987291] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 6844.989956] ? update_process_times+0x6d/0xc0 [ 6844.991811] ? tick_sched_handle.isra.18+0x27/0x90 [ 6844.993529] ? tick_sched_timer+0x53/0xb0 [ 6844.996890] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6845.000319] ? tick_sched_handle.isra.18+0x90/0x90 [ 6845.002902] ? hrtimer_interrupt+0x128/0x2c0 [ 6845.005552] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6845.008721] ? apic_timer_interrupt+0xf/0x20 [ 6845.010573] [ 6845.012093] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6845.016171] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6845.018841] ? do_raw_spin_lock+0x177/0x190 [ 6845.020775] ? _raw_spin_lock+0x21/0x30 [ 6845.022743] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6845.024997] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 6845.027573] ? mdc_commit_open+0x91/0xb0 [mdc] [ 6845.029899] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6845.032871] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6845.036507] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 6845.039030] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6845.041734] ? __next_timer_interrupt+0x160/0x160 [ 6845.044429] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6845.046900] ? do_wait_intr+0xf0/0xf0 [ 6845.048732] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6845.051933] ? kthread+0x1d1/0x200 [ 6845.054066] ? set_kthread_struct+0x70/0x70 [ 6845.056135] ? ret_from_fork+0x1f/0x30 [ 6849.504634] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4918s! [ 6849.508467] Showing busy workqueues and worker pools: [ 6849.510494] workqueue events: flags=0x0 [ 6849.512188] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6849.516337] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 6849.522165] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6849.525811] pending: kfree_rcu_monitor [ 6849.528338] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6849.532681] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 6849.537351] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6849.544354] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 6849.548859] workqueue events_long: flags=0x0 [ 6849.551702] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6849.556128] pending: lu_objects_flush [obdclass] [ 6849.560965] workqueue events_unbound: flags=0x2 [ 6849.565776] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6849.570764] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6849.578853] workqueue events_power_efficient: flags=0x80 [ 6849.582883] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6849.586725] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 6849.591180] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6849.595710] pending: neigh_periodic_work [ 6849.598326] workqueue mm_percpu_wq: flags=0x8 [ 6849.602456] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6849.607131] pending: vmstat_update, lru_add_drain_per_cpu [ 6849.610193] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6849.615689] pending: vmstat_update [ 6849.618822] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6849.623881] pending: lru_add_drain_per_cpu [ 6849.626358] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6849.630683] pending: lru_add_drain_per_cpu BAR(48) [ 6849.633076] workqueue writeback: flags=0x4a [ 6849.636109] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 6849.639813] pending: wb_workfn [ 6849.642143] workqueue xprtiod: flags=0xa [ 6849.644614] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6849.648645] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6849.654319] workqueue ipv6_addrconf: flags=0x40008 [ 6849.656922] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6849.660810] pending: addrconf_verify_work [ 6849.663522] workqueue knbd0-recv: flags=0x1a [ 6849.666511] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6849.671113] in-flight: 658:recv_work [ 6849.673733] workqueue ptlrpc_pinger: flags=0x2 [ 6849.676541] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6849.680362] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6849.687096] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4919s workers=4 idle: 32984 37 8 [ 6849.691933] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6834s workers=2 idle: 661 [ 6860.059130] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 6860.063514] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6860.109193] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6860.116705] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6860.126148] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6860.132047] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6860.149402] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6860.156130] RAX: 0000000000004184 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6860.161289] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 6860.168442] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6860.171295] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 6860.176146] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 6860.184443] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 6860.191543] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6860.204970] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 6860.210237] Call Trace: [ 6860.212269] [ 6860.213201] ? show_regs.cold.9+0x22/0x2f [ 6860.222579] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6860.224092] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6860.225550] ? watchdog+0x40/0x40 [ 6860.228172] ? hrtimer_interrupt+0x128/0x2c0 [ 6860.229433] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6860.233749] ? apic_timer_interrupt+0xf/0x20 [ 6860.235082] [ 6860.235651] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6860.247304] do_raw_spin_lock+0x177/0x190 [ 6860.249365] _raw_spin_lock+0x21/0x30 [ 6860.251997] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6860.258292] ? do_raw_spin_unlock+0x75/0x190 [ 6860.268797] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6860.272063] ? do_wait_intr+0xf0/0xf0 [ 6860.274915] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6860.277273] mdc_close+0x53b/0xd70 [mdc] [ 6860.279073] lmv_close+0x18d/0x490 [lmv] [ 6860.280197] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6860.287471] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6860.296336] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6860.300948] ll_md_close+0x274/0x800 [lustre] [ 6860.306047] ll_file_release+0x128/0xc50 [lustre] [ 6860.309419] __fput+0x10d/0x440 [ 6860.312863] ____fput+0x12/0x20 [ 6860.316294] task_work_run+0xc6/0x110 [ 6860.317836] exit_to_usermode_loop+0x1dd/0x1f0 [ 6860.322024] do_syscall_64+0x3d6/0x440 [ 6860.324456] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6860.329014] RIP: 0033:0x7f151bee3cd8 [ 6860.330344] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6860.342968] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6860.352043] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 6860.365308] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 6860.375854] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6860.384124] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6860.390486] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6864.056009] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6864.059413] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6864.108145] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6864.113192] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6864.121800] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6864.128055] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 6864.140762] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 6864.149867] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6864.153382] RAX: 0000000000004cfe RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6864.156682] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 6864.160011] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 6864.163031] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 6864.165942] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 6864.168671] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 6864.171945] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6864.174346] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6864.177384] Call Trace: [ 6864.178412] [ 6864.179851] ? show_regs.cold.9+0x22/0x2f [ 6864.181658] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6864.184152] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6864.186114] ? watchdog+0x40/0x40 [ 6864.188716] ? hrtimer_interrupt+0x128/0x2c0 [ 6864.190433] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6864.192024] ? apic_timer_interrupt+0xf/0x20 [ 6864.194441] [ 6864.195921] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 6864.198727] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6864.201208] do_raw_spin_lock+0x177/0x190 [ 6864.203047] _raw_spin_lock+0x21/0x30 [ 6864.204578] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6864.207319] process_one_work+0x2c8/0x700 [ 6864.209312] worker_thread+0x296/0x6e0 [ 6864.211066] ? rescuer_thread+0x570/0x570 [ 6864.213355] kthread+0x1d1/0x200 [ 6864.214990] ? set_kthread_struct+0x70/0x70 [ 6864.216533] ret_from_fork+0x1f/0x30 [ 6872.055097] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 6872.062849] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6872.110099] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6872.137040] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6872.143959] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6872.150311] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6872.172842] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6872.184833] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 6872.189829] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 6872.194676] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 6872.200872] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 6872.204054] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6872.208809] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 6872.217633] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6872.222252] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6872.229922] Call Trace: [ 6872.231914] [ 6872.233365] ? show_regs.cold.9+0x22/0x2f [ 6872.237230] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6872.240495] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6872.245694] ? watchdog+0x40/0x40 [ 6872.248153] ? hrtimer_interrupt+0x128/0x2c0 [ 6872.250754] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6872.252500] ? apic_timer_interrupt+0xf/0x20 [ 6872.255925] [ 6872.257670] ? native_safe_halt+0x2e/0x50 [ 6872.260189] kvm_wait+0xcd/0xe0 [ 6872.263191] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6872.267795] do_raw_spin_lock+0x177/0x190 [ 6872.270050] _raw_spin_lock+0x21/0x30 [ 6872.273142] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6872.281484] obd_mod_free+0x46/0x140 [ptlrpc] [ 6872.293113] mdc_commit_open+0x91/0xb0 [mdc] [ 6872.296247] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6872.305257] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6872.317784] after_reply+0xe2c/0x1780 [ptlrpc] [ 6872.323040] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6872.330468] ? __next_timer_interrupt+0x160/0x160 [ 6872.332651] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6872.334764] ? do_wait_intr+0xf0/0xf0 [ 6872.336715] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6872.341794] kthread+0x1d1/0x200 [ 6872.343072] ? set_kthread_struct+0x70/0x70 [ 6872.347381] ret_from_fork+0x1f/0x30 [ 6876.052046] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 6876.059165] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6876.093850] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6876.102174] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6876.110865] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6876.114193] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6876.127167] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 6876.132156] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6876.138308] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 6876.144203] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 6876.154068] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 6876.164406] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 6876.174316] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 6876.179246] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6876.184239] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 6876.190155] Call Trace: [ 6876.191814] [ 6876.193067] ? show_regs.cold.9+0x22/0x2f [ 6876.202823] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6876.212837] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6876.214210] ? watchdog+0x40/0x40 [ 6876.217854] ? hrtimer_interrupt+0x128/0x2c0 [ 6876.222044] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6876.225639] ? apic_timer_interrupt+0xf/0x20 [ 6876.230205] [ 6876.233719] ? native_safe_halt+0x2e/0x50 [ 6876.240160] kvm_wait+0xcd/0xe0 [ 6876.242012] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6876.245100] do_raw_spin_lock+0x177/0x190 [ 6876.247413] _raw_spin_lock+0x21/0x30 [ 6876.249609] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6876.254227] ? do_raw_spin_unlock+0x75/0x190 [ 6876.256474] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6876.259863] ? do_wait_intr+0xf0/0xf0 [ 6876.262564] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6876.265587] mdc_reint+0x2d/0x110 [mdc] [ 6876.269987] mdc_unlink+0x658/0x870 [mdc] [ 6876.274780] lmv_unlink+0x34d/0xbd0 [lmv] [ 6876.278657] ll_unlink+0x24d/0x7f0 [lustre] [ 6876.288303] vfs_unlink+0x202/0x330 [ 6876.290973] do_unlinkat+0x2fa/0x460 [ 6876.293584] __x64_sys_unlink+0x37/0x50 [ 6876.296265] do_syscall_64+0xc1/0x440 [ 6876.299025] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6876.303256] RIP: 0033:0x7f151bdee10b [ 6876.305189] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6876.317134] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6876.329417] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 6876.339311] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 6876.346159] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6876.354060] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6876.359577] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6879.714405] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4949s! [ 6879.722670] Showing busy workqueues and worker pools: [ 6879.726062] workqueue events: flags=0x0 [ 6879.731307] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6879.733370] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 6879.753957] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6879.756201] pending: kfree_rcu_monitor [ 6879.762787] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6879.777022] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 6879.785950] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6879.789636] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 6879.801556] workqueue events_long: flags=0x0 [ 6879.803116] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6879.812339] pending: lu_objects_flush [obdclass] [ 6879.830555] workqueue events_unbound: flags=0x2 [ 6879.843851] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6879.850069] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6879.856293] workqueue events_power_efficient: flags=0x80 [ 6879.859827] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6879.862875] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 6879.877627] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6879.883072] pending: neigh_periodic_work [ 6879.892858] workqueue mm_percpu_wq: flags=0x8 [ 6879.898560] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6879.903436] pending: vmstat_update, lru_add_drain_per_cpu [ 6879.908541] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6879.915554] pending: vmstat_update [ 6879.920481] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6879.927517] pending: lru_add_drain_per_cpu [ 6879.933124] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6879.936603] pending: lru_add_drain_per_cpu BAR(48) [ 6879.940395] workqueue writeback: flags=0x4a [ 6879.942661] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 6879.946464] pending: wb_workfn [ 6879.948772] workqueue xprtiod: flags=0xa [ 6879.950669] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6879.955397] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6879.964890] workqueue ipv6_addrconf: flags=0x40008 [ 6879.971285] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6879.976897] pending: addrconf_verify_work [ 6879.986299] workqueue knbd0-recv: flags=0x1a [ 6879.991283] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6879.996298] in-flight: 658:recv_work [ 6879.999038] workqueue ptlrpc_pinger: flags=0x2 [ 6880.002237] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6880.005991] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6880.014154] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4949s workers=4 idle: 32984 37 8 [ 6880.023156] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6864s workers=2 idle: 661 [ 6888.057042] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 6888.060288] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6888.107400] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6888.117718] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6888.125924] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6888.132958] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6888.147136] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6888.150866] RAX: 0000000000005365 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6888.153889] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 6888.158627] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6888.160925] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 6888.166981] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 6888.169859] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 6888.173732] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6888.176956] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 6888.189167] Call Trace: [ 6888.190801] [ 6888.191656] ? show_regs.cold.9+0x22/0x2f [ 6888.196003] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6888.200660] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6888.203845] ? watchdog+0x40/0x40 [ 6888.210854] ? hrtimer_interrupt+0x128/0x2c0 [ 6888.218517] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6888.221599] ? apic_timer_interrupt+0xf/0x20 [ 6888.223310] [ 6888.223883] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6888.226428] do_raw_spin_lock+0x177/0x190 [ 6888.227738] _raw_spin_lock+0x21/0x30 [ 6888.229480] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6888.231128] ? do_raw_spin_unlock+0x75/0x190 [ 6888.232301] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6888.235938] ? do_wait_intr+0xf0/0xf0 [ 6888.240831] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6888.244016] mdc_close+0x53b/0xd70 [mdc] [ 6888.251147] lmv_close+0x18d/0x490 [lmv] [ 6888.254897] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6888.258820] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6888.266520] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6888.268872] ll_md_close+0x274/0x800 [lustre] [ 6888.270929] ll_file_release+0x128/0xc50 [lustre] [ 6888.280083] __fput+0x10d/0x440 [ 6888.281819] ____fput+0x12/0x20 [ 6888.283090] task_work_run+0xc6/0x110 [ 6888.286985] exit_to_usermode_loop+0x1dd/0x1f0 [ 6888.292792] do_syscall_64+0x3d6/0x440 [ 6888.294211] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6888.295971] RIP: 0033:0x7f151bee3cd8 [ 6888.298831] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6888.307099] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6888.314182] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 6888.316934] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 6888.320551] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6888.323751] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6888.328144] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6892.056018] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6892.060672] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6892.100814] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6892.108884] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6892.119446] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6892.125883] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6892.130742] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6892.160098] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6892.169279] RAX: 0000000000006688 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6892.172094] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 6892.175255] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 6892.179012] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 6892.182653] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 6892.185007] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 6892.189501] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6892.192814] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6892.196719] Call Trace: [ 6892.197941] [ 6892.199142] ? show_regs.cold.9+0x22/0x2f [ 6892.201425] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6892.203944] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6892.206045] ? watchdog+0x40/0x40 [ 6892.207636] ? hrtimer_interrupt+0x128/0x2c0 [ 6892.209912] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6892.212384] ? apic_timer_interrupt+0xf/0x20 [ 6892.214447] [ 6892.214991] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6892.216288] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6892.218984] do_raw_spin_lock+0x177/0x190 [ 6892.221057] _raw_spin_lock+0x21/0x30 [ 6892.223486] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6892.226279] process_one_work+0x2c8/0x700 [ 6892.228017] worker_thread+0x296/0x6e0 [ 6892.230027] ? rescuer_thread+0x570/0x570 [ 6892.232592] kthread+0x1d1/0x200 [ 6892.233858] ? set_kthread_struct+0x70/0x70 [ 6892.236067] ret_from_fork+0x1f/0x30 [ 6900.055079] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 6900.058749] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6900.077450] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6900.081798] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6900.085619] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6900.087945] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6900.095100] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6900.098740] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 6900.102291] RDX: 00000000000003b1 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 6900.105025] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 6900.108520] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 6900.111672] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6900.115259] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 6900.119775] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6900.123039] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6900.126880] Call Trace: [ 6900.129441] [ 6900.130967] ? show_regs.cold.9+0x22/0x2f [ 6900.133414] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6900.136030] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6900.138812] ? watchdog+0x40/0x40 [ 6900.141227] ? hrtimer_interrupt+0x128/0x2c0 [ 6900.143087] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6900.145348] ? apic_timer_interrupt+0xf/0x20 [ 6900.147033] [ 6900.147804] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6900.149894] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6900.152186] do_raw_spin_lock+0x177/0x190 [ 6900.153645] _raw_spin_lock+0x21/0x30 [ 6900.155412] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6900.158150] obd_mod_free+0x46/0x140 [ptlrpc] [ 6900.160313] mdc_commit_open+0x91/0xb0 [mdc] [ 6900.161770] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6900.164779] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6900.168147] after_reply+0xe2c/0x1780 [ptlrpc] [ 6900.170752] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6900.172901] ? __next_timer_interrupt+0x160/0x160 [ 6900.174954] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6900.177096] ? do_wait_intr+0xf0/0xf0 [ 6900.178644] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6900.181699] kthread+0x1d1/0x200 [ 6900.183012] ? set_kthread_struct+0x70/0x70 [ 6900.185748] ret_from_fork+0x1f/0x30 [ 6904.052011] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 6904.055706] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6904.089587] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6904.101945] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6904.108553] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 6904.112432] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 6904.122534] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6904.126520] RAX: 000000000000106e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6904.130760] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 6904.134967] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 6904.141378] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 6904.146709] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 6904.151167] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 6904.155633] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6904.159957] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 6904.163647] Call Trace: [ 6904.164854] [ 6904.165732] ? show_regs.cold.9+0x22/0x2f [ 6904.167555] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6904.173383] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6904.175589] ? watchdog+0x40/0x40 [ 6904.178806] ? hrtimer_interrupt+0x128/0x2c0 [ 6904.181993] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6904.187562] ? apic_timer_interrupt+0xf/0x20 [ 6904.189393] [ 6904.190331] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 6904.192670] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6904.195510] do_raw_spin_lock+0x177/0x190 [ 6904.198487] _raw_spin_lock+0x21/0x30 [ 6904.200946] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6904.209766] ? do_raw_spin_unlock+0x75/0x190 [ 6904.213858] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6904.222735] ? do_wait_intr+0xf0/0xf0 [ 6904.227824] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6904.233719] mdc_reint+0x2d/0x110 [mdc] [ 6904.236957] mdc_unlink+0x658/0x870 [mdc] [ 6904.240580] lmv_unlink+0x34d/0xbd0 [lmv] [ 6904.246624] ll_unlink+0x24d/0x7f0 [lustre] [ 6904.252674] vfs_unlink+0x202/0x330 [ 6904.260275] do_unlinkat+0x2fa/0x460 [ 6904.261700] __x64_sys_unlink+0x37/0x50 [ 6904.263152] do_syscall_64+0xc1/0x440 [ 6904.270660] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6904.274773] RIP: 0033:0x7f151bdee10b [ 6904.281482] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6904.302316] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6904.317428] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 6904.325266] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 6904.331650] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6904.334901] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6904.338641] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6910.438724] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4979s! [ 6910.454299] Showing busy workqueues and worker pools: [ 6910.457503] workqueue events: flags=0x0 [ 6910.459988] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6910.464091] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 6910.472968] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6910.477081] pending: kfree_rcu_monitor [ 6910.488292] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6910.496570] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 6910.509411] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6910.517470] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 6910.523036] workqueue events_long: flags=0x0 [ 6910.528369] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6910.535792] pending: lu_objects_flush [obdclass] [ 6910.540561] workqueue events_unbound: flags=0x2 [ 6910.544564] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6910.550725] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6910.557146] workqueue events_power_efficient: flags=0x80 [ 6910.561407] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6910.566915] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 6910.573620] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6910.576829] pending: neigh_periodic_work [ 6910.581785] workqueue mm_percpu_wq: flags=0x8 [ 6910.583087] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6910.589977] pending: vmstat_update, lru_add_drain_per_cpu [ 6910.594645] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6910.600900] pending: vmstat_update [ 6910.604163] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6910.610032] pending: lru_add_drain_per_cpu [ 6910.612602] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6910.616983] pending: lru_add_drain_per_cpu BAR(48) [ 6910.619391] workqueue writeback: flags=0x4a [ 6910.621552] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 6910.628769] pending: wb_workfn [ 6910.631897] workqueue xprtiod: flags=0xa [ 6910.633884] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6910.639732] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6910.642772] workqueue ipv6_addrconf: flags=0x40008 [ 6910.645847] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6910.650377] pending: addrconf_verify_work [ 6910.655129] workqueue knbd0-recv: flags=0x1a [ 6910.658553] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6910.663215] in-flight: 658:recv_work [ 6910.667162] workqueue ptlrpc_pinger: flags=0x2 [ 6910.671802] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6910.681727] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6910.687886] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4980s workers=4 idle: 32984 37 8 [ 6910.695946] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6895s workers=2 idle: 661 [ 6916.058475] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 6916.073494] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6916.108561] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6916.122650] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6916.131387] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6916.135470] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6916.157871] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6916.171429] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6916.179278] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 6916.183269] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6916.192741] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 6916.198317] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 6916.202005] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 6916.207395] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6916.210514] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 6916.214331] Call Trace: [ 6916.216268] [ 6916.218143] ? show_regs.cold.9+0x22/0x2f [ 6916.220863] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6916.227618] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6916.230939] ? watchdog+0x40/0x40 [ 6916.237257] ? hrtimer_interrupt+0x128/0x2c0 [ 6916.239616] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6916.244648] ? apic_timer_interrupt+0xf/0x20 [ 6916.249501] [ 6916.251933] ? native_safe_halt+0x2e/0x50 [ 6916.254465] kvm_wait+0xcd/0xe0 [ 6916.257274] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6916.258999] do_raw_spin_lock+0x177/0x190 [ 6916.261783] _raw_spin_lock+0x21/0x30 [ 6916.263620] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6916.266546] ? do_raw_spin_unlock+0x75/0x190 [ 6916.269866] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6916.273426] ? do_wait_intr+0xf0/0xf0 [ 6916.275378] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6916.277936] mdc_close+0x53b/0xd70 [mdc] [ 6916.279957] lmv_close+0x18d/0x490 [lmv] [ 6916.282385] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6916.285052] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6916.287914] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6916.290366] ll_md_close+0x274/0x800 [lustre] [ 6916.292524] ll_file_release+0x128/0xc50 [lustre] [ 6916.295354] __fput+0x10d/0x440 [ 6916.296802] ____fput+0x12/0x20 [ 6916.299930] task_work_run+0xc6/0x110 [ 6916.303938] exit_to_usermode_loop+0x1dd/0x1f0 [ 6916.315241] do_syscall_64+0x3d6/0x440 [ 6916.317625] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6916.323577] RIP: 0033:0x7f151bee3cd8 [ 6916.326012] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6916.348930] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6916.361472] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 6916.366876] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 6916.374320] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6916.380197] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6916.383521] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6920.056009] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6920.059706] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6920.082226] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6920.087904] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6920.092189] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6920.096135] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6920.099037] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6920.107622] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6920.111381] RAX: 0000000000005b08 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6920.114791] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 6920.117820] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 6920.121068] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 6920.124471] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 6920.127725] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 6920.131250] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6920.134136] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6920.137434] Call Trace: [ 6920.138799] [ 6920.139777] ? show_regs.cold.9+0x22/0x2f [ 6920.141583] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6920.145891] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6920.148045] ? watchdog+0x40/0x40 [ 6920.150350] ? hrtimer_interrupt+0x128/0x2c0 [ 6920.155736] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6920.158574] ? apic_timer_interrupt+0xf/0x20 [ 6920.162423] [ 6920.163581] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6920.166188] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6920.169164] do_raw_spin_lock+0x177/0x190 [ 6920.171374] _raw_spin_lock+0x21/0x30 [ 6920.173380] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6920.176822] process_one_work+0x2c8/0x700 [ 6920.179778] worker_thread+0x296/0x6e0 [ 6920.181875] ? rescuer_thread+0x570/0x570 [ 6920.184686] kthread+0x1d1/0x200 [ 6920.188074] ? set_kthread_struct+0x70/0x70 [ 6920.191013] ret_from_fork+0x1f/0x30 [ 6928.055008] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 6928.062044] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6928.105906] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6928.114676] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6928.119746] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6928.122376] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6928.135407] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6928.140110] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 6928.143221] RDX: 0000000000000a3b RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 6928.147397] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 6928.150675] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 6928.153581] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6928.156962] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 6928.160647] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6928.163076] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6928.166938] Call Trace: [ 6928.168074] [ 6928.168941] ? show_regs.cold.9+0x22/0x2f [ 6928.170807] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6928.173099] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6928.179973] ? watchdog+0x40/0x40 [ 6928.181671] ? hrtimer_interrupt+0x128/0x2c0 [ 6928.186520] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6928.189194] ? apic_timer_interrupt+0xf/0x20 [ 6928.195094] [ 6928.196154] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6928.201689] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6928.204941] do_raw_spin_lock+0x177/0x190 [ 6928.213293] _raw_spin_lock+0x21/0x30 [ 6928.215687] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6928.222817] obd_mod_free+0x46/0x140 [ptlrpc] [ 6928.225787] mdc_commit_open+0x91/0xb0 [mdc] [ 6928.229996] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6928.235076] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6928.238754] after_reply+0xe2c/0x1780 [ptlrpc] [ 6928.242084] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6928.245725] ? __next_timer_interrupt+0x160/0x160 [ 6928.248382] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6928.250996] ? do_wait_intr+0xf0/0xf0 [ 6928.253257] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6928.260041] kthread+0x1d1/0x200 [ 6928.261782] ? set_kthread_struct+0x70/0x70 [ 6928.264767] ret_from_fork+0x1f/0x30 [ 6936.053064] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 6936.058494] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6936.089446] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6936.095818] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6936.104381] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6936.108223] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6936.120198] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6936.124056] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6936.130694] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 6936.132919] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 6936.138141] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 6936.146556] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 6936.152437] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 6936.158131] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6936.160719] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 6936.166083] Call Trace: [ 6936.168030] [ 6936.169116] ? show_regs.cold.9+0x22/0x2f [ 6936.171275] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6936.178678] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6936.180444] ? watchdog+0x40/0x40 [ 6936.181628] ? hrtimer_interrupt+0x128/0x2c0 [ 6936.184073] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6936.186142] ? apic_timer_interrupt+0xf/0x20 [ 6936.189735] [ 6936.190671] ? native_safe_halt+0x2e/0x50 [ 6936.192979] kvm_wait+0xcd/0xe0 [ 6936.193987] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6936.195796] do_raw_spin_lock+0x177/0x190 [ 6936.200908] _raw_spin_lock+0x21/0x30 [ 6936.201870] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6936.205054] ? do_raw_spin_unlock+0x75/0x190 [ 6936.207632] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6936.210031] ? do_wait_intr+0xf0/0xf0 [ 6936.214256] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6936.216757] mdc_reint+0x2d/0x110 [mdc] [ 6936.218266] mdc_unlink+0x658/0x870 [mdc] [ 6936.220191] lmv_unlink+0x34d/0xbd0 [lmv] [ 6936.223982] ll_unlink+0x24d/0x7f0 [lustre] [ 6936.225942] vfs_unlink+0x202/0x330 [ 6936.227481] do_unlinkat+0x2fa/0x460 [ 6936.230170] __x64_sys_unlink+0x37/0x50 [ 6936.231103] do_syscall_64+0xc1/0x440 [ 6936.232183] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6936.236151] RIP: 0033:0x7f151bdee10b [ 6936.237376] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6936.253745] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6936.260256] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 6936.264593] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 6936.274550] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6936.276636] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6936.279146] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6941.158089] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5010s! [ 6941.166709] Showing busy workqueues and worker pools: [ 6941.169719] workqueue events: flags=0x0 [ 6941.172610] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6941.180431] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 6941.187682] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6941.192456] pending: kfree_rcu_monitor [ 6941.194537] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6941.206248] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 6941.218419] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6941.225787] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 6941.232199] workqueue events_long: flags=0x0 [ 6941.233556] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6941.236249] pending: lu_objects_flush [obdclass] [ 6941.242650] workqueue events_unbound: flags=0x2 [ 6941.245832] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6941.248427] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6941.261721] workqueue events_power_efficient: flags=0x80 [ 6941.266284] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6941.271848] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 6941.275507] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6941.283841] pending: neigh_periodic_work [ 6941.285436] workqueue mm_percpu_wq: flags=0x8 [ 6941.289285] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6941.293336] pending: vmstat_update, lru_add_drain_per_cpu [ 6941.295625] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6941.299048] pending: vmstat_update [ 6941.300655] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6941.303956] pending: lru_add_drain_per_cpu [ 6941.305484] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6941.307663] pending: lru_add_drain_per_cpu BAR(48) [ 6941.309169] workqueue writeback: flags=0x4a [ 6941.310448] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 6941.313333] pending: wb_workfn [ 6941.315030] workqueue xprtiod: flags=0xa [ 6941.316642] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6941.320664] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6941.323640] workqueue ipv6_addrconf: flags=0x40008 [ 6941.325924] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6941.328645] pending: addrconf_verify_work [ 6941.330468] workqueue knbd0-recv: flags=0x1a [ 6941.333598] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6941.337430] in-flight: 658:recv_work [ 6941.343826] workqueue ptlrpc_pinger: flags=0x2 [ 6941.346960] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6941.349709] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6941.352920] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5010s workers=4 idle: 32984 37 8 [ 6941.359204] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6926s workers=2 idle: 661 [ 6944.057024] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 6944.062441] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6944.111701] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6944.135460] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6944.146331] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6944.147543] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6944.173719] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6944.177283] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6944.187304] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 6944.192537] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6944.199498] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 6944.202664] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 6944.208400] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 6944.212913] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6944.220853] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 6944.224785] Call Trace: [ 6944.225675] [ 6944.229627] ? show_regs.cold.9+0x22/0x2f [ 6944.232846] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6944.233956] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6944.238227] ? watchdog+0x40/0x40 [ 6944.241559] ? hrtimer_interrupt+0x128/0x2c0 [ 6944.245168] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6944.247009] ? apic_timer_interrupt+0xf/0x20 [ 6944.258541] [ 6944.259316] ? native_safe_halt+0x2e/0x50 [ 6944.260627] kvm_wait+0xcd/0xe0 [ 6944.261731] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6944.266896] do_raw_spin_lock+0x177/0x190 [ 6944.271173] _raw_spin_lock+0x21/0x30 [ 6944.276157] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6944.288251] ? do_raw_spin_unlock+0x75/0x190 [ 6944.289702] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6944.300870] ? do_wait_intr+0xf0/0xf0 [ 6944.302193] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6944.307409] mdc_close+0x53b/0xd70 [mdc] [ 6944.310647] lmv_close+0x18d/0x490 [lmv] [ 6944.312414] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6944.317203] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6944.322512] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6944.327375] ll_md_close+0x274/0x800 [lustre] [ 6944.334510] ll_file_release+0x128/0xc50 [lustre] [ 6944.339079] __fput+0x10d/0x440 [ 6944.342603] ____fput+0x12/0x20 [ 6944.345329] task_work_run+0xc6/0x110 [ 6944.346666] exit_to_usermode_loop+0x1dd/0x1f0 [ 6944.350298] do_syscall_64+0x3d6/0x440 [ 6944.351426] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6944.354975] RIP: 0033:0x7f151bee3cd8 [ 6944.357574] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6944.371093] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6944.384629] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 6944.387203] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 6944.400733] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6944.407938] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6944.411937] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6948.058138] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6948.064187] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6948.104089] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6948.110759] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6948.116027] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6948.120000] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6948.123509] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6948.146086] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6948.153307] RAX: 0000000000007687 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6948.156265] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 6948.160589] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 6948.162725] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 6948.165259] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 6948.168114] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 6948.171836] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6948.174527] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6948.178543] Call Trace: [ 6948.179963] [ 6948.181076] ? show_regs.cold.9+0x22/0x2f [ 6948.182569] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6948.185235] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6948.186896] ? watchdog+0x40/0x40 [ 6948.188328] ? hrtimer_interrupt+0x128/0x2c0 [ 6948.190457] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6948.193221] ? apic_timer_interrupt+0xf/0x20 [ 6948.195523] [ 6948.196337] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6948.199273] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6948.201872] do_raw_spin_lock+0x177/0x190 [ 6948.203732] _raw_spin_lock+0x21/0x30 [ 6948.205134] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6948.207714] process_one_work+0x2c8/0x700 [ 6948.209580] worker_thread+0x296/0x6e0 [ 6948.211277] ? rescuer_thread+0x570/0x570 [ 6948.213124] kthread+0x1d1/0x200 [ 6948.214629] ? set_kthread_struct+0x70/0x70 [ 6948.216536] ret_from_fork+0x1f/0x30 [ 6956.055064] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 6956.058995] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6956.079555] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6956.084289] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6956.087125] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6956.088534] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6956.094570] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6956.097378] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 6956.099393] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 6956.101624] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 6956.103794] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 6956.105742] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6956.108279] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 6956.111245] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6956.113235] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6956.115117] Call Trace: [ 6956.115867] [ 6956.116361] ? show_regs.cold.9+0x22/0x2f [ 6956.117456] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6956.118960] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6956.120209] ? watchdog+0x40/0x40 [ 6956.121106] ? hrtimer_interrupt+0x128/0x2c0 [ 6956.122263] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6956.123445] ? apic_timer_interrupt+0xf/0x20 [ 6956.124841] [ 6956.125334] ? native_safe_halt+0x2e/0x50 [ 6956.126379] kvm_wait+0xcd/0xe0 [ 6956.127231] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6956.128739] do_raw_spin_lock+0x177/0x190 [ 6956.130108] _raw_spin_lock+0x21/0x30 [ 6956.131065] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6956.132569] obd_mod_free+0x46/0x140 [ptlrpc] [ 6956.134975] mdc_commit_open+0x91/0xb0 [mdc] [ 6956.137900] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6956.140335] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6956.145150] after_reply+0xe2c/0x1780 [ptlrpc] [ 6956.148321] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6956.151725] ? __next_timer_interrupt+0x160/0x160 [ 6956.154318] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6956.156562] ? do_wait_intr+0xf0/0xf0 [ 6956.158308] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6956.161239] kthread+0x1d1/0x200 [ 6956.162869] ? set_kthread_struct+0x70/0x70 [ 6956.164927] ret_from_fork+0x1f/0x30 [ 6968.052212] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 6968.054847] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6968.086301] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6968.098039] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6968.104614] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6968.107366] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6968.124145] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6968.132099] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6968.135247] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 6968.140326] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 6968.144402] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 6968.150714] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 6968.156243] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 6968.163259] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6968.165560] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 6968.168844] Call Trace: [ 6968.169766] [ 6968.170440] ? show_regs.cold.9+0x22/0x2f [ 6968.171870] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6968.176267] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6968.177525] ? watchdog+0x40/0x40 [ 6968.180363] ? hrtimer_interrupt+0x128/0x2c0 [ 6968.181534] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6968.188992] ? apic_timer_interrupt+0xf/0x20 [ 6968.190467] [ 6968.191116] ? native_safe_halt+0x2e/0x50 [ 6968.192295] kvm_wait+0xcd/0xe0 [ 6968.203316] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6968.206580] do_raw_spin_lock+0x177/0x190 [ 6968.215083] _raw_spin_lock+0x21/0x30 [ 6968.216070] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6968.222081] ? do_raw_spin_unlock+0x75/0x190 [ 6968.226852] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6968.234911] ? do_wait_intr+0xf0/0xf0 [ 6968.236121] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6968.244355] mdc_reint+0x2d/0x110 [mdc] [ 6968.248798] mdc_unlink+0x658/0x870 [mdc] [ 6968.256218] lmv_unlink+0x34d/0xbd0 [lmv] [ 6968.259211] ll_unlink+0x24d/0x7f0 [lustre] [ 6968.266929] vfs_unlink+0x202/0x330 [ 6968.271366] do_unlinkat+0x2fa/0x460 [ 6968.272601] __x64_sys_unlink+0x37/0x50 [ 6968.282058] do_syscall_64+0xc1/0x440 [ 6968.283367] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6968.285023] RIP: 0033:0x7f151bdee10b [ 6968.293084] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6968.307385] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6968.313912] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 6968.318401] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 6968.322551] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6968.328405] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6968.334199] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6971.872565] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5041s! [ 6971.876242] Showing busy workqueues and worker pools: [ 6971.885898] workqueue events: flags=0x0 [ 6971.888012] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6971.893712] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 6971.898968] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6971.907864] pending: kfree_rcu_monitor [ 6971.910250] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6971.913935] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 6971.922982] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6971.928344] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 6971.940262] workqueue events_long: flags=0x0 [ 6971.943757] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6971.949822] pending: lu_objects_flush [obdclass] [ 6971.957253] workqueue events_unbound: flags=0x2 [ 6971.962470] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 6971.968256] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 6971.977141] workqueue events_power_efficient: flags=0x80 [ 6971.983406] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6971.985543] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 6971.991016] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6971.994677] pending: neigh_periodic_work [ 6971.996123] workqueue mm_percpu_wq: flags=0x8 [ 6971.997841] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6972.000786] pending: vmstat_update, lru_add_drain_per_cpu [ 6972.005012] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6972.009896] pending: vmstat_update [ 6972.012080] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6972.015254] pending: lru_add_drain_per_cpu [ 6972.019303] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6972.026752] pending: lru_add_drain_per_cpu BAR(48) [ 6972.028289] workqueue writeback: flags=0x4a [ 6972.031011] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 6972.037173] pending: wb_workfn [ 6972.040266] workqueue xprtiod: flags=0xa [ 6972.042257] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 6972.046762] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 6972.050653] workqueue ipv6_addrconf: flags=0x40008 [ 6972.053086] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6972.055834] pending: addrconf_verify_work [ 6972.057194] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 6972.058966] workqueue knbd0-recv: flags=0x1a [ 6972.063097] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6972.063301] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6972.063309] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6972.063313] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6972.063324] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6972.063329] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 6972.063333] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6972.063335] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 6972.063337] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6972.063340] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 6972.063342] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 6972.063345] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 6972.063348] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6972.063350] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 6972.063359] Call Trace: [ 6972.063364] [ 6972.063369] ? show_regs.cold.9+0x22/0x2f [ 6972.063378] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6972.063384] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6972.063388] ? watchdog+0x40/0x40 [ 6972.063393] ? hrtimer_interrupt+0x128/0x2c0 [ 6972.063397] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6972.063403] ? apic_timer_interrupt+0xf/0x20 [ 6972.063407] [ 6972.063410] ? native_safe_halt+0x2e/0x50 [ 6972.063414] kvm_wait+0xcd/0xe0 [ 6972.063624] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6972.063633] do_raw_spin_lock+0x177/0x190 [ 6972.063638] _raw_spin_lock+0x21/0x30 [ 6972.063643] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6972.068841] pwq 9: [ 6972.104900] ? do_raw_spin_unlock+0x75/0x190 [ 6972.104921] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6972.105640] ? do_wait_intr+0xf0/0xf0 [ 6972.105649] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6972.105848] mdc_close+0x53b/0xd70 [mdc] [ 6972.105884] lmv_close+0x18d/0x490 [lmv] [ 6972.105912] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 6972.120855] cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6972.120875] in-flight: 658:recv_work [ 6972.120917] workqueue ptlrpc_pinger: flags=0x2 [ 6972.120921] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6972.120928] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 6972.127805] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 6972.130051] [ 6972.130091] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5041s workers=4 idle: 32984 37 8 [ 6972.130112] pool 9: cpus=0-3 flags=0x4 nice=-20 [ 6972.142799] ll_md_real_close+0xa5/0x2a0 [lustre] [ 6972.150996] hung=6956s workers=2 idle: 661 [ 6972.255604] ll_md_close+0x274/0x800 [lustre] [ 6972.257770] ll_file_release+0x128/0xc50 [lustre] [ 6972.259652] __fput+0x10d/0x440 [ 6972.260606] ____fput+0x12/0x20 [ 6972.261596] task_work_run+0xc6/0x110 [ 6972.262821] exit_to_usermode_loop+0x1dd/0x1f0 [ 6972.264104] do_syscall_64+0x3d6/0x440 [ 6972.265225] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6972.266881] RIP: 0033:0x7f151bee3cd8 [ 6972.268017] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 6972.273770] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 6972.275968] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 6972.278772] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 6972.282949] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 6972.287088] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 6972.290157] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6976.057994] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 6976.067759] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6976.122254] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6976.140618] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6976.155627] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6976.165343] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6976.168170] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6976.186453] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6976.194132] RAX: 0000000000002737 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 6976.201029] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 6976.208158] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 6976.215159] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 6976.220744] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 6976.230185] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 6976.234630] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6976.239105] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6976.243996] Call Trace: [ 6976.253025] [ 6976.254562] ? show_regs.cold.9+0x22/0x2f [ 6976.258160] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6976.263967] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6976.268910] ? watchdog+0x40/0x40 [ 6976.271711] ? hrtimer_interrupt+0x128/0x2c0 [ 6976.275157] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6976.280087] ? apic_timer_interrupt+0xf/0x20 [ 6976.284796] [ 6976.285358] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6976.287744] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6976.291009] do_raw_spin_lock+0x177/0x190 [ 6976.292708] _raw_spin_lock+0x21/0x30 [ 6976.296553] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6976.301110] process_one_work+0x2c8/0x700 [ 6976.302758] worker_thread+0x296/0x6e0 [ 6976.304223] ? rescuer_thread+0x570/0x570 [ 6976.306012] kthread+0x1d1/0x200 [ 6976.307246] ? set_kthread_struct+0x70/0x70 [ 6976.309142] ret_from_fork+0x1f/0x30 [ 6984.056240] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 6984.065939] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6984.092125] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6984.099169] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6984.105085] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6984.110891] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6984.129637] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6984.137153] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 6984.142032] RDX: 0000000000001244 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 6984.145598] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 6984.148992] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 6984.152764] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6984.155728] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 6984.159519] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6984.161842] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 6984.164942] Call Trace: [ 6984.166729] [ 6984.167802] ? show_regs.cold.9+0x22/0x2f [ 6984.169713] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6984.175816] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6984.179790] ? watchdog+0x40/0x40 [ 6984.183543] ? hrtimer_interrupt+0x128/0x2c0 [ 6984.186173] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6984.190996] ? apic_timer_interrupt+0xf/0x20 [ 6984.194925] [ 6984.197169] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6984.200845] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6984.207208] do_raw_spin_lock+0x177/0x190 [ 6984.210694] _raw_spin_lock+0x21/0x30 [ 6984.212044] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6984.222732] obd_mod_free+0x46/0x140 [ptlrpc] [ 6984.226911] mdc_commit_open+0x91/0xb0 [mdc] [ 6984.228883] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6984.235560] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6984.240466] after_reply+0xe2c/0x1780 [ptlrpc] [ 6984.243905] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6984.246065] ? __next_timer_interrupt+0x160/0x160 [ 6984.253698] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6984.257451] ? do_wait_intr+0xf0/0xf0 [ 6984.259228] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6984.264639] kthread+0x1d1/0x200 [ 6984.265521] ? set_kthread_struct+0x70/0x70 [ 6984.273859] ret_from_fork+0x1f/0x30 [ 7000.052061] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 7000.057031] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 7000.058735] Modules linked in: lustre(O) osp(O) [ 7000.063607] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7000.063684] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7000.063691] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7000.063695] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7000.063708] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7000.063712] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 7000.063717] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7000.063719] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 7000.063722] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7000.063724] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 7000.063727] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 7000.063730] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 7000.063733] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7000.063735] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 7000.063743] Call Trace: [ 7000.063749] [ 7000.063754] ? show_regs.cold.9+0x22/0x2f [ 7000.063764] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7000.063771] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7000.063775] ? watchdog+0x40/0x40 [ 7000.063779] ? hrtimer_interrupt+0x128/0x2c0 [ 7000.063784] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7000.063790] ? apic_timer_interrupt+0xf/0x20 [ 7000.063794] [ 7000.063797] ? native_safe_halt+0x2e/0x50 [ 7000.063800] kvm_wait+0xcd/0xe0 [ 7000.063807] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7000.063815] do_raw_spin_lock+0x177/0x190 [ 7000.063820] _raw_spin_lock+0x21/0x30 [ 7000.063824] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7000.065617] ? do_raw_spin_unlock+0x75/0x190 [ 7000.065625] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7000.065816] ? do_wait_intr+0xf0/0xf0 [ 7000.065824] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7000.066422] mdc_close+0x53b/0xd70 [mdc] [ 7000.066462] lmv_close+0x18d/0x490 [lmv] [ 7000.066489] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7000.066649] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7000.066757] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7000.066836] ll_md_close+0x274/0x800 [lustre] [ 7000.066934] ll_file_release+0x128/0xc50 [lustre] [ 7000.067071] __fput+0x10d/0x440 [ 7000.067080] ____fput+0x12/0x20 [ 7000.067085] task_work_run+0xc6/0x110 [ 7000.067092] exit_to_usermode_loop+0x1dd/0x1f0 [ 7000.067100] do_syscall_64+0x3d6/0x440 [ 7000.067104] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7000.067111] RIP: 0033:0x7f151bee3cd8 [ 7000.067119] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7000.067123] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7000.067128] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 7000.067130] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 7000.067132] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7000.067134] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7000.067136] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7000.420621] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7000.437627] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7000.443049] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7000.447623] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7000.449861] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7000.463922] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7000.475883] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7000.484378] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 7000.491945] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 7000.496715] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 7000.503298] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 7000.510491] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 7000.516348] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7000.519514] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 7000.526038] Call Trace: [ 7000.527317] [ 7000.528313] ? show_regs.cold.9+0x22/0x2f [ 7000.530358] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7000.532315] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7000.533949] ? watchdog+0x40/0x40 [ 7000.535387] ? hrtimer_interrupt+0x128/0x2c0 [ 7000.536997] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7000.538921] ? apic_timer_interrupt+0xf/0x20 [ 7000.540713] [ 7000.541577] ? native_safe_halt+0x2e/0x50 [ 7000.543343] kvm_wait+0xcd/0xe0 [ 7000.544556] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7000.546489] do_raw_spin_lock+0x177/0x190 [ 7000.547851] _raw_spin_lock+0x21/0x30 [ 7000.549287] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7000.551953] ? do_raw_spin_unlock+0x75/0x190 [ 7000.553773] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7000.556361] ? do_wait_intr+0xf0/0xf0 [ 7000.557920] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7000.563191] mdc_reint+0x2d/0x110 [mdc] [ 7000.567767] mdc_unlink+0x658/0x870 [mdc] [ 7000.570032] lmv_unlink+0x34d/0xbd0 [lmv] [ 7000.571851] ll_unlink+0x24d/0x7f0 [lustre] [ 7000.575539] vfs_unlink+0x202/0x330 [ 7000.581017] do_unlinkat+0x2fa/0x460 [ 7000.582374] __x64_sys_unlink+0x37/0x50 [ 7000.584833] do_syscall_64+0xc1/0x440 [ 7000.586807] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7000.589311] RIP: 0033:0x7f151bdee10b [ 7000.590983] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7000.599706] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7000.603512] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 7000.606825] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 7000.615479] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7000.618670] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7000.621788] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7002.599408] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5071s! [ 7002.619776] Showing busy workqueues and worker pools: [ 7002.621785] workqueue events: flags=0x0 [ 7002.635764] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7002.654884] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 7002.668590] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7002.671603] pending: kfree_rcu_monitor [ 7002.673522] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7002.676140] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 7002.679940] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7002.682749] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 7002.685967] workqueue events_long: flags=0x0 [ 7002.688336] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7002.691447] pending: lu_objects_flush [obdclass] [ 7002.694323] workqueue events_unbound: flags=0x2 [ 7002.696494] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7002.699210] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7002.704399] workqueue events_power_efficient: flags=0x80 [ 7002.706904] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7002.709965] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 7002.713606] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7002.716490] pending: neigh_periodic_work [ 7002.718760] workqueue mm_percpu_wq: flags=0x8 [ 7002.720672] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7002.723864] pending: vmstat_update, lru_add_drain_per_cpu [ 7002.727497] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7002.731004] pending: vmstat_update [ 7002.732309] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7002.735192] pending: lru_add_drain_per_cpu [ 7002.737220] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7002.740603] pending: lru_add_drain_per_cpu BAR(48) [ 7002.743436] workqueue writeback: flags=0x4a [ 7002.745392] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 7002.748332] pending: wb_workfn [ 7002.750193] workqueue xprtiod: flags=0xa [ 7002.752072] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7002.755172] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7002.762596] workqueue ipv6_addrconf: flags=0x40008 [ 7002.765720] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7002.773493] pending: addrconf_verify_work [ 7002.775057] workqueue knbd0-recv: flags=0x1a [ 7002.780688] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7002.784024] in-flight: 658:recv_work [ 7002.787490] workqueue ptlrpc_pinger: flags=0x2 [ 7002.794687] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7002.808706] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7002.820185] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5072s workers=4 idle: 32984 37 8 [ 7002.833446] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6987s workers=2 idle: 661 [ 7004.056020] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 7004.062911] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7004.111485] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7004.125576] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7004.130612] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7004.138831] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7004.147252] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7004.163680] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7004.173953] RAX: 000000000000728f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7004.187150] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 7004.191837] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 7004.201249] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 7004.209031] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 7004.219364] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 7004.224097] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7004.236082] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7004.247713] Call Trace: [ 7004.251032] [ 7004.255782] ? show_regs.cold.9+0x22/0x2f [ 7004.261109] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7004.265525] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7004.270205] ? watchdog+0x40/0x40 [ 7004.272301] ? hrtimer_interrupt+0x128/0x2c0 [ 7004.273885] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7004.289271] ? apic_timer_interrupt+0xf/0x20 [ 7004.295055] [ 7004.295795] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7004.308017] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7004.318387] do_raw_spin_lock+0x177/0x190 [ 7004.321464] _raw_spin_lock+0x21/0x30 [ 7004.325372] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7004.337148] process_one_work+0x2c8/0x700 [ 7004.343546] worker_thread+0x296/0x6e0 [ 7004.354081] ? rescuer_thread+0x570/0x570 [ 7004.362215] kthread+0x1d1/0x200 [ 7004.367253] ? set_kthread_struct+0x70/0x70 [ 7004.373680] ret_from_fork+0x1f/0x30 [ 7012.056232] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 7012.063223] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7012.131195] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7012.151482] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7012.168541] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7012.182461] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7012.212724] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7012.221677] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 7012.230506] RDX: 000000000000063b RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 7012.234604] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 7012.236827] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 7012.240778] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7012.246513] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 7012.255412] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7012.261620] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7012.270387] Call Trace: [ 7012.271918] [ 7012.272881] ? show_regs.cold.9+0x22/0x2f [ 7012.275495] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7012.280594] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7012.285432] ? watchdog+0x40/0x40 [ 7012.288243] ? hrtimer_interrupt+0x128/0x2c0 [ 7012.290808] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7012.304727] ? apic_timer_interrupt+0xf/0x20 [ 7012.312441] [ 7012.316853] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7012.318604] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7012.335002] do_raw_spin_lock+0x177/0x190 [ 7012.339838] _raw_spin_lock+0x21/0x30 [ 7012.342609] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7012.350854] obd_mod_free+0x46/0x140 [ptlrpc] [ 7012.360080] mdc_commit_open+0x91/0xb0 [mdc] [ 7012.366951] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7012.371432] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7012.374867] after_reply+0xe2c/0x1780 [ptlrpc] [ 7012.381952] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7012.389129] ? __next_timer_interrupt+0x160/0x160 [ 7012.411582] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7012.413334] ? do_wait_intr+0xf0/0xf0 [ 7012.416451] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7012.418413] kthread+0x1d1/0x200 [ 7012.420480] ? set_kthread_struct+0x70/0x70 [ 7012.421804] ret_from_fork+0x1f/0x30 [ 7024.755478] rcu: INFO: rcu_sched self-detected stall on CPU [ 7024.764717] rcu: 1-...!: (2884769 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 7024.773020] (t=5100108 jiffies g=249657 q=3833) [ 7024.786123] rcu: rcu_sched kthread starved for 5095060 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 7024.795606] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7024.811326] rcu: RCU grace-period kthread stack dump: [ 7024.817296] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7024.826355] Call Trace: [ 7024.830792] __schedule+0x351/0xcb0 [ 7024.838338] schedule+0xc0/0x180 [ 7024.840215] schedule_timeout+0xb4/0x190 [ 7024.841703] ? do_raw_spin_unlock+0x75/0x190 [ 7024.858997] ? __next_timer_interrupt+0x160/0x160 [ 7024.868757] ? prepare_to_swait_event+0xf3/0x140 [ 7024.871528] rcu_gp_kthread+0x8dd/0x15b0 [ 7024.872964] ? force_qs_rnp+0x2c0/0x2c0 [ 7024.879679] kthread+0x1d1/0x200 [ 7024.883972] ? set_kthread_struct+0x70/0x70 [ 7024.885813] ret_from_fork+0x1f/0x30 [ 7024.887137] rcu: Stack dump where RCU GP kthread last ran: [ 7024.890801] Sending NMI from CPU 1 to CPUs 3: [ 7024.896414] NMI backtrace for cpu 3 [ 7024.896423] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7024.896425] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7024.896427] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7024.896429] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7024.896432] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 [ 7024.896434] RAX: 0000000000005ea7 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7024.896435] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 7024.896437] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7024.896438] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 7024.896440] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 7024.896441] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 7024.896443] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7024.896444] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 7024.896445] Call Trace: [ 7024.896446] [ 7024.896447] ? show_regs.cold.9+0x22/0x2f [ 7024.896449] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 7024.896450] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 7024.896451] ? nmi_handle+0xbd/0x210 [ 7024.896452] ? default_do_nmi+0x4b/0x1b0 [ 7024.896454] ? do_nmi+0x280/0x380 [ 7024.896455] ? end_repeat_nmi+0x16/0x1f [ 7024.896456] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7024.896458] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7024.896459] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7024.896461] [ 7024.896461] do_raw_spin_lock+0x177/0x190 [ 7024.896463] _raw_spin_lock+0x21/0x30 [ 7024.896464] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7024.896465] ? do_raw_spin_unlock+0x75/0x190 [ 7024.896466] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7024.896468] ? do_wait_intr+0xf0/0xf0 [ 7024.896469] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7024.896470] mdc_close+0x53b/0xd70 [mdc] [ 7024.896471] lmv_close+0x18d/0x490 [lmv] [ 7024.896484] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7024.896485] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7024.896487] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7024.896488] ll_md_close+0x274/0x800 [lustre] [ 7024.896489] ll_file_release+0x128/0xc50 [lustre] [ 7024.896491] __fput+0x10d/0x440 [ 7024.896492] ____fput+0x12/0x20 [ 7024.896493] task_work_run+0xc6/0x110 [ 7024.896494] exit_to_usermode_loop+0x1dd/0x1f0 [ 7024.896495] do_syscall_64+0x3d6/0x440 [ 7024.896496] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7024.896497] RIP: 0033:0x7f151bee3cd8 [ 7024.896498] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7024.896501] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7024.896503] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 7024.896505] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 7024.896507] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7024.896509] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7024.896510] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7024.901064] NMI backtrace for cpu 1 [ 7025.216951] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7025.233473] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7025.238425] Call Trace: [ 7025.242841] [ 7025.243644] ? dump_stack+0xbb/0x10e [ 7025.244816] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7025.249216] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7025.253608] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7025.255375] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7025.263547] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7025.267518] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 7025.270437] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 7025.273473] ? update_process_times+0x6d/0xc0 [ 7025.275688] ? tick_sched_handle.isra.18+0x27/0x90 [ 7025.281547] ? tick_sched_timer+0x53/0xb0 [ 7025.283125] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7025.285956] ? tick_sched_handle.isra.18+0x90/0x90 [ 7025.289860] ? hrtimer_interrupt+0x128/0x2c0 [ 7025.298099] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7025.301833] ? apic_timer_interrupt+0xf/0x20 [ 7025.302887] [ 7025.305130] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7025.308398] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7025.313049] ? do_raw_spin_lock+0x177/0x190 [ 7025.314490] ? _raw_spin_lock+0x21/0x30 [ 7025.320479] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7025.325618] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7025.327709] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7025.336986] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7025.338600] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7025.344430] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7025.346357] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7025.352770] ? __next_timer_interrupt+0x160/0x160 [ 7025.354195] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7025.364168] ? do_wait_intr+0xf0/0xf0 [ 7025.365749] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7025.375406] ? kthread+0x1d1/0x200 [ 7025.377694] ? set_kthread_struct+0x70/0x70 [ 7025.379944] ? ret_from_fork+0x1f/0x30 [ 7028.052033] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 7028.054602] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7028.103717] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7028.109787] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7028.112635] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7028.122173] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7028.142176] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7028.147498] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7028.152326] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 7028.154853] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 7028.158854] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 7028.162979] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 7028.166137] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 7028.169632] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7028.171742] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 7028.175717] Call Trace: [ 7028.176729] [ 7028.178127] ? show_regs.cold.9+0x22/0x2f [ 7028.183129] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7028.185505] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7028.190431] ? watchdog+0x40/0x40 [ 7028.207705] ? hrtimer_interrupt+0x128/0x2c0 [ 7028.209705] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7028.217754] ? apic_timer_interrupt+0xf/0x20 [ 7028.224124] [ 7028.224817] ? native_safe_halt+0x2e/0x50 [ 7028.226185] kvm_wait+0xcd/0xe0 [ 7028.235574] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7028.240413] do_raw_spin_lock+0x177/0x190 [ 7028.242534] _raw_spin_lock+0x21/0x30 [ 7028.244280] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7028.246632] ? do_raw_spin_unlock+0x75/0x190 [ 7028.248170] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7028.251306] ? do_wait_intr+0xf0/0xf0 [ 7028.254461] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7028.258356] mdc_reint+0x2d/0x110 [mdc] [ 7028.259718] mdc_unlink+0x658/0x870 [mdc] [ 7028.266519] lmv_unlink+0x34d/0xbd0 [lmv] [ 7028.270911] ll_unlink+0x24d/0x7f0 [lustre] [ 7028.276752] vfs_unlink+0x202/0x330 [ 7028.277921] do_unlinkat+0x2fa/0x460 [ 7028.281233] __x64_sys_unlink+0x37/0x50 [ 7028.285313] do_syscall_64+0xc1/0x440 [ 7028.286537] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7028.291253] RIP: 0033:0x7f151bdee10b [ 7028.292502] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7028.300232] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7028.304807] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 7028.307292] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 7028.313043] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7028.316049] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7028.320838] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7032.059175] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 7032.069859] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7032.119679] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7032.133022] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7032.144933] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7032.155024] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7032.157310] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7032.184223] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7032.193066] RAX: 000000000000048f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7032.198359] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 7032.205013] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 7032.212438] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 7032.216778] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 7032.223510] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 7032.228414] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7032.232581] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7032.240090] Call Trace: [ 7032.243148] [ 7032.245672] ? show_regs.cold.9+0x22/0x2f [ 7032.247213] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7032.251357] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7032.257905] ? watchdog+0x40/0x40 [ 7032.259206] ? hrtimer_interrupt+0x128/0x2c0 [ 7032.260869] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7032.267113] ? apic_timer_interrupt+0xf/0x20 [ 7032.271396] [ 7032.273827] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7032.282695] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7032.287282] do_raw_spin_lock+0x177/0x190 [ 7032.290765] _raw_spin_lock+0x21/0x30 [ 7032.291785] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7032.296615] process_one_work+0x2c8/0x700 [ 7032.297812] worker_thread+0x296/0x6e0 [ 7032.301632] ? rescuer_thread+0x570/0x570 [ 7032.303169] kthread+0x1d1/0x200 [ 7032.304258] ? set_kthread_struct+0x70/0x70 [ 7032.307460] ret_from_fork+0x1f/0x30 [ 7033.313371] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5102s! [ 7033.321135] Showing busy workqueues and worker pools: [ 7033.326226] workqueue events: flags=0x0 [ 7033.333221] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7033.342131] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 7033.347331] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7033.350479] pending: kfree_rcu_monitor [ 7033.353701] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7033.357815] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 7033.377519] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7033.379786] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 7033.391986] workqueue events_long: flags=0x0 [ 7033.403935] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7033.412815] pending: lu_objects_flush [obdclass] [ 7033.422832] workqueue events_unbound: flags=0x2 [ 7033.428459] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7033.435549] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7033.448362] workqueue events_power_efficient: flags=0x80 [ 7033.450245] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7033.456425] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 7033.459514] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7033.465676] pending: neigh_periodic_work [ 7033.470212] workqueue mm_percpu_wq: flags=0x8 [ 7033.479619] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7033.483739] pending: vmstat_update, lru_add_drain_per_cpu [ 7033.488886] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7033.493690] pending: vmstat_update [ 7033.498569] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7033.503714] pending: lru_add_drain_per_cpu [ 7033.509222] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7033.511530] pending: lru_add_drain_per_cpu BAR(48) [ 7033.517735] workqueue writeback: flags=0x4a [ 7033.519183] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 7033.521658] pending: wb_workfn [ 7033.529545] workqueue xprtiod: flags=0xa [ 7033.536343] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7033.540811] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7033.554400] workqueue ipv6_addrconf: flags=0x40008 [ 7033.558598] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7033.569418] pending: addrconf_verify_work [ 7033.571723] workqueue knbd0-recv: flags=0x1a [ 7033.574259] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7033.579740] in-flight: 658:recv_work [ 7033.583457] workqueue ptlrpc_pinger: flags=0x2 [ 7033.585512] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7033.596659] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7033.603836] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5103s workers=4 idle: 32984 37 8 [ 7033.610798] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7018s workers=2 idle: 661 [ 7052.055295] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 7052.057049] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 7052.069285] Modules linked in: [ 7052.073489] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) [ 7052.075602] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7052.075673] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7052.075679] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7052.075682] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7052.075697] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7052.075701] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7052.075705] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 7052.075708] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 7052.075710] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 7052.075712] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 7052.075714] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7052.075717] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 7052.075719] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7052.075721] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7052.077135] Call Trace: [ 7052.077144] [ 7052.077150] ? show_regs.cold.9+0x22/0x2f [ 7052.077163] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7052.077170] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7052.077175] ? watchdog+0x40/0x40 [ 7052.077180] ? hrtimer_interrupt+0x128/0x2c0 [ 7052.077183] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7052.077191] ? apic_timer_interrupt+0xf/0x20 [ 7052.077195] [ 7052.077198] ? native_safe_halt+0x2e/0x50 [ 7052.077202] kvm_wait+0xcd/0xe0 [ 7052.077268] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7052.077276] do_raw_spin_lock+0x177/0x190 [ 7052.077280] _raw_spin_lock+0x21/0x30 [ 7052.077287] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7052.089464] lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7052.089937] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7052.089944] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7052.089947] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7052.089960] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7052.089964] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7052.089968] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7052.089970] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 7052.089972] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7052.089974] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 7052.089976] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 7052.089979] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 7052.089981] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7052.089984] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 7052.089992] Call Trace: [ 7052.089998] [ 7052.090003] ? show_regs.cold.9+0x22/0x2f [ 7052.090011] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7052.090017] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7052.090022] ? watchdog+0x40/0x40 [ 7052.090027] ? hrtimer_interrupt+0x128/0x2c0 [ 7052.090032] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7052.090039] ? apic_timer_interrupt+0xf/0x20 [ 7052.090044] [ 7052.090047] ? native_safe_halt+0x2e/0x50 [ 7052.090050] kvm_wait+0xcd/0xe0 [ 7052.090057] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7052.090362] do_raw_spin_lock+0x177/0x190 [ 7052.090367] _raw_spin_lock+0x21/0x30 [ 7052.090371] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7052.135540] obd_mod_free+0x46/0x140 [ptlrpc] [ 7052.135888] mdc_commit_open+0x91/0xb0 [mdc] [ 7052.135926] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7052.136124] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7052.149481] ? do_raw_spin_unlock+0x75/0x190 [ 7052.149501] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7052.149814] ? do_wait_intr+0xf0/0xf0 [ 7052.149822] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7052.150081] mdc_close+0x53b/0xd70 [mdc] [ 7052.150121] lmv_close+0x18d/0x490 [lmv] [ 7052.150148] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7052.150313] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7052.150428] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7052.150512] ll_md_close+0x274/0x800 [lustre] [ 7052.150598] ll_file_release+0x128/0xc50 [lustre] [ 7052.150696] __fput+0x10d/0x440 [ 7052.150704] ____fput+0x12/0x20 [ 7052.150709] task_work_run+0xc6/0x110 [ 7052.150716] exit_to_usermode_loop+0x1dd/0x1f0 [ 7052.150722] do_syscall_64+0x3d6/0x440 [ 7052.150727] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7052.150735] RIP: 0033:0x7f151bee3cd8 [ 7052.150743] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7052.150747] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7052.150753] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 7052.150755] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 7052.150758] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7052.150760] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7052.150762] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7052.730751] after_reply+0xe2c/0x1780 [ptlrpc] [ 7052.739390] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7052.744267] ? __next_timer_interrupt+0x160/0x160 [ 7052.751640] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7052.754866] ? do_wait_intr+0xf0/0xf0 [ 7052.761918] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7052.771116] kthread+0x1d1/0x200 [ 7052.773079] ? set_kthread_struct+0x70/0x70 [ 7052.778757] ret_from_fork+0x1f/0x30 [ 7060.055590] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 7060.058781] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 7060.061801] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7060.061876] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7060.061884] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7060.061886] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7060.061900] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7060.061904] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7060.061908] RAX: 0000000000007e5d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7060.061910] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 7060.061912] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 7060.061914] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 7060.061916] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 7060.061919] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 7060.061921] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7060.061923] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 7060.061933] Call Trace: [ 7060.061938] [ 7060.061955] ? show_regs.cold.9+0x22/0x2f [ 7060.061965] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7060.061973] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7060.061977] ? watchdog+0x40/0x40 [ 7060.061982] ? hrtimer_interrupt+0x128/0x2c0 [ 7060.061986] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7060.061995] ? apic_timer_interrupt+0xf/0x20 [ 7060.061999] [ 7060.062003] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7060.062007] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7060.062011] do_raw_spin_lock+0x177/0x190 [ 7060.062016] _raw_spin_lock+0x21/0x30 [ 7060.062022] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7060.062307] ? do_raw_spin_unlock+0x75/0x190 [ 7060.062313] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7060.062507] ? do_wait_intr+0xf0/0xf0 [ 7060.062515] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7060.063046] mdc_reint+0x2d/0x110 [mdc] [ 7060.063088] mdc_unlink+0x658/0x870 [mdc] [ 7060.063112] lmv_unlink+0x34d/0xbd0 [lmv] [ 7060.063142] ll_unlink+0x24d/0x7f0 [lustre] [ 7060.063364] vfs_unlink+0x202/0x330 [ 7060.063373] do_unlinkat+0x2fa/0x460 [ 7060.063377] __x64_sys_unlink+0x37/0x50 [ 7060.063381] do_syscall_64+0xc1/0x440 [ 7060.063387] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7060.063396] RIP: 0033:0x7f151bdee10b [ 7060.063404] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7060.063408] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7060.063413] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 7060.063416] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 7060.063418] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7060.063420] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7060.063422] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7060.441096] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7060.491099] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7060.504014] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7060.511390] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7060.516440] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7060.523034] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7060.549253] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7060.556029] RAX: 0000000000007f89 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7060.562341] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 7060.570438] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 7060.579396] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 7060.588461] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 7060.605706] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 7060.620853] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7060.627464] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7060.638262] Call Trace: [ 7060.639163] [ 7060.639850] ? show_regs.cold.9+0x22/0x2f [ 7060.650386] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7060.652385] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7060.662473] ? watchdog+0x40/0x40 [ 7060.673430] ? hrtimer_interrupt+0x128/0x2c0 [ 7060.675000] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7060.676662] ? apic_timer_interrupt+0xf/0x20 [ 7060.684351] [ 7060.693472] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7060.698346] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7060.703831] do_raw_spin_lock+0x177/0x190 [ 7060.706123] _raw_spin_lock+0x21/0x30 [ 7060.708170] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7060.712502] process_one_work+0x2c8/0x700 [ 7060.714373] worker_thread+0x296/0x6e0 [ 7060.717362] ? rescuer_thread+0x570/0x570 [ 7060.721501] kthread+0x1d1/0x200 [ 7060.727431] ? set_kthread_struct+0x70/0x70 [ 7060.734141] ret_from_fork+0x1f/0x30 [ 7064.045988] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5133s! [ 7064.049756] Showing busy workqueues and worker pools: [ 7064.060121] workqueue events: flags=0x0 [ 7064.072347] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7064.088362] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 7064.111170] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7064.137674] pending: kfree_rcu_monitor [ 7064.139066] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7064.152106] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 7064.168255] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7064.181204] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 7064.197745] workqueue events_long: flags=0x0 [ 7064.208136] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7064.215034] pending: lu_objects_flush [obdclass] [ 7064.218457] workqueue events_unbound: flags=0x2 [ 7064.227226] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7064.239837] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7064.249519] workqueue events_power_efficient: flags=0x80 [ 7064.252423] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7064.256146] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 7064.262646] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7064.272176] pending: neigh_periodic_work [ 7064.282232] workqueue mm_percpu_wq: flags=0x8 [ 7064.283705] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7064.302348] pending: vmstat_update, lru_add_drain_per_cpu [ 7064.304431] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7064.316508] pending: vmstat_update [ 7064.324149] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7064.326553] pending: lru_add_drain_per_cpu [ 7064.333851] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7064.339386] pending: lru_add_drain_per_cpu BAR(48) [ 7064.351884] workqueue writeback: flags=0x4a [ 7064.353687] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 7064.357525] pending: wb_workfn [ 7064.360673] workqueue xprtiod: flags=0xa [ 7064.375174] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7064.377346] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7064.409853] workqueue ipv6_addrconf: flags=0x40008 [ 7064.411411] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7064.421229] pending: addrconf_verify_work [ 7064.424383] workqueue knbd0-recv: flags=0x1a [ 7064.433383] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7064.450465] in-flight: 658:recv_work [ 7064.451909] workqueue ptlrpc_pinger: flags=0x2 [ 7064.457918] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7064.473464] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7064.484962] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5133s workers=4 idle: 32984 37 8 [ 7064.497852] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7049s workers=2 idle: 661 [ 7080.057014] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 7080.057089] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 7080.062047] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7080.062127] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7080.062134] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7080.062137] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7080.062151] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7080.062222] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7080.062227] RAX: 0000000000000a9b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7080.062229] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 7080.062232] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7080.062234] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 7080.062237] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 7080.062240] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 7080.062243] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7080.062246] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 7080.062253] Call Trace: [ 7080.062259] [ 7080.062264] ? show_regs.cold.9+0x22/0x2f [ 7080.062274] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7080.062280] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7080.062284] ? watchdog+0x40/0x40 [ 7080.062289] ? hrtimer_interrupt+0x128/0x2c0 [ 7080.062293] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7080.062301] ? apic_timer_interrupt+0xf/0x20 [ 7080.062306] [ 7080.062310] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7080.062524] do_raw_spin_lock+0x177/0x190 [ 7080.062532] _raw_spin_lock+0x21/0x30 [ 7080.062536] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7080.063243] ? do_raw_spin_unlock+0x75/0x190 [ 7080.063249] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7080.063719] ? do_wait_intr+0xf0/0xf0 [ 7080.063727] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7080.084927] Modules linked in: [ 7080.110112] mdc_close+0x53b/0xd70 [mdc] [ 7080.110163] lmv_close+0x18d/0x490 [lmv] [ 7080.110193] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7080.110347] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7080.111157] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7080.111246] ll_md_close+0x274/0x800 [lustre] [ 7080.111328] ll_file_release+0x128/0xc50 [lustre] [ 7080.112489] __fput+0x10d/0x440 [ 7080.112498] ____fput+0x12/0x20 [ 7080.112514] task_work_run+0xc6/0x110 [ 7080.112522] exit_to_usermode_loop+0x1dd/0x1f0 [ 7080.112529] do_syscall_64+0x3d6/0x440 [ 7080.112534] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7080.112542] RIP: 0033:0x7f151bee3cd8 [ 7080.112550] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7080.112555] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7080.112560] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 7080.112562] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 7080.112565] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7080.112567] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7080.112569] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7080.379585] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7080.413995] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7080.419809] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7080.428702] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7080.432682] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7080.442194] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7080.448726] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 7080.453052] RDX: 0000000000006409 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 7080.461517] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 7080.466963] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 7080.475563] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7080.479729] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 7080.484999] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7080.488510] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7080.500242] Call Trace: [ 7080.501678] [ 7080.502795] ? show_regs.cold.9+0x22/0x2f [ 7080.505018] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7080.507601] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7080.509844] ? watchdog+0x40/0x40 [ 7080.511705] ? hrtimer_interrupt+0x128/0x2c0 [ 7080.513754] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7080.515252] ? apic_timer_interrupt+0xf/0x20 [ 7080.521070] [ 7080.524460] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7080.529000] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7080.533895] do_raw_spin_lock+0x177/0x190 [ 7080.536027] _raw_spin_lock+0x21/0x30 [ 7080.539554] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7080.546745] obd_mod_free+0x46/0x140 [ptlrpc] [ 7080.553627] mdc_commit_open+0x91/0xb0 [mdc] [ 7080.557864] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7080.560232] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7080.570864] after_reply+0xe2c/0x1780 [ptlrpc] [ 7080.574657] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7080.579697] ? __next_timer_interrupt+0x160/0x160 [ 7080.581358] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7080.588795] ? do_wait_intr+0xf0/0xf0 [ 7080.592543] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7080.596755] kthread+0x1d1/0x200 [ 7080.598038] ? set_kthread_struct+0x70/0x70 [ 7080.600636] ret_from_fork+0x1f/0x30 [ 7088.056084] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 7088.070877] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7088.093587] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7088.098546] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7088.102709] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7088.105906] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7088.109415] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7088.142971] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7088.157003] RAX: 000000000000766a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7088.165036] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 7088.176602] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 7088.186908] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 7088.194517] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 7088.197071] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 7088.204770] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7088.207091] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7088.209548] Call Trace: [ 7088.210377] [ 7088.212203] ? show_regs.cold.9+0x22/0x2f [ 7088.214934] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7088.217599] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7088.219901] ? watchdog+0x40/0x40 [ 7088.221570] ? hrtimer_interrupt+0x128/0x2c0 [ 7088.223747] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7088.226154] ? apic_timer_interrupt+0xf/0x20 [ 7088.229782] [ 7088.231569] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7088.235569] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7088.241789] do_raw_spin_lock+0x177/0x190 [ 7088.242944] _raw_spin_lock+0x21/0x30 [ 7088.244244] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7088.251699] process_one_work+0x2c8/0x700 [ 7088.255450] worker_thread+0x296/0x6e0 [ 7088.258186] ? rescuer_thread+0x570/0x570 [ 7088.260027] kthread+0x1d1/0x200 [ 7088.264662] ? set_kthread_struct+0x70/0x70 [ 7088.268419] ret_from_fork+0x1f/0x30 [ 7092.053088] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 7092.061953] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7092.151064] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7092.173155] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7092.193810] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7092.205815] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7092.227582] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7092.234929] RAX: 000000000000762c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7092.245070] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 7092.255968] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 7092.264754] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 7092.273730] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 7092.282732] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 7092.291951] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7092.298366] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 7092.304739] Call Trace: [ 7092.308318] [ 7092.309180] ? show_regs.cold.9+0x22/0x2f [ 7092.313843] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7092.322673] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7092.324370] ? watchdog+0x40/0x40 [ 7092.325769] ? hrtimer_interrupt+0x128/0x2c0 [ 7092.329152] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7092.336916] ? apic_timer_interrupt+0xf/0x20 [ 7092.341093] [ 7092.343913] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7092.350028] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7092.354908] do_raw_spin_lock+0x177/0x190 [ 7092.356217] _raw_spin_lock+0x21/0x30 [ 7092.360572] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7092.364905] ? do_raw_spin_unlock+0x75/0x190 [ 7092.370480] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7092.375872] ? do_wait_intr+0xf0/0xf0 [ 7092.378420] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7092.384930] mdc_reint+0x2d/0x110 [mdc] [ 7092.386756] mdc_unlink+0x658/0x870 [mdc] [ 7092.391017] lmv_unlink+0x34d/0xbd0 [lmv] [ 7092.392936] ll_unlink+0x24d/0x7f0 [lustre] [ 7092.398423] vfs_unlink+0x202/0x330 [ 7092.401708] do_unlinkat+0x2fa/0x460 [ 7092.406272] __x64_sys_unlink+0x37/0x50 [ 7092.407678] do_syscall_64+0xc1/0x440 [ 7092.409075] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7092.414868] RIP: 0033:0x7f151bdee10b [ 7092.418659] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7092.444210] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7092.452846] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 7092.466463] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 7092.476493] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7092.485190] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7092.488518] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7094.764265] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5164s! [ 7094.773811] Showing busy workqueues and worker pools: [ 7094.779223] workqueue events: flags=0x0 [ 7094.781552] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7094.785306] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 7094.805306] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7094.817060] pending: kfree_rcu_monitor [ 7094.824621] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7094.833892] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 7094.862810] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7094.880017] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 7094.887789] workqueue events_long: flags=0x0 [ 7094.891718] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7094.904841] pending: lu_objects_flush [obdclass] [ 7094.912049] workqueue events_unbound: flags=0x2 [ 7094.915543] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7094.919865] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7094.936707] workqueue events_power_efficient: flags=0x80 [ 7094.943942] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7094.947986] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 7094.957750] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7094.962779] pending: neigh_periodic_work [ 7094.965449] workqueue mm_percpu_wq: flags=0x8 [ 7094.968784] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7094.984865] pending: vmstat_update, lru_add_drain_per_cpu [ 7094.997458] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7095.015803] pending: vmstat_update [ 7095.017197] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7095.023560] pending: lru_add_drain_per_cpu [ 7095.027539] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7095.038054] pending: lru_add_drain_per_cpu BAR(48) [ 7095.039922] workqueue writeback: flags=0x4a [ 7095.041210] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 7095.053021] pending: wb_workfn [ 7095.058446] workqueue xprtiod: flags=0xa [ 7095.060413] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7095.066629] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7095.071873] workqueue ipv6_addrconf: flags=0x40008 [ 7095.075157] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7095.081109] pending: addrconf_verify_work [ 7095.086940] workqueue knbd0-recv: flags=0x1a [ 7095.089979] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7095.096124] in-flight: 658:recv_work [ 7095.100343] workqueue ptlrpc_pinger: flags=0x2 [ 7095.106385] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7095.109693] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7095.119464] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5164s workers=4 idle: 32984 37 8 [ 7095.143603] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7079s workers=2 idle: 661 [ 7108.055052] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 7108.056735] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7108.057020] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 7108.111753] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7108.111769] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7108.111773] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7108.111787] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7108.111791] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7108.111798] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 7108.111801] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 7108.111803] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 7108.111805] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 7108.111808] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7108.111812] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 7108.111815] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7108.111818] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7108.111827] Call Trace: [ 7108.111833] [ 7108.111841] ? show_regs.cold.9+0x22/0x2f [ 7108.111851] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7108.111858] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7108.111863] ? watchdog+0x40/0x40 [ 7108.111867] ? hrtimer_interrupt+0x128/0x2c0 [ 7108.111871] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7108.111879] ? apic_timer_interrupt+0xf/0x20 [ 7108.111884] [ 7108.111887] ? native_safe_halt+0x2e/0x50 [ 7108.111891] kvm_wait+0xcd/0xe0 [ 7108.111900] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7108.111908] do_raw_spin_lock+0x177/0x190 [ 7108.111913] _raw_spin_lock+0x21/0x30 [ 7108.111920] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7108.114584] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7108.114641] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7108.114647] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7108.114649] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7108.114659] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7108.114662] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7108.114665] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7108.114666] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 7108.114668] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7108.114669] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 7108.114671] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 7108.114674] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 7108.114676] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7108.114677] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 7108.114684] Call Trace: [ 7108.114687] [ 7108.114691] ? show_regs.cold.9+0x22/0x2f [ 7108.114699] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7108.114704] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7108.114706] ? watchdog+0x40/0x40 [ 7108.114710] ? hrtimer_interrupt+0x128/0x2c0 [ 7108.114712] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7108.114717] ? apic_timer_interrupt+0xf/0x20 [ 7108.114720] [ 7108.114722] ? native_safe_halt+0x2e/0x50 [ 7108.114724] kvm_wait+0xcd/0xe0 [ 7108.114730] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7108.114736] do_raw_spin_lock+0x177/0x190 [ 7108.114739] _raw_spin_lock+0x21/0x30 [ 7108.114741] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7108.114989] ? do_raw_spin_unlock+0x75/0x190 [ 7108.114994] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7108.115184] ? do_wait_intr+0xf0/0xf0 [ 7108.115192] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7108.115491] mdc_close+0x53b/0xd70 [mdc] [ 7108.115522] lmv_close+0x18d/0x490 [lmv] [ 7108.115540] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7108.115666] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7108.115760] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7108.115811] ll_md_close+0x274/0x800 [lustre] [ 7108.117103] ll_file_release+0x128/0xc50 [lustre] [ 7108.117183] __fput+0x10d/0x440 [ 7108.117192] ____fput+0x12/0x20 [ 7108.117196] task_work_run+0xc6/0x110 [ 7108.117203] exit_to_usermode_loop+0x1dd/0x1f0 [ 7108.117209] do_syscall_64+0x3d6/0x440 [ 7108.117213] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7108.117221] RIP: 0033:0x7f151bee3cd8 [ 7108.117228] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7108.117244] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7108.117250] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 7108.117252] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 7108.117255] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7108.117257] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7108.117259] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7108.856961] obd_mod_free+0x46/0x140 [ptlrpc] [ 7108.861413] mdc_commit_open+0x91/0xb0 [mdc] [ 7108.867151] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7108.869103] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7108.883070] after_reply+0xe2c/0x1780 [ptlrpc] [ 7108.890318] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7108.892203] ? __next_timer_interrupt+0x160/0x160 [ 7108.896935] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7108.904335] ? do_wait_intr+0xf0/0xf0 [ 7108.909284] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7108.919049] kthread+0x1d1/0x200 [ 7108.920110] ? set_kthread_struct+0x70/0x70 [ 7108.926427] ret_from_fork+0x1f/0x30 [ 7116.056085] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 7116.060329] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7116.094230] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7116.100111] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7116.105317] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7116.109364] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7116.112101] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7116.123255] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7116.127953] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7116.133153] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 7116.138099] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 7116.142576] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 7116.146765] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 7116.151733] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 7116.156591] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7116.159616] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7116.163637] Call Trace: [ 7116.165417] [ 7116.166736] ? show_regs.cold.9+0x22/0x2f [ 7116.169292] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7116.171718] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7116.173838] ? watchdog+0x40/0x40 [ 7116.175703] ? hrtimer_interrupt+0x128/0x2c0 [ 7116.178237] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7116.180781] ? apic_timer_interrupt+0xf/0x20 [ 7116.182942] [ 7116.183726] ? native_safe_halt+0x2e/0x50 [ 7116.185976] kvm_wait+0xcd/0xe0 [ 7116.187944] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7116.190942] do_raw_spin_lock+0x177/0x190 [ 7116.193871] _raw_spin_lock+0x21/0x30 [ 7116.196886] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7116.200393] process_one_work+0x2c8/0x700 [ 7116.203057] worker_thread+0x296/0x6e0 [ 7116.205123] ? rescuer_thread+0x570/0x570 [ 7116.207581] kthread+0x1d1/0x200 [ 7116.209305] ? set_kthread_struct+0x70/0x70 [ 7116.211404] ret_from_fork+0x1f/0x30 [ 7120.052107] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 7120.057247] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7120.081895] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7120.087711] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7120.092071] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7120.095300] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7120.104258] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7120.108308] RAX: 0000000000006ad0 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7120.112356] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 7120.116693] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 7120.121325] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 7120.125151] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 7120.128499] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 7120.132988] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7120.136516] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 7120.140439] Call Trace: [ 7120.142344] [ 7120.143338] ? show_regs.cold.9+0x22/0x2f [ 7120.146437] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7120.149047] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7120.152653] ? watchdog+0x40/0x40 [ 7120.154230] ? hrtimer_interrupt+0x128/0x2c0 [ 7120.157147] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7120.158943] ? apic_timer_interrupt+0xf/0x20 [ 7120.161228] [ 7120.162190] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7120.165133] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7120.168321] do_raw_spin_lock+0x177/0x190 [ 7120.170494] _raw_spin_lock+0x21/0x30 [ 7120.172435] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7120.176274] ? do_raw_spin_unlock+0x75/0x190 [ 7120.178391] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7120.181113] ? do_wait_intr+0xf0/0xf0 [ 7120.182873] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7120.186196] mdc_reint+0x2d/0x110 [mdc] [ 7120.188275] mdc_unlink+0x658/0x870 [mdc] [ 7120.190702] lmv_unlink+0x34d/0xbd0 [lmv] [ 7120.193312] ll_unlink+0x24d/0x7f0 [lustre] [ 7120.195526] vfs_unlink+0x202/0x330 [ 7120.197303] do_unlinkat+0x2fa/0x460 [ 7120.198867] __x64_sys_unlink+0x37/0x50 [ 7120.201137] do_syscall_64+0xc1/0x440 [ 7120.202816] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7120.205859] RIP: 0033:0x7f151bdee10b [ 7120.207737] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7120.218664] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7120.222891] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 7120.226975] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 7120.231443] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7120.235491] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7120.239253] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7125.474311] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5194s! [ 7125.493586] Showing busy workqueues and worker pools: [ 7125.508903] workqueue events: flags=0x0 [ 7125.517942] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7125.527351] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 7125.542679] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7125.553628] pending: kfree_rcu_monitor [ 7125.557074] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7125.563900] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 7125.580306] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7125.585903] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 7125.590297] workqueue events_long: flags=0x0 [ 7125.592951] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7125.597914] pending: lu_objects_flush [obdclass] [ 7125.601418] workqueue events_unbound: flags=0x2 [ 7125.603889] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7125.607913] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7125.619534] workqueue events_power_efficient: flags=0x80 [ 7125.626892] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7125.637158] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 7125.643348] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7125.651898] pending: neigh_periodic_work [ 7125.657318] workqueue mm_percpu_wq: flags=0x8 [ 7125.662217] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7125.676319] pending: vmstat_update, lru_add_drain_per_cpu [ 7125.684086] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7125.689822] pending: vmstat_update [ 7125.692428] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7125.702449] pending: lru_add_drain_per_cpu [ 7125.707022] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7125.714078] pending: lru_add_drain_per_cpu BAR(48) [ 7125.716907] workqueue writeback: flags=0x4a [ 7125.718397] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 7125.724303] pending: wb_workfn [ 7125.726581] workqueue xprtiod: flags=0xa [ 7125.733259] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7125.740114] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7125.754299] workqueue ipv6_addrconf: flags=0x40008 [ 7125.760478] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7125.774256] pending: addrconf_verify_work [ 7125.779430] workqueue knbd0-recv: flags=0x1a [ 7125.786108] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7125.790812] in-flight: 658:recv_work [ 7125.797791] workqueue ptlrpc_pinger: flags=0x2 [ 7125.799013] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7125.804834] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7125.809889] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5195s workers=4 idle: 32984 37 8 [ 7125.817578] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7110s workers=2 idle: 661 [ 7136.055015] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 7136.057872] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7136.061498] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 7136.142383] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7136.142396] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7136.142400] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7136.142413] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7136.142417] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7136.142423] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 7136.142425] RDX: 000000000000709b RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 7136.142428] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 7136.142430] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 7136.142433] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7136.142436] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 7136.142439] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7136.142441] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7136.142449] Call Trace: [ 7136.142454] [ 7136.142459] ? show_regs.cold.9+0x22/0x2f [ 7136.142468] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7136.142475] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7136.142479] ? watchdog+0x40/0x40 [ 7136.142483] ? hrtimer_interrupt+0x128/0x2c0 [ 7136.142488] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7136.142495] ? apic_timer_interrupt+0xf/0x20 [ 7136.142499] [ 7136.142502] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7136.142506] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7136.142511] do_raw_spin_lock+0x177/0x190 [ 7136.142515] _raw_spin_lock+0x21/0x30 [ 7136.142521] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7136.142807] obd_mod_free+0x46/0x140 [ptlrpc] [ 7136.143672] mdc_commit_open+0x91/0xb0 [mdc] [ 7136.143715] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7136.144578] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7136.146449] after_reply+0xe2c/0x1780 [ptlrpc] [ 7136.147723] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7136.148671] ? __next_timer_interrupt+0x160/0x160 [ 7136.148680] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7136.150185] ? do_wait_intr+0xf0/0xf0 [ 7136.150197] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7136.152179] kthread+0x1d1/0x200 [ 7136.152185] ? set_kthread_struct+0x70/0x70 [ 7136.152189] ret_from_fork+0x1f/0x30 [ 7136.299701] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7136.312528] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7136.315360] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7136.317652] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7136.319187] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7136.324170] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7136.326925] RAX: 0000000000004d07 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7136.329760] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 7136.331545] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7136.333632] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 7136.335724] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 7136.338919] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 7136.342437] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7136.344877] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 7136.348408] Call Trace: [ 7136.349444] [ 7136.350331] ? show_regs.cold.9+0x22/0x2f [ 7136.352183] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7136.354255] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7136.356159] ? watchdog+0x40/0x40 [ 7136.357617] ? hrtimer_interrupt+0x128/0x2c0 [ 7136.359595] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7136.361769] ? apic_timer_interrupt+0xf/0x20 [ 7136.363741] [ 7136.364941] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7136.367424] do_raw_spin_lock+0x177/0x190 [ 7136.369263] _raw_spin_lock+0x21/0x30 [ 7136.370773] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7136.373265] ? do_raw_spin_unlock+0x75/0x190 [ 7136.375185] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7136.377564] ? do_wait_intr+0xf0/0xf0 [ 7136.379731] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7136.382378] mdc_close+0x53b/0xd70 [mdc] [ 7136.384239] lmv_close+0x18d/0x490 [lmv] [ 7136.385986] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7136.388625] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7136.391182] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7136.393339] ll_md_close+0x274/0x800 [lustre] [ 7136.395407] ll_file_release+0x128/0xc50 [lustre] [ 7136.397398] __fput+0x10d/0x440 [ 7136.399294] ____fput+0x12/0x20 [ 7136.400675] task_work_run+0xc6/0x110 [ 7136.402392] exit_to_usermode_loop+0x1dd/0x1f0 [ 7136.404300] do_syscall_64+0x3d6/0x440 [ 7136.406026] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7136.408087] RIP: 0033:0x7f151bee3cd8 [ 7136.409522] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7136.417064] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7136.420337] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 7136.423372] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 7136.426227] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7136.429000] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7136.432420] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7144.059221] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7144.071136] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7144.128114] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7144.137213] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7144.149838] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7144.155893] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7144.161442] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7144.179426] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7144.186087] RAX: 0000000000004b83 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7144.190308] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 7144.194390] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 7144.201184] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 7144.207388] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 7144.214156] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 7144.217939] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7144.220995] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7144.224246] Call Trace: [ 7144.225479] [ 7144.226303] ? show_regs.cold.9+0x22/0x2f [ 7144.229656] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7144.232507] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7144.240455] ? watchdog+0x40/0x40 [ 7144.248396] ? hrtimer_interrupt+0x128/0x2c0 [ 7144.251056] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7144.254069] ? apic_timer_interrupt+0xf/0x20 [ 7144.255641] [ 7144.262379] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7144.271475] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7144.280374] do_raw_spin_lock+0x177/0x190 [ 7144.283178] _raw_spin_lock+0x21/0x30 [ 7144.284961] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7144.292500] process_one_work+0x2c8/0x700 [ 7144.297667] worker_thread+0x296/0x6e0 [ 7144.301217] ? rescuer_thread+0x570/0x570 [ 7144.308420] kthread+0x1d1/0x200 [ 7144.313217] ? set_kthread_struct+0x70/0x70 [ 7144.316958] ret_from_fork+0x1f/0x30 [ 7152.052670] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 7152.056655] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7152.076634] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7152.080902] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7152.084326] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7152.086282] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7152.092930] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7152.095179] RAX: 000000000000380b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7152.097357] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 7152.099877] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 7152.101950] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 7152.104204] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 7152.106651] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 7152.109361] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7152.111395] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 7152.113922] Call Trace: [ 7152.114776] [ 7152.115437] ? show_regs.cold.9+0x22/0x2f [ 7152.116876] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7152.118580] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7152.120107] ? watchdog+0x40/0x40 [ 7152.121115] ? hrtimer_interrupt+0x128/0x2c0 [ 7152.122786] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7152.124332] ? apic_timer_interrupt+0xf/0x20 [ 7152.125872] [ 7152.126656] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7152.129208] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7152.131305] do_raw_spin_lock+0x177/0x190 [ 7152.132840] _raw_spin_lock+0x21/0x30 [ 7152.134345] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7152.136653] ? do_raw_spin_unlock+0x75/0x190 [ 7152.138889] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7152.140823] ? do_wait_intr+0xf0/0xf0 [ 7152.142899] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7152.144956] mdc_reint+0x2d/0x110 [mdc] [ 7152.146546] mdc_unlink+0x658/0x870 [mdc] [ 7152.148243] lmv_unlink+0x34d/0xbd0 [lmv] [ 7152.149825] ll_unlink+0x24d/0x7f0 [lustre] [ 7152.152893] vfs_unlink+0x202/0x330 [ 7152.154267] do_unlinkat+0x2fa/0x460 [ 7152.155332] __x64_sys_unlink+0x37/0x50 [ 7152.157209] do_syscall_64+0xc1/0x440 [ 7152.158192] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7152.159923] RIP: 0033:0x7f151bdee10b [ 7152.161162] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7152.168199] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7152.170586] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 7152.173017] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 7152.175368] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7152.178203] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7152.180864] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7156.208481] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5225s! [ 7156.219076] Showing busy workqueues and worker pools: [ 7156.225345] workqueue events: flags=0x0 [ 7156.226898] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7156.234546] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 7156.242946] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7156.246974] pending: kfree_rcu_monitor [ 7156.250338] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7156.253112] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 7156.257862] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7156.264884] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 7156.269180] workqueue events_long: flags=0x0 [ 7156.272961] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7156.275142] pending: lu_objects_flush [obdclass] [ 7156.277698] workqueue events_unbound: flags=0x2 [ 7156.279854] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7156.283080] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7156.296676] workqueue events_power_efficient: flags=0x80 [ 7156.300114] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7156.303065] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 7156.306852] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7156.310453] pending: neigh_periodic_work [ 7156.314193] workqueue mm_percpu_wq: flags=0x8 [ 7156.315879] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7156.319009] pending: vmstat_update, lru_add_drain_per_cpu [ 7156.321793] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7156.330200] pending: vmstat_update [ 7156.331760] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7156.334564] pending: lru_add_drain_per_cpu [ 7156.336018] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7156.339520] pending: lru_add_drain_per_cpu BAR(48) [ 7156.342046] workqueue writeback: flags=0x4a [ 7156.343531] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 7156.347178] pending: wb_workfn [ 7156.348544] workqueue xprtiod: flags=0xa [ 7156.351929] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7156.353934] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7156.357461] workqueue ipv6_addrconf: flags=0x40008 [ 7156.361948] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7156.369135] pending: addrconf_verify_work [ 7156.371393] workqueue knbd0-recv: flags=0x1a [ 7156.373722] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7156.378095] in-flight: 658:recv_work [ 7156.381738] workqueue ptlrpc_pinger: flags=0x2 [ 7156.385014] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7156.388787] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7156.390383] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5225s workers=4 idle: 32984 37 8 [ 7156.392654] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7141s workers=2 idle: 661 [ 7164.055072] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 7164.057047] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 7164.060773] Modules linked in: lustre(O) [ 7164.065945] Modules linked in: lustre(O) [ 7164.067304] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) [ 7164.071071] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7164.071151] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7164.074464] obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7164.074525] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7164.098544] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7164.104411] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7164.104424] RIP: 0010:__pv_queued_spin_lock_slowpath+0x17c/0x6d0 [ 7164.104439] Code: 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 48 83 05 5c d6 dc 02 01 <83> ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 75 e6 48 83 05 2c d6 dc [ 7164.104443] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7164.104449] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 7164.104452] RDX: 0000000000000339 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 7164.104455] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 7164.104458] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 7164.104460] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7164.104464] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 7164.104467] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7164.104470] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7164.104478] Call Trace: [ 7164.104483] [ 7164.104489] ? show_regs.cold.9+0x22/0x2f [ 7164.104498] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7164.104504] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7164.104509] ? watchdog+0x40/0x40 [ 7164.104514] ? hrtimer_interrupt+0x128/0x2c0 [ 7164.104518] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7164.104527] ? apic_timer_interrupt+0xf/0x20 [ 7164.104532] [ 7164.104535] ? __pv_queued_spin_lock_slowpath+0x17c/0x6d0 [ 7164.104539] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7164.104545] do_raw_spin_lock+0x177/0x190 [ 7164.104550] _raw_spin_lock+0x21/0x30 [ 7164.104807] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7164.105077] obd_mod_free+0x46/0x140 [ptlrpc] [ 7164.105290] mdc_commit_open+0x91/0xb0 [mdc] [ 7164.105329] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7164.105524] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7164.145799] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7164.145821] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7164.145825] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7164.145830] RAX: 0000000000005c82 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7164.145832] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 7164.145834] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7164.145837] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 7164.145839] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 7164.145842] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 7164.145846] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7164.145848] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 7164.145856] Call Trace: [ 7164.153953] after_reply+0xe2c/0x1780 [ptlrpc] [ 7164.154749] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7164.155000] ? __next_timer_interrupt+0x160/0x160 [ 7164.155011] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7164.175115] [ 7164.175129] ? show_regs.cold.9+0x22/0x2f [ 7164.175143] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7164.175150] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7164.175154] ? watchdog+0x40/0x40 [ 7164.175159] ? hrtimer_interrupt+0x128/0x2c0 [ 7164.175163] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7164.175171] ? apic_timer_interrupt+0xf/0x20 [ 7164.175175] [ 7164.175178] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7164.175188] do_raw_spin_lock+0x177/0x190 [ 7164.175194] _raw_spin_lock+0x21/0x30 [ 7164.175198] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7164.180299] ? do_wait_intr+0xf0/0xf0 [ 7164.180321] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7164.180722] kthread+0x1d1/0x200 [ 7164.180729] ? set_kthread_struct+0x70/0x70 [ 7164.180733] ret_from_fork+0x1f/0x30 [ 7164.571165] ? do_raw_spin_unlock+0x75/0x190 [ 7164.573813] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7164.580881] ? do_wait_intr+0xf0/0xf0 [ 7164.583140] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7164.588189] mdc_close+0x53b/0xd70 [mdc] [ 7164.596196] lmv_close+0x18d/0x490 [lmv] [ 7164.601201] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7164.603503] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7164.605532] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7164.613162] ll_md_close+0x274/0x800 [lustre] [ 7164.620283] ll_file_release+0x128/0xc50 [lustre] [ 7164.625445] __fput+0x10d/0x440 [ 7164.627614] ____fput+0x12/0x20 [ 7164.629201] task_work_run+0xc6/0x110 [ 7164.631439] exit_to_usermode_loop+0x1dd/0x1f0 [ 7164.634063] do_syscall_64+0x3d6/0x440 [ 7164.638223] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7164.641856] RIP: 0033:0x7f151bee3cd8 [ 7164.644157] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7164.662093] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7164.676193] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 7164.689385] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 7164.702942] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7164.716056] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7164.725094] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7172.056025] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7172.059890] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7172.083880] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7172.089113] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7172.093745] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7172.097579] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7172.099883] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7172.110331] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 7172.114015] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7172.117607] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 7172.121053] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 7172.124983] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 7172.128289] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 7172.131942] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 7172.136119] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7172.139037] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7172.142683] Call Trace: [ 7172.143743] [ 7172.144819] ? show_regs.cold.9+0x22/0x2f [ 7172.146946] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7172.149794] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7172.152248] ? watchdog+0x40/0x40 [ 7172.154054] ? hrtimer_interrupt+0x128/0x2c0 [ 7172.156595] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7172.158865] ? apic_timer_interrupt+0xf/0x20 [ 7172.160829] [ 7172.161909] ? native_safe_halt+0x2e/0x50 [ 7172.163813] kvm_wait+0xcd/0xe0 [ 7172.165276] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7172.167250] do_raw_spin_lock+0x177/0x190 [ 7172.169280] _raw_spin_lock+0x21/0x30 [ 7172.170985] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7172.174271] process_one_work+0x2c8/0x700 [ 7172.176623] worker_thread+0x296/0x6e0 [ 7172.178960] ? rescuer_thread+0x570/0x570 [ 7172.180723] kthread+0x1d1/0x200 [ 7172.182146] ? set_kthread_struct+0x70/0x70 [ 7172.184259] ret_from_fork+0x1f/0x30 [ 7184.053209] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 7184.062179] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7184.117127] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7184.129148] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7184.148076] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7184.150187] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7184.170135] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7184.176174] RAX: 0000000000007f89 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7184.181143] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 7184.185606] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 7184.192279] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 7184.196569] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 7184.201883] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 7184.206653] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7184.213385] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 7184.219040] Call Trace: [ 7184.220053] [ 7184.224967] ? show_regs.cold.9+0x22/0x2f [ 7184.230226] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7184.233028] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7184.235467] ? watchdog+0x40/0x40 [ 7184.237274] ? hrtimer_interrupt+0x128/0x2c0 [ 7184.238778] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7184.250239] ? apic_timer_interrupt+0xf/0x20 [ 7184.262031] [ 7184.262838] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7184.264663] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7184.273634] do_raw_spin_lock+0x177/0x190 [ 7184.282795] _raw_spin_lock+0x21/0x30 [ 7184.287193] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7184.298665] ? do_raw_spin_unlock+0x75/0x190 [ 7184.308520] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7184.310517] ? do_wait_intr+0xf0/0xf0 [ 7184.325409] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7184.326685] mdc_reint+0x2d/0x110 [mdc] [ 7184.327511] mdc_unlink+0x658/0x870 [mdc] [ 7184.337244] lmv_unlink+0x34d/0xbd0 [lmv] [ 7184.342630] ll_unlink+0x24d/0x7f0 [lustre] [ 7184.344175] vfs_unlink+0x202/0x330 [ 7184.349888] do_unlinkat+0x2fa/0x460 [ 7184.355231] __x64_sys_unlink+0x37/0x50 [ 7184.360287] do_syscall_64+0xc1/0x440 [ 7184.361614] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7184.368740] RIP: 0033:0x7f151bdee10b [ 7184.371252] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7184.387517] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7184.392626] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 7184.398524] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 7184.403547] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7184.409717] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7184.414215] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7186.406088] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5255s! [ 7186.412442] Showing busy workqueues and worker pools: [ 7186.417657] workqueue events: flags=0x0 [ 7186.420142] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7186.428811] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 7186.446116] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7186.453768] pending: kfree_rcu_monitor [ 7186.461599] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7186.472190] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 7186.492446] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7186.496909] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 7186.499856] workqueue events_long: flags=0x0 [ 7186.502254] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7186.508899] pending: lu_objects_flush [obdclass] [ 7186.511398] workqueue events_unbound: flags=0x2 [ 7186.517841] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7186.522118] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7186.530914] workqueue events_power_efficient: flags=0x80 [ 7186.534484] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7186.541095] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 7186.554870] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7186.565626] pending: neigh_periodic_work [ 7186.569012] workqueue mm_percpu_wq: flags=0x8 [ 7186.571251] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7186.575232] pending: vmstat_update, lru_add_drain_per_cpu [ 7186.578891] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7186.586941] pending: vmstat_update [ 7186.590277] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7186.594241] pending: lru_add_drain_per_cpu [ 7186.596354] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7186.604283] pending: lru_add_drain_per_cpu BAR(48) [ 7186.608313] workqueue writeback: flags=0x4a [ 7186.610974] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 7186.615205] pending: wb_workfn [ 7186.618123] workqueue xprtiod: flags=0xa [ 7186.620853] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7186.627594] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7186.633129] workqueue ipv6_addrconf: flags=0x40008 [ 7186.635823] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7186.640160] pending: addrconf_verify_work [ 7186.644873] workqueue knbd0-recv: flags=0x1a [ 7186.647124] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7186.652620] in-flight: 658:recv_work [ 7186.655143] workqueue ptlrpc_pinger: flags=0x2 [ 7186.657933] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7186.661318] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7186.664610] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5256s workers=4 idle: 32984 37 8 [ 7186.672032] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7171s workers=2 idle: 661 [ 7192.057637] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 7192.057682] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 7192.057691] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7192.057770] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7192.057776] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7192.057779] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7192.057793] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7192.057798] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7192.057802] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 7192.057804] RDX: 000000000000208a RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 7192.057807] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 7192.057809] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 7192.057812] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7192.057815] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 7192.057818] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7192.057821] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7192.057829] Call Trace: [ 7192.057835] [ 7192.057840] ? show_regs.cold.9+0x22/0x2f [ 7192.057849] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7192.057856] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7192.057860] ? watchdog+0x40/0x40 [ 7192.057866] ? hrtimer_interrupt+0x128/0x2c0 [ 7192.057870] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7192.057879] ? apic_timer_interrupt+0xf/0x20 [ 7192.057883] [ 7192.057887] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7192.057892] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7192.057897] do_raw_spin_lock+0x177/0x190 [ 7192.057902] _raw_spin_lock+0x21/0x30 [ 7192.057909] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7192.071556] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7192.071653] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7192.071660] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7192.071663] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7192.071676] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7192.071679] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 7192.071683] RAX: 0000000000004d88 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7192.071685] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 7192.071688] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7192.071690] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 7192.071692] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 7192.071696] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 7192.071699] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7192.071701] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 7192.071710] Call Trace: [ 7192.071715] [ 7192.071722] ? show_regs.cold.9+0x22/0x2f [ 7192.071732] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7192.071738] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7192.071742] ? watchdog+0x40/0x40 [ 7192.071747] ? hrtimer_interrupt+0x128/0x2c0 [ 7192.071750] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7192.071758] ? apic_timer_interrupt+0xf/0x20 [ 7192.071762] [ 7192.071765] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7192.071771] do_raw_spin_lock+0x177/0x190 [ 7192.071776] _raw_spin_lock+0x21/0x30 [ 7192.071781] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7192.072053] ? do_raw_spin_unlock+0x75/0x190 [ 7192.072059] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7192.072372] ? do_wait_intr+0xf0/0xf0 [ 7192.072380] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7192.072766] mdc_close+0x53b/0xd70 [mdc] [ 7192.072810] lmv_close+0x18d/0x490 [lmv] [ 7192.072845] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7192.073135] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7192.073277] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7192.073369] ll_md_close+0x274/0x800 [lustre] [ 7192.073473] ll_file_release+0x128/0xc50 [lustre] [ 7192.073563] __fput+0x10d/0x440 [ 7192.073577] ____fput+0x12/0x20 [ 7192.073582] task_work_run+0xc6/0x110 [ 7192.073992] exit_to_usermode_loop+0x1dd/0x1f0 [ 7192.074004] do_syscall_64+0x3d6/0x440 [ 7192.074010] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7192.074018] RIP: 0033:0x7f151bee3cd8 [ 7192.074026] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7192.074030] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7192.074035] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 7192.074038] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 7192.074040] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7192.074042] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7192.074044] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7192.469772] obd_mod_free+0x46/0x140 [ptlrpc] [ 7192.471449] mdc_commit_open+0x91/0xb0 [mdc] [ 7192.472922] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7192.474753] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7192.477182] after_reply+0xe2c/0x1780 [ptlrpc] [ 7192.479650] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7192.481667] ? __next_timer_interrupt+0x160/0x160 [ 7192.483219] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7192.484930] ? do_wait_intr+0xf0/0xf0 [ 7192.486148] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7192.488073] kthread+0x1d1/0x200 [ 7192.489183] ? set_kthread_struct+0x70/0x70 [ 7192.490535] ret_from_fork+0x1f/0x30 [ 7200.056358] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7200.064001] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7200.082256] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7200.085380] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7200.088102] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7200.090386] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7200.092286] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7200.099879] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7200.107960] RAX: 0000000000007e59 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7200.114651] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 7200.125041] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 7200.130845] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 7200.138988] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 7200.143367] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 7200.149419] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7200.155746] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7200.163570] Call Trace: [ 7200.165793] [ 7200.167862] ? show_regs.cold.9+0x22/0x2f [ 7200.173849] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7200.176893] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7200.181072] ? watchdog+0x40/0x40 [ 7200.184456] ? hrtimer_interrupt+0x128/0x2c0 [ 7200.185971] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7200.194181] ? apic_timer_interrupt+0xf/0x20 [ 7200.198575] [ 7200.200306] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7200.202064] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7200.203796] do_raw_spin_lock+0x177/0x190 [ 7200.209657] _raw_spin_lock+0x21/0x30 [ 7200.210828] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7200.214356] process_one_work+0x2c8/0x700 [ 7200.215568] worker_thread+0x296/0x6e0 [ 7200.218895] ? rescuer_thread+0x570/0x570 [ 7200.221353] kthread+0x1d1/0x200 [ 7200.224315] ? set_kthread_struct+0x70/0x70 [ 7200.226928] ret_from_fork+0x1f/0x30 [ 7204.758991] rcu: INFO: rcu_sched self-detected stall on CPU [ 7204.765559] rcu: 1-...!: (2977968 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 7204.780369] (t=5280111 jiffies g=249657 q=3936) [ 7204.786703] rcu: rcu_sched kthread starved for 5275063 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 7204.800628] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7204.816786] rcu: RCU grace-period kthread stack dump: [ 7204.825370] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7204.839005] Call Trace: [ 7204.840734] __schedule+0x351/0xcb0 [ 7204.841877] schedule+0xc0/0x180 [ 7204.850529] schedule_timeout+0xb4/0x190 [ 7204.854438] ? do_raw_spin_unlock+0x75/0x190 [ 7204.863501] ? __next_timer_interrupt+0x160/0x160 [ 7204.871117] ? prepare_to_swait_event+0xf3/0x140 [ 7204.873895] rcu_gp_kthread+0x8dd/0x15b0 [ 7204.877766] ? force_qs_rnp+0x2c0/0x2c0 [ 7204.884068] kthread+0x1d1/0x200 [ 7204.890437] ? set_kthread_struct+0x70/0x70 [ 7204.895517] ret_from_fork+0x1f/0x30 [ 7204.901996] rcu: Stack dump where RCU GP kthread last ran: [ 7204.913599] Sending NMI from CPU 1 to CPUs 3: [ 7204.917318] NMI backtrace for cpu 3 skipped: idling at native_safe_halt+0x2e/0x50 [ 7204.921538] NMI backtrace for cpu 1 [ 7204.932687] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7204.942542] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7204.947699] Call Trace: [ 7204.949146] [ 7204.950471] ? dump_stack+0xbb/0x10e [ 7204.951923] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7204.955269] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7204.956474] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7204.960057] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7204.967450] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7204.973576] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 7204.979515] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 7204.987782] ? update_process_times+0x6d/0xc0 [ 7204.991601] ? tick_sched_handle.isra.18+0x27/0x90 [ 7204.995547] ? tick_sched_timer+0x53/0xb0 [ 7205.000434] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7205.006606] ? tick_sched_handle.isra.18+0x90/0x90 [ 7205.014029] ? hrtimer_interrupt+0x128/0x2c0 [ 7205.020318] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7205.023331] ? apic_timer_interrupt+0xf/0x20 [ 7205.026024] [ 7205.027637] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7205.032484] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7205.035614] ? do_raw_spin_lock+0x177/0x190 [ 7205.039747] ? _raw_spin_lock+0x21/0x30 [ 7205.042449] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7205.046567] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7205.058130] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7205.061930] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7205.068717] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7205.075411] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7205.081480] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7205.087708] ? __next_timer_interrupt+0x160/0x160 [ 7205.092749] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7205.099065] ? do_wait_intr+0xf0/0xf0 [ 7205.102972] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7205.107584] ? kthread+0x1d1/0x200 [ 7205.109017] ? set_kthread_struct+0x70/0x70 [ 7205.115434] ? ret_from_fork+0x1f/0x30 [ 7212.054402] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 7212.062620] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7212.120468] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7212.129375] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7212.138883] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7212.146695] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7212.169533] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7212.176199] RAX: 0000000000004f88 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7212.179999] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 7212.186520] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 7212.193399] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 7212.203491] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 7212.208443] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 7212.214501] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7212.221402] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 7212.228377] Call Trace: [ 7212.229630] [ 7212.230676] ? show_regs.cold.9+0x22/0x2f [ 7212.239307] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7212.243583] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7212.247811] ? watchdog+0x40/0x40 [ 7212.254245] ? hrtimer_interrupt+0x128/0x2c0 [ 7212.256868] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7212.258716] ? apic_timer_interrupt+0xf/0x20 [ 7212.264142] [ 7212.264946] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7212.267738] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7212.273574] do_raw_spin_lock+0x177/0x190 [ 7212.276283] _raw_spin_lock+0x21/0x30 [ 7212.280520] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7212.286848] ? do_raw_spin_unlock+0x75/0x190 [ 7212.291468] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7212.297651] ? do_wait_intr+0xf0/0xf0 [ 7212.301361] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7212.304698] mdc_reint+0x2d/0x110 [mdc] [ 7212.307466] mdc_unlink+0x658/0x870 [mdc] [ 7212.310789] lmv_unlink+0x34d/0xbd0 [lmv] [ 7212.312913] ll_unlink+0x24d/0x7f0 [lustre] [ 7212.316068] vfs_unlink+0x202/0x330 [ 7212.317721] do_unlinkat+0x2fa/0x460 [ 7212.319910] __x64_sys_unlink+0x37/0x50 [ 7212.321199] do_syscall_64+0xc1/0x440 [ 7212.325180] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7212.328771] RIP: 0033:0x7f151bdee10b [ 7212.338671] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7212.352603] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7212.357472] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 7212.363190] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 7212.372320] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7212.377696] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7212.381311] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7217.121045] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5286s! [ 7217.123934] Showing busy workqueues and worker pools: [ 7217.126146] workqueue events: flags=0x0 [ 7217.127476] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7217.133362] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 7217.143669] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7217.147262] pending: kfree_rcu_monitor [ 7217.151882] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7217.160945] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 7217.172527] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7217.178969] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 7217.182012] workqueue events_long: flags=0x0 [ 7217.186364] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7217.189789] pending: lu_objects_flush [obdclass] [ 7217.195886] workqueue events_unbound: flags=0x2 [ 7217.200952] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7217.211860] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7217.222464] workqueue events_power_efficient: flags=0x80 [ 7217.227546] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7217.232544] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 7217.240270] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7217.243911] pending: neigh_periodic_work [ 7217.246470] workqueue mm_percpu_wq: flags=0x8 [ 7217.249460] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7217.257041] pending: vmstat_update, lru_add_drain_per_cpu [ 7217.262525] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7217.267119] pending: vmstat_update [ 7217.270472] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7217.273884] pending: lru_add_drain_per_cpu [ 7217.276523] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7217.280462] pending: lru_add_drain_per_cpu BAR(48) [ 7217.285929] workqueue writeback: flags=0x4a [ 7217.289555] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 7217.297819] pending: wb_workfn [ 7217.302865] workqueue xprtiod: flags=0xa [ 7217.309964] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7217.325327] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7217.328133] workqueue ipv6_addrconf: flags=0x40008 [ 7217.339396] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7217.349213] pending: addrconf_verify_work [ 7217.354895] workqueue knbd0-recv: flags=0x1a [ 7217.356880] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7217.366824] in-flight: 658:recv_work [ 7217.374572] workqueue ptlrpc_pinger: flags=0x2 [ 7217.379348] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7217.386620] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7217.395510] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5286s workers=4 idle: 32984 37 8 [ 7217.412675] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7202s workers=2 idle: 661 [ 7228.056600] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7228.064192] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7228.105091] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7228.115337] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7228.119364] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7228.125951] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7228.132460] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7228.147618] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7228.155155] RAX: 0000000000007e99 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7228.162547] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 7228.168015] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 7228.171082] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 7228.177929] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 7228.184825] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 7228.191621] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7228.196226] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7228.203371] Call Trace: [ 7228.210121] [ 7228.213218] ? show_regs.cold.9+0x22/0x2f [ 7228.218250] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7228.224732] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7228.231194] ? watchdog+0x40/0x40 [ 7228.234671] ? hrtimer_interrupt+0x128/0x2c0 [ 7228.239232] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7228.245384] ? apic_timer_interrupt+0xf/0x20 [ 7228.250459] [ 7228.251249] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7228.257837] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7228.262245] do_raw_spin_lock+0x177/0x190 [ 7228.264060] _raw_spin_lock+0x21/0x30 [ 7228.266463] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7228.269323] process_one_work+0x2c8/0x700 [ 7228.277169] worker_thread+0x296/0x6e0 [ 7228.282677] ? rescuer_thread+0x570/0x570 [ 7228.286342] kthread+0x1d1/0x200 [ 7228.288468] ? set_kthread_struct+0x70/0x70 [ 7228.293430] ret_from_fork+0x1f/0x30 [ 7232.055057] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 7232.058701] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7232.061023] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 7232.105565] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7232.112342] Modules linked in: [ 7232.118055] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7232.118067] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7232.122300] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7232.122376] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7232.122385] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7232.122388] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7232.122402] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7232.122406] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7232.122411] RAX: 0000000000002507 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7232.122413] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 7232.122416] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7232.122418] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 7232.122420] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 7232.122424] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 7232.122427] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7232.122429] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 7232.122439] Call Trace: [ 7232.122443] [ 7232.122449] ? show_regs.cold.9+0x22/0x2f [ 7232.122458] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7232.122465] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7232.122469] ? watchdog+0x40/0x40 [ 7232.122474] ? hrtimer_interrupt+0x128/0x2c0 [ 7232.122478] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7232.122486] ? apic_timer_interrupt+0xf/0x20 [ 7232.122491] [ 7232.122493] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7232.122500] do_raw_spin_lock+0x177/0x190 [ 7232.122506] _raw_spin_lock+0x21/0x30 [ 7232.122510] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7232.122780] ? do_raw_spin_unlock+0x75/0x190 [ 7232.122786] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7232.134023] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7232.134046] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7232.134054] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 7232.134056] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 7232.134063] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 7232.134066] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 7232.134068] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7232.134071] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 7232.134078] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7232.134080] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7232.134088] Call Trace: [ 7232.134093] [ 7232.134098] ? show_regs.cold.9+0x22/0x2f [ 7232.134111] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7232.134123] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7232.134128] ? watchdog+0x40/0x40 [ 7232.134132] ? hrtimer_interrupt+0x128/0x2c0 [ 7232.134135] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7232.134144] ? apic_timer_interrupt+0xf/0x20 [ 7232.134148] [ 7232.134151] ? native_safe_halt+0x2e/0x50 [ 7232.134156] kvm_wait+0xcd/0xe0 [ 7232.134163] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7232.134170] do_raw_spin_lock+0x177/0x190 [ 7232.134174] _raw_spin_lock+0x21/0x30 [ 7232.134181] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7232.134655] obd_mod_free+0x46/0x140 [ptlrpc] [ 7232.134849] mdc_commit_open+0x91/0xb0 [mdc] [ 7232.135221] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7232.138790] ? do_wait_intr+0xf0/0xf0 [ 7232.138814] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7232.139191] mdc_close+0x53b/0xd70 [mdc] [ 7232.139230] lmv_close+0x18d/0x490 [lmv] [ 7232.139262] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7232.139422] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7232.139532] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7232.139613] ll_md_close+0x274/0x800 [lustre] [ 7232.139699] ll_file_release+0x128/0xc50 [lustre] [ 7232.139782] __fput+0x10d/0x440 [ 7232.139793] ____fput+0x12/0x20 [ 7232.139797] task_work_run+0xc6/0x110 [ 7232.139806] exit_to_usermode_loop+0x1dd/0x1f0 [ 7232.139815] do_syscall_64+0x3d6/0x440 [ 7232.139820] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7232.139830] RIP: 0033:0x7f151bee3cd8 [ 7232.139839] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7232.139843] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7232.139847] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 7232.139849] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 7232.139852] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7232.139854] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7232.139856] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7232.554135] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7232.557461] after_reply+0xe2c/0x1780 [ptlrpc] [ 7232.560449] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7232.564925] ? __next_timer_interrupt+0x160/0x160 [ 7232.566608] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7232.571997] ? do_wait_intr+0xf0/0xf0 [ 7232.576616] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7232.580210] kthread+0x1d1/0x200 [ 7232.581653] ? set_kthread_struct+0x70/0x70 [ 7232.585301] ret_from_fork+0x1f/0x30 [ 7244.062089] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 7244.065081] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7244.129646] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7244.147695] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7244.151830] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7244.158430] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7244.172083] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7244.179008] RAX: 0000000000007e27 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7244.183433] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 7244.195154] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 7244.215070] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 7244.230129] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 7244.245909] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 7244.255233] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7244.267326] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 7244.277261] Call Trace: [ 7244.278234] [ 7244.279049] ? show_regs.cold.9+0x22/0x2f [ 7244.290126] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7244.295410] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7244.298711] ? watchdog+0x40/0x40 [ 7244.307732] ? hrtimer_interrupt+0x128/0x2c0 [ 7244.311086] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7244.312829] ? apic_timer_interrupt+0xf/0x20 [ 7244.330325] [ 7244.331141] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7244.357825] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7244.359715] do_raw_spin_lock+0x177/0x190 [ 7244.384837] _raw_spin_lock+0x21/0x30 [ 7244.386309] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7244.400298] ? do_raw_spin_unlock+0x75/0x190 [ 7244.401829] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7244.436597] ? do_wait_intr+0xf0/0xf0 [ 7244.440699] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7244.447296] mdc_reint+0x2d/0x110 [mdc] [ 7244.448415] mdc_unlink+0x658/0x870 [mdc] [ 7244.459343] lmv_unlink+0x34d/0xbd0 [lmv] [ 7244.467776] ll_unlink+0x24d/0x7f0 [lustre] [ 7244.477402] vfs_unlink+0x202/0x330 [ 7244.480649] do_unlinkat+0x2fa/0x460 [ 7244.490418] __x64_sys_unlink+0x37/0x50 [ 7244.491884] do_syscall_64+0xc1/0x440 [ 7244.502517] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7244.505603] RIP: 0033:0x7f151bdee10b [ 7244.506859] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7244.531560] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7244.539829] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 7244.547528] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 7244.557192] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7244.568551] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7244.577902] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7247.846085] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5317s! [ 7247.856209] Showing busy workqueues and worker pools: [ 7247.866034] workqueue events: flags=0x0 [ 7247.887070] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7247.896654] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 7247.902263] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7247.911124] pending: kfree_rcu_monitor [ 7247.914418] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7247.919138] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 7247.927706] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7247.930661] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 7247.938159] workqueue events_long: flags=0x0 [ 7247.940059] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7247.944350] pending: lu_objects_flush [obdclass] [ 7247.947808] workqueue events_unbound: flags=0x2 [ 7247.950522] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7247.954288] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7247.960366] workqueue events_power_efficient: flags=0x80 [ 7247.964286] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7247.968487] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 7247.972505] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7247.976560] pending: neigh_periodic_work [ 7247.979248] workqueue mm_percpu_wq: flags=0x8 [ 7247.982022] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7247.993664] pending: vmstat_update, lru_add_drain_per_cpu [ 7248.001095] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7248.006394] pending: vmstat_update [ 7248.010635] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7248.025255] pending: lru_add_drain_per_cpu [ 7248.034040] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7248.041907] pending: lru_add_drain_per_cpu BAR(48) [ 7248.046069] workqueue writeback: flags=0x4a [ 7248.050500] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 7248.057182] pending: wb_workfn [ 7248.064656] workqueue xprtiod: flags=0xa [ 7248.069617] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7248.078487] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7248.106439] workqueue ipv6_addrconf: flags=0x40008 [ 7248.111687] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7248.121080] pending: addrconf_verify_work [ 7248.130593] workqueue knbd0-recv: flags=0x1a [ 7248.131965] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7248.139342] in-flight: 658:recv_work [ 7248.150933] workqueue ptlrpc_pinger: flags=0x2 [ 7248.165128] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7248.169957] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7248.177892] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5317s workers=4 idle: 32984 37 8 [ 7248.194728] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7232s workers=2 idle: 661 [ 7256.059460] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7256.069339] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7256.116727] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7256.123231] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7256.127631] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7256.131032] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7256.133813] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7256.148127] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7256.152789] RAX: 0000000000007799 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7256.156855] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 7256.160018] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 7256.163945] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 7256.172386] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 7256.180569] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 7256.193201] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7256.199495] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7256.208196] Call Trace: [ 7256.209558] [ 7256.211978] ? show_regs.cold.9+0x22/0x2f [ 7256.222529] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7256.226105] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7256.228637] ? watchdog+0x40/0x40 [ 7256.230632] ? hrtimer_interrupt+0x128/0x2c0 [ 7256.234110] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7256.238198] ? apic_timer_interrupt+0xf/0x20 [ 7256.240935] [ 7256.241956] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7256.246409] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7256.248811] do_raw_spin_lock+0x177/0x190 [ 7256.250482] _raw_spin_lock+0x21/0x30 [ 7256.252038] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7256.261152] process_one_work+0x2c8/0x700 [ 7256.263200] worker_thread+0x296/0x6e0 [ 7256.268231] ? rescuer_thread+0x570/0x570 [ 7256.269820] kthread+0x1d1/0x200 [ 7256.279891] ? set_kthread_struct+0x70/0x70 [ 7256.287085] ret_from_fork+0x1f/0x30 [ 7260.055108] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 7260.058199] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 7260.058986] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7260.059227] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7260.059234] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7260.059237] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7260.059251] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7260.059255] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7260.059259] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 7260.059261] RDX: 000000000000454c RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 7260.059263] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 7260.059266] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 7260.059268] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7260.059271] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 7260.059274] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7260.059277] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7260.059285] Call Trace: [ 7260.059291] [ 7260.059297] ? show_regs.cold.9+0x22/0x2f [ 7260.059364] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7260.059371] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7260.059375] ? watchdog+0x40/0x40 [ 7260.059379] ? hrtimer_interrupt+0x128/0x2c0 [ 7260.059383] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7260.059391] ? apic_timer_interrupt+0xf/0x20 [ 7260.059396] [ 7260.059399] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7260.059403] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7260.059407] do_raw_spin_lock+0x177/0x190 [ 7260.059412] _raw_spin_lock+0x21/0x30 [ 7260.059417] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7260.059698] obd_mod_free+0x46/0x140 [ptlrpc] [ 7260.060154] mdc_commit_open+0x91/0xb0 [mdc] [ 7260.060194] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7260.060469] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7260.060676] after_reply+0xe2c/0x1780 [ptlrpc] [ 7260.061158] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7260.061523] ? __next_timer_interrupt+0x160/0x160 [ 7260.061534] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7260.061865] ? do_wait_intr+0xf0/0xf0 [ 7260.061875] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7260.062168] kthread+0x1d1/0x200 [ 7260.062175] ? set_kthread_struct+0x70/0x70 [ 7260.062179] ret_from_fork+0x1f/0x30 [ 7260.235607] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7260.267340] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7260.271411] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7260.274557] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7260.277412] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7260.286288] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7260.289986] RAX: 0000000000000e91 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7260.293565] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 7260.297370] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7260.300998] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 7260.304397] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 7260.307712] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 7260.311316] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7260.314005] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 7260.317515] Call Trace: [ 7260.318907] [ 7260.319659] ? show_regs.cold.9+0x22/0x2f [ 7260.322615] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7260.324739] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7260.327078] ? watchdog+0x40/0x40 [ 7260.328507] ? hrtimer_interrupt+0x128/0x2c0 [ 7260.330586] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7260.333234] ? apic_timer_interrupt+0xf/0x20 [ 7260.335269] [ 7260.336426] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7260.339458] do_raw_spin_lock+0x177/0x190 [ 7260.342183] _raw_spin_lock+0x21/0x30 [ 7260.343941] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7260.346730] ? do_raw_spin_unlock+0x75/0x190 [ 7260.349016] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7260.351585] ? do_wait_intr+0xf0/0xf0 [ 7260.353387] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7260.355958] mdc_close+0x53b/0xd70 [mdc] [ 7260.358149] lmv_close+0x18d/0x490 [lmv] [ 7260.359569] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7260.363056] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7260.366288] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7260.368249] ll_md_close+0x274/0x800 [lustre] [ 7260.370473] ll_file_release+0x128/0xc50 [lustre] [ 7260.373039] __fput+0x10d/0x440 [ 7260.374731] ____fput+0x12/0x20 [ 7260.375975] task_work_run+0xc6/0x110 [ 7260.377910] exit_to_usermode_loop+0x1dd/0x1f0 [ 7260.379908] do_syscall_64+0x3d6/0x440 [ 7260.381976] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7260.384277] RIP: 0033:0x7f151bee3cd8 [ 7260.386106] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7260.395469] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7260.398613] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 7260.401984] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 7260.405424] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7260.408970] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7260.412611] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7276.052251] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 7276.063062] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7276.087200] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7276.091687] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7276.097374] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7276.099953] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7276.108672] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7276.112572] RAX: 0000000000007e16 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7276.117259] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 7276.120641] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 7276.125987] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 7276.130340] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 7276.134387] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 7276.139776] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7276.142930] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 7276.147948] Call Trace: [ 7276.150649] [ 7276.151831] ? show_regs.cold.9+0x22/0x2f [ 7276.154323] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7276.158672] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7276.162898] ? watchdog+0x40/0x40 [ 7276.164461] ? hrtimer_interrupt+0x128/0x2c0 [ 7276.166600] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7276.168367] ? apic_timer_interrupt+0xf/0x20 [ 7276.170624] [ 7276.172328] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7276.177006] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7276.180593] do_raw_spin_lock+0x177/0x190 [ 7276.182621] _raw_spin_lock+0x21/0x30 [ 7276.184291] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7276.188021] ? do_raw_spin_unlock+0x75/0x190 [ 7276.190805] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7276.193510] ? do_wait_intr+0xf0/0xf0 [ 7276.195978] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7276.199893] mdc_reint+0x2d/0x110 [mdc] [ 7276.202431] mdc_unlink+0x658/0x870 [mdc] [ 7276.205766] lmv_unlink+0x34d/0xbd0 [lmv] [ 7276.208162] ll_unlink+0x24d/0x7f0 [lustre] [ 7276.211416] vfs_unlink+0x202/0x330 [ 7276.213588] do_unlinkat+0x2fa/0x460 [ 7276.214984] __x64_sys_unlink+0x37/0x50 [ 7276.216990] do_syscall_64+0xc1/0x440 [ 7276.219982] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7276.223009] RIP: 0033:0x7f151bdee10b [ 7276.224207] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7276.234768] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7276.238708] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 7276.242742] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 7276.247275] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7276.250704] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7276.254176] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7278.560106] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5347s! [ 7278.563207] Showing busy workqueues and worker pools: [ 7278.568299] workqueue events: flags=0x0 [ 7278.570336] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7278.574915] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 7278.581660] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7278.587604] pending: kfree_rcu_monitor [ 7278.590876] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7278.594510] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 7278.603526] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7278.610165] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 7278.618353] workqueue events_long: flags=0x0 [ 7278.622792] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7278.633967] pending: lu_objects_flush [obdclass] [ 7278.641342] workqueue events_unbound: flags=0x2 [ 7278.643672] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7278.650826] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7278.663025] workqueue events_power_efficient: flags=0x80 [ 7278.670868] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7278.681273] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 7278.688161] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7278.692587] pending: neigh_periodic_work [ 7278.695027] workqueue mm_percpu_wq: flags=0x8 [ 7278.697880] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7278.704222] pending: vmstat_update, lru_add_drain_per_cpu [ 7278.708851] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7278.711966] pending: vmstat_update [ 7278.713777] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7278.716759] pending: lru_add_drain_per_cpu [ 7278.718928] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7278.722268] pending: lru_add_drain_per_cpu BAR(48) [ 7278.729167] workqueue writeback: flags=0x4a [ 7278.737060] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 7278.739434] pending: wb_workfn [ 7278.742454] workqueue xprtiod: flags=0xa [ 7278.743880] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7278.750205] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7278.760015] workqueue ipv6_addrconf: flags=0x40008 [ 7278.762244] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7278.768045] pending: addrconf_verify_work [ 7278.773526] workqueue knbd0-recv: flags=0x1a [ 7278.776961] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7278.783511] in-flight: 658:recv_work [ 7278.787307] workqueue ptlrpc_pinger: flags=0x2 [ 7278.792257] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7278.800322] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7278.811292] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5348s workers=4 idle: 32984 37 8 [ 7278.827370] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7263s workers=2 idle: 661 [ 7284.056514] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7284.063096] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7284.110300] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7284.127740] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7284.138765] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7284.144965] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7284.148335] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7284.166514] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7284.174156] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7284.180594] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 7284.187334] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 7284.192017] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 7284.198950] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 7284.212339] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 7284.223329] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7284.231051] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7284.238567] Call Trace: [ 7284.241111] [ 7284.243797] ? show_regs.cold.9+0x22/0x2f [ 7284.247884] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7284.252362] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7284.258147] ? watchdog+0x40/0x40 [ 7284.262125] ? hrtimer_interrupt+0x128/0x2c0 [ 7284.267679] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7284.271302] ? apic_timer_interrupt+0xf/0x20 [ 7284.275042] [ 7284.276404] ? native_safe_halt+0x2e/0x50 [ 7284.282348] kvm_wait+0xcd/0xe0 [ 7284.284335] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7284.289782] do_raw_spin_lock+0x177/0x190 [ 7284.294620] _raw_spin_lock+0x21/0x30 [ 7284.298228] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7284.307134] process_one_work+0x2c8/0x700 [ 7284.311681] worker_thread+0x296/0x6e0 [ 7284.315450] ? rescuer_thread+0x570/0x570 [ 7284.319749] kthread+0x1d1/0x200 [ 7284.322962] ? set_kthread_struct+0x70/0x70 [ 7284.326353] ret_from_fork+0x1f/0x30 [ 7288.055523] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 7288.062226] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 7288.063579] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7288.064045] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7288.064114] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7288.064116] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7288.064133] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7288.064235] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7288.064242] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 7288.064244] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 7288.064246] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 7288.064248] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 7288.064250] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7288.064252] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 7288.064471] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7288.064473] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7288.064596] Call Trace: [ 7288.064603] [ 7288.064744] ? show_regs.cold.9+0x22/0x2f [ 7288.064760] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7288.064767] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7288.064772] ? watchdog+0x40/0x40 [ 7288.064776] ? hrtimer_interrupt+0x128/0x2c0 [ 7288.064780] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7288.065251] ? apic_timer_interrupt+0xf/0x20 [ 7288.065257] [ 7288.065260] ? native_safe_halt+0x2e/0x50 [ 7288.065265] kvm_wait+0xcd/0xe0 [ 7288.065318] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7288.065329] do_raw_spin_lock+0x177/0x190 [ 7288.065334] _raw_spin_lock+0x21/0x30 [ 7288.065340] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7288.066716] obd_mod_free+0x46/0x140 [ptlrpc] [ 7288.066906] mdc_commit_open+0x91/0xb0 [mdc] [ 7288.068657] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7288.069018] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7288.069529] after_reply+0xe2c/0x1780 [ptlrpc] [ 7288.069727] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7288.078792] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7288.078868] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7288.078876] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7288.078878] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7288.078893] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7288.078897] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7288.078902] RAX: 0000000000007e92 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7288.078905] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 7288.078907] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7288.078908] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 7288.078911] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 7288.078914] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 7288.078917] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7288.078918] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 7288.078927] Call Trace: [ 7288.078933] [ 7288.078939] ? show_regs.cold.9+0x22/0x2f [ 7288.078948] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7288.078955] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7288.078959] ? watchdog+0x40/0x40 [ 7288.078963] ? hrtimer_interrupt+0x128/0x2c0 [ 7288.078968] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7288.078976] ? apic_timer_interrupt+0xf/0x20 [ 7288.078981] [ 7288.078984] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7288.078991] do_raw_spin_lock+0x177/0x190 [ 7288.078997] _raw_spin_lock+0x21/0x30 [ 7288.079002] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7288.128922] ? __next_timer_interrupt+0x160/0x160 [ 7288.128959] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7288.129475] ? do_wait_intr+0xf0/0xf0 [ 7288.129495] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7288.129693] kthread+0x1d1/0x200 [ 7288.129699] ? set_kthread_struct+0x70/0x70 [ 7288.129703] ret_from_fork+0x1f/0x30 [ 7288.612291] ? do_raw_spin_unlock+0x75/0x190 [ 7288.615933] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7288.618946] ? do_wait_intr+0xf0/0xf0 [ 7288.620203] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7288.625012] mdc_close+0x53b/0xd70 [mdc] [ 7288.626865] lmv_close+0x18d/0x490 [lmv] [ 7288.632259] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7288.638122] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7288.644790] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7288.647524] ll_md_close+0x274/0x800 [lustre] [ 7288.651913] ll_file_release+0x128/0xc50 [lustre] [ 7288.659894] __fput+0x10d/0x440 [ 7288.661208] ____fput+0x12/0x20 [ 7288.664209] task_work_run+0xc6/0x110 [ 7288.666819] exit_to_usermode_loop+0x1dd/0x1f0 [ 7288.672341] do_syscall_64+0x3d6/0x440 [ 7288.675731] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7288.677555] RIP: 0033:0x7f151bee3cd8 [ 7288.679975] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7288.699556] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7288.707468] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 7288.718086] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 7288.723489] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7288.729672] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7288.734866] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7304.060610] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 7304.068917] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7304.119100] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7304.127625] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7304.134394] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7304.138968] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7304.173200] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7304.186453] RAX: 000000000000330a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7304.198665] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 7304.211493] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 7304.218772] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 7304.224624] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 7304.230380] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 7304.235152] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7304.238789] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 7304.242100] Call Trace: [ 7304.249900] [ 7304.252905] ? show_regs.cold.9+0x22/0x2f [ 7304.257443] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7304.262340] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7304.267772] ? watchdog+0x40/0x40 [ 7304.270777] ? hrtimer_interrupt+0x128/0x2c0 [ 7304.274945] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7304.277848] ? apic_timer_interrupt+0xf/0x20 [ 7304.286365] [ 7304.287131] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7304.296737] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7304.300117] do_raw_spin_lock+0x177/0x190 [ 7304.303641] _raw_spin_lock+0x21/0x30 [ 7304.306898] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7304.310264] ? do_raw_spin_unlock+0x75/0x190 [ 7304.311740] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7304.319728] ? do_wait_intr+0xf0/0xf0 [ 7304.322791] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7304.327091] mdc_reint+0x2d/0x110 [mdc] [ 7304.331907] mdc_unlink+0x658/0x870 [mdc] [ 7304.337124] lmv_unlink+0x34d/0xbd0 [lmv] [ 7304.344644] ll_unlink+0x24d/0x7f0 [lustre] [ 7304.349891] vfs_unlink+0x202/0x330 [ 7304.351140] do_unlinkat+0x2fa/0x460 [ 7304.353446] __x64_sys_unlink+0x37/0x50 [ 7304.357431] do_syscall_64+0xc1/0x440 [ 7304.359518] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7304.365828] RIP: 0033:0x7f151bdee10b [ 7304.375522] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7304.391834] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7304.405304] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 7304.412692] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 7304.417099] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7304.424714] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7304.429827] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7309.281332] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5378s! [ 7309.289093] Showing busy workqueues and worker pools: [ 7309.291990] workqueue events: flags=0x0 [ 7309.299118] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7309.303380] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 7309.308046] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7309.310552] pending: kfree_rcu_monitor [ 7309.314699] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7309.317977] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 7309.322352] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7309.325860] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 7309.337024] workqueue events_long: flags=0x0 [ 7309.339464] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7309.347764] pending: lu_objects_flush [obdclass] [ 7309.352166] workqueue events_unbound: flags=0x2 [ 7309.354598] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7309.358126] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7309.365629] workqueue events_power_efficient: flags=0x80 [ 7309.373548] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7309.378859] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 7309.383166] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7309.386541] pending: neigh_periodic_work [ 7309.388173] workqueue mm_percpu_wq: flags=0x8 [ 7309.395741] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7309.400394] pending: vmstat_update, lru_add_drain_per_cpu [ 7309.409095] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7309.416043] pending: vmstat_update [ 7309.420484] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7309.427039] pending: lru_add_drain_per_cpu [ 7309.429768] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7309.433875] pending: lru_add_drain_per_cpu BAR(48) [ 7309.439051] workqueue writeback: flags=0x4a [ 7309.446739] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 7309.451544] pending: wb_workfn [ 7309.452822] workqueue xprtiod: flags=0xa [ 7309.460490] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7309.470690] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7309.478070] workqueue ipv6_addrconf: flags=0x40008 [ 7309.481503] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7309.485370] pending: addrconf_verify_work [ 7309.489423] workqueue knbd0-recv: flags=0x1a [ 7309.491850] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7309.496054] in-flight: 658:recv_work [ 7309.498460] workqueue ptlrpc_pinger: flags=0x2 [ 7309.501375] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7309.505615] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7309.510448] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5378s workers=4 idle: 32984 37 8 [ 7309.513604] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7294s workers=2 idle: 661 [ 7312.056454] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7312.062164] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7312.088721] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7312.100618] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7312.107729] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7312.111745] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7312.115224] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7312.137323] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7312.144291] RAX: 0000000000004f92 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7312.149714] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 7312.153662] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 7312.160713] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 7312.171059] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 7312.176384] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 7312.181697] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7312.183714] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7312.189483] Call Trace: [ 7312.192842] [ 7312.193927] ? show_regs.cold.9+0x22/0x2f [ 7312.197507] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7312.200372] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7312.202534] ? watchdog+0x40/0x40 [ 7312.204139] ? hrtimer_interrupt+0x128/0x2c0 [ 7312.205829] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7312.209302] ? apic_timer_interrupt+0xf/0x20 [ 7312.210975] [ 7312.212937] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7312.218553] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7312.220850] do_raw_spin_lock+0x177/0x190 [ 7312.222506] _raw_spin_lock+0x21/0x30 [ 7312.223672] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7312.226004] process_one_work+0x2c8/0x700 [ 7312.230826] worker_thread+0x296/0x6e0 [ 7312.231960] ? rescuer_thread+0x570/0x570 [ 7312.234241] kthread+0x1d1/0x200 [ 7312.235340] ? set_kthread_struct+0x70/0x70 [ 7312.236680] ret_from_fork+0x1f/0x30 [ 7316.055120] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 7316.060317] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7316.061670] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 7316.086605] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7316.086620] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7316.086635] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7316.086650] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7316.086654] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7316.086660] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 7316.086662] RDX: 00000000000016b4 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 7316.086664] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 7316.086667] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 7316.086669] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7316.086673] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 7316.086676] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7316.086678] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7316.086686] Call Trace: [ 7316.086692] [ 7316.086698] ? show_regs.cold.9+0x22/0x2f [ 7316.086846] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7316.086930] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7316.086937] ? watchdog+0x40/0x40 [ 7316.086942] ? hrtimer_interrupt+0x128/0x2c0 [ 7316.086945] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7316.086953] ? apic_timer_interrupt+0xf/0x20 [ 7316.086957] [ 7316.086959] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7316.086964] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7316.086968] do_raw_spin_lock+0x177/0x190 [ 7316.086972] _raw_spin_lock+0x21/0x30 [ 7316.086978] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7316.087346] obd_mod_free+0x46/0x140 [ptlrpc] [ 7316.087553] mdc_commit_open+0x91/0xb0 [mdc] [ 7316.087597] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7316.088160] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7316.088373] after_reply+0xe2c/0x1780 [ptlrpc] [ 7316.088577] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7316.089242] ? __next_timer_interrupt+0x160/0x160 [ 7316.089252] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7316.089459] ? do_wait_intr+0xf0/0xf0 [ 7316.089468] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7316.089670] kthread+0x1d1/0x200 [ 7316.089676] ? set_kthread_struct+0x70/0x70 [ 7316.089679] ret_from_fork+0x1f/0x30 [ 7316.283064] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7316.351594] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7316.357897] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7316.363797] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7316.366514] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7316.377479] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7316.380464] RAX: 000000000000784c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7316.383976] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 7316.387807] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7316.391200] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 7316.394310] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 7316.399297] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 7316.404186] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7316.406782] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 7316.410143] Call Trace: [ 7316.411269] [ 7316.412036] ? show_regs.cold.9+0x22/0x2f [ 7316.415983] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7316.422627] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7316.424309] ? watchdog+0x40/0x40 [ 7316.429507] ? hrtimer_interrupt+0x128/0x2c0 [ 7316.432047] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7316.434480] ? apic_timer_interrupt+0xf/0x20 [ 7316.441621] [ 7316.442572] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7316.445425] do_raw_spin_lock+0x177/0x190 [ 7316.447239] _raw_spin_lock+0x21/0x30 [ 7316.448798] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7316.451563] ? do_raw_spin_unlock+0x75/0x190 [ 7316.453762] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7316.457693] ? do_wait_intr+0xf0/0xf0 [ 7316.462115] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7316.468313] mdc_close+0x53b/0xd70 [mdc] [ 7316.473734] lmv_close+0x18d/0x490 [lmv] [ 7316.482166] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7316.488685] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7316.500318] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7316.501930] ll_md_close+0x274/0x800 [lustre] [ 7316.509620] ll_file_release+0x128/0xc50 [lustre] [ 7316.514783] __fput+0x10d/0x440 [ 7316.517679] ____fput+0x12/0x20 [ 7316.520166] task_work_run+0xc6/0x110 [ 7316.527283] exit_to_usermode_loop+0x1dd/0x1f0 [ 7316.533060] do_syscall_64+0x3d6/0x440 [ 7316.537468] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7316.543697] RIP: 0033:0x7f151bee3cd8 [ 7316.550499] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7316.585500] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7316.596690] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 7316.609842] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 7316.627468] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7316.631697] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7316.636555] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7336.056077] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 7336.061152] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7336.108924] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7336.115313] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7336.126774] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7336.128963] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7336.141928] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7336.150643] RAX: 0000000000004088 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7336.154323] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 7336.158642] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 7336.161433] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 7336.166652] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 7336.171752] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 7336.176366] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7336.181446] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 7336.185440] Call Trace: [ 7336.188413] [ 7336.191741] ? show_regs.cold.9+0x22/0x2f [ 7336.194941] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7336.199730] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7336.202206] ? watchdog+0x40/0x40 [ 7336.204078] ? hrtimer_interrupt+0x128/0x2c0 [ 7336.205956] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7336.209986] ? apic_timer_interrupt+0xf/0x20 [ 7336.213097] [ 7336.214069] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7336.217492] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7336.220050] do_raw_spin_lock+0x177/0x190 [ 7336.223203] _raw_spin_lock+0x21/0x30 [ 7336.224704] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7336.228645] ? do_raw_spin_unlock+0x75/0x190 [ 7336.232696] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7336.236148] ? do_wait_intr+0xf0/0xf0 [ 7336.237542] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7336.241009] mdc_reint+0x2d/0x110 [mdc] [ 7336.245787] mdc_unlink+0x658/0x870 [mdc] [ 7336.247942] lmv_unlink+0x34d/0xbd0 [lmv] [ 7336.253401] ll_unlink+0x24d/0x7f0 [lustre] [ 7336.256181] vfs_unlink+0x202/0x330 [ 7336.260339] do_unlinkat+0x2fa/0x460 [ 7336.264354] __x64_sys_unlink+0x37/0x50 [ 7336.269376] do_syscall_64+0xc1/0x440 [ 7336.270295] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7336.271710] RIP: 0033:0x7f151bdee10b [ 7336.276803] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7336.292286] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7336.297577] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 7336.303713] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 7336.308274] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7336.314140] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7336.318974] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7340.000115] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5409s! [ 7340.005278] Showing busy workqueues and worker pools: [ 7340.014295] workqueue events: flags=0x0 [ 7340.018379] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7340.024107] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 7340.037443] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7340.039651] pending: kfree_rcu_monitor [ 7340.053372] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7340.056029] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7340.064469] pending: nfs4_renew_state [nfsv4] [ 7340.068740] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7340.068818] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7340.068825] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7340.068831] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7340.077485] , key_garbage_collector, osc_grant_work_handler [osc] [ 7340.101582] [ 7340.101592] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7340.102039] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7340.102043] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7340.102048] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7340.102050] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 7340.102052] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 7340.102054] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 7340.102057] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 7340.102060] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 7340.102062] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7340.102064] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7340.102073] Call Trace: [ 7340.102079] [ 7340.102085] ? show_regs.cold.9+0x22/0x2f [ 7340.102095] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7340.102103] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7340.102108] ? watchdog+0x40/0x40 [ 7340.102113] ? hrtimer_interrupt+0x128/0x2c0 [ 7340.102117] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7340.102124] ? apic_timer_interrupt+0xf/0x20 [ 7340.102129] [ 7340.102132] ? native_safe_halt+0x2e/0x50 [ 7340.102137] kvm_wait+0xcd/0xe0 [ 7340.102144] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7340.102155] do_raw_spin_lock+0x177/0x190 [ 7340.102161] _raw_spin_lock+0x21/0x30 [ 7340.102333] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7340.102819] process_one_work+0x2c8/0x700 [ 7340.102826] worker_thread+0x296/0x6e0 [ 7340.102830] ? rescuer_thread+0x570/0x570 [ 7340.102833] kthread+0x1d1/0x200 [ 7340.102838] ? set_kthread_struct+0x70/0x70 [ 7340.102842] ret_from_fork+0x1f/0x30 [ 7340.292026] [ 7340.293039] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7340.299153] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 7340.310688] workqueue events_long: flags=0x0 [ 7340.312653] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7340.319657] pending: lu_objects_flush [obdclass] [ 7340.329969] workqueue events_unbound: flags=0x2 [ 7340.331681] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7340.349503] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7340.357927] workqueue events_power_efficient: flags=0x80 [ 7340.366762] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7340.383217] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 7340.390399] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7340.398993] pending: neigh_periodic_work [ 7340.401203] workqueue mm_percpu_wq: flags=0x8 [ 7340.406549] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7340.413172] pending: vmstat_update, lru_add_drain_per_cpu [ 7340.419746] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7340.426938] pending: vmstat_update [ 7340.433589] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7340.440358] pending: lru_add_drain_per_cpu [ 7340.443642] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7340.449769] pending: lru_add_drain_per_cpu BAR(48) [ 7340.455054] workqueue writeback: flags=0x4a [ 7340.456623] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 7340.465921] pending: wb_workfn [ 7340.467683] workqueue xprtiod: flags=0xa [ 7340.471997] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7340.479468] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7340.489218] workqueue ipv6_addrconf: flags=0x40008 [ 7340.491295] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7340.497824] pending: addrconf_verify_work [ 7340.507425] workqueue knbd0-recv: flags=0x1a [ 7340.512406] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7340.520453] in-flight: 658:recv_work [ 7340.523436] workqueue ptlrpc_pinger: flags=0x2 [ 7340.528126] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7340.533767] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7340.541777] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5409s workers=4 idle: 32984 37 8 [ 7340.548695] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7325s workers=2 idle: 661 [ 7344.055242] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 7344.057061] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 7344.069228] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7344.069311] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7344.069319] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7344.069322] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7344.069334] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7344.069339] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7344.069343] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 7344.069346] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 7344.069348] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 7344.069350] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 7344.069353] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7344.069356] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 7344.069359] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7344.069361] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7344.069369] Call Trace: [ 7344.069375] [ 7344.069380] ? show_regs.cold.9+0x22/0x2f [ 7344.069390] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7344.069395] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7344.069400] ? watchdog+0x40/0x40 [ 7344.069404] ? hrtimer_interrupt+0x128/0x2c0 [ 7344.069409] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7344.069415] ? apic_timer_interrupt+0xf/0x20 [ 7344.069420] [ 7344.069423] ? native_safe_halt+0x2e/0x50 [ 7344.069427] kvm_wait+0xcd/0xe0 [ 7344.069434] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7344.069441] do_raw_spin_lock+0x177/0x190 [ 7344.069445] _raw_spin_lock+0x21/0x30 [ 7344.069450] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7344.069731] obd_mod_free+0x46/0x140 [ptlrpc] [ 7344.074217] mdc_commit_open+0x91/0xb0 [mdc] [ 7344.074257] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7344.074472] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7344.074960] after_reply+0xe2c/0x1780 [ptlrpc] [ 7344.075275] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7344.078527] Modules linked in: [ 7344.094405] ? __next_timer_interrupt+0x160/0x160 [ 7344.098990] lustre(O) [ 7344.106793] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7344.109111] osp(O) [ 7344.114802] ? do_wait_intr+0xf0/0xf0 [ 7344.118645] ofd(O) [ 7344.122315] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7344.125577] lod(O) [ 7344.129280] kthread+0x1d1/0x200 [ 7344.129299] ? set_kthread_struct+0x70/0x70 [ 7344.129304] ret_from_fork+0x1f/0x30 [ 7344.213369] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7344.243481] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7344.248798] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7344.253326] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7344.255973] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7344.267424] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7344.271654] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7344.275154] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 7344.278150] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7344.281591] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 7344.286372] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 7344.290559] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 7344.295024] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7344.297619] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 7344.300282] Call Trace: [ 7344.301054] [ 7344.301569] ? show_regs.cold.9+0x22/0x2f [ 7344.304188] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7344.313386] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7344.323036] ? watchdog+0x40/0x40 [ 7344.326048] ? hrtimer_interrupt+0x128/0x2c0 [ 7344.328130] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7344.330411] ? apic_timer_interrupt+0xf/0x20 [ 7344.332666] [ 7344.334037] ? native_safe_halt+0x2e/0x50 [ 7344.336163] kvm_wait+0xcd/0xe0 [ 7344.339557] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7344.344327] do_raw_spin_lock+0x177/0x190 [ 7344.345835] _raw_spin_lock+0x21/0x30 [ 7344.350015] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7344.352883] ? do_raw_spin_unlock+0x75/0x190 [ 7344.355083] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7344.357788] ? do_wait_intr+0xf0/0xf0 [ 7344.359561] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7344.362657] mdc_close+0x53b/0xd70 [mdc] [ 7344.364545] lmv_close+0x18d/0x490 [lmv] [ 7344.366453] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7344.369263] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7344.372751] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7344.376076] ll_md_close+0x274/0x800 [lustre] [ 7344.378330] ll_file_release+0x128/0xc50 [lustre] [ 7344.380563] __fput+0x10d/0x440 [ 7344.382268] ____fput+0x12/0x20 [ 7344.383897] task_work_run+0xc6/0x110 [ 7344.385309] exit_to_usermode_loop+0x1dd/0x1f0 [ 7344.386474] do_syscall_64+0x3d6/0x440 [ 7344.387648] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7344.390230] RIP: 0033:0x7f151bee3cd8 [ 7344.391494] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7344.406497] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7344.410633] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 7344.414498] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 7344.418027] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7344.422256] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7344.428056] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7368.058867] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 7368.061689] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 7368.069692] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7368.069795] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7368.069802] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7368.069805] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7368.069819] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7368.069824] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7368.069828] RAX: 0000000000006789 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7368.069831] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 7368.069833] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 7368.069836] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 7368.069838] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 7368.069842] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 7368.069845] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7368.069848] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 7368.069857] Call Trace: [ 7368.069863] [ 7368.069868] ? show_regs.cold.9+0x22/0x2f [ 7368.069878] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7368.069885] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7368.069889] ? watchdog+0x40/0x40 [ 7368.069893] ? hrtimer_interrupt+0x128/0x2c0 [ 7368.069897] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7368.069906] ? apic_timer_interrupt+0xf/0x20 [ 7368.069911] [ 7368.069914] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7368.069919] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7368.069923] do_raw_spin_lock+0x177/0x190 [ 7368.069928] _raw_spin_lock+0x21/0x30 [ 7368.069934] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7368.070219] ? do_raw_spin_unlock+0x75/0x190 [ 7368.070225] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7368.070415] ? do_wait_intr+0xf0/0xf0 [ 7368.070424] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7368.071362] mdc_reint+0x2d/0x110 [mdc] [ 7368.071403] mdc_unlink+0x658/0x870 [mdc] [ 7368.071428] lmv_unlink+0x34d/0xbd0 [lmv] [ 7368.071461] ll_unlink+0x24d/0x7f0 [lustre] [ 7368.071737] vfs_unlink+0x202/0x330 [ 7368.071748] do_unlinkat+0x2fa/0x460 [ 7368.071753] __x64_sys_unlink+0x37/0x50 [ 7368.071758] do_syscall_64+0xc1/0x440 [ 7368.071765] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7368.071842] RIP: 0033:0x7f151bdee10b [ 7368.071850] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7368.071854] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7368.071860] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 7368.071863] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 7368.071865] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7368.071867] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7368.071870] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7368.494139] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7368.563526] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7368.589180] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7368.596104] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7368.608823] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7368.610760] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7368.634247] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7368.638755] RAX: 0000000000006d34 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7368.642430] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 7368.653667] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 7368.664149] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 7368.683151] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 7368.685641] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 7368.696934] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7368.709242] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7368.720926] Call Trace: [ 7368.724356] [ 7368.728050] ? show_regs.cold.9+0x22/0x2f [ 7368.733782] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7368.740652] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7368.753925] ? watchdog+0x40/0x40 [ 7368.762053] ? hrtimer_interrupt+0x128/0x2c0 [ 7368.766759] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7368.773806] ? apic_timer_interrupt+0xf/0x20 [ 7368.782928] [ 7368.786206] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7368.795199] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7368.800086] do_raw_spin_lock+0x177/0x190 [ 7368.807810] _raw_spin_lock+0x21/0x30 [ 7368.814285] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7368.822437] process_one_work+0x2c8/0x700 [ 7368.831220] worker_thread+0x296/0x6e0 [ 7368.837153] ? rescuer_thread+0x570/0x570 [ 7368.841680] kthread+0x1d1/0x200 [ 7368.843486] ? set_kthread_struct+0x70/0x70 [ 7368.852430] ret_from_fork+0x1f/0x30 [ 7370.721070] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5440s! [ 7370.725084] Showing busy workqueues and worker pools: [ 7370.727523] workqueue events: flags=0x0 [ 7370.729389] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7370.732833] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 7370.737029] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7370.739733] pending: kfree_rcu_monitor [ 7370.741376] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7370.744400] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 7370.749136] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7370.752228] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 7370.756435] workqueue events_long: flags=0x0 [ 7370.758653] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7370.761982] pending: lu_objects_flush [obdclass] [ 7370.764305] workqueue events_unbound: flags=0x2 [ 7370.766386] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7370.769454] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7370.774466] workqueue events_power_efficient: flags=0x80 [ 7370.776905] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7370.780317] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 7370.783496] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7370.786756] pending: neigh_periodic_work [ 7370.788296] workqueue mm_percpu_wq: flags=0x8 [ 7370.790479] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7370.793345] pending: vmstat_update, lru_add_drain_per_cpu [ 7370.796032] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7370.799240] pending: vmstat_update [ 7370.801333] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7370.804282] pending: lru_add_drain_per_cpu [ 7370.806461] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7370.810221] pending: lru_add_drain_per_cpu BAR(48) [ 7370.812879] workqueue writeback: flags=0x4a [ 7370.814818] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 7370.818108] pending: wb_workfn [ 7370.819667] workqueue xprtiod: flags=0xa [ 7370.821329] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7370.823992] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7370.827819] workqueue ipv6_addrconf: flags=0x40008 [ 7370.829908] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7370.833060] pending: addrconf_verify_work [ 7370.834716] workqueue knbd0-recv: flags=0x1a [ 7370.836473] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7370.839046] in-flight: 658:recv_work [ 7370.840378] workqueue ptlrpc_pinger: flags=0x2 [ 7370.841946] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7370.844301] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7370.847165] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5440s workers=4 idle: 32984 37 8 [ 7370.850488] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7355s workers=2 idle: 661 [ 7372.055025] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 7372.057022] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 7372.058426] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7372.058508] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7372.058516] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7372.058519] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7372.058534] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7372.058538] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7372.058542] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 7372.058545] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 7372.058547] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 7372.058550] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 7372.058553] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7372.058556] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 7372.058559] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7372.058562] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7372.058651] Call Trace: [ 7372.058658] [ 7372.058663] ? show_regs.cold.9+0x22/0x2f [ 7372.058688] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7372.058695] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7372.058700] ? watchdog+0x40/0x40 [ 7372.058705] ? hrtimer_interrupt+0x128/0x2c0 [ 7372.058709] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7372.058717] ? apic_timer_interrupt+0xf/0x20 [ 7372.058722] [ 7372.058725] ? native_safe_halt+0x2e/0x50 [ 7372.058729] kvm_wait+0xcd/0xe0 [ 7372.058737] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7372.058745] do_raw_spin_lock+0x177/0x190 [ 7372.058750] _raw_spin_lock+0x21/0x30 [ 7372.058755] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7372.059040] obd_mod_free+0x46/0x140 [ptlrpc] [ 7372.059857] mdc_commit_open+0x91/0xb0 [mdc] [ 7372.059901] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7372.060122] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7372.060517] after_reply+0xe2c/0x1780 [ptlrpc] [ 7372.060865] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7372.061089] ? __next_timer_interrupt+0x160/0x160 [ 7372.061099] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7372.061474] ? do_wait_intr+0xf0/0xf0 [ 7372.061485] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7372.061824] kthread+0x1d1/0x200 [ 7372.061830] ? set_kthread_struct+0x70/0x70 [ 7372.061834] ret_from_fork+0x1f/0x30 [ 7372.365473] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7372.411335] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7372.429338] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7372.434444] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7372.435972] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7372.462315] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7372.482785] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7372.484439] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 7372.500190] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7372.505846] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 7372.527949] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 7372.535400] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 7372.543438] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7372.549963] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 7372.560898] Call Trace: [ 7372.564038] [ 7372.565360] ? show_regs.cold.9+0x22/0x2f [ 7372.568131] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7372.572435] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7372.575310] ? watchdog+0x40/0x40 [ 7372.582092] ? hrtimer_interrupt+0x128/0x2c0 [ 7372.587251] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7372.592067] ? apic_timer_interrupt+0xf/0x20 [ 7372.594363] [ 7372.596438] ? native_safe_halt+0x2e/0x50 [ 7372.602986] kvm_wait+0xcd/0xe0 [ 7372.604465] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7372.612826] do_raw_spin_lock+0x177/0x190 [ 7372.615095] _raw_spin_lock+0x21/0x30 [ 7372.620082] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7372.624031] ? do_raw_spin_unlock+0x75/0x190 [ 7372.625283] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7372.628513] ? do_wait_intr+0xf0/0xf0 [ 7372.629955] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7372.635934] mdc_close+0x53b/0xd70 [mdc] [ 7372.637338] lmv_close+0x18d/0x490 [lmv] [ 7372.648066] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7372.656752] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7372.658869] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7372.668219] ll_md_close+0x274/0x800 [lustre] [ 7372.674794] ll_file_release+0x128/0xc50 [lustre] [ 7372.681353] __fput+0x10d/0x440 [ 7372.682554] ____fput+0x12/0x20 [ 7372.683581] task_work_run+0xc6/0x110 [ 7372.693037] exit_to_usermode_loop+0x1dd/0x1f0 [ 7372.699979] do_syscall_64+0x3d6/0x440 [ 7372.701482] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7372.707179] RIP: 0033:0x7f151bee3cd8 [ 7372.708464] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7372.745451] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7372.760031] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 7372.769514] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 7372.777875] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7372.788613] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7372.794355] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7384.768336] rcu: INFO: rcu_sched self-detected stall on CPU [ 7384.770604] rcu: 1-...!: (3077827 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 7384.778204] (t=5460121 jiffies g=249657 q=3995) [ 7384.780217] rcu: rcu_sched kthread starved for 5455073 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 7384.791604] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7384.800127] rcu: RCU grace-period kthread stack dump: [ 7384.803162] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7384.811171] Call Trace: [ 7384.814620] __schedule+0x351/0xcb0 [ 7384.819300] schedule+0xc0/0x180 [ 7384.820498] schedule_timeout+0xb4/0x190 [ 7384.829095] ? do_raw_spin_unlock+0x75/0x190 [ 7384.832040] ? __next_timer_interrupt+0x160/0x160 [ 7384.841855] ? prepare_to_swait_event+0xf3/0x140 [ 7384.846360] rcu_gp_kthread+0x8dd/0x15b0 [ 7384.852102] ? force_qs_rnp+0x2c0/0x2c0 [ 7384.859430] kthread+0x1d1/0x200 [ 7384.862918] ? set_kthread_struct+0x70/0x70 [ 7384.869066] ret_from_fork+0x1f/0x30 [ 7384.873175] rcu: Stack dump where RCU GP kthread last ran: [ 7384.878922] Sending NMI from CPU 1 to CPUs 3: [ 7384.883348] NMI backtrace for cpu 3 [ 7384.883356] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7384.883359] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7384.883361] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7384.883362] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7384.883364] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 [ 7384.883367] RAX: 00000000000062cc RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7384.883368] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 7384.883370] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7384.883371] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 7384.883372] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 7384.883374] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 7384.883376] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7384.883377] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 7384.883378] Call Trace: [ 7384.883379] [ 7384.883380] ? show_regs.cold.9+0x22/0x2f [ 7384.883382] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 7384.883383] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 7384.883384] ? nmi_handle+0xbd/0x210 [ 7384.883385] ? default_do_nmi+0x4b/0x1b0 [ 7384.883386] ? do_nmi+0x280/0x380 [ 7384.883387] ? end_repeat_nmi+0x16/0x1f [ 7384.883389] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7384.883390] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7384.883391] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7384.883392] [ 7384.883393] do_raw_spin_lock+0x177/0x190 [ 7384.883394] _raw_spin_lock+0x21/0x30 [ 7384.883395] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7384.883396] ? do_raw_spin_unlock+0x75/0x190 [ 7384.883397] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7384.883398] ? do_wait_intr+0xf0/0xf0 [ 7384.883399] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7384.883401] mdc_close+0x53b/0xd70 [mdc] [ 7384.883402] lmv_close+0x18d/0x490 [lmv] [ 7384.883404] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7384.883405] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7384.883406] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7384.883407] ll_md_close+0x274/0x800 [lustre] [ 7384.883409] ll_file_release+0x128/0xc50 [lustre] [ 7384.883409] __fput+0x10d/0x440 [ 7384.883410] ____fput+0x12/0x20 [ 7384.883411] task_work_run+0xc6/0x110 [ 7384.883412] exit_to_usermode_loop+0x1dd/0x1f0 [ 7384.883413] do_syscall_64+0x3d6/0x440 [ 7384.883414] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7384.883415] RIP: 0033:0x7f151bee3cd8 [ 7384.883417] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7384.883419] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7384.883422] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 7384.883424] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 7384.883425] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7384.883427] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7384.883428] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7384.884175] NMI backtrace for cpu 1 [ 7385.162608] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7385.169264] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7385.177913] Call Trace: [ 7385.180362] [ 7385.183781] ? dump_stack+0xbb/0x10e [ 7385.187210] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7385.189783] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7385.191143] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7385.192965] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7385.195620] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7385.198877] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 7385.201132] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 7385.204043] ? update_process_times+0x6d/0xc0 [ 7385.206616] ? tick_sched_handle.isra.18+0x27/0x90 [ 7385.209325] ? tick_sched_timer+0x53/0xb0 [ 7385.211579] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7385.213881] ? tick_sched_handle.isra.18+0x90/0x90 [ 7385.216309] ? hrtimer_interrupt+0x128/0x2c0 [ 7385.218802] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7385.222237] ? apic_timer_interrupt+0xf/0x20 [ 7385.225616] [ 7385.227048] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7385.232634] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7385.236045] ? do_raw_spin_lock+0x177/0x190 [ 7385.238018] ? _raw_spin_lock+0x21/0x30 [ 7385.243874] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7385.247459] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7385.251900] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7385.254247] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7385.257198] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7385.268052] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7385.273466] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7385.278289] ? __next_timer_interrupt+0x160/0x160 [ 7385.284398] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7385.288573] ? do_wait_intr+0xf0/0xf0 [ 7385.291319] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7385.295927] ? kthread+0x1d1/0x200 [ 7385.297650] ? set_kthread_struct+0x70/0x70 [ 7385.299378] ? ret_from_fork+0x1f/0x30 [ 7396.052024] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 7396.054066] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7396.056029] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 7396.072533] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7396.072548] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7396.072551] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7396.072565] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7396.072570] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7396.072575] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7396.072578] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 7396.072580] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 7396.072583] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 7396.072585] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 7396.072589] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 7396.072592] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7396.072594] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 7396.072605] Call Trace: [ 7396.072610] [ 7396.072616] ? show_regs.cold.9+0x22/0x2f [ 7396.072627] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7396.072633] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7396.072638] ? watchdog+0x40/0x40 [ 7396.072643] ? hrtimer_interrupt+0x128/0x2c0 [ 7396.072647] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7396.072653] ? apic_timer_interrupt+0xf/0x20 [ 7396.072658] [ 7396.072662] ? native_safe_halt+0x2e/0x50 [ 7396.072666] kvm_wait+0xcd/0xe0 [ 7396.072674] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7396.072682] do_raw_spin_lock+0x177/0x190 [ 7396.072687] _raw_spin_lock+0x21/0x30 [ 7396.072694] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7396.073065] ? do_raw_spin_unlock+0x75/0x190 [ 7396.073072] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7396.073271] ? do_wait_intr+0xf0/0xf0 [ 7396.073280] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7396.073570] mdc_reint+0x2d/0x110 [mdc] [ 7396.073610] mdc_unlink+0x658/0x870 [mdc] [ 7396.073639] lmv_unlink+0x34d/0xbd0 [lmv] [ 7396.073671] ll_unlink+0x24d/0x7f0 [lustre] [ 7396.081085] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7396.081171] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7396.081179] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7396.081185] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7396.089170] vfs_unlink+0x202/0x330 [ 7396.089193] do_unlinkat+0x2fa/0x460 [ 7396.089199] __x64_sys_unlink+0x37/0x50 [ 7396.089203] do_syscall_64+0xc1/0x440 [ 7396.089210] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7396.089221] RIP: 0033:0x7f151bdee10b [ 7396.089229] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7396.089234] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7396.089240] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 7396.089242] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 7396.089244] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7396.089246] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7396.089249] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7396.373125] [ 7396.373606] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7396.374903] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7396.393664] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7396.399397] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7396.405591] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 7396.412832] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 7396.416621] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 7396.419933] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 7396.423610] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 7396.428490] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7396.431589] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7396.439714] Call Trace: [ 7396.440951] [ 7396.443078] ? show_regs.cold.9+0x22/0x2f [ 7396.448795] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7396.450943] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7396.452386] ? watchdog+0x40/0x40 [ 7396.455155] ? hrtimer_interrupt+0x128/0x2c0 [ 7396.457058] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7396.463451] ? apic_timer_interrupt+0xf/0x20 [ 7396.465255] [ 7396.466901] ? native_safe_halt+0x2e/0x50 [ 7396.470222] kvm_wait+0xcd/0xe0 [ 7396.477333] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7396.479974] do_raw_spin_lock+0x177/0x190 [ 7396.483613] _raw_spin_lock+0x21/0x30 [ 7396.490935] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7396.500108] process_one_work+0x2c8/0x700 [ 7396.504054] worker_thread+0x296/0x6e0 [ 7396.507143] ? rescuer_thread+0x570/0x570 [ 7396.510098] kthread+0x1d1/0x200 [ 7396.517641] ? set_kthread_struct+0x70/0x70 [ 7396.520141] ret_from_fork+0x1f/0x30 [ 7400.933420] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5470s! [ 7400.937703] Showing busy workqueues and worker pools: [ 7400.940151] workqueue events: flags=0x0 [ 7400.942089] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7400.946117] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 7400.958058] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7400.973993] pending: kfree_rcu_monitor [ 7400.988042] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7400.990564] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 7401.002466] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7401.021974] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 7401.034843] workqueue events_long: flags=0x0 [ 7401.044714] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7401.054955] pending: lu_objects_flush [obdclass] [ 7401.066530] workqueue events_unbound: flags=0x2 [ 7401.068086] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7401.079543] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7401.090485] workqueue events_power_efficient: flags=0x80 [ 7401.100990] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7401.109402] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 7401.114139] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7401.120023] pending: neigh_periodic_work [ 7401.122115] workqueue mm_percpu_wq: flags=0x8 [ 7401.125217] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7401.132651] pending: vmstat_update, lru_add_drain_per_cpu [ 7401.138810] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7401.143740] pending: vmstat_update [ 7401.147647] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7401.153162] pending: lru_add_drain_per_cpu [ 7401.159673] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7401.163160] pending: lru_add_drain_per_cpu BAR(48) [ 7401.165853] workqueue writeback: flags=0x4a [ 7401.169071] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 7401.173224] pending: wb_workfn [ 7401.178496] workqueue xprtiod: flags=0xa [ 7401.180190] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7401.184085] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7401.189960] workqueue ipv6_addrconf: flags=0x40008 [ 7401.194072] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7401.199944] pending: addrconf_verify_work [ 7401.202425] workqueue knbd0-recv: flags=0x1a [ 7401.209834] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7401.218844] in-flight: 658:recv_work [ 7401.220977] workqueue ptlrpc_pinger: flags=0x2 [ 7401.223741] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7401.229754] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7401.238840] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5470s workers=4 idle: 32984 37 8 [ 7401.243572] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7385s workers=2 idle: 661 [ 7412.057013] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 7412.059304] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 7412.061970] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7412.062056] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7412.062063] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7412.062066] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7412.062086] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7412.062090] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7412.062095] RAX: 0000000000004fd2 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7412.062098] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 7412.062100] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7412.062103] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 7412.062105] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 7412.062109] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 7412.062112] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7412.062114] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 7412.062122] Call Trace: [ 7412.062127] [ 7412.062133] ? show_regs.cold.9+0x22/0x2f [ 7412.062155] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7412.062161] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7412.062166] ? watchdog+0x40/0x40 [ 7412.062170] ? hrtimer_interrupt+0x128/0x2c0 [ 7412.062174] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7412.062182] ? apic_timer_interrupt+0xf/0x20 [ 7412.062187] [ 7412.062190] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7412.062196] do_raw_spin_lock+0x177/0x190 [ 7412.062201] _raw_spin_lock+0x21/0x30 [ 7412.062206] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7412.062519] ? do_raw_spin_unlock+0x75/0x190 [ 7412.062525] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7412.063049] ? do_wait_intr+0xf0/0xf0 [ 7412.063057] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7412.063291] mdc_close+0x53b/0xd70 [mdc] [ 7412.063331] lmv_close+0x18d/0x490 [lmv] [ 7412.063361] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7412.063530] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7412.063914] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7412.063995] ll_md_close+0x274/0x800 [lustre] [ 7412.064171] ll_file_release+0x128/0xc50 [lustre] [ 7412.064253] __fput+0x10d/0x440 [ 7412.064263] ____fput+0x12/0x20 [ 7412.064267] task_work_run+0xc6/0x110 [ 7412.064276] exit_to_usermode_loop+0x1dd/0x1f0 [ 7412.064296] do_syscall_64+0x3d6/0x440 [ 7412.064301] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7412.064310] RIP: 0033:0x7f151bee3cd8 [ 7412.064317] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7412.064321] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7412.064327] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 7412.064329] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 7412.064331] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7412.064334] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7412.064336] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7412.385077] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7412.414569] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7412.419196] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7412.423731] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7412.425868] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7412.449142] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7412.461098] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 7412.465450] RDX: 000000000000735a RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 7412.468822] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 7412.473927] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 7412.479643] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7412.481660] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 7412.486472] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7412.489411] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7412.491648] Call Trace: [ 7412.493160] [ 7412.493900] ? show_regs.cold.9+0x22/0x2f [ 7412.495269] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7412.497647] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7412.499823] ? watchdog+0x40/0x40 [ 7412.501502] ? hrtimer_interrupt+0x128/0x2c0 [ 7412.504259] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7412.507023] ? apic_timer_interrupt+0xf/0x20 [ 7412.514378] [ 7412.516612] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7412.520885] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7412.523640] do_raw_spin_lock+0x177/0x190 [ 7412.525604] _raw_spin_lock+0x21/0x30 [ 7412.531530] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7412.536556] obd_mod_free+0x46/0x140 [ptlrpc] [ 7412.540270] mdc_commit_open+0x91/0xb0 [mdc] [ 7412.542353] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7412.546053] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7412.551627] after_reply+0xe2c/0x1780 [ptlrpc] [ 7412.553730] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7412.556181] ? __next_timer_interrupt+0x160/0x160 [ 7412.558356] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7412.560698] ? do_wait_intr+0xf0/0xf0 [ 7412.562739] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7412.565477] kthread+0x1d1/0x200 [ 7412.566866] ? set_kthread_struct+0x70/0x70 [ 7412.570938] ret_from_fork+0x1f/0x30 [ 7424.056008] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 7424.062914] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7424.091991] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7424.104954] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7424.116533] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7424.122566] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7424.126431] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7424.141463] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7424.146017] RAX: 0000000000005f5d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7424.151065] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 7424.154039] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 7424.159426] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 7424.163527] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 7424.170610] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 7424.176990] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7424.180663] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7424.184833] Call Trace: [ 7424.189701] [ 7424.190563] ? show_regs.cold.9+0x22/0x2f [ 7424.194232] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7424.204826] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7424.207482] ? watchdog+0x40/0x40 [ 7424.209554] ? hrtimer_interrupt+0x128/0x2c0 [ 7424.212836] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7424.215314] ? apic_timer_interrupt+0xf/0x20 [ 7424.218397] [ 7424.219698] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7424.229554] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7424.231593] do_raw_spin_lock+0x177/0x190 [ 7424.235384] _raw_spin_lock+0x21/0x30 [ 7424.237292] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7424.240847] process_one_work+0x2c8/0x700 [ 7424.243457] worker_thread+0x296/0x6e0 [ 7424.246223] ? rescuer_thread+0x570/0x570 [ 7424.248004] kthread+0x1d1/0x200 [ 7424.250005] ? set_kthread_struct+0x70/0x70 [ 7424.251581] ret_from_fork+0x1f/0x30 [ 7428.052038] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 7428.057477] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7428.087072] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7428.093107] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7428.097747] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7428.102745] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7428.113256] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7428.118331] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7428.123531] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 7428.126342] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 7428.128812] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 7428.132196] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 7428.136438] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 7428.139972] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7428.142280] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 7428.148061] Call Trace: [ 7428.149269] [ 7428.150462] ? show_regs.cold.9+0x22/0x2f [ 7428.153172] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7428.155481] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7428.158243] ? watchdog+0x40/0x40 [ 7428.160097] ? hrtimer_interrupt+0x128/0x2c0 [ 7428.162451] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7428.165255] ? apic_timer_interrupt+0xf/0x20 [ 7428.167835] [ 7428.169196] ? native_safe_halt+0x2e/0x50 [ 7428.171544] kvm_wait+0xcd/0xe0 [ 7428.173702] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7428.176870] do_raw_spin_lock+0x177/0x190 [ 7428.180444] _raw_spin_lock+0x21/0x30 [ 7428.182679] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7428.186379] ? do_raw_spin_unlock+0x75/0x190 [ 7428.190288] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7428.192821] ? do_wait_intr+0xf0/0xf0 [ 7428.194656] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7428.197614] mdc_reint+0x2d/0x110 [mdc] [ 7428.199747] mdc_unlink+0x658/0x870 [mdc] [ 7428.201935] lmv_unlink+0x34d/0xbd0 [lmv] [ 7428.204810] ll_unlink+0x24d/0x7f0 [lustre] [ 7428.207847] vfs_unlink+0x202/0x330 [ 7428.209769] do_unlinkat+0x2fa/0x460 [ 7428.211264] __x64_sys_unlink+0x37/0x50 [ 7428.212831] do_syscall_64+0xc1/0x440 [ 7428.216280] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7428.221748] RIP: 0033:0x7f151bdee10b [ 7428.224202] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7428.236485] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7428.243871] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 7428.247648] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 7428.253379] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7428.259592] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7428.263003] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7431.651706] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5501s! [ 7431.657779] Showing busy workqueues and worker pools: [ 7431.662525] workqueue events: flags=0x0 [ 7431.667616] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7431.673603] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 7431.681413] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7431.683380] pending: kfree_rcu_monitor [ 7431.693571] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7431.700915] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 7431.710695] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7431.718595] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 7431.726787] workqueue events_long: flags=0x0 [ 7431.731654] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7431.735421] pending: lu_objects_flush [obdclass] [ 7431.740798] workqueue events_unbound: flags=0x2 [ 7431.742478] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7431.746046] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7431.751795] workqueue events_power_efficient: flags=0x80 [ 7431.761915] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7431.765221] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 7431.769454] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7431.772718] pending: neigh_periodic_work [ 7431.777266] workqueue mm_percpu_wq: flags=0x8 [ 7431.778592] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7431.786989] pending: vmstat_update, lru_add_drain_per_cpu [ 7431.791992] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7431.797802] pending: vmstat_update [ 7431.801424] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7431.807693] pending: lru_add_drain_per_cpu [ 7431.811045] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7431.813775] pending: lru_add_drain_per_cpu BAR(48) [ 7431.816400] workqueue writeback: flags=0x4a [ 7431.821090] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 7431.824983] pending: wb_workfn [ 7431.829311] workqueue xprtiod: flags=0xa [ 7431.830516] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7431.831973] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7431.838038] workqueue ipv6_addrconf: flags=0x40008 [ 7431.839728] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7431.841592] pending: addrconf_verify_work [ 7431.842708] workqueue knbd0-recv: flags=0x1a [ 7431.844572] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7431.850908] in-flight: 658:recv_work [ 7431.852658] workqueue ptlrpc_pinger: flags=0x2 [ 7431.855398] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7431.857337] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7431.860213] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5501s workers=4 idle: 32984 37 8 [ 7431.863717] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7416s workers=2 idle: 661 [ 7440.055211] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 7440.061348] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7440.065750] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 7440.093278] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7440.093293] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7440.093297] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7440.093310] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7440.093315] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7440.093321] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 7440.093324] RDX: 0000000000007f89 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 7440.093327] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 7440.093329] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 7440.093331] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7440.093335] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 7440.093338] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7440.093340] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7440.093349] Call Trace: [ 7440.093356] [ 7440.093361] ? show_regs.cold.9+0x22/0x2f [ 7440.093372] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7440.093378] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7440.093382] ? watchdog+0x40/0x40 [ 7440.093387] ? hrtimer_interrupt+0x128/0x2c0 [ 7440.093390] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7440.093400] ? apic_timer_interrupt+0xf/0x20 [ 7440.093405] [ 7440.093408] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7440.093413] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7440.093418] do_raw_spin_lock+0x177/0x190 [ 7440.093423] _raw_spin_lock+0x21/0x30 [ 7440.093431] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7440.093694] obd_mod_free+0x46/0x140 [ptlrpc] [ 7440.094441] mdc_commit_open+0x91/0xb0 [mdc] [ 7440.094481] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7440.094729] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7440.094933] after_reply+0xe2c/0x1780 [ptlrpc] [ 7440.095268] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7440.095482] ? __next_timer_interrupt+0x160/0x160 [ 7440.095491] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7440.095702] ? do_wait_intr+0xf0/0xf0 [ 7440.095712] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7440.095987] kthread+0x1d1/0x200 [ 7440.095992] ? set_kthread_struct+0x70/0x70 [ 7440.095997] ret_from_fork+0x1f/0x30 [ 7440.283333] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7440.312138] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7440.319578] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7440.323550] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7440.326735] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7440.335420] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7440.340423] RAX: 0000000000007b08 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7440.346427] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 7440.352239] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7440.355534] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 7440.363674] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 7440.373704] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 7440.386445] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7440.394398] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 7440.399626] Call Trace: [ 7440.401231] [ 7440.401978] ? show_regs.cold.9+0x22/0x2f [ 7440.403411] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7440.405422] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7440.407113] ? watchdog+0x40/0x40 [ 7440.409558] ? hrtimer_interrupt+0x128/0x2c0 [ 7440.412030] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7440.414592] ? apic_timer_interrupt+0xf/0x20 [ 7440.417283] [ 7440.418597] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7440.423524] do_raw_spin_lock+0x177/0x190 [ 7440.426532] _raw_spin_lock+0x21/0x30 [ 7440.428553] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7440.431471] ? do_raw_spin_unlock+0x75/0x190 [ 7440.436440] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7440.441970] ? do_wait_intr+0xf0/0xf0 [ 7440.443576] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7440.449400] mdc_close+0x53b/0xd70 [mdc] [ 7440.452727] lmv_close+0x18d/0x490 [lmv] [ 7440.454556] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7440.458304] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7440.464587] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7440.467933] ll_md_close+0x274/0x800 [lustre] [ 7440.472264] ll_file_release+0x128/0xc50 [lustre] [ 7440.475756] __fput+0x10d/0x440 [ 7440.478403] ____fput+0x12/0x20 [ 7440.480860] task_work_run+0xc6/0x110 [ 7440.482233] exit_to_usermode_loop+0x1dd/0x1f0 [ 7440.489855] do_syscall_64+0x3d6/0x440 [ 7440.494089] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7440.496729] RIP: 0033:0x7f151bee3cd8 [ 7440.500709] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7440.519783] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7440.527202] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 7440.534156] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 7440.536802] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7440.543337] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7440.550065] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7452.058101] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 7452.070241] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7452.108036] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7452.122841] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7452.128110] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7452.133489] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7452.141014] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7452.155696] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7452.168420] RAX: 0000000000007f0a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7452.181329] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 7452.185995] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 7452.192507] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 7452.196278] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 7452.207444] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 7452.213947] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7452.216655] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7452.219626] Call Trace: [ 7452.220875] [ 7452.221643] ? show_regs.cold.9+0x22/0x2f [ 7452.223156] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7452.226157] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7452.229831] ? watchdog+0x40/0x40 [ 7452.232168] ? hrtimer_interrupt+0x128/0x2c0 [ 7452.235360] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7452.239914] ? apic_timer_interrupt+0xf/0x20 [ 7452.241461] [ 7452.246222] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7452.250443] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7452.254330] do_raw_spin_lock+0x177/0x190 [ 7452.257361] _raw_spin_lock+0x21/0x30 [ 7452.259953] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7452.274097] process_one_work+0x2c8/0x700 [ 7452.278834] worker_thread+0x296/0x6e0 [ 7452.281743] ? rescuer_thread+0x570/0x570 [ 7452.287386] kthread+0x1d1/0x200 [ 7452.293428] ? set_kthread_struct+0x70/0x70 [ 7452.295690] ret_from_fork+0x1f/0x30 [ 7456.055602] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 7456.066341] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7456.122640] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7456.142557] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7456.154104] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7456.167317] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7456.194336] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7456.207705] RAX: 000000000000068b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7456.213045] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 7456.217144] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 7456.221198] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 7456.228778] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 7456.240323] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 7456.254107] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7456.259383] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 7456.261794] Call Trace: [ 7456.269619] [ 7456.270514] ? show_regs.cold.9+0x22/0x2f [ 7456.278480] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7456.281619] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7456.287825] ? watchdog+0x40/0x40 [ 7456.289946] ? hrtimer_interrupt+0x128/0x2c0 [ 7456.292434] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7456.298899] ? apic_timer_interrupt+0xf/0x20 [ 7456.300032] [ 7456.300560] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7456.302077] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7456.303951] do_raw_spin_lock+0x177/0x190 [ 7456.305583] _raw_spin_lock+0x21/0x30 [ 7456.307399] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7456.310536] ? do_raw_spin_unlock+0x75/0x190 [ 7456.312307] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7456.314836] ? do_wait_intr+0xf0/0xf0 [ 7456.316037] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7456.320949] mdc_reint+0x2d/0x110 [mdc] [ 7456.322591] mdc_unlink+0x658/0x870 [mdc] [ 7456.324525] lmv_unlink+0x34d/0xbd0 [lmv] [ 7456.326914] ll_unlink+0x24d/0x7f0 [lustre] [ 7456.331051] vfs_unlink+0x202/0x330 [ 7456.331997] do_unlinkat+0x2fa/0x460 [ 7456.333205] __x64_sys_unlink+0x37/0x50 [ 7456.334913] do_syscall_64+0xc1/0x440 [ 7456.338391] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7456.340669] RIP: 0033:0x7f151bdee10b [ 7456.342529] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7456.358093] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7456.360998] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 7456.365027] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 7456.369315] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7456.372499] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7456.378227] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7462.380826] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5531s! [ 7462.384489] Showing busy workqueues and worker pools: [ 7462.385700] workqueue events: flags=0x0 [ 7462.391071] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7462.405323] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 7462.414523] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7462.423045] pending: kfree_rcu_monitor [ 7462.426243] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7462.437171] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 7462.451264] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7462.457466] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 7462.462478] workqueue events_long: flags=0x0 [ 7462.463495] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7462.466086] pending: lu_objects_flush [obdclass] [ 7462.470108] workqueue events_unbound: flags=0x2 [ 7462.476137] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7462.483362] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7462.492348] workqueue events_power_efficient: flags=0x80 [ 7462.498381] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7462.504072] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 7462.511017] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7462.516479] pending: neigh_periodic_work [ 7462.520295] workqueue mm_percpu_wq: flags=0x8 [ 7462.521715] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7462.534971] pending: vmstat_update, lru_add_drain_per_cpu [ 7462.538102] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7462.545454] pending: vmstat_update [ 7462.551405] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7462.557803] pending: lru_add_drain_per_cpu [ 7462.561447] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7462.568114] pending: lru_add_drain_per_cpu BAR(48) [ 7462.571776] workqueue writeback: flags=0x4a [ 7462.575455] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 7462.585465] pending: wb_workfn [ 7462.591416] workqueue xprtiod: flags=0xa [ 7462.597127] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7462.602021] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7462.613148] workqueue ipv6_addrconf: flags=0x40008 [ 7462.614744] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7462.623248] pending: addrconf_verify_work [ 7462.630354] workqueue knbd0-recv: flags=0x1a [ 7462.638594] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7462.648655] in-flight: 658:recv_work [ 7462.654091] workqueue ptlrpc_pinger: flags=0x2 [ 7462.656343] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7462.665302] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7462.673561] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5532s workers=4 idle: 32984 37 8 [ 7462.680166] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7447s workers=2 idle: 661 [ 7468.055424] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 7468.057543] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 7468.060223] Modules linked in: [ 7468.065379] Modules linked in: [ 7468.065383] lustre(O) osp(O) [ 7468.067012] lustre(O) [ 7468.068356] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7468.068427] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7468.068435] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7468.068438] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7468.068452] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7468.068456] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7468.068461] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 7468.068463] RDX: 00000000000029cd RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 7468.068466] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 7468.068468] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 7468.068471] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7468.068474] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 7468.068476] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7468.068478] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7468.068486] Call Trace: [ 7468.068492] [ 7468.068497] ? show_regs.cold.9+0x22/0x2f [ 7468.068506] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7468.068512] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7468.068516] ? watchdog+0x40/0x40 [ 7468.068520] ? hrtimer_interrupt+0x128/0x2c0 [ 7468.068524] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7468.068532] ? apic_timer_interrupt+0xf/0x20 [ 7468.068654] [ 7468.068660] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7468.068665] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7468.068670] do_raw_spin_lock+0x177/0x190 [ 7468.068675] _raw_spin_lock+0x21/0x30 [ 7468.068681] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7468.072140] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7468.072215] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7468.072222] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7468.072225] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7468.072239] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7468.072243] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7468.072248] RAX: 00000000000027f1 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7468.072251] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 7468.076489] obd_mod_free+0x46/0x140 [ptlrpc] [ 7468.077009] mdc_commit_open+0x91/0xb0 [mdc] [ 7468.077056] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7468.077258] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7468.077459] after_reply+0xe2c/0x1780 [ptlrpc] [ 7468.077873] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7468.078081] ? __next_timer_interrupt+0x160/0x160 [ 7468.078091] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7468.098211] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7468.098220] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 7468.098223] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 7468.098226] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 7468.098229] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7468.098232] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 7468.098241] Call Trace: [ 7468.098247] [ 7468.098339] ? show_regs.cold.9+0x22/0x2f [ 7468.098352] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7468.098358] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7468.098362] ? watchdog+0x40/0x40 [ 7468.098370] ? hrtimer_interrupt+0x128/0x2c0 [ 7468.098374] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7468.098382] ? apic_timer_interrupt+0xf/0x20 [ 7468.098387] [ 7468.098390] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7468.098405] do_raw_spin_lock+0x177/0x190 [ 7468.098414] _raw_spin_lock+0x21/0x30 [ 7468.098419] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7468.098922] ? do_raw_spin_unlock+0x75/0x190 [ 7468.098933] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7468.099338] ? do_wait_intr+0xf0/0xf0 [ 7468.099346] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7468.099585] mdc_close+0x53b/0xd70 [mdc] [ 7468.099690] lmv_close+0x18d/0x490 [lmv] [ 7468.099786] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7468.099957] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7468.100070] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7468.100157] ll_md_close+0x274/0x800 [lustre] [ 7468.100245] ll_file_release+0x128/0xc50 [lustre] [ 7468.100331] __fput+0x10d/0x440 [ 7468.100341] ____fput+0x12/0x20 [ 7468.100345] task_work_run+0xc6/0x110 [ 7468.100352] exit_to_usermode_loop+0x1dd/0x1f0 [ 7468.100443] do_syscall_64+0x3d6/0x440 [ 7468.100447] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7468.100455] RIP: 0033:0x7f151bee3cd8 [ 7468.100462] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7468.100466] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7468.100471] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 7468.100473] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 7468.100475] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7468.100477] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7468.100479] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7468.482989] ? do_wait_intr+0xf0/0xf0 [ 7468.485516] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7468.489200] kthread+0x1d1/0x200 [ 7468.490378] ? set_kthread_struct+0x70/0x70 [ 7468.492282] ret_from_fork+0x1f/0x30 [ 7480.059954] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 7480.068011] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7480.103026] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7480.115679] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7480.124128] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7480.130363] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7480.132548] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7480.172222] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7480.181956] RAX: 000000000000628e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7480.185179] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 7480.198131] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 7480.205595] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 7480.211101] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 7480.214334] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 7480.218073] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7480.220649] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7480.224020] Call Trace: [ 7480.228109] [ 7480.230038] ? show_regs.cold.9+0x22/0x2f [ 7480.233144] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7480.238096] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7480.244084] ? watchdog+0x40/0x40 [ 7480.248731] ? hrtimer_interrupt+0x128/0x2c0 [ 7480.252250] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7480.264872] ? apic_timer_interrupt+0xf/0x20 [ 7480.273775] [ 7480.276832] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7480.286617] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7480.294880] do_raw_spin_lock+0x177/0x190 [ 7480.299082] _raw_spin_lock+0x21/0x30 [ 7480.302959] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7480.310429] process_one_work+0x2c8/0x700 [ 7480.312241] worker_thread+0x296/0x6e0 [ 7480.313821] ? rescuer_thread+0x570/0x570 [ 7480.316385] kthread+0x1d1/0x200 [ 7480.318603] ? set_kthread_struct+0x70/0x70 [ 7480.320447] ret_from_fork+0x1f/0x30 [ 7488.054220] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 7488.064046] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7488.102635] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7488.108884] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7488.119781] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7488.122778] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7488.139356] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7488.143977] RAX: 0000000000007f7f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7488.147556] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 7488.151198] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 7488.154524] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 7488.157792] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 7488.162498] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 7488.167070] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7488.169694] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 7488.172925] Call Trace: [ 7488.174143] [ 7488.175157] ? show_regs.cold.9+0x22/0x2f [ 7488.177202] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7488.179449] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7488.181584] ? watchdog+0x40/0x40 [ 7488.183230] ? hrtimer_interrupt+0x128/0x2c0 [ 7488.185256] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7488.187937] ? apic_timer_interrupt+0xf/0x20 [ 7488.190074] [ 7488.191156] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7488.193719] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7488.196350] do_raw_spin_lock+0x177/0x190 [ 7488.198225] _raw_spin_lock+0x21/0x30 [ 7488.199957] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7488.202247] ? do_raw_spin_unlock+0x75/0x190 [ 7488.203946] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7488.207330] ? do_wait_intr+0xf0/0xf0 [ 7488.209312] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7488.212074] mdc_reint+0x2d/0x110 [mdc] [ 7488.213191] mdc_unlink+0x658/0x870 [mdc] [ 7488.214381] lmv_unlink+0x34d/0xbd0 [lmv] [ 7488.216145] ll_unlink+0x24d/0x7f0 [lustre] [ 7488.218404] vfs_unlink+0x202/0x330 [ 7488.219940] do_unlinkat+0x2fa/0x460 [ 7488.221768] __x64_sys_unlink+0x37/0x50 [ 7488.223700] do_syscall_64+0xc1/0x440 [ 7488.225151] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7488.227447] RIP: 0033:0x7f151bdee10b [ 7488.228923] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7488.237954] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7488.241860] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 7488.245500] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 7488.249309] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7488.253271] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7488.256902] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7493.088823] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5562s! [ 7493.096723] Showing busy workqueues and worker pools: [ 7493.099833] workqueue events: flags=0x0 [ 7493.102485] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7493.106916] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 7493.112950] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7493.116090] pending: kfree_rcu_monitor [ 7493.118727] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7493.123125] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 7493.129014] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7493.133317] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 7493.138914] workqueue events_long: flags=0x0 [ 7493.141348] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7493.145866] pending: lu_objects_flush [obdclass] [ 7493.149049] workqueue events_unbound: flags=0x2 [ 7493.151271] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7493.155161] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7493.161985] workqueue events_power_efficient: flags=0x80 [ 7493.165303] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7493.169153] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 7493.173538] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7493.176888] pending: neigh_periodic_work [ 7493.179381] workqueue mm_percpu_wq: flags=0x8 [ 7493.182804] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7493.186355] pending: vmstat_update, lru_add_drain_per_cpu [ 7493.189917] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7493.195691] pending: vmstat_update [ 7493.198505] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7493.202814] pending: lru_add_drain_per_cpu [ 7493.205790] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7493.209806] pending: lru_add_drain_per_cpu BAR(48) [ 7493.212717] workqueue writeback: flags=0x4a [ 7493.216133] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 7493.220532] pending: wb_workfn [ 7493.222973] workqueue xprtiod: flags=0xa [ 7493.225720] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7493.229797] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7493.234666] workqueue ipv6_addrconf: flags=0x40008 [ 7493.237810] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7493.241591] pending: addrconf_verify_work [ 7493.243777] workqueue knbd0-recv: flags=0x1a [ 7493.246317] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7493.250393] in-flight: 658:recv_work [ 7493.252700] workqueue ptlrpc_pinger: flags=0x2 [ 7493.256837] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7493.260492] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7493.263512] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5562s workers=4 idle: 32984 37 8 [ 7493.267986] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7478s workers=2 idle: 661 [ 7496.055404] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 7496.057047] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 7496.061307] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7496.062049] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7496.062057] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7496.062060] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7496.062074] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7496.062079] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7496.062084] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 7496.062086] RDX: 00000000000024b7 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 7496.062089] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 7496.062091] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 7496.062093] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7496.062097] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 7496.062100] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7496.062102] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7496.062111] Call Trace: [ 7496.062117] [ 7496.062122] ? show_regs.cold.9+0x22/0x2f [ 7496.062131] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7496.062138] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7496.062142] ? watchdog+0x40/0x40 [ 7496.062147] ? hrtimer_interrupt+0x128/0x2c0 [ 7496.062151] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7496.062160] ? apic_timer_interrupt+0xf/0x20 [ 7496.062164] [ 7496.062167] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7496.062172] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7496.062177] do_raw_spin_lock+0x177/0x190 [ 7496.062182] _raw_spin_lock+0x21/0x30 [ 7496.062188] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7496.066565] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 [ 7496.101015] obd_mod_free+0x46/0x140 [ptlrpc] [ 7496.101287] mdc_commit_open+0x91/0xb0 [mdc] [ 7496.101323] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7496.101721] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7496.101929] after_reply+0xe2c/0x1780 [ptlrpc] [ 7496.102216] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7496.103108] ? __next_timer_interrupt+0x160/0x160 [ 7496.103121] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7496.103330] ? do_wait_intr+0xf0/0xf0 [ 7496.103515] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7496.103648] kthread+0x1d1/0x200 [ 7496.103653] ? set_kthread_struct+0x70/0x70 [ 7496.103656] ret_from_fork+0x1f/0x30 [ 7496.248515] auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7496.261088] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7496.266911] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7496.273550] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7496.276489] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7496.292139] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7496.296207] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7496.299157] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 7496.305232] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7496.309131] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 7496.314187] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 7496.316694] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 7496.322305] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7496.325830] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 7496.336852] Call Trace: [ 7496.337804] [ 7496.338685] ? show_regs.cold.9+0x22/0x2f [ 7496.341236] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7496.344575] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7496.346214] ? watchdog+0x40/0x40 [ 7496.353205] ? hrtimer_interrupt+0x128/0x2c0 [ 7496.357026] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7496.358306] ? apic_timer_interrupt+0xf/0x20 [ 7496.360635] [ 7496.363090] ? native_safe_halt+0x2e/0x50 [ 7496.368124] kvm_wait+0xcd/0xe0 [ 7496.369906] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7496.373461] do_raw_spin_lock+0x177/0x190 [ 7496.377098] _raw_spin_lock+0x21/0x30 [ 7496.378233] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7496.379827] ? do_raw_spin_unlock+0x75/0x190 [ 7496.380836] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7496.383823] ? do_wait_intr+0xf0/0xf0 [ 7496.386426] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7496.389702] mdc_close+0x53b/0xd70 [mdc] [ 7496.394463] lmv_close+0x18d/0x490 [lmv] [ 7496.396794] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7496.400823] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7496.406848] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7496.408713] ll_md_close+0x274/0x800 [lustre] [ 7496.411567] ll_file_release+0x128/0xc50 [lustre] [ 7496.414860] __fput+0x10d/0x440 [ 7496.417257] ____fput+0x12/0x20 [ 7496.421025] task_work_run+0xc6/0x110 [ 7496.423297] exit_to_usermode_loop+0x1dd/0x1f0 [ 7496.425352] do_syscall_64+0x3d6/0x440 [ 7496.427774] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7496.430729] RIP: 0033:0x7f151bee3cd8 [ 7496.433456] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7496.455619] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7496.466772] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 7496.472173] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 7496.487461] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7496.496542] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7496.508353] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7508.056025] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7508.062656] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7508.112096] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7508.120234] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7508.128781] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7508.133335] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7508.136580] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7508.153748] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7508.159453] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7508.166146] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 7508.173432] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 7508.180559] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 7508.185511] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 7508.192007] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 7508.197693] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7508.202162] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7508.206874] Call Trace: [ 7508.208909] [ 7508.212224] ? show_regs.cold.9+0x22/0x2f [ 7508.214850] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7508.216893] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7508.218292] ? watchdog+0x40/0x40 [ 7508.219273] ? hrtimer_interrupt+0x128/0x2c0 [ 7508.227185] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7508.229480] ? apic_timer_interrupt+0xf/0x20 [ 7508.232691] [ 7508.233506] ? native_safe_halt+0x2e/0x50 [ 7508.235170] kvm_wait+0xcd/0xe0 [ 7508.236769] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7508.238791] do_raw_spin_lock+0x177/0x190 [ 7508.240063] _raw_spin_lock+0x21/0x30 [ 7508.241301] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7508.247728] process_one_work+0x2c8/0x700 [ 7508.250935] worker_thread+0x296/0x6e0 [ 7508.252714] ? rescuer_thread+0x570/0x570 [ 7508.255162] kthread+0x1d1/0x200 [ 7508.258289] ? set_kthread_struct+0x70/0x70 [ 7508.261179] ret_from_fork+0x1f/0x30 [ 7520.052513] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 7520.059038] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7520.085934] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7520.098986] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7520.113868] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 7520.124989] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 7520.148229] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7520.156056] RAX: 0000000000006085 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7520.161745] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 7520.168610] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 7520.173398] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 7520.179291] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 7520.182959] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 7520.190378] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7520.201465] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 7520.207339] Call Trace: [ 7520.210479] [ 7520.211336] ? show_regs.cold.9+0x22/0x2f [ 7520.213485] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7520.218333] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7520.221033] ? watchdog+0x40/0x40 [ 7520.224847] ? hrtimer_interrupt+0x128/0x2c0 [ 7520.227085] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7520.231490] ? apic_timer_interrupt+0xf/0x20 [ 7520.233845] [ 7520.235337] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 7520.240674] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7520.244505] do_raw_spin_lock+0x177/0x190 [ 7520.246819] _raw_spin_lock+0x21/0x30 [ 7520.255449] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7520.262191] ? do_raw_spin_unlock+0x75/0x190 [ 7520.265675] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7520.273648] ? do_wait_intr+0xf0/0xf0 [ 7520.274979] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7520.279526] mdc_reint+0x2d/0x110 [mdc] [ 7520.283330] mdc_unlink+0x658/0x870 [mdc] [ 7520.286816] lmv_unlink+0x34d/0xbd0 [lmv] [ 7520.288230] ll_unlink+0x24d/0x7f0 [lustre] [ 7520.292728] vfs_unlink+0x202/0x330 [ 7520.293989] do_unlinkat+0x2fa/0x460 [ 7520.296457] __x64_sys_unlink+0x37/0x50 [ 7520.299355] do_syscall_64+0xc1/0x440 [ 7520.302591] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7520.305372] RIP: 0033:0x7f151bdee10b [ 7520.307780] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7520.319258] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7520.322836] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 7520.328695] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 7520.341219] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7520.347261] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7520.353311] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7523.300894] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5592s! [ 7523.316306] Showing busy workqueues and worker pools: [ 7523.318142] workqueue events: flags=0x0 [ 7523.322978] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7523.327511] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 7523.337263] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7523.345958] pending: kfree_rcu_monitor [ 7523.347505] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7523.356925] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 7523.365523] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7523.373582] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 7523.377657] workqueue events_long: flags=0x0 [ 7523.388748] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7523.395476] pending: lu_objects_flush [obdclass] [ 7523.400585] workqueue events_unbound: flags=0x2 [ 7523.405196] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7523.414182] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7523.427371] workqueue events_power_efficient: flags=0x80 [ 7523.435856] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7523.437347] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 7523.444755] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7523.447196] pending: neigh_periodic_work [ 7523.463422] workqueue mm_percpu_wq: flags=0x8 [ 7523.470227] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7523.472688] pending: vmstat_update, lru_add_drain_per_cpu [ 7523.485681] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7523.497773] pending: vmstat_update [ 7523.501671] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7523.514689] pending: lru_add_drain_per_cpu [ 7523.521796] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7523.527552] pending: lru_add_drain_per_cpu BAR(48) [ 7523.530599] workqueue writeback: flags=0x4a [ 7523.533792] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 7523.542456] pending: wb_workfn [ 7523.549780] workqueue xprtiod: flags=0xa [ 7523.552341] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7523.556712] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7523.565342] workqueue ipv6_addrconf: flags=0x40008 [ 7523.576923] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7523.584154] pending: addrconf_verify_work [ 7523.588841] workqueue knbd0-recv: flags=0x1a [ 7523.593841] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7523.597711] in-flight: 658:recv_work [ 7523.600437] workqueue ptlrpc_pinger: flags=0x2 [ 7523.603133] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7523.609402] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7523.613980] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5593s workers=4 idle: 32984 37 8 [ 7523.619614] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7508s workers=2 idle: 661 [ 7524.055438] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 7524.060304] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 7524.061252] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7524.061342] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7524.061352] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7524.061355] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7524.061369] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7524.061373] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7524.061377] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 7524.061383] RDX: 00000000000072fc RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 7524.061385] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 7524.061387] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 7524.061390] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7524.061393] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 7524.061397] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7524.061399] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7524.061407] Call Trace: [ 7524.061412] [ 7524.061418] ? show_regs.cold.9+0x22/0x2f [ 7524.061429] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7524.061436] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7524.061441] ? watchdog+0x40/0x40 [ 7524.061445] ? hrtimer_interrupt+0x128/0x2c0 [ 7524.061449] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7524.061456] ? apic_timer_interrupt+0xf/0x20 [ 7524.061460] [ 7524.061463] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7524.061467] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7524.061471] do_raw_spin_lock+0x177/0x190 [ 7524.061489] _raw_spin_lock+0x21/0x30 [ 7524.061495] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7524.062131] obd_mod_free+0x46/0x140 [ptlrpc] [ 7524.062543] mdc_commit_open+0x91/0xb0 [mdc] [ 7524.062580] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7524.062799] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7524.063162] after_reply+0xe2c/0x1780 [ptlrpc] [ 7524.063531] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7524.063793] ? __next_timer_interrupt+0x160/0x160 [ 7524.063803] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7524.064104] ? do_wait_intr+0xf0/0xf0 [ 7524.064119] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7524.064467] kthread+0x1d1/0x200 [ 7524.064538] ? set_kthread_struct+0x70/0x70 [ 7524.064543] ret_from_fork+0x1f/0x30 [ 7524.277859] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7524.314855] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7524.319627] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7524.323500] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7524.326039] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7524.333036] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7524.335988] RAX: 0000000000000a81 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7524.339418] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 7524.342417] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7524.346314] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 7524.349345] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 7524.351895] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 7524.355360] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7524.357531] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 7524.360150] Call Trace: [ 7524.361302] [ 7524.361818] ? show_regs.cold.9+0x22/0x2f [ 7524.362867] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7524.364079] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7524.365495] ? watchdog+0x40/0x40 [ 7524.367777] ? hrtimer_interrupt+0x128/0x2c0 [ 7524.369105] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7524.370928] ? apic_timer_interrupt+0xf/0x20 [ 7524.372538] [ 7524.373511] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7524.375671] do_raw_spin_lock+0x177/0x190 [ 7524.377387] _raw_spin_lock+0x21/0x30 [ 7524.378907] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7524.381335] ? do_raw_spin_unlock+0x75/0x190 [ 7524.384019] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7524.386998] ? do_wait_intr+0xf0/0xf0 [ 7524.389485] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7524.392372] mdc_close+0x53b/0xd70 [mdc] [ 7524.394618] lmv_close+0x18d/0x490 [lmv] [ 7524.397395] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7524.399895] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7524.402554] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7524.405264] ll_md_close+0x274/0x800 [lustre] [ 7524.407205] ll_file_release+0x128/0xc50 [lustre] [ 7524.409586] __fput+0x10d/0x440 [ 7524.411036] ____fput+0x12/0x20 [ 7524.412743] task_work_run+0xc6/0x110 [ 7524.413813] exit_to_usermode_loop+0x1dd/0x1f0 [ 7524.415173] do_syscall_64+0x3d6/0x440 [ 7524.416430] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7524.419698] RIP: 0033:0x7f151bee3cd8 [ 7524.422100] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7524.432693] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7524.436776] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 7524.440833] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 7524.447869] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7524.455302] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7524.459530] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7536.056095] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7536.058807] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7536.077905] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7536.082034] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7536.084929] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7536.087243] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7536.088831] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7536.094034] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7536.095823] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7536.097545] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 7536.099439] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 7536.101163] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 7536.102798] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 7536.104790] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 7536.106809] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7536.108451] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7536.110135] Call Trace: [ 7536.110643] [ 7536.112338] ? show_regs.cold.9+0x22/0x2f [ 7536.113332] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7536.115352] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7536.119675] ? watchdog+0x40/0x40 [ 7536.120451] ? hrtimer_interrupt+0x128/0x2c0 [ 7536.121502] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7536.123660] ? apic_timer_interrupt+0xf/0x20 [ 7536.126933] [ 7536.127457] ? native_safe_halt+0x2e/0x50 [ 7536.128425] kvm_wait+0xcd/0xe0 [ 7536.130327] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7536.135689] do_raw_spin_lock+0x177/0x190 [ 7536.138902] _raw_spin_lock+0x21/0x30 [ 7536.142504] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7536.146492] process_one_work+0x2c8/0x700 [ 7536.148000] worker_thread+0x296/0x6e0 [ 7536.151470] ? rescuer_thread+0x570/0x570 [ 7536.155284] kthread+0x1d1/0x200 [ 7536.157742] ? set_kthread_struct+0x70/0x70 [ 7536.160649] ret_from_fork+0x1f/0x30 [ 7548.053427] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 7548.064754] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7548.122451] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7548.132995] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7548.150582] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7548.161062] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7548.181551] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7548.195178] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7548.197614] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 7548.210382] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 7548.231013] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 7548.236008] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 7548.242581] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 7548.247153] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7548.256691] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 7548.260882] Call Trace: [ 7548.261733] [ 7548.262288] ? show_regs.cold.9+0x22/0x2f [ 7548.266543] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7548.270197] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7548.274724] ? watchdog+0x40/0x40 [ 7548.282251] ? hrtimer_interrupt+0x128/0x2c0 [ 7548.286202] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7548.300229] ? apic_timer_interrupt+0xf/0x20 [ 7548.304440] [ 7548.305298] ? native_safe_halt+0x2e/0x50 [ 7548.306706] kvm_wait+0xcd/0xe0 [ 7548.309031] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7548.312130] do_raw_spin_lock+0x177/0x190 [ 7548.314538] _raw_spin_lock+0x21/0x30 [ 7548.317110] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7548.321237] ? do_raw_spin_unlock+0x75/0x190 [ 7548.328286] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7548.333505] ? do_wait_intr+0xf0/0xf0 [ 7548.337351] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7548.345403] mdc_reint+0x2d/0x110 [mdc] [ 7548.350778] mdc_unlink+0x658/0x870 [mdc] [ 7548.354363] lmv_unlink+0x34d/0xbd0 [lmv] [ 7548.361159] ll_unlink+0x24d/0x7f0 [lustre] [ 7548.367588] vfs_unlink+0x202/0x330 [ 7548.370480] do_unlinkat+0x2fa/0x460 [ 7548.374527] __x64_sys_unlink+0x37/0x50 [ 7548.377513] do_syscall_64+0xc1/0x440 [ 7548.380412] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7548.386735] RIP: 0033:0x7f151bdee10b [ 7548.388593] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7548.404618] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7548.407610] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 7548.410168] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 7548.413529] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7548.416909] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7548.419616] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7552.058477] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 7552.059336] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 7552.070741] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7552.070900] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7552.070908] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7552.070911] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7552.070932] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7552.070937] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7552.070941] RAX: 0000000000006a3c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7552.070944] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 7552.070946] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7552.070948] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 7552.070951] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 7552.070954] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 7552.070958] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7552.070960] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 7552.070968] Call Trace: [ 7552.070975] [ 7552.070980] ? show_regs.cold.9+0x22/0x2f [ 7552.072456] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7552.072463] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7552.073711] ? watchdog+0x40/0x40 [ 7552.073720] ? hrtimer_interrupt+0x128/0x2c0 [ 7552.073724] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7552.073732] ? apic_timer_interrupt+0xf/0x20 [ 7552.091435] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7552.091518] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7552.091527] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7552.091530] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7552.091545] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7552.091550] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7552.091555] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 7552.091558] RDX: 0000000000005ef2 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 7552.091561] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 7552.091563] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 7552.091566] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7552.091569] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 7552.091572] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7552.091575] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7552.091584] Call Trace: [ 7552.091590] [ 7552.091596] ? show_regs.cold.9+0x22/0x2f [ 7552.091608] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7552.091617] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7552.091621] ? watchdog+0x40/0x40 [ 7552.091626] ? hrtimer_interrupt+0x128/0x2c0 [ 7552.091631] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7552.091641] ? apic_timer_interrupt+0xf/0x20 [ 7552.091647] [ 7552.091650] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7552.091656] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7552.091662] do_raw_spin_lock+0x177/0x190 [ 7552.091667] _raw_spin_lock+0x21/0x30 [ 7552.091673] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7552.091984] obd_mod_free+0x46/0x140 [ptlrpc] [ 7552.092185] mdc_commit_open+0x91/0xb0 [mdc] [ 7552.092223] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7552.092423] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7552.092620] after_reply+0xe2c/0x1780 [ptlrpc] [ 7552.092837] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7552.093054] ? __next_timer_interrupt+0x160/0x160 [ 7552.093065] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7552.093264] ? do_wait_intr+0xf0/0xf0 [ 7552.093273] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7552.093475] kthread+0x1d1/0x200 [ 7552.093481] ? set_kthread_struct+0x70/0x70 [ 7552.093484] ret_from_fork+0x1f/0x30 [ 7552.521881] [ 7552.523080] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7552.528818] do_raw_spin_lock+0x177/0x190 [ 7552.530850] _raw_spin_lock+0x21/0x30 [ 7552.535160] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7552.541440] ? do_raw_spin_unlock+0x75/0x190 [ 7552.543434] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7552.548672] ? do_wait_intr+0xf0/0xf0 [ 7552.551822] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7552.555452] mdc_close+0x53b/0xd70 [mdc] [ 7552.558788] lmv_close+0x18d/0x490 [lmv] [ 7552.561389] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7552.564478] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7552.570469] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7552.573728] ll_md_close+0x274/0x800 [lustre] [ 7552.576902] ll_file_release+0x128/0xc50 [lustre] [ 7552.579091] __fput+0x10d/0x440 [ 7552.583009] ____fput+0x12/0x20 [ 7552.584884] task_work_run+0xc6/0x110 [ 7552.587262] exit_to_usermode_loop+0x1dd/0x1f0 [ 7552.591714] do_syscall_64+0x3d6/0x440 [ 7552.593645] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7552.597522] RIP: 0033:0x7f151bee3cd8 [ 7552.600541] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7552.612640] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7552.618035] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 7552.622258] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 7552.627244] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7552.636241] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7552.641406] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7554.017122] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5623s! [ 7554.023059] Showing busy workqueues and worker pools: [ 7554.025901] workqueue events: flags=0x0 [ 7554.028353] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7554.032281] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 7554.037165] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7554.041400] pending: kfree_rcu_monitor [ 7554.044059] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7554.047841] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 7554.054918] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7554.059309] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 7554.063683] workqueue events_long: flags=0x0 [ 7554.066651] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7554.070548] pending: lu_objects_flush [obdclass] [ 7554.074029] workqueue events_unbound: flags=0x2 [ 7554.076453] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7554.080238] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7554.085731] workqueue events_power_efficient: flags=0x80 [ 7554.088658] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7554.092656] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 7554.096359] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7554.102229] pending: neigh_periodic_work [ 7554.105269] workqueue mm_percpu_wq: flags=0x8 [ 7554.108927] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7554.112380] pending: vmstat_update, lru_add_drain_per_cpu [ 7554.115669] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7554.119928] pending: vmstat_update [ 7554.121668] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7554.125354] pending: lru_add_drain_per_cpu [ 7554.127704] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7554.131635] pending: lru_add_drain_per_cpu BAR(48) [ 7554.134163] workqueue writeback: flags=0x4a [ 7554.137059] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 7554.140780] pending: wb_workfn [ 7554.142109] workqueue xprtiod: flags=0xa [ 7554.144221] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7554.147112] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7554.151635] workqueue ipv6_addrconf: flags=0x40008 [ 7554.154459] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7554.158838] pending: addrconf_verify_work [ 7554.160475] workqueue knbd0-recv: flags=0x1a [ 7554.162807] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7554.166833] in-flight: 658:recv_work [ 7554.169149] workqueue ptlrpc_pinger: flags=0x2 [ 7554.171546] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7554.176635] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7554.184744] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5623s workers=4 idle: 32984 37 8 [ 7554.192394] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7538s workers=2 idle: 661 [ 7564.057222] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7564.062610] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7564.087052] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7564.091128] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7564.093847] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7564.097219] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7564.100421] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7564.114284] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7564.120501] RAX: 0000000000002396 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7564.131363] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 7564.137817] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 7564.141804] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 7564.146418] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 7564.156409] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 7564.162013] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7564.165891] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7564.170407] Call Trace: [ 7564.171956] [ 7564.177426] ? show_regs.cold.9+0x22/0x2f [ 7564.180667] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7564.185260] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7564.186548] ? watchdog+0x40/0x40 [ 7564.192033] ? hrtimer_interrupt+0x128/0x2c0 [ 7564.194936] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7564.197896] ? apic_timer_interrupt+0xf/0x20 [ 7564.200761] [ 7564.201631] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7564.205170] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7564.208437] do_raw_spin_lock+0x177/0x190 [ 7564.210322] _raw_spin_lock+0x21/0x30 [ 7564.212235] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7564.215750] process_one_work+0x2c8/0x700 [ 7564.218990] worker_thread+0x296/0x6e0 [ 7564.220633] ? rescuer_thread+0x570/0x570 [ 7564.223128] kthread+0x1d1/0x200 [ 7564.224891] ? set_kthread_struct+0x70/0x70 [ 7564.227228] ret_from_fork+0x1f/0x30 [ 7564.771177] rcu: INFO: rcu_sched self-detected stall on CPU [ 7564.774795] rcu: 1-...!: (3190045 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 7564.783796] (t=5640124 jiffies g=249657 q=4022) [ 7564.788574] rcu: rcu_sched kthread starved for 5635076 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 7564.796514] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7564.801353] rcu: RCU grace-period kthread stack dump: [ 7564.803367] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7564.809332] Call Trace: [ 7564.810234] __schedule+0x351/0xcb0 [ 7564.815231] schedule+0xc0/0x180 [ 7564.816550] schedule_timeout+0xb4/0x190 [ 7564.821125] ? do_raw_spin_unlock+0x75/0x190 [ 7564.822817] ? __next_timer_interrupt+0x160/0x160 [ 7564.827543] ? prepare_to_swait_event+0xf3/0x140 [ 7564.829015] rcu_gp_kthread+0x8dd/0x15b0 [ 7564.831512] ? force_qs_rnp+0x2c0/0x2c0 [ 7564.835092] kthread+0x1d1/0x200 [ 7564.839173] ? set_kthread_struct+0x70/0x70 [ 7564.841886] ret_from_fork+0x1f/0x30 [ 7564.844531] rcu: Stack dump where RCU GP kthread last ran: [ 7564.850032] Sending NMI from CPU 1 to CPUs 0: [ 7564.861409] NMI backtrace for cpu 0 [ 7564.861418] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7564.861421] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7564.861422] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7564.861424] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7564.861426] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 [ 7564.861429] RAX: 0000000000004638 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7564.861431] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 7564.861432] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 7564.861434] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 7564.861436] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 7564.861437] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 7564.861439] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7564.861440] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 7564.861442] Call Trace: [ 7564.861442] [ 7564.861443] ? show_regs.cold.9+0x22/0x2f [ 7564.861444] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 7564.861445] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 7564.861447] ? nmi_handle+0xbd/0x210 [ 7564.861447] ? default_do_nmi+0x4b/0x1b0 [ 7564.861448] ? do_nmi+0x280/0x380 [ 7564.861449] ? end_repeat_nmi+0x16/0x1f [ 7564.861450] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7564.861451] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7564.861452] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7564.861453] [ 7564.861454] do_raw_spin_lock+0x177/0x190 [ 7564.861455] _raw_spin_lock+0x21/0x30 [ 7564.861456] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7564.861457] ? do_raw_spin_unlock+0x75/0x190 [ 7564.861458] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7564.861459] ? do_wait_intr+0xf0/0xf0 [ 7564.861460] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7564.861462] mdc_reint+0x2d/0x110 [mdc] [ 7564.861463] mdc_unlink+0x658/0x870 [mdc] [ 7564.861464] lmv_unlink+0x34d/0xbd0 [lmv] [ 7564.861465] ll_unlink+0x24d/0x7f0 [lustre] [ 7564.861466] vfs_unlink+0x202/0x330 [ 7564.861467] do_unlinkat+0x2fa/0x460 [ 7564.861468] __x64_sys_unlink+0x37/0x50 [ 7564.861469] do_syscall_64+0xc1/0x440 [ 7564.861470] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7564.861472] RIP: 0033:0x7f151bdee10b [ 7564.861473] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7564.861476] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7564.861478] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 7564.861479] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 7564.861482] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7564.861483] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7564.861485] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7564.862834] NMI backtrace for cpu 1 [ 7565.126688] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7565.135423] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7565.142365] Call Trace: [ 7565.145034] [ 7565.147365] ? dump_stack+0xbb/0x10e [ 7565.148822] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7565.150344] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7565.152284] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7565.154105] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7565.156959] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7565.158548] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 7565.165322] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 7565.169134] ? update_process_times+0x6d/0xc0 [ 7565.173329] ? tick_sched_handle.isra.18+0x27/0x90 [ 7565.178287] ? tick_sched_timer+0x53/0xb0 [ 7565.181353] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7565.185930] ? tick_sched_handle.isra.18+0x90/0x90 [ 7565.189172] ? hrtimer_interrupt+0x128/0x2c0 [ 7565.192723] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7565.196573] ? apic_timer_interrupt+0xf/0x20 [ 7565.200912] [ 7565.201930] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7565.205946] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7565.209918] ? do_raw_spin_lock+0x177/0x190 [ 7565.213063] ? _raw_spin_lock+0x21/0x30 [ 7565.216648] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7565.221004] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7565.226602] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7565.229860] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7565.233290] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7565.236380] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7565.241018] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7565.248043] ? __next_timer_interrupt+0x160/0x160 [ 7565.250790] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7565.256285] ? do_wait_intr+0xf0/0xf0 [ 7565.259339] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7565.262979] ? kthread+0x1d1/0x200 [ 7565.264492] ? set_kthread_struct+0x70/0x70 [ 7565.266831] ? ret_from_fork+0x1f/0x30 [ 7580.059840] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 7580.070171] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7580.113194] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7580.124627] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7580.133091] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7580.136906] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7580.147583] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 7580.150276] RAX: 0000000000005873 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7580.155292] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 7580.159274] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7580.163589] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 7580.168903] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 7580.172209] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 7580.175105] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7580.186422] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 7580.190062] Call Trace: [ 7580.191027] [ 7580.193228] ? show_regs.cold.9+0x22/0x2f [ 7580.195233] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7580.202841] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7580.209039] ? watchdog+0x40/0x40 [ 7580.210346] ? hrtimer_interrupt+0x128/0x2c0 [ 7580.212302] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7580.216148] ? apic_timer_interrupt+0xf/0x20 [ 7580.217875] [ 7580.220186] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7580.223056] do_raw_spin_lock+0x177/0x190 [ 7580.224918] _raw_spin_lock+0x21/0x30 [ 7580.227918] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7580.230020] ? do_raw_spin_unlock+0x75/0x190 [ 7580.234070] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7580.240169] ? do_wait_intr+0xf0/0xf0 [ 7580.241786] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7580.245642] mdc_close+0x53b/0xd70 [mdc] [ 7580.248163] lmv_close+0x18d/0x490 [lmv] [ 7580.250250] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7580.252526] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7580.255800] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7580.258234] ll_md_close+0x274/0x800 [lustre] [ 7580.262102] ll_file_release+0x128/0xc50 [lustre] [ 7580.265827] __fput+0x10d/0x440 [ 7580.267034] ____fput+0x12/0x20 [ 7580.270720] task_work_run+0xc6/0x110 [ 7580.272741] exit_to_usermode_loop+0x1dd/0x1f0 [ 7580.275264] do_syscall_64+0x3d6/0x440 [ 7580.277396] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7580.284290] RIP: 0033:0x7f151bee3cd8 [ 7580.288206] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7580.297878] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7580.306917] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 7580.310249] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 7580.313901] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7580.318446] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7580.322734] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7584.227371] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5653s! [ 7584.241360] Showing busy workqueues and worker pools: [ 7584.248611] workqueue events: flags=0x0 [ 7584.253950] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7584.261691] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 7584.270695] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7584.277307] pending: kfree_rcu_monitor [ 7584.283324] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7584.300299] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 7584.306848] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7584.310158] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 7584.315335] workqueue events_long: flags=0x0 [ 7584.317927] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7584.321226] pending: lu_objects_flush [obdclass] [ 7584.328514] workqueue events_unbound: flags=0x2 [ 7584.330222] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7584.334072] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7584.342689] workqueue events_power_efficient: flags=0x80 [ 7584.346392] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7584.352237] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 7584.354306] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7584.360845] pending: neigh_periodic_work [ 7584.363379] workqueue mm_percpu_wq: flags=0x8 [ 7584.372730] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7584.379341] pending: vmstat_update, lru_add_drain_per_cpu [ 7584.383252] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7584.390511] pending: vmstat_update [ 7584.393281] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7584.410108] pending: lru_add_drain_per_cpu [ 7584.411458] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7584.421735] pending: lru_add_drain_per_cpu BAR(48) [ 7584.425236] workqueue writeback: flags=0x4a [ 7584.433628] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 7584.442884] pending: wb_workfn [ 7584.451164] workqueue xprtiod: flags=0xa [ 7584.452408] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7584.462075] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7584.477122] workqueue ipv6_addrconf: flags=0x40008 [ 7584.478851] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7584.483402] pending: addrconf_verify_work [ 7584.488679] workqueue knbd0-recv: flags=0x1a [ 7584.492425] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7584.500277] in-flight: 658:recv_work [ 7584.505242] workqueue ptlrpc_pinger: flags=0x2 [ 7584.507383] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7584.510584] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7584.518861] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5653s workers=4 idle: 32984 37 8 [ 7584.530031] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7569s workers=2 idle: 661 [ 7592.055739] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 7592.056087] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7592.063281] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7592.063360] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7592.063368] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7592.063370] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7592.063383] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7592.063387] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7592.063390] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 7592.063392] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 7592.063394] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 7592.063396] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 7592.063398] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7592.063412] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 7592.063415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7592.063417] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7592.063424] Call Trace: [ 7592.063429] [ 7592.063435] ? show_regs.cold.9+0x22/0x2f [ 7592.063444] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7592.063451] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7592.063455] ? watchdog+0x40/0x40 [ 7592.063458] ? hrtimer_interrupt+0x128/0x2c0 [ 7592.063462] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7592.063468] ? apic_timer_interrupt+0xf/0x20 [ 7592.063472] [ 7592.063475] ? native_safe_halt+0x2e/0x50 [ 7592.063478] kvm_wait+0xcd/0xe0 [ 7592.063484] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7592.063491] do_raw_spin_lock+0x177/0x190 [ 7592.063495] _raw_spin_lock+0x21/0x30 [ 7592.063499] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7592.063785] obd_mod_free+0x46/0x140 [ptlrpc] [ 7592.063980] mdc_commit_open+0x91/0xb0 [mdc] [ 7592.064021] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7592.064217] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7592.064414] after_reply+0xe2c/0x1780 [ptlrpc] [ 7592.064616] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7592.064810] ? __next_timer_interrupt+0x160/0x160 [ 7592.064821] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7592.065022] ? do_wait_intr+0xf0/0xf0 [ 7592.065033] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7592.065196] kthread+0x1d1/0x200 [ 7592.065200] ? set_kthread_struct+0x70/0x70 [ 7592.065202] ret_from_fork+0x1f/0x30 [ 7592.287116] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7592.312014] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7592.317974] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7592.321325] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7592.323989] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7592.325759] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7592.333301] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7592.339677] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7592.345839] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 7592.350366] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 7592.355310] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 7592.360215] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 7592.365287] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 7592.371989] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7592.375478] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7592.379114] Call Trace: [ 7592.380496] [ 7592.381447] ? show_regs.cold.9+0x22/0x2f [ 7592.383409] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7592.385165] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7592.387510] ? watchdog+0x40/0x40 [ 7592.388837] ? hrtimer_interrupt+0x128/0x2c0 [ 7592.390932] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7592.392963] ? apic_timer_interrupt+0xf/0x20 [ 7592.394885] [ 7592.395849] ? native_safe_halt+0x2e/0x50 [ 7592.397643] kvm_wait+0xcd/0xe0 [ 7592.398772] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7592.401358] do_raw_spin_lock+0x177/0x190 [ 7592.403255] _raw_spin_lock+0x21/0x30 [ 7592.404270] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7592.407613] process_one_work+0x2c8/0x700 [ 7592.408898] worker_thread+0x296/0x6e0 [ 7592.410061] ? rescuer_thread+0x570/0x570 [ 7592.412070] kthread+0x1d1/0x200 [ 7592.413197] ? set_kthread_struct+0x70/0x70 [ 7592.415175] ret_from_fork+0x1f/0x30 [ 7608.057114] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 7608.060413] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7608.084547] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7608.089379] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7608.093546] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7608.096623] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7608.110437] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7608.114174] RAX: 000000000000659b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7608.117703] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 7608.121679] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7608.125544] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 7608.128749] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 7608.131413] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 7608.134314] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7608.136106] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 7608.138325] Call Trace: [ 7608.139128] [ 7608.139716] ? show_regs.cold.9+0x22/0x2f [ 7608.141162] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7608.142792] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7608.144518] ? watchdog+0x40/0x40 [ 7608.145667] ? hrtimer_interrupt+0x128/0x2c0 [ 7608.147219] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7608.148785] ? apic_timer_interrupt+0xf/0x20 [ 7608.150441] [ 7608.151155] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7608.152992] do_raw_spin_lock+0x177/0x190 [ 7608.154524] _raw_spin_lock+0x21/0x30 [ 7608.155755] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7608.157832] ? do_raw_spin_unlock+0x75/0x190 [ 7608.159511] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7608.161544] ? do_wait_intr+0xf0/0xf0 [ 7608.162804] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7608.164831] mdc_close+0x53b/0xd70 [mdc] [ 7608.166335] lmv_close+0x18d/0x490 [lmv] [ 7608.167640] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7608.169658] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7608.172221] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7608.174751] ll_md_close+0x274/0x800 [lustre] [ 7608.177568] ll_file_release+0x128/0xc50 [lustre] [ 7608.180320] __fput+0x10d/0x440 [ 7608.181728] ____fput+0x12/0x20 [ 7608.183350] task_work_run+0xc6/0x110 [ 7608.185446] exit_to_usermode_loop+0x1dd/0x1f0 [ 7608.187410] do_syscall_64+0x3d6/0x440 [ 7608.189427] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7608.191776] RIP: 0033:0x7f151bee3cd8 [ 7608.193687] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7608.202819] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7608.206523] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 7608.210308] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 7608.213756] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7608.217413] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7608.220827] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7612.052283] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 7612.064154] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7612.115985] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7612.125357] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7612.138961] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7612.151793] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7612.175642] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7612.184783] RAX: 00000000000061c6 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7612.196432] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 7612.205306] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 7612.211826] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 7612.217852] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 7612.222763] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 7612.229013] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7612.232371] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 7612.237761] Call Trace: [ 7612.240699] [ 7612.242878] ? show_regs.cold.9+0x22/0x2f [ 7612.247652] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7612.254136] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7612.256349] ? watchdog+0x40/0x40 [ 7612.261121] ? hrtimer_interrupt+0x128/0x2c0 [ 7612.264806] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7612.269856] ? apic_timer_interrupt+0xf/0x20 [ 7612.274754] [ 7612.276954] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7612.284999] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7612.288948] do_raw_spin_lock+0x177/0x190 [ 7612.290858] _raw_spin_lock+0x21/0x30 [ 7612.293010] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7612.298150] ? do_raw_spin_unlock+0x75/0x190 [ 7612.301545] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7612.307055] ? do_wait_intr+0xf0/0xf0 [ 7612.308797] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7612.313054] mdc_reint+0x2d/0x110 [mdc] [ 7612.316055] mdc_unlink+0x658/0x870 [mdc] [ 7612.319932] lmv_unlink+0x34d/0xbd0 [lmv] [ 7612.322672] ll_unlink+0x24d/0x7f0 [lustre] [ 7612.325635] vfs_unlink+0x202/0x330 [ 7612.330518] do_unlinkat+0x2fa/0x460 [ 7612.331968] __x64_sys_unlink+0x37/0x50 [ 7612.333854] do_syscall_64+0xc1/0x440 [ 7612.335375] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7612.338509] RIP: 0033:0x7f151bdee10b [ 7612.340032] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7612.351422] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7612.357175] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 7612.360618] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 7612.363942] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7612.367590] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7612.373205] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7614.944636] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5684s! [ 7614.950742] Showing busy workqueues and worker pools: [ 7614.952715] workqueue events: flags=0x0 [ 7614.954023] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7614.956222] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 7614.959800] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7614.963177] pending: kfree_rcu_monitor [ 7614.966035] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7614.970665] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 7614.976880] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7614.981987] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 7614.986447] workqueue events_long: flags=0x0 [ 7614.988736] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7614.991830] pending: lu_objects_flush [obdclass] [ 7614.994474] workqueue events_unbound: flags=0x2 [ 7614.997015] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7615.000693] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7615.004919] workqueue events_power_efficient: flags=0x80 [ 7615.006922] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7615.010114] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 7615.013462] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7615.015610] pending: neigh_periodic_work [ 7615.017338] workqueue mm_percpu_wq: flags=0x8 [ 7615.018928] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7615.021734] pending: vmstat_update, lru_add_drain_per_cpu [ 7615.024008] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7615.026933] pending: vmstat_update [ 7615.028800] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7615.033701] pending: lru_add_drain_per_cpu [ 7615.038645] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7615.044062] pending: lru_add_drain_per_cpu BAR(48) [ 7615.047055] workqueue writeback: flags=0x4a [ 7615.048191] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 7615.051461] pending: wb_workfn [ 7615.053131] workqueue xprtiod: flags=0xa [ 7615.055461] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7615.059674] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7615.064940] workqueue ipv6_addrconf: flags=0x40008 [ 7615.068676] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7615.072854] pending: addrconf_verify_work [ 7615.078160] workqueue knbd0-recv: flags=0x1a [ 7615.080126] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7615.087198] in-flight: 658:recv_work [ 7615.089741] workqueue ptlrpc_pinger: flags=0x2 [ 7615.091919] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7615.094743] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7615.098036] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5684s workers=4 idle: 32984 37 8 [ 7615.103474] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7599s workers=2 idle: 661 [ 7620.058204] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7620.063187] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 7620.068444] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7620.070253] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7620.070261] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7620.070276] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7620.070592] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7620.070602] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7620.070606] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7620.070611] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7620.070614] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 7620.070616] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 7620.070618] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 7620.070622] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 7620.070625] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 7620.070627] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7620.070629] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7620.070638] Call Trace: [ 7620.070644] [ 7620.071070] ? show_regs.cold.9+0x22/0x2f [ 7620.071082] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7620.071089] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7620.071093] ? watchdog+0x40/0x40 [ 7620.071101] ? hrtimer_interrupt+0x128/0x2c0 [ 7620.071106] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7620.071112] ? apic_timer_interrupt+0xf/0x20 [ 7620.071117] [ 7620.071121] ? native_safe_halt+0x2e/0x50 [ 7620.071125] kvm_wait+0xcd/0xe0 [ 7620.071132] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7620.071140] do_raw_spin_lock+0x177/0x190 [ 7620.071144] _raw_spin_lock+0x21/0x30 [ 7620.071156] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7620.071445] process_one_work+0x2c8/0x700 [ 7620.071453] worker_thread+0x296/0x6e0 [ 7620.071457] ? rescuer_thread+0x570/0x570 [ 7620.071461] kthread+0x1d1/0x200 [ 7620.071465] ? set_kthread_struct+0x70/0x70 [ 7620.071469] ret_from_fork+0x1f/0x30 [ 7620.289008] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7620.338583] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7620.346115] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7620.354008] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7620.357766] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7620.373034] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7620.376892] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 7620.381329] RDX: 0000000000003b26 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 7620.391751] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 7620.405283] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 7620.414788] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7620.428021] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 7620.439296] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7620.446373] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7620.448943] Call Trace: [ 7620.460910] [ 7620.464360] ? show_regs.cold.9+0x22/0x2f [ 7620.469683] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7620.476982] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7620.483549] ? watchdog+0x40/0x40 [ 7620.488746] ? hrtimer_interrupt+0x128/0x2c0 [ 7620.492921] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7620.498053] ? apic_timer_interrupt+0xf/0x20 [ 7620.502899] [ 7620.506782] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7620.513591] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7620.521349] do_raw_spin_lock+0x177/0x190 [ 7620.524816] _raw_spin_lock+0x21/0x30 [ 7620.530690] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7620.533092] obd_mod_free+0x46/0x140 [ptlrpc] [ 7620.540101] mdc_commit_open+0x91/0xb0 [mdc] [ 7620.548815] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7620.556847] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7620.565631] after_reply+0xe2c/0x1780 [ptlrpc] [ 7620.571079] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7620.576282] ? __next_timer_interrupt+0x160/0x160 [ 7620.581853] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7620.585230] ? do_wait_intr+0xf0/0xf0 [ 7620.592726] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7620.601546] kthread+0x1d1/0x200 [ 7620.606327] ? set_kthread_struct+0x70/0x70 [ 7620.615390] ret_from_fork+0x1f/0x30 [ 7636.060514] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 7636.067182] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7636.093690] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7636.099732] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7636.103861] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7636.106063] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7636.116638] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7636.123580] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7636.127144] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 7636.131600] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7636.138232] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 7636.141504] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 7636.145643] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 7636.149896] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7636.152806] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 7636.156471] Call Trace: [ 7636.157744] [ 7636.158705] ? show_regs.cold.9+0x22/0x2f [ 7636.161160] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7636.164125] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7636.166747] ? watchdog+0x40/0x40 [ 7636.168443] ? hrtimer_interrupt+0x128/0x2c0 [ 7636.170916] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7636.172851] ? apic_timer_interrupt+0xf/0x20 [ 7636.175318] [ 7636.176749] ? native_safe_halt+0x2e/0x50 [ 7636.179126] kvm_wait+0xcd/0xe0 [ 7636.181749] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7636.184238] do_raw_spin_lock+0x177/0x190 [ 7636.186264] _raw_spin_lock+0x21/0x30 [ 7636.188316] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7636.191572] ? do_raw_spin_unlock+0x75/0x190 [ 7636.194651] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7636.197683] ? do_wait_intr+0xf0/0xf0 [ 7636.199932] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7636.202764] mdc_close+0x53b/0xd70 [mdc] [ 7636.204842] lmv_close+0x18d/0x490 [lmv] [ 7636.206717] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7636.209735] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7636.212689] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7636.214882] ll_md_close+0x274/0x800 [lustre] [ 7636.217215] ll_file_release+0x128/0xc50 [lustre] [ 7636.219690] __fput+0x10d/0x440 [ 7636.221275] ____fput+0x12/0x20 [ 7636.222994] task_work_run+0xc6/0x110 [ 7636.224739] exit_to_usermode_loop+0x1dd/0x1f0 [ 7636.226840] do_syscall_64+0x3d6/0x440 [ 7636.228503] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7636.230756] RIP: 0033:0x7f151bee3cd8 [ 7636.231994] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7636.239797] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7636.243172] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 7636.246547] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 7636.249701] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7636.253499] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7636.256411] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7640.052519] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 7640.058236] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7640.108585] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7640.115107] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7640.120220] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7640.127463] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7640.146122] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7640.153933] RAX: 0000000000007f11 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7640.159592] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 7640.173153] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 7640.179333] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 7640.191080] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 7640.203599] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 7640.215343] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7640.221645] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 7640.230465] Call Trace: [ 7640.235892] [ 7640.236673] ? show_regs.cold.9+0x22/0x2f [ 7640.239339] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7640.243164] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7640.245981] ? watchdog+0x40/0x40 [ 7640.247693] ? hrtimer_interrupt+0x128/0x2c0 [ 7640.250438] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7640.257473] ? apic_timer_interrupt+0xf/0x20 [ 7640.265298] [ 7640.271081] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7640.277607] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7640.279493] do_raw_spin_lock+0x177/0x190 [ 7640.280797] _raw_spin_lock+0x21/0x30 [ 7640.285064] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7640.290910] ? do_raw_spin_unlock+0x75/0x190 [ 7640.293694] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7640.297912] ? do_wait_intr+0xf0/0xf0 [ 7640.303205] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7640.309490] mdc_reint+0x2d/0x110 [mdc] [ 7640.311887] mdc_unlink+0x658/0x870 [mdc] [ 7640.313254] lmv_unlink+0x34d/0xbd0 [lmv] [ 7640.318287] ll_unlink+0x24d/0x7f0 [lustre] [ 7640.321306] vfs_unlink+0x202/0x330 [ 7640.324151] do_unlinkat+0x2fa/0x460 [ 7640.330492] __x64_sys_unlink+0x37/0x50 [ 7640.334749] do_syscall_64+0xc1/0x440 [ 7640.341111] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7640.346478] RIP: 0033:0x7f151bdee10b [ 7640.351066] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7640.391811] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7640.402408] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 7640.417349] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 7640.426282] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7640.433499] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7640.440227] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7645.152771] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5714s! [ 7645.158673] Showing busy workqueues and worker pools: [ 7645.160918] workqueue events: flags=0x0 [ 7645.163242] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7645.169255] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 7645.177110] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7645.180386] pending: kfree_rcu_monitor [ 7645.183393] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7645.191227] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 7645.197352] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7645.201355] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 7645.206444] workqueue events_long: flags=0x0 [ 7645.208813] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7645.214849] pending: lu_objects_flush [obdclass] [ 7645.219397] workqueue events_unbound: flags=0x2 [ 7645.222514] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7645.225423] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7645.242822] workqueue events_power_efficient: flags=0x80 [ 7645.251664] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7645.254146] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 7645.268139] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7645.278183] pending: neigh_periodic_work [ 7645.282786] workqueue mm_percpu_wq: flags=0x8 [ 7645.287918] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7645.296328] pending: vmstat_update, lru_add_drain_per_cpu [ 7645.306631] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7645.315171] pending: vmstat_update [ 7645.318765] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7645.333155] pending: lru_add_drain_per_cpu [ 7645.336134] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7645.345498] pending: lru_add_drain_per_cpu BAR(48) [ 7645.353120] workqueue writeback: flags=0x4a [ 7645.359278] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 7645.365337] pending: wb_workfn [ 7645.368632] workqueue xprtiod: flags=0xa [ 7645.371269] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7645.377201] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7645.394333] workqueue ipv6_addrconf: flags=0x40008 [ 7645.400958] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7645.414848] pending: addrconf_verify_work [ 7645.419385] workqueue knbd0-recv: flags=0x1a [ 7645.422029] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7645.427251] in-flight: 658:recv_work [ 7645.429891] workqueue ptlrpc_pinger: flags=0x2 [ 7645.432466] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7645.438835] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7645.449175] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5714s workers=4 idle: 32984 37 8 [ 7645.460740] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7630s workers=2 idle: 661 [ 7648.055581] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 7648.056143] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7648.058514] Modules linked in: [ 7648.064902] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7648.064989] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7648.065009] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7648.065016] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7648.065508] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7648.065519] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7648.065523] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7648.065528] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7648.065530] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 7648.065532] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 7648.065534] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 7648.065536] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 7648.065539] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 7648.065542] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7648.065544] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7648.065633] Call Trace: [ 7648.065640] [ 7648.065645] ? show_regs.cold.9+0x22/0x2f [ 7648.065656] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7648.065662] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7648.065666] ? watchdog+0x40/0x40 [ 7648.065671] ? hrtimer_interrupt+0x128/0x2c0 [ 7648.065675] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7648.065683] ? apic_timer_interrupt+0xf/0x20 [ 7648.065688] [ 7648.065691] ? native_safe_halt+0x2e/0x50 [ 7648.065695] kvm_wait+0xcd/0xe0 [ 7648.065703] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7648.065712] do_raw_spin_lock+0x177/0x190 [ 7648.065716] _raw_spin_lock+0x21/0x30 [ 7648.065721] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7648.066089] process_one_work+0x2c8/0x700 [ 7648.066096] worker_thread+0x296/0x6e0 [ 7648.066101] ? rescuer_thread+0x570/0x570 [ 7648.066105] kthread+0x1d1/0x200 [ 7648.066109] ? set_kthread_struct+0x70/0x70 [ 7648.066113] ret_from_fork+0x1f/0x30 [ 7648.236788] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7648.256171] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7648.268359] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7648.273730] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7648.278145] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7648.296706] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7648.301481] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 7648.308466] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 7648.319643] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 7648.329462] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 7648.333522] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7648.347176] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 7648.356551] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7648.359142] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7648.365363] Call Trace: [ 7648.366808] [ 7648.368723] ? show_regs.cold.9+0x22/0x2f [ 7648.370821] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7648.373153] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7648.376358] ? watchdog+0x40/0x40 [ 7648.377801] ? hrtimer_interrupt+0x128/0x2c0 [ 7648.380967] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7648.383364] ? apic_timer_interrupt+0xf/0x20 [ 7648.386245] [ 7648.388834] ? native_safe_halt+0x2e/0x50 [ 7648.391766] kvm_wait+0xcd/0xe0 [ 7648.395463] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7648.398770] do_raw_spin_lock+0x177/0x190 [ 7648.402363] _raw_spin_lock+0x21/0x30 [ 7648.404158] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7648.409038] obd_mod_free+0x46/0x140 [ptlrpc] [ 7648.410875] mdc_commit_open+0x91/0xb0 [mdc] [ 7648.414323] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7648.416824] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7648.419658] after_reply+0xe2c/0x1780 [ptlrpc] [ 7648.424064] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7648.426803] ? __next_timer_interrupt+0x160/0x160 [ 7648.430791] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7648.432785] ? do_wait_intr+0xf0/0xf0 [ 7648.435488] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7648.438436] kthread+0x1d1/0x200 [ 7648.440531] ? set_kthread_struct+0x70/0x70 [ 7648.443488] ret_from_fork+0x1f/0x30 [ 7664.057012] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 7664.062872] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7664.088401] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7664.094500] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7664.099400] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30a/0x6d0 [ 7664.104907] Code: 03 00 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 [ 7664.134563] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 7664.140554] RAX: 00000000000006a8 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7664.146161] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 7664.149000] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7664.153001] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 7664.155909] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 7664.158982] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 7664.165447] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7664.172925] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 7664.179050] Call Trace: [ 7664.183627] [ 7664.187548] ? show_regs.cold.9+0x22/0x2f [ 7664.194823] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7664.201544] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7664.204470] ? watchdog+0x40/0x40 [ 7664.207190] ? hrtimer_interrupt+0x128/0x2c0 [ 7664.209559] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7664.213443] ? apic_timer_interrupt+0xf/0x20 [ 7664.216990] [ 7664.218535] ? __pv_queued_spin_lock_slowpath+0x30a/0x6d0 [ 7664.223551] do_raw_spin_lock+0x177/0x190 [ 7664.226167] _raw_spin_lock+0x21/0x30 [ 7664.228442] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7664.233146] ? do_raw_spin_unlock+0x75/0x190 [ 7664.235465] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7664.238663] ? do_wait_intr+0xf0/0xf0 [ 7664.240668] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7664.243960] mdc_close+0x53b/0xd70 [mdc] [ 7664.246048] lmv_close+0x18d/0x490 [lmv] [ 7664.248353] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7664.250559] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7664.253874] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7664.256518] ll_md_close+0x274/0x800 [lustre] [ 7664.258938] ll_file_release+0x128/0xc50 [lustre] [ 7664.263310] __fput+0x10d/0x440 [ 7664.267646] ____fput+0x12/0x20 [ 7664.269567] task_work_run+0xc6/0x110 [ 7664.271680] exit_to_usermode_loop+0x1dd/0x1f0 [ 7664.274239] do_syscall_64+0x3d6/0x440 [ 7664.279988] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7664.285549] RIP: 0033:0x7f151bee3cd8 [ 7664.287453] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7664.315167] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7664.319446] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 7664.327330] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 7664.333651] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7664.345330] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7664.351365] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7672.052011] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 7672.067376] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7672.118750] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7672.129881] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7672.136798] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7672.140409] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7672.151402] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7672.155823] RAX: 000000000000282d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7672.160388] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 7672.164581] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 7672.168780] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 7672.172459] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 7672.176478] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 7672.179699] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7672.182836] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 7672.186377] Call Trace: [ 7672.187813] [ 7672.188803] ? show_regs.cold.9+0x22/0x2f [ 7672.191376] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7672.194424] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7672.196617] ? watchdog+0x40/0x40 [ 7672.198382] ? hrtimer_interrupt+0x128/0x2c0 [ 7672.199931] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7672.201867] ? apic_timer_interrupt+0xf/0x20 [ 7672.203823] [ 7672.204735] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7672.207463] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7672.211121] do_raw_spin_lock+0x177/0x190 [ 7672.213797] _raw_spin_lock+0x21/0x30 [ 7672.215739] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7672.219004] ? do_raw_spin_unlock+0x75/0x190 [ 7672.221725] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7672.224786] ? do_wait_intr+0xf0/0xf0 [ 7672.226847] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7672.230360] mdc_reint+0x2d/0x110 [mdc] [ 7672.233111] mdc_unlink+0x658/0x870 [mdc] [ 7672.239327] lmv_unlink+0x34d/0xbd0 [lmv] [ 7672.241713] ll_unlink+0x24d/0x7f0 [lustre] [ 7672.244846] vfs_unlink+0x202/0x330 [ 7672.246826] do_unlinkat+0x2fa/0x460 [ 7672.250471] __x64_sys_unlink+0x37/0x50 [ 7672.251964] do_syscall_64+0xc1/0x440 [ 7672.253158] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7672.257385] RIP: 0033:0x7f151bdee10b [ 7672.259600] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7672.268869] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7672.272583] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 7672.275931] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 7672.279423] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7672.284778] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7672.291914] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7675.873314] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5745s! [ 7675.883403] Showing busy workqueues and worker pools: [ 7675.889678] workqueue events: flags=0x0 [ 7675.896786] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7675.904396] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 7675.913883] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7675.922784] pending: kfree_rcu_monitor [ 7675.925259] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7675.930274] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 7675.937453] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7675.941630] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 7675.945510] workqueue events_long: flags=0x0 [ 7675.948080] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7675.956888] pending: lu_objects_flush [obdclass] [ 7675.959875] workqueue events_unbound: flags=0x2 [ 7675.961875] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7675.966449] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7675.974940] workqueue events_power_efficient: flags=0x80 [ 7675.978211] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7675.982042] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 7675.985339] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7675.989272] pending: neigh_periodic_work [ 7675.992223] workqueue mm_percpu_wq: flags=0x8 [ 7675.998438] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7676.005149] pending: vmstat_update, lru_add_drain_per_cpu [ 7676.008807] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7676.012645] pending: vmstat_update [ 7676.014862] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7676.020896] pending: lru_add_drain_per_cpu [ 7676.023019] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7676.029925] pending: lru_add_drain_per_cpu BAR(48) [ 7676.036157] workqueue writeback: flags=0x4a [ 7676.039471] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 7676.046522] pending: wb_workfn [ 7676.051065] workqueue xprtiod: flags=0xa [ 7676.054464] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7676.055020] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 7676.056385] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7676.056947] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7676.057022] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7676.057027] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7676.057034] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7676.059783] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7676.059798] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7676.059803] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7676.059807] RAX: 0000000000007f12 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7676.059809] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 7676.059811] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 7676.059813] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 7676.059815] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 7676.059818] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 7676.059820] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7676.059823] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7676.059832] Call Trace: [ 7676.059838] [ 7676.059843] ? show_regs.cold.9+0x22/0x2f [ 7676.059851] pending: [ 7676.059855] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7676.059856] xs_stream_data_receive_workfn [ 7676.059865] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7676.059868] , xprt_autoclose [ 7676.059873] ? watchdog+0x40/0x40 [ 7676.059878] , xs_error_handle [ 7676.059883] ? hrtimer_interrupt+0x128/0x2c0 [ 7676.059892] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7676.059900] ? apic_timer_interrupt+0xf/0x20 [ 7676.059905] [ 7676.059908] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7676.059912] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7676.059919] do_raw_spin_lock+0x177/0x190 [ 7676.059923] _raw_spin_lock+0x21/0x30 [ 7676.059928] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7676.060184] process_one_work+0x2c8/0x700 [ 7676.060192] worker_thread+0x296/0x6e0 [ 7676.060197] ? rescuer_thread+0x570/0x570 [ 7676.060202] kthread+0x1d1/0x200 [ 7676.060207] ? set_kthread_struct+0x70/0x70 [ 7676.060222] ret_from_fork+0x1f/0x30 [ 7676.065970] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7676.066071] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7676.066079] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7676.066086] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7676.066101] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7676.066105] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7676.066110] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 7676.066113] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 7676.066116] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 7676.066118] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 7676.066121] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7676.066125] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 7676.066128] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7676.066130] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7676.066140] Call Trace: [ 7676.066146] [ 7676.066152] ? show_regs.cold.9+0x22/0x2f [ 7676.066164] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7676.066170] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7676.066175] ? watchdog+0x40/0x40 [ 7676.066179] ? hrtimer_interrupt+0x128/0x2c0 [ 7676.066183] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7676.066190] ? apic_timer_interrupt+0xf/0x20 [ 7676.066194] [ 7676.066197] ? native_safe_halt+0x2e/0x50 [ 7676.066202] kvm_wait+0xcd/0xe0 [ 7676.066210] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7676.066218] do_raw_spin_lock+0x177/0x190 [ 7676.066361] _raw_spin_lock+0x21/0x30 [ 7676.066367] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7676.066789] obd_mod_free+0x46/0x140 [ptlrpc] [ 7676.067161] mdc_commit_open+0x91/0xb0 [mdc] [ 7676.067196] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7676.076575] workqueue ipv6_addrconf: flags=0x40008 [ 7676.076643] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7676.076656] pending: addrconf_verify_work [ 7676.076672] workqueue knbd0-recv: flags=0x1a [ 7676.076674] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7676.076681] in-flight: 658:recv_work [ 7676.076691] workqueue ptlrpc_pinger: flags=0x2 [ 7676.076694] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7676.076700] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7676.077166] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5745s workers=4 idle: 32984 37 8 [ 7676.077180] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7660s workers=2 idle: 661 [ 7676.547680] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7676.549917] after_reply+0xe2c/0x1780 [ptlrpc] [ 7676.552091] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7676.554306] ? __next_timer_interrupt+0x160/0x160 [ 7676.556285] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7676.558305] ? do_wait_intr+0xf0/0xf0 [ 7676.559997] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7676.562354] kthread+0x1d1/0x200 [ 7676.563875] ? set_kthread_struct+0x70/0x70 [ 7676.565775] ret_from_fork+0x1f/0x30 [ 7692.057046] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 7692.061126] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7692.117187] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7692.124953] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7692.129446] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7692.136007] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7692.154602] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7692.162718] RAX: 0000000000005a11 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7692.175375] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 7692.180026] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7692.187033] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 7692.195290] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 7692.199742] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 7692.211118] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7692.214416] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 7692.221865] Call Trace: [ 7692.223462] [ 7692.226765] ? show_regs.cold.9+0x22/0x2f [ 7692.230505] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7692.240732] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7692.242053] ? watchdog+0x40/0x40 [ 7692.243196] ? hrtimer_interrupt+0x128/0x2c0 [ 7692.251330] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7692.253193] ? apic_timer_interrupt+0xf/0x20 [ 7692.270637] [ 7692.277764] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7692.283057] do_raw_spin_lock+0x177/0x190 [ 7692.289706] _raw_spin_lock+0x21/0x30 [ 7692.295005] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7692.302081] ? do_raw_spin_unlock+0x75/0x190 [ 7692.312184] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7692.318321] ? do_wait_intr+0xf0/0xf0 [ 7692.327399] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7692.333222] mdc_close+0x53b/0xd70 [mdc] [ 7692.339589] lmv_close+0x18d/0x490 [lmv] [ 7692.344964] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7692.347056] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7692.363593] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7692.378444] ll_md_close+0x274/0x800 [lustre] [ 7692.381928] ll_file_release+0x128/0xc50 [lustre] [ 7692.388336] __fput+0x10d/0x440 [ 7692.390156] ____fput+0x12/0x20 [ 7692.391915] task_work_run+0xc6/0x110 [ 7692.397394] exit_to_usermode_loop+0x1dd/0x1f0 [ 7692.401421] do_syscall_64+0x3d6/0x440 [ 7692.411464] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7692.420381] RIP: 0033:0x7f151bee3cd8 [ 7692.424734] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7692.448964] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7692.452373] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 7692.461475] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 7692.471167] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7692.475471] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7692.482438] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7704.053427] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 7704.055057] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 7704.059217] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7704.059232] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7704.059304] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7704.059309] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7704.059315] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7704.059617] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7704.059628] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7704.059633] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7704.059638] RAX: 0000000000006317 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7704.059641] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 7704.059643] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 7704.059646] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 7704.059648] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 7704.059652] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 7704.059655] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7704.059657] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7704.059668] Call Trace: [ 7704.059673] [ 7704.059679] ? show_regs.cold.9+0x22/0x2f [ 7704.059689] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7704.059696] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7704.059701] ? watchdog+0x40/0x40 [ 7704.059706] ? hrtimer_interrupt+0x128/0x2c0 [ 7704.059709] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7704.059718] ? apic_timer_interrupt+0xf/0x20 [ 7704.059723] [ 7704.059725] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7704.059730] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7704.059736] do_raw_spin_lock+0x177/0x190 [ 7704.059740] _raw_spin_lock+0x21/0x30 [ 7704.059746] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7704.059970] process_one_work+0x2c8/0x700 [ 7704.059978] worker_thread+0x296/0x6e0 [ 7704.059982] ? rescuer_thread+0x570/0x570 [ 7704.059986] kthread+0x1d1/0x200 [ 7704.059991] ? set_kthread_struct+0x70/0x70 [ 7704.059994] ret_from_fork+0x1f/0x30 [ 7704.063875] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs [ 7704.067374] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7704.067470] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7704.067480] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7704.067483] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7704.067501] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7704.067505] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7704.067513] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 7704.067516] RDX: 0000000000001ecb RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 7704.067518] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 7704.067521] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 7704.067524] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7704.067526] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 7704.067529] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7704.067531] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7704.067539] Call Trace: [ 7704.067549] [ 7704.067554] ? show_regs.cold.9+0x22/0x2f [ 7704.067568] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7704.067575] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7704.067624] ? watchdog+0x40/0x40 [ 7704.067634] ? hrtimer_interrupt+0x128/0x2c0 [ 7704.067639] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7704.067648] ? apic_timer_interrupt+0xf/0x20 [ 7704.067652] [ 7704.067656] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7704.067660] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7704.067665] do_raw_spin_lock+0x177/0x190 [ 7704.067669] _raw_spin_lock+0x21/0x30 [ 7704.067674] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7704.067981] obd_mod_free+0x46/0x140 [ptlrpc] [ 7704.068768] mdc_commit_open+0x91/0xb0 [mdc] [ 7704.068805] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7704.077110] ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7704.077148] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7704.077158] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7704.077161] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7704.077178] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7704.111206] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7704.111536] after_reply+0xe2c/0x1780 [ptlrpc] [ 7704.113256] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7704.113582] ? __next_timer_interrupt+0x160/0x160 [ 7704.113593] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7704.114004] ? do_wait_intr+0xf0/0xf0 [ 7704.114014] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7704.114285] kthread+0x1d1/0x200 [ 7704.114292] ? set_kthread_struct+0x70/0x70 [ 7704.114296] ret_from_fork+0x1f/0x30 [ 7704.613863] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7704.618766] RAX: 000000000000258e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7704.623106] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 7704.629919] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 7704.638631] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 7704.649939] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 7704.657230] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 7704.662008] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7704.665379] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 7704.670954] Call Trace: [ 7704.671791] [ 7704.672442] ? show_regs.cold.9+0x22/0x2f [ 7704.679150] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7704.682003] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7704.684819] ? watchdog+0x40/0x40 [ 7704.689299] ? hrtimer_interrupt+0x128/0x2c0 [ 7704.691951] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7704.697364] ? apic_timer_interrupt+0xf/0x20 [ 7704.700208] [ 7704.701036] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7704.703693] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7704.707909] do_raw_spin_lock+0x177/0x190 [ 7704.710837] _raw_spin_lock+0x21/0x30 [ 7704.713137] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7704.716134] ? do_raw_spin_unlock+0x75/0x190 [ 7704.718640] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7704.727885] ? do_wait_intr+0xf0/0xf0 [ 7704.729240] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7704.731177] mdc_reint+0x2d/0x110 [mdc] [ 7704.735739] mdc_unlink+0x658/0x870 [mdc] [ 7704.737318] lmv_unlink+0x34d/0xbd0 [lmv] [ 7704.744931] ll_unlink+0x24d/0x7f0 [lustre] [ 7704.747206] vfs_unlink+0x202/0x330 [ 7704.748898] do_unlinkat+0x2fa/0x460 [ 7704.750076] __x64_sys_unlink+0x37/0x50 [ 7704.752568] do_syscall_64+0xc1/0x440 [ 7704.754241] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7704.759535] RIP: 0033:0x7f151bdee10b [ 7704.762051] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7704.771686] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7704.777058] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 7704.782371] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 7704.789055] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7704.795729] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7704.804462] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7706.080097] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5775s! [ 7706.085693] Showing busy workqueues and worker pools: [ 7706.088667] workqueue events: flags=0x0 [ 7706.092791] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7706.096669] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 7706.106028] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7706.110250] pending: kfree_rcu_monitor [ 7706.113827] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7706.117838] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 7706.124500] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7706.128023] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 7706.133644] workqueue events_long: flags=0x0 [ 7706.135129] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7706.138581] pending: lu_objects_flush [obdclass] [ 7706.141326] workqueue events_unbound: flags=0x2 [ 7706.144282] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7706.147967] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7706.154267] workqueue events_power_efficient: flags=0x80 [ 7706.158415] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7706.161971] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 7706.165783] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7706.169267] pending: neigh_periodic_work [ 7706.171554] workqueue mm_percpu_wq: flags=0x8 [ 7706.173408] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7706.180541] pending: vmstat_update, lru_add_drain_per_cpu [ 7706.185483] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7706.188918] pending: vmstat_update [ 7706.191067] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7706.194411] pending: lru_add_drain_per_cpu [ 7706.196810] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7706.200381] pending: lru_add_drain_per_cpu BAR(48) [ 7706.203099] workqueue writeback: flags=0x4a [ 7706.205600] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 7706.209437] pending: wb_workfn [ 7706.211415] workqueue xprtiod: flags=0xa [ 7706.213516] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7706.217573] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7706.222585] workqueue ipv6_addrconf: flags=0x40008 [ 7706.225377] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7706.229474] pending: addrconf_verify_work [ 7706.232133] workqueue knbd0-recv: flags=0x1a [ 7706.234573] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7706.237162] in-flight: 658:recv_work [ 7706.239409] workqueue ptlrpc_pinger: flags=0x2 [ 7706.241732] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7706.246546] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7706.250352] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5775s workers=4 idle: 32984 37 8 [ 7706.254375] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7691s workers=2 idle: 661 [ 7720.057068] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 7720.061491] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7720.103697] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7720.109501] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7720.116307] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7720.120400] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7720.133819] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 7720.138053] RAX: 0000000000006e8e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7720.145220] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 7720.150282] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7720.153625] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 7720.157384] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 7720.161088] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 7720.165059] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7720.167772] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 7720.171102] Call Trace: [ 7720.172086] [ 7720.172947] ? show_regs.cold.9+0x22/0x2f [ 7720.174857] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7720.176669] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7720.178972] ? watchdog+0x40/0x40 [ 7720.180469] ? hrtimer_interrupt+0x128/0x2c0 [ 7720.182068] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7720.192058] ? apic_timer_interrupt+0xf/0x20 [ 7720.197024] [ 7720.198691] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7720.202675] do_raw_spin_lock+0x177/0x190 [ 7720.205110] _raw_spin_lock+0x21/0x30 [ 7720.207131] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7720.210955] ? do_raw_spin_unlock+0x75/0x190 [ 7720.214093] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7720.217155] ? do_wait_intr+0xf0/0xf0 [ 7720.219356] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7720.223031] mdc_close+0x53b/0xd70 [mdc] [ 7720.225681] lmv_close+0x18d/0x490 [lmv] [ 7720.227650] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7720.231369] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7720.235013] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7720.239992] ll_md_close+0x274/0x800 [lustre] [ 7720.242664] ll_file_release+0x128/0xc50 [lustre] [ 7720.246782] __fput+0x10d/0x440 [ 7720.247950] ____fput+0x12/0x20 [ 7720.253932] task_work_run+0xc6/0x110 [ 7720.255411] exit_to_usermode_loop+0x1dd/0x1f0 [ 7720.259795] do_syscall_64+0x3d6/0x440 [ 7720.262321] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7720.265969] RIP: 0033:0x7f151bee3cd8 [ 7720.267389] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7720.280124] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7720.286048] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 7720.290497] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 7720.294264] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7720.297738] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7720.301888] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7732.052017] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 7732.054927] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7732.062116] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 7732.062498] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 7732.062514] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7732.062595] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7732.062601] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7732.062604] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7732.062619] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7732.062622] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7732.062625] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 7732.062627] RDX: 0000000000007e85 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 7732.062629] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 7732.062631] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 7732.062632] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7732.062635] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 7732.062638] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7732.062640] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7732.062649] Call Trace: [ 7732.062654] [ 7732.062660] ? show_regs.cold.9+0x22/0x2f [ 7732.062681] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7732.062688] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7732.062692] ? watchdog+0x40/0x40 [ 7732.062696] ? hrtimer_interrupt+0x128/0x2c0 [ 7732.062700] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7732.062708] ? apic_timer_interrupt+0xf/0x20 [ 7732.062713] [ 7732.062717] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7732.062721] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7732.062726] do_raw_spin_lock+0x177/0x190 [ 7732.062731] _raw_spin_lock+0x21/0x30 [ 7732.062737] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7732.062969] obd_mod_free+0x46/0x140 [ptlrpc] [ 7732.063092] mdc_commit_open+0x91/0xb0 [mdc] [ 7732.063138] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7732.063334] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7732.063817] after_reply+0xe2c/0x1780 [ptlrpc] [ 7732.064008] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7732.064839] ? __next_timer_interrupt+0x160/0x160 [ 7732.064850] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7732.065029] ? do_wait_intr+0xf0/0xf0 [ 7732.065037] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7732.065217] kthread+0x1d1/0x200 [ 7732.065224] ? set_kthread_struct+0x70/0x70 [ 7732.065227] ret_from_fork+0x1f/0x30 [ 7732.105406] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7732.112004] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7732.112088] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7732.112094] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7732.112100] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7732.112395] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7732.112406] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7732.112410] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7732.112415] RAX: 0000000000000721 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7732.112417] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 7732.112420] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 7732.112422] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 7732.112424] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 7732.112427] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 7732.112430] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7732.112432] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7732.112441] Call Trace: [ 7732.112447] [ 7732.112452] ? show_regs.cold.9+0x22/0x2f [ 7732.112462] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7732.112469] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7732.112474] ? watchdog+0x40/0x40 [ 7732.112478] ? hrtimer_interrupt+0x128/0x2c0 [ 7732.112482] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7732.112491] ? apic_timer_interrupt+0xf/0x20 [ 7732.112495] [ 7732.112497] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7732.112502] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7732.112507] do_raw_spin_lock+0x177/0x190 [ 7732.112512] _raw_spin_lock+0x21/0x30 [ 7732.112517] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7732.112718] process_one_work+0x2c8/0x700 [ 7732.112725] worker_thread+0x296/0x6e0 [ 7732.112730] ? rescuer_thread+0x570/0x570 [ 7732.112734] kthread+0x1d1/0x200 [ 7732.112738] ? set_kthread_struct+0x70/0x70 [ 7732.112742] ret_from_fork+0x1f/0x30 [ 7732.593528] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7732.597376] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7732.599959] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7732.606960] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 7732.609868] RAX: 00000000000063c7 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7732.614389] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 7732.619745] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 7732.622982] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 7732.625940] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 7732.633073] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 7732.637181] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7732.642470] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 7732.648189] Call Trace: [ 7732.648887] [ 7732.649447] ? show_regs.cold.9+0x22/0x2f [ 7732.651939] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7732.655883] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7732.658803] ? watchdog+0x40/0x40 [ 7732.663568] ? hrtimer_interrupt+0x128/0x2c0 [ 7732.669293] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7732.676056] ? apic_timer_interrupt+0xf/0x20 [ 7732.681701] [ 7732.682992] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7732.688016] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7732.694411] do_raw_spin_lock+0x177/0x190 [ 7732.698330] _raw_spin_lock+0x21/0x30 [ 7732.699273] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7732.703820] ? do_raw_spin_unlock+0x75/0x190 [ 7732.708521] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7732.714959] ? do_wait_intr+0xf0/0xf0 [ 7732.716059] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7732.726252] mdc_reint+0x2d/0x110 [mdc] [ 7732.730260] mdc_unlink+0x658/0x870 [mdc] [ 7732.732677] lmv_unlink+0x34d/0xbd0 [lmv] [ 7732.737411] ll_unlink+0x24d/0x7f0 [lustre] [ 7732.740906] vfs_unlink+0x202/0x330 [ 7732.742818] do_unlinkat+0x2fa/0x460 [ 7732.746341] __x64_sys_unlink+0x37/0x50 [ 7732.749625] do_syscall_64+0xc1/0x440 [ 7732.750695] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7732.753485] RIP: 0033:0x7f151bdee10b [ 7732.755879] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7732.764136] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7732.767236] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 7732.770758] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 7732.774227] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7732.777226] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7732.780394] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7736.288371] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5805s! [ 7736.295519] Showing busy workqueues and worker pools: [ 7736.299494] workqueue events: flags=0x0 [ 7736.301740] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7736.305711] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 7736.312582] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7736.317771] pending: kfree_rcu_monitor [ 7736.320732] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7736.324902] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 7736.333314] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7736.342516] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 7736.348500] workqueue events_long: flags=0x0 [ 7736.350786] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7736.355747] pending: lu_objects_flush [obdclass] [ 7736.362482] workqueue events_unbound: flags=0x2 [ 7736.366440] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7736.372786] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7736.381893] workqueue events_power_efficient: flags=0x80 [ 7736.383978] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7736.388537] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 7736.392764] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7736.396656] pending: neigh_periodic_work [ 7736.403460] workqueue mm_percpu_wq: flags=0x8 [ 7736.408962] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7736.413327] pending: vmstat_update, lru_add_drain_per_cpu [ 7736.415795] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7736.419678] pending: vmstat_update [ 7736.421641] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7736.425116] pending: lru_add_drain_per_cpu [ 7736.426914] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7736.430405] pending: lru_add_drain_per_cpu BAR(48) [ 7736.433221] workqueue writeback: flags=0x4a [ 7736.434887] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 7736.441266] pending: wb_workfn [ 7736.443777] workqueue xprtiod: flags=0xa [ 7736.446950] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7736.449795] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7736.466515] workqueue ipv6_addrconf: flags=0x40008 [ 7736.468350] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7736.476295] pending: addrconf_verify_work [ 7736.478465] workqueue knbd0-recv: flags=0x1a [ 7736.480567] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7736.484576] in-flight: 658:recv_work [ 7736.487432] workqueue ptlrpc_pinger: flags=0x2 [ 7736.489729] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7736.498933] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7736.506864] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5805s workers=4 idle: 32984 37 8 [ 7736.512776] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7721s workers=2 idle: 661 [ 7744.774673] rcu: INFO: rcu_sched self-detected stall on CPU [ 7744.783248] rcu: 1-...!: (3292604 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 7744.789783] (t=5820127 jiffies g=249657 q=4056) [ 7744.794155] rcu: rcu_sched kthread starved for 5815079 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 7744.800988] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7744.808705] rcu: RCU grace-period kthread stack dump: [ 7744.817366] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7744.826006] Call Trace: [ 7744.826678] __schedule+0x351/0xcb0 [ 7744.830549] schedule+0xc0/0x180 [ 7744.831828] schedule_timeout+0xb4/0x190 [ 7744.836789] ? do_raw_spin_unlock+0x75/0x190 [ 7744.843718] ? __next_timer_interrupt+0x160/0x160 [ 7744.844959] ? prepare_to_swait_event+0xf3/0x140 [ 7744.846439] rcu_gp_kthread+0x8dd/0x15b0 [ 7744.852121] ? force_qs_rnp+0x2c0/0x2c0 [ 7744.853711] kthread+0x1d1/0x200 [ 7744.855340] ? set_kthread_struct+0x70/0x70 [ 7744.856787] ret_from_fork+0x1f/0x30 [ 7744.859475] rcu: Stack dump where RCU GP kthread last ran: [ 7744.864249] Sending NMI from CPU 1 to CPUs 3: [ 7744.867450] NMI backtrace for cpu 3 [ 7744.867457] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7744.867460] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7744.867461] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7744.867463] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7744.867465] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 [ 7744.867468] RAX: 00000000000066a4 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7744.867469] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 7744.867471] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7744.867472] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 7744.867473] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 7744.867475] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 7744.867476] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7744.867478] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 7744.867479] Call Trace: [ 7744.867480] [ 7744.867481] ? show_regs.cold.9+0x22/0x2f [ 7744.867482] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 7744.867483] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 7744.867485] ? nmi_handle+0xbd/0x210 [ 7744.867486] ? default_do_nmi+0x4b/0x1b0 [ 7744.867487] ? do_nmi+0x280/0x380 [ 7744.867488] ? end_repeat_nmi+0x16/0x1f [ 7744.867489] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7744.867490] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7744.867491] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7744.867492] [ 7744.867493] do_raw_spin_lock+0x177/0x190 [ 7744.867494] _raw_spin_lock+0x21/0x30 [ 7744.867496] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7744.867497] ? do_raw_spin_unlock+0x75/0x190 [ 7744.867498] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7744.867500] ? do_wait_intr+0xf0/0xf0 [ 7744.867501] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7744.867502] mdc_close+0x53b/0xd70 [mdc] [ 7744.867503] lmv_close+0x18d/0x490 [lmv] [ 7744.867504] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7744.867506] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7744.867507] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7744.867508] ll_md_close+0x274/0x800 [lustre] [ 7744.867509] ll_file_release+0x128/0xc50 [lustre] [ 7744.867510] __fput+0x10d/0x440 [ 7744.867511] ____fput+0x12/0x20 [ 7744.867513] task_work_run+0xc6/0x110 [ 7744.867514] exit_to_usermode_loop+0x1dd/0x1f0 [ 7744.867515] do_syscall_64+0x3d6/0x440 [ 7744.867516] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7744.867517] RIP: 0033:0x7f151bee3cd8 [ 7744.867518] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7744.867521] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7744.867523] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 7744.867525] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 7744.867526] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7744.867527] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7744.867529] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7744.874716] NMI backtrace for cpu 1 [ 7745.178650] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7745.184331] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7745.187266] Call Trace: [ 7745.188328] [ 7745.190176] ? dump_stack+0xbb/0x10e [ 7745.193455] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7745.201125] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7745.205376] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7745.206839] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7745.218902] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7745.223734] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 7745.225777] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 7745.239169] ? update_process_times+0x6d/0xc0 [ 7745.244242] ? tick_sched_handle.isra.18+0x27/0x90 [ 7745.249370] ? tick_sched_timer+0x53/0xb0 [ 7745.253229] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7745.259875] ? tick_sched_handle.isra.18+0x90/0x90 [ 7745.262562] ? hrtimer_interrupt+0x128/0x2c0 [ 7745.265661] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7745.270703] ? apic_timer_interrupt+0xf/0x20 [ 7745.277376] [ 7745.278442] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7745.283138] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7745.288864] ? do_raw_spin_lock+0x177/0x190 [ 7745.293835] ? _raw_spin_lock+0x21/0x30 [ 7745.295950] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7745.299443] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7745.307295] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7745.309963] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7745.313922] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7745.319453] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7745.322147] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7745.327869] ? __next_timer_interrupt+0x160/0x160 [ 7745.332294] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7745.338842] ? do_wait_intr+0xf0/0xf0 [ 7745.342302] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7745.344827] ? kthread+0x1d1/0x200 [ 7745.345913] ? set_kthread_struct+0x70/0x70 [ 7745.347006] ? ret_from_fork+0x1f/0x30 [ 7760.058875] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 7760.067055] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7760.091944] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7760.097722] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7760.101744] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7760.104351] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7760.109108] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7760.119834] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7760.127408] RAX: 0000000000007f14 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7760.132031] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 7760.136430] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 7760.140752] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 7760.144964] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 7760.149122] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 7760.153411] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7760.156795] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7760.159562] Call Trace: [ 7760.161158] [ 7760.162015] ? show_regs.cold.9+0x22/0x2f [ 7760.163685] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7760.168374] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7760.172739] ? watchdog+0x40/0x40 [ 7760.176480] ? hrtimer_interrupt+0x128/0x2c0 [ 7760.179603] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7760.185164] ? apic_timer_interrupt+0xf/0x20 [ 7760.189762] [ 7760.190880] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7760.193288] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7760.196701] do_raw_spin_lock+0x177/0x190 [ 7760.199505] _raw_spin_lock+0x21/0x30 [ 7760.204464] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7760.208352] process_one_work+0x2c8/0x700 [ 7760.211682] worker_thread+0x296/0x6e0 [ 7760.213828] ? rescuer_thread+0x570/0x570 [ 7760.216340] kthread+0x1d1/0x200 [ 7760.219119] ? set_kthread_struct+0x70/0x70 [ 7760.221050] ret_from_fork+0x1f/0x30 [ 7764.053677] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 7764.059207] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7764.090428] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7764.095215] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7764.099433] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7764.103038] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7764.112892] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7764.117956] RAX: 0000000000000206 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7764.121328] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 7764.124649] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 7764.133557] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 7764.137097] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 7764.140101] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 7764.144120] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7764.146531] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 7764.149117] Call Trace: [ 7764.150002] [ 7764.150663] ? show_regs.cold.9+0x22/0x2f [ 7764.152037] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7764.153699] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7764.156156] ? watchdog+0x40/0x40 [ 7764.158103] ? hrtimer_interrupt+0x128/0x2c0 [ 7764.165561] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7764.166896] ? apic_timer_interrupt+0xf/0x20 [ 7764.171185] [ 7764.172251] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7764.177337] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7764.179201] do_raw_spin_lock+0x177/0x190 [ 7764.182456] _raw_spin_lock+0x21/0x30 [ 7764.184568] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7764.191308] ? do_raw_spin_unlock+0x75/0x190 [ 7764.193661] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7764.196571] ? do_wait_intr+0xf0/0xf0 [ 7764.198516] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7764.200964] mdc_reint+0x2d/0x110 [mdc] [ 7764.203551] mdc_unlink+0x658/0x870 [mdc] [ 7764.207555] lmv_unlink+0x34d/0xbd0 [lmv] [ 7764.211031] ll_unlink+0x24d/0x7f0 [lustre] [ 7764.213467] vfs_unlink+0x202/0x330 [ 7764.215412] do_unlinkat+0x2fa/0x460 [ 7764.217451] __x64_sys_unlink+0x37/0x50 [ 7764.218921] do_syscall_64+0xc1/0x440 [ 7764.220448] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7764.223015] RIP: 0033:0x7f151bdee10b [ 7764.224347] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7764.238160] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7764.249875] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 7764.254498] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 7764.263390] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7764.267385] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7764.274724] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7767.008098] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5836s! [ 7767.011875] Showing busy workqueues and worker pools: [ 7767.014287] workqueue events: flags=0x0 [ 7767.016234] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7767.019796] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 7767.026306] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7767.031114] pending: kfree_rcu_monitor [ 7767.034610] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7767.040049] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 7767.047292] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7767.052231] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 7767.056641] workqueue events_long: flags=0x0 [ 7767.061199] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7767.073733] pending: lu_objects_flush [obdclass] [ 7767.077116] workqueue events_unbound: flags=0x2 [ 7767.079354] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7767.087093] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7767.100603] workqueue events_power_efficient: flags=0x80 [ 7767.104865] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7767.113241] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 7767.122306] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7767.130098] pending: neigh_periodic_work [ 7767.135831] workqueue mm_percpu_wq: flags=0x8 [ 7767.141214] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7767.145820] pending: vmstat_update, lru_add_drain_per_cpu [ 7767.151133] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7767.156864] pending: vmstat_update [ 7767.159846] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7767.174342] pending: lru_add_drain_per_cpu [ 7767.177251] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7767.184695] pending: lru_add_drain_per_cpu BAR(48) [ 7767.186578] workqueue writeback: flags=0x4a [ 7767.202135] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 7767.206511] pending: wb_workfn [ 7767.213030] workqueue xprtiod: flags=0xa [ 7767.219014] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7767.220994] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7767.237012] workqueue ipv6_addrconf: flags=0x40008 [ 7767.238369] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7767.254394] pending: addrconf_verify_work [ 7767.261479] workqueue knbd0-recv: flags=0x1a [ 7767.266137] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7767.274508] in-flight: 658:recv_work [ 7767.275985] workqueue ptlrpc_pinger: flags=0x2 [ 7767.286412] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7767.295020] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7767.307536] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5836s workers=4 idle: 32984 37 8 [ 7767.317108] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7752s workers=2 idle: 661 [ 7772.055248] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 7772.061579] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7772.061835] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 7772.112221] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7772.115673] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) [ 7772.121161] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7772.121173] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7772.121189] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7772.121193] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7772.121198] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 7772.121200] RDX: 0000000000001f08 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 7772.121203] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 7772.121205] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 7772.121208] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7772.121210] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 7772.121214] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7772.121216] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7772.121224] Call Trace: [ 7772.121230] [ 7772.121236] ? show_regs.cold.9+0x22/0x2f [ 7772.121246] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7772.121254] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7772.121259] ? watchdog+0x40/0x40 [ 7772.121264] ? hrtimer_interrupt+0x128/0x2c0 [ 7772.121268] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7772.121291] ? apic_timer_interrupt+0xf/0x20 [ 7772.121296] [ 7772.121299] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7772.121304] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7772.121309] do_raw_spin_lock+0x177/0x190 [ 7772.121314] _raw_spin_lock+0x21/0x30 [ 7772.121321] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7772.121608] obd_mod_free+0x46/0x140 [ptlrpc] [ 7772.122363] mdc_commit_open+0x91/0xb0 [mdc] [ 7772.122400] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7772.123118] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7772.123453] after_reply+0xe2c/0x1780 [ptlrpc] [ 7772.124096] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7772.124406] ? __next_timer_interrupt+0x160/0x160 [ 7772.124418] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7772.124620] ? do_wait_intr+0xf0/0xf0 [ 7772.124629] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7772.130824] fid(O) [ 7772.134394] kthread+0x1d1/0x200 [ 7772.134409] ? set_kthread_struct+0x70/0x70 [ 7772.134414] ret_from_fork+0x1f/0x30 [ 7772.247897] fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7772.279089] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7772.296494] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7772.303850] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7772.306308] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7772.314123] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7772.318206] RAX: 0000000000007e0c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7772.321551] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 7772.327232] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7772.329879] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 7772.334800] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 7772.337739] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 7772.341138] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7772.343919] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 7772.346708] Call Trace: [ 7772.347719] [ 7772.348582] ? show_regs.cold.9+0x22/0x2f [ 7772.350148] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7772.351915] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7772.353666] ? watchdog+0x40/0x40 [ 7772.354963] ? hrtimer_interrupt+0x128/0x2c0 [ 7772.356624] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7772.358481] ? apic_timer_interrupt+0xf/0x20 [ 7772.360863] [ 7772.362194] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7772.364204] do_raw_spin_lock+0x177/0x190 [ 7772.370748] _raw_spin_lock+0x21/0x30 [ 7772.372204] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7772.377897] ? do_raw_spin_unlock+0x75/0x190 [ 7772.381743] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7772.384782] ? do_wait_intr+0xf0/0xf0 [ 7772.387375] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7772.394815] mdc_close+0x53b/0xd70 [mdc] [ 7772.396542] lmv_close+0x18d/0x490 [lmv] [ 7772.398358] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7772.401049] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7772.403829] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7772.406098] ll_md_close+0x274/0x800 [lustre] [ 7772.408106] ll_file_release+0x128/0xc50 [lustre] [ 7772.410279] __fput+0x10d/0x440 [ 7772.417909] ____fput+0x12/0x20 [ 7772.422246] task_work_run+0xc6/0x110 [ 7772.425279] exit_to_usermode_loop+0x1dd/0x1f0 [ 7772.427004] do_syscall_64+0x3d6/0x440 [ 7772.428506] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7772.430458] RIP: 0033:0x7f151bee3cd8 [ 7772.431883] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7772.439831] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7772.442700] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 7772.445357] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 7772.448059] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7772.450943] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7772.454033] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7788.056137] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 7788.064981] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7788.135041] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7788.150740] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7788.165901] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7788.172790] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7788.178322] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7788.205792] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7788.219452] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7788.234891] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 7788.248698] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 7788.269743] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 7788.272454] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 7788.293329] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 7788.304263] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7788.314707] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7788.318486] Call Trace: [ 7788.327743] [ 7788.328491] ? show_regs.cold.9+0x22/0x2f [ 7788.329951] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7788.340216] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7788.347125] ? watchdog+0x40/0x40 [ 7788.351758] ? hrtimer_interrupt+0x128/0x2c0 [ 7788.357273] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7788.364745] ? apic_timer_interrupt+0xf/0x20 [ 7788.371759] [ 7788.373947] ? native_safe_halt+0x2e/0x50 [ 7788.379361] kvm_wait+0xcd/0xe0 [ 7788.381922] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7788.387987] do_raw_spin_lock+0x177/0x190 [ 7788.396419] _raw_spin_lock+0x21/0x30 [ 7788.401887] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7788.411464] process_one_work+0x2c8/0x700 [ 7788.415443] worker_thread+0x296/0x6e0 [ 7788.420599] ? rescuer_thread+0x570/0x570 [ 7788.424235] kthread+0x1d1/0x200 [ 7788.427683] ? set_kthread_struct+0x70/0x70 [ 7788.430798] ret_from_fork+0x1f/0x30 [ 7792.052016] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 7792.054456] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7792.116061] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7792.125345] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7792.130267] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7792.134272] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7792.147327] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 7792.151568] RAX: 0000000000006039 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7792.159079] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 7792.163682] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 7792.168085] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 7792.172786] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 7792.177739] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 7792.183798] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7792.185551] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 7792.188172] Call Trace: [ 7792.189384] [ 7792.190955] ? show_regs.cold.9+0x22/0x2f [ 7792.192731] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7792.200529] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7792.206824] ? watchdog+0x40/0x40 [ 7792.210225] ? hrtimer_interrupt+0x128/0x2c0 [ 7792.213368] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7792.215775] ? apic_timer_interrupt+0xf/0x20 [ 7792.217783] [ 7792.218382] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7792.220293] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7792.229091] do_raw_spin_lock+0x177/0x190 [ 7792.230684] _raw_spin_lock+0x21/0x30 [ 7792.235101] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7792.241680] ? do_raw_spin_unlock+0x75/0x190 [ 7792.246782] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7792.252897] ? do_wait_intr+0xf0/0xf0 [ 7792.256083] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7792.258574] mdc_reint+0x2d/0x110 [mdc] [ 7792.260733] mdc_unlink+0x658/0x870 [mdc] [ 7792.262394] lmv_unlink+0x34d/0xbd0 [lmv] [ 7792.269288] ll_unlink+0x24d/0x7f0 [lustre] [ 7792.271432] vfs_unlink+0x202/0x330 [ 7792.274224] do_unlinkat+0x2fa/0x460 [ 7792.276400] __x64_sys_unlink+0x37/0x50 [ 7792.277881] do_syscall_64+0xc1/0x440 [ 7792.280284] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7792.286221] RIP: 0033:0x7f151bdee10b [ 7792.289419] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7792.308535] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7792.310990] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 7792.318390] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 7792.322178] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7792.327391] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7792.335397] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7797.729029] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5867s! [ 7797.734017] Showing busy workqueues and worker pools: [ 7797.736741] workqueue events: flags=0x0 [ 7797.738539] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7797.742156] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 7797.746611] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7797.749700] pending: kfree_rcu_monitor [ 7797.751736] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7797.754013] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 7797.758471] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7797.764674] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 7797.769055] workqueue events_long: flags=0x0 [ 7797.770432] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7797.776746] pending: lu_objects_flush [obdclass] [ 7797.779826] workqueue events_unbound: flags=0x2 [ 7797.781849] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7797.786740] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7797.791756] workqueue events_power_efficient: flags=0x80 [ 7797.794294] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7797.797163] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 7797.802365] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7797.808112] pending: neigh_periodic_work [ 7797.809918] workqueue mm_percpu_wq: flags=0x8 [ 7797.812283] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7797.817321] pending: vmstat_update, lru_add_drain_per_cpu [ 7797.821655] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7797.827011] pending: vmstat_update [ 7797.832262] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7797.839262] pending: lru_add_drain_per_cpu [ 7797.840947] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7797.844499] pending: lru_add_drain_per_cpu BAR(48) [ 7797.849875] workqueue writeback: flags=0x4a [ 7797.861846] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 7797.872898] pending: wb_workfn [ 7797.875242] workqueue xprtiod: flags=0xa [ 7797.884440] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7797.887715] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7797.893290] workqueue ipv6_addrconf: flags=0x40008 [ 7797.898560] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7797.902387] pending: addrconf_verify_work [ 7797.905747] workqueue knbd0-recv: flags=0x1a [ 7797.908246] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7797.911126] in-flight: 658:recv_work [ 7797.912165] workqueue ptlrpc_pinger: flags=0x2 [ 7797.915130] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7797.916744] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7797.925991] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5867s workers=4 idle: 32984 37 8 [ 7797.939214] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7782s workers=2 idle: 661 [ 7800.055029] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 7800.057540] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 7800.057927] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7800.057999] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7800.058006] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7800.058008] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7800.058022] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7800.058027] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7800.058033] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 7800.058035] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 7800.058037] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 7800.058040] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 7800.058043] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7800.058046] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 7800.058049] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7800.058051] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7800.058059] Call Trace: [ 7800.058063] [ 7800.058072] ? show_regs.cold.9+0x22/0x2f [ 7800.058081] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7800.058086] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7800.058089] ? watchdog+0x40/0x40 [ 7800.058092] ? hrtimer_interrupt+0x128/0x2c0 [ 7800.058094] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7800.058099] ? apic_timer_interrupt+0xf/0x20 [ 7800.058102] [ 7800.058104] ? native_safe_halt+0x2e/0x50 [ 7800.058107] kvm_wait+0xcd/0xe0 [ 7800.058112] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7800.058116] do_raw_spin_lock+0x177/0x190 [ 7800.058119] _raw_spin_lock+0x21/0x30 [ 7800.058123] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7800.058341] obd_mod_free+0x46/0x140 [ptlrpc] [ 7800.058490] mdc_commit_open+0x91/0xb0 [mdc] [ 7800.058539] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7800.058710] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7800.058874] after_reply+0xe2c/0x1780 [ptlrpc] [ 7800.059020] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7800.059144] ? __next_timer_interrupt+0x160/0x160 [ 7800.059151] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7800.059283] ? do_wait_intr+0xf0/0xf0 [ 7800.059288] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7800.059443] kthread+0x1d1/0x200 [ 7800.059447] ? set_kthread_struct+0x70/0x70 [ 7800.059449] ret_from_fork+0x1f/0x30 [ 7800.202735] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7800.226902] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7800.233429] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7800.237961] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7800.241477] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7800.249942] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7800.254038] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7800.257604] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 7800.260528] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7800.264164] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 7800.268143] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 7800.272094] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 7800.276625] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7800.280696] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 7800.284867] Call Trace: [ 7800.286473] [ 7800.288044] ? show_regs.cold.9+0x22/0x2f [ 7800.290031] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7800.293000] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7800.295185] ? watchdog+0x40/0x40 [ 7800.297028] ? hrtimer_interrupt+0x128/0x2c0 [ 7800.298767] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7800.301599] ? apic_timer_interrupt+0xf/0x20 [ 7800.303368] [ 7800.304496] ? native_safe_halt+0x2e/0x50 [ 7800.306091] kvm_wait+0xcd/0xe0 [ 7800.307731] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7800.310137] do_raw_spin_lock+0x177/0x190 [ 7800.312163] _raw_spin_lock+0x21/0x30 [ 7800.313931] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7800.316843] ? do_raw_spin_unlock+0x75/0x190 [ 7800.319837] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7800.323858] ? do_wait_intr+0xf0/0xf0 [ 7800.326176] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7800.330471] mdc_close+0x53b/0xd70 [mdc] [ 7800.333398] lmv_close+0x18d/0x490 [lmv] [ 7800.335124] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7800.338391] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7800.342099] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7800.344331] ll_md_close+0x274/0x800 [lustre] [ 7800.346667] ll_file_release+0x128/0xc50 [lustre] [ 7800.348341] __fput+0x10d/0x440 [ 7800.349348] ____fput+0x12/0x20 [ 7800.351252] task_work_run+0xc6/0x110 [ 7800.353026] exit_to_usermode_loop+0x1dd/0x1f0 [ 7800.355992] do_syscall_64+0x3d6/0x440 [ 7800.358053] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7800.361092] RIP: 0033:0x7f151bee3cd8 [ 7800.362592] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7800.371072] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7800.374453] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 7800.379042] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 7800.382436] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7800.386047] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7800.389117] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7816.056165] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 7816.067080] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7816.138958] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7816.149269] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7816.160907] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7816.169484] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7816.172447] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7816.183806] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7816.188095] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7816.197706] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 7816.203903] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 7816.209554] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 7816.213540] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 7816.218414] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 7816.226058] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7816.231384] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7816.240511] Call Trace: [ 7816.244352] [ 7816.245776] ? show_regs.cold.9+0x22/0x2f [ 7816.250677] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7816.253523] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7816.256409] ? watchdog+0x40/0x40 [ 7816.259383] ? hrtimer_interrupt+0x128/0x2c0 [ 7816.261645] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7816.268099] ? apic_timer_interrupt+0xf/0x20 [ 7816.270854] [ 7816.272075] ? native_safe_halt+0x2e/0x50 [ 7816.275338] kvm_wait+0xcd/0xe0 [ 7816.276983] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7816.283712] do_raw_spin_lock+0x177/0x190 [ 7816.286324] _raw_spin_lock+0x21/0x30 [ 7816.288396] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7816.294073] process_one_work+0x2c8/0x700 [ 7816.295254] worker_thread+0x296/0x6e0 [ 7816.296225] ? rescuer_thread+0x570/0x570 [ 7816.303623] kthread+0x1d1/0x200 [ 7816.308538] ? set_kthread_struct+0x70/0x70 [ 7816.311422] ret_from_fork+0x1f/0x30 [ 7824.052028] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 7824.054852] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7824.078039] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7824.084593] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7824.088637] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7824.090730] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7824.099089] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7824.103684] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7824.108356] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 7824.111876] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 7824.115325] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 7824.118858] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 7824.122184] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 7824.125750] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7824.128445] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 7824.133264] Call Trace: [ 7824.134360] [ 7824.135082] ? show_regs.cold.9+0x22/0x2f [ 7824.138528] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7824.140586] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7824.146461] ? watchdog+0x40/0x40 [ 7824.147753] ? hrtimer_interrupt+0x128/0x2c0 [ 7824.149629] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7824.151671] ? apic_timer_interrupt+0xf/0x20 [ 7824.155114] [ 7824.155930] ? native_safe_halt+0x2e/0x50 [ 7824.158867] kvm_wait+0xcd/0xe0 [ 7824.160882] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7824.164180] do_raw_spin_lock+0x177/0x190 [ 7824.165854] _raw_spin_lock+0x21/0x30 [ 7824.169110] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7824.171534] ? do_raw_spin_unlock+0x75/0x190 [ 7824.173749] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7824.176378] ? do_wait_intr+0xf0/0xf0 [ 7824.177739] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7824.181430] mdc_reint+0x2d/0x110 [mdc] [ 7824.182970] mdc_unlink+0x658/0x870 [mdc] [ 7824.185521] lmv_unlink+0x34d/0xbd0 [lmv] [ 7824.186798] ll_unlink+0x24d/0x7f0 [lustre] [ 7824.188691] vfs_unlink+0x202/0x330 [ 7824.190004] do_unlinkat+0x2fa/0x460 [ 7824.191910] __x64_sys_unlink+0x37/0x50 [ 7824.193375] do_syscall_64+0xc1/0x440 [ 7824.195709] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7824.197971] RIP: 0033:0x7f151bdee10b [ 7824.199206] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7824.206880] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7824.209978] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 7824.213662] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 7824.216738] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7824.219562] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7824.222568] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7828.055057] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 7828.057328] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 7828.058634] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7828.058713] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7828.058721] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7828.058724] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7828.058893] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7828.058900] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7828.058905] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 7828.058908] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 7828.058910] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 7828.058913] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 7828.058916] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7828.058919] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 7828.058922] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7828.058924] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7828.058932] Call Trace: [ 7828.058938] [ 7828.058943] ? show_regs.cold.9+0x22/0x2f [ 7828.058955] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7828.058961] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7828.058977] ? watchdog+0x40/0x40 [ 7828.058982] ? hrtimer_interrupt+0x128/0x2c0 [ 7828.058987] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7828.058994] ? apic_timer_interrupt+0xf/0x20 [ 7828.059007] [ 7828.059010] ? native_safe_halt+0x2e/0x50 [ 7828.059015] kvm_wait+0xcd/0xe0 [ 7828.059022] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7828.059030] do_raw_spin_lock+0x177/0x190 [ 7828.059035] _raw_spin_lock+0x21/0x30 [ 7828.059041] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7828.059355] obd_mod_free+0x46/0x140 [ptlrpc] [ 7828.059566] mdc_commit_open+0x91/0xb0 [mdc] [ 7828.059608] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7828.059829] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7828.060030] after_reply+0xe2c/0x1780 [ptlrpc] [ 7828.060231] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7828.060472] ? __next_timer_interrupt+0x160/0x160 [ 7828.060482] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7828.060699] ? do_wait_intr+0xf0/0xf0 [ 7828.060709] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7828.060924] kthread+0x1d1/0x200 [ 7828.060930] ? set_kthread_struct+0x70/0x70 [ 7828.060935] ret_from_fork+0x1f/0x30 [ 7828.203865] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7828.223929] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7828.228680] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7828.232415] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7828.234064] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7828.240680] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7828.245559] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7828.247865] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 7828.249868] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7828.251749] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 7828.255218] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 7828.258618] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 7828.261457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7828.263769] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 7828.266881] Call Trace: [ 7828.267892] [ 7828.268757] ? show_regs.cold.9+0x22/0x2f [ 7828.270481] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7828.272497] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7828.274970] ? watchdog+0x40/0x40 [ 7828.276377] ? hrtimer_interrupt+0x128/0x2c0 [ 7828.278411] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7828.279934] ? apic_timer_interrupt+0xf/0x20 [ 7828.282478] [ 7828.283439] ? native_safe_halt+0x2e/0x50 [ 7828.285541] kvm_wait+0xcd/0xe0 [ 7828.287228] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7828.290373] do_raw_spin_lock+0x177/0x190 [ 7828.293569] _raw_spin_lock+0x21/0x30 [ 7828.295421] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7828.297863] ? do_raw_spin_unlock+0x75/0x190 [ 7828.299871] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7828.302672] ? do_wait_intr+0xf0/0xf0 [ 7828.304320] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7828.307425] mdc_close+0x53b/0xd70 [mdc] [ 7828.308963] lmv_close+0x18d/0x490 [lmv] [ 7828.310888] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7828.313820] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7828.316729] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7828.318970] ll_md_close+0x274/0x800 [lustre] [ 7828.320910] ll_file_release+0x128/0xc50 [lustre] [ 7828.325481] __fput+0x10d/0x440 [ 7828.326848] ____fput+0x12/0x20 [ 7828.328018] task_work_run+0xc6/0x110 [ 7828.329985] exit_to_usermode_loop+0x1dd/0x1f0 [ 7828.334590] do_syscall_64+0x3d6/0x440 [ 7828.336476] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7828.338690] RIP: 0033:0x7f151bee3cd8 [ 7828.341496] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7828.352932] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7828.356770] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 7828.361344] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 7828.364721] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7828.370880] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7828.374020] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7828.448070] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5897s! [ 7828.456134] Showing busy workqueues and worker pools: [ 7828.460508] workqueue events: flags=0x0 [ 7828.462442] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7828.471656] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 7828.476448] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7828.480219] pending: kfree_rcu_monitor [ 7828.482644] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7828.486357] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 7828.491194] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7828.494506] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 7828.497879] workqueue events_long: flags=0x0 [ 7828.499553] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7828.501910] pending: lu_objects_flush [obdclass] [ 7828.504426] workqueue events_unbound: flags=0x2 [ 7828.505936] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7828.508894] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7828.519755] workqueue events_power_efficient: flags=0x80 [ 7828.525593] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7828.535898] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 7828.544039] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7828.548031] pending: neigh_periodic_work [ 7828.551557] workqueue mm_percpu_wq: flags=0x8 [ 7828.554108] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7828.558916] pending: vmstat_update, lru_add_drain_per_cpu [ 7828.562531] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7828.565986] pending: vmstat_update [ 7828.567790] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7828.570007] pending: lru_add_drain_per_cpu [ 7828.571468] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7828.573779] pending: lru_add_drain_per_cpu BAR(48) [ 7828.575159] workqueue writeback: flags=0x4a [ 7828.578799] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 7828.583704] pending: wb_workfn [ 7828.588523] workqueue xprtiod: flags=0xa [ 7828.591806] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7828.595418] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7828.599702] workqueue ipv6_addrconf: flags=0x40008 [ 7828.601976] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7828.607766] pending: addrconf_verify_work [ 7828.610737] workqueue knbd0-recv: flags=0x1a [ 7828.613648] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7828.616947] in-flight: 658:recv_work [ 7828.619678] workqueue ptlrpc_pinger: flags=0x2 [ 7828.623566] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7828.626518] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7828.629716] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5898s workers=4 idle: 32984 37 8 [ 7828.634757] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7813s workers=2 idle: 661 [ 7844.056024] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 7844.059565] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7844.100081] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7844.109162] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7844.115619] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7844.127865] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7844.135873] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7844.151862] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 7844.156089] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7844.159444] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 7844.163285] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 7844.171038] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 7844.177136] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 7844.184385] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 7844.195336] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7844.200291] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7844.204701] Call Trace: [ 7844.206081] [ 7844.209484] ? show_regs.cold.9+0x22/0x2f [ 7844.213539] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7844.221142] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7844.226981] ? watchdog+0x40/0x40 [ 7844.228431] ? hrtimer_interrupt+0x128/0x2c0 [ 7844.232115] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7844.234438] ? apic_timer_interrupt+0xf/0x20 [ 7844.237262] [ 7844.238264] ? native_safe_halt+0x2e/0x50 [ 7844.240426] kvm_wait+0xcd/0xe0 [ 7844.241747] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7844.246168] do_raw_spin_lock+0x177/0x190 [ 7844.248559] _raw_spin_lock+0x21/0x30 [ 7844.251253] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7844.255681] process_one_work+0x2c8/0x700 [ 7844.257159] worker_thread+0x296/0x6e0 [ 7844.260291] ? rescuer_thread+0x570/0x570 [ 7844.265192] kthread+0x1d1/0x200 [ 7844.269016] ? set_kthread_struct+0x70/0x70 [ 7844.273312] ret_from_fork+0x1f/0x30 [ 7856.052332] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 7856.055024] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 7856.060784] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 7856.060800] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7856.060871] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7856.060876] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7856.060878] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7856.060891] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7856.060895] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7856.060899] RAX: 0000000000003095 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7856.060901] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 7856.060903] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7856.060905] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 7856.060907] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 7856.060910] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 7856.060913] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7856.060915] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 7856.060923] Call Trace: [ 7856.060936] [ 7856.060943] ? show_regs.cold.9+0x22/0x2f [ 7856.060952] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7856.060957] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7856.060961] ? watchdog+0x40/0x40 [ 7856.060964] ? hrtimer_interrupt+0x128/0x2c0 [ 7856.060968] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7856.060983] ? apic_timer_interrupt+0xf/0x20 [ 7856.060988] [ 7856.060995] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7856.061002] do_raw_spin_lock+0x177/0x190 [ 7856.061006] _raw_spin_lock+0x21/0x30 [ 7856.061012] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7856.061555] ? do_raw_spin_unlock+0x75/0x190 [ 7856.061561] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7856.061761] ? do_wait_intr+0xf0/0xf0 [ 7856.061769] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7856.061991] mdc_close+0x53b/0xd70 [mdc] [ 7856.062034] lmv_close+0x18d/0x490 [lmv] [ 7856.062062] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7856.063044] Modules linked in: lustre(O) [ 7856.062234] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7856.065733] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7856.065815] ll_md_close+0x274/0x800 [lustre] [ 7856.065897] ll_file_release+0x128/0xc50 [lustre] [ 7856.066113] __fput+0x10d/0x440 [ 7856.066124] ____fput+0x12/0x20 [ 7856.066129] task_work_run+0xc6/0x110 [ 7856.066137] exit_to_usermode_loop+0x1dd/0x1f0 [ 7856.066144] do_syscall_64+0x3d6/0x440 [ 7856.066149] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7856.066158] RIP: 0033:0x7f151bee3cd8 [ 7856.066167] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7856.066171] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7856.066177] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 7856.066179] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 7856.066181] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7856.066184] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7856.066186] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7856.067292] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7856.067366] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7856.067374] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7856.067377] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7856.067387] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7856.067391] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7856.067395] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 7856.067398] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 7856.067400] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 7856.067402] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 7856.067404] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7856.067408] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 7856.067411] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7856.067413] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7856.067422] Call Trace: [ 7856.067426] [ 7856.067430] ? show_regs.cold.9+0x22/0x2f [ 7856.067439] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7856.067446] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7856.067449] ? watchdog+0x40/0x40 [ 7856.067453] ? hrtimer_interrupt+0x128/0x2c0 [ 7856.067457] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7856.067463] ? apic_timer_interrupt+0xf/0x20 [ 7856.067468] [ 7856.067471] ? native_safe_halt+0x2e/0x50 [ 7856.067476] kvm_wait+0xcd/0xe0 [ 7856.067483] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7856.067490] do_raw_spin_lock+0x177/0x190 [ 7856.067495] _raw_spin_lock+0x21/0x30 [ 7856.067500] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7856.068061] obd_mod_free+0x46/0x140 [ptlrpc] [ 7856.068340] mdc_commit_open+0x91/0xb0 [mdc] [ 7856.068375] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7856.068571] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7856.076278] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) [ 7856.106212] after_reply+0xe2c/0x1780 [ptlrpc] [ 7856.106716] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7856.107188] ? __next_timer_interrupt+0x160/0x160 [ 7856.107202] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7856.107457] ? do_wait_intr+0xf0/0xf0 [ 7856.107467] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7856.108025] kthread+0x1d1/0x200 [ 7856.108032] ? set_kthread_struct+0x70/0x70 [ 7856.108036] ret_from_fork+0x1f/0x30 [ 7856.690622] mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7856.727737] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7856.739210] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7856.742562] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7856.746297] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7856.755586] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7856.760666] RAX: 00000000000042df RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7856.772560] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 7856.774916] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 7856.794364] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 7856.800537] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 7856.806431] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 7856.813395] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7856.820372] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 7856.827064] Call Trace: [ 7856.827923] [ 7856.828630] ? show_regs.cold.9+0x22/0x2f [ 7856.832897] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7856.834531] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7856.841648] ? watchdog+0x40/0x40 [ 7856.843072] ? hrtimer_interrupt+0x128/0x2c0 [ 7856.845112] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7856.846684] ? apic_timer_interrupt+0xf/0x20 [ 7856.848425] [ 7856.850397] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7856.856722] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7856.860879] do_raw_spin_lock+0x177/0x190 [ 7856.869652] _raw_spin_lock+0x21/0x30 [ 7856.870889] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7856.897166] ? do_raw_spin_unlock+0x75/0x190 [ 7856.902308] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7856.905326] ? do_wait_intr+0xf0/0xf0 [ 7856.913035] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7856.923039] mdc_reint+0x2d/0x110 [mdc] [ 7856.933315] mdc_unlink+0x658/0x870 [mdc] [ 7856.942335] lmv_unlink+0x34d/0xbd0 [lmv] [ 7856.946392] ll_unlink+0x24d/0x7f0 [lustre] [ 7856.948637] vfs_unlink+0x202/0x330 [ 7856.949795] do_unlinkat+0x2fa/0x460 [ 7856.952181] __x64_sys_unlink+0x37/0x50 [ 7856.954418] do_syscall_64+0xc1/0x440 [ 7856.959145] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7856.961931] RIP: 0033:0x7f151bdee10b [ 7856.963475] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7856.975617] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7856.987914] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 7856.991063] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 7856.996521] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7857.002420] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7857.005781] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7858.661404] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5928s! [ 7858.665234] Showing busy workqueues and worker pools: [ 7858.667091] workqueue events: flags=0x0 [ 7858.668506] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7858.670917] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 7858.674258] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7858.676525] pending: kfree_rcu_monitor [ 7858.678049] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7858.680811] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 7858.685767] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7858.695737] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 7858.699886] workqueue events_long: flags=0x0 [ 7858.702037] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7858.707958] pending: lu_objects_flush [obdclass] [ 7858.710495] workqueue events_unbound: flags=0x2 [ 7858.712369] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7858.716370] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7858.723772] workqueue events_power_efficient: flags=0x80 [ 7858.727690] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7858.730788] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 7858.734192] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7858.736559] pending: neigh_periodic_work [ 7858.737925] workqueue mm_percpu_wq: flags=0x8 [ 7858.739350] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7858.741869] pending: vmstat_update, lru_add_drain_per_cpu [ 7858.744751] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7858.749704] pending: vmstat_update [ 7858.755821] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7858.759963] pending: lru_add_drain_per_cpu [ 7858.764346] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7858.767687] pending: lru_add_drain_per_cpu BAR(48) [ 7858.771669] workqueue writeback: flags=0x4a [ 7858.774728] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 7858.778895] pending: wb_workfn [ 7858.780524] workqueue xprtiod: flags=0xa [ 7858.782388] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7858.785450] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7858.790321] workqueue ipv6_addrconf: flags=0x40008 [ 7858.793323] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7858.800364] pending: addrconf_verify_work [ 7858.805728] workqueue knbd0-recv: flags=0x1a [ 7858.808294] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7858.813159] in-flight: 658:recv_work [ 7858.815147] workqueue ptlrpc_pinger: flags=0x2 [ 7858.816987] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7858.819776] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7858.822678] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5928s workers=4 idle: 32984 37 8 [ 7858.827623] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7843s workers=2 idle: 661 [ 7872.056424] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7872.062977] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7872.096427] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7872.101021] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7872.104549] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7872.107511] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7872.110021] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7872.117491] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7872.120655] RAX: 0000000000007e8d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7872.125119] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 7872.131721] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 7872.138127] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 7872.144617] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 7872.151411] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 7872.158414] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7872.164380] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7872.172849] Call Trace: [ 7872.174767] [ 7872.178109] ? show_regs.cold.9+0x22/0x2f [ 7872.181878] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7872.187486] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7872.190087] ? watchdog+0x40/0x40 [ 7872.195533] ? hrtimer_interrupt+0x128/0x2c0 [ 7872.197894] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7872.203499] ? apic_timer_interrupt+0xf/0x20 [ 7872.207679] [ 7872.212041] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7872.221970] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7872.225829] do_raw_spin_lock+0x177/0x190 [ 7872.227563] _raw_spin_lock+0x21/0x30 [ 7872.228880] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7872.235052] process_one_work+0x2c8/0x700 [ 7872.238860] worker_thread+0x296/0x6e0 [ 7872.242560] ? rescuer_thread+0x570/0x570 [ 7872.246150] kthread+0x1d1/0x200 [ 7872.249803] ? set_kthread_struct+0x70/0x70 [ 7872.253848] ret_from_fork+0x1f/0x30 [ 7884.052293] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 7884.059965] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7884.060760] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 7884.060858] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 7884.060864] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7884.060926] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7884.060933] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7884.060936] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7884.060951] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7884.060955] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7884.060960] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 7884.060963] RDX: 000000000000432a RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 7884.060965] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 7884.060968] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 7884.060970] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7884.060974] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 7884.060977] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7884.060979] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7884.060988] Call Trace: [ 7884.060993] [ 7884.060998] ? show_regs.cold.9+0x22/0x2f [ 7884.061008] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7884.061015] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7884.061020] ? watchdog+0x40/0x40 [ 7884.061024] ? hrtimer_interrupt+0x128/0x2c0 [ 7884.061029] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7884.061039] ? apic_timer_interrupt+0xf/0x20 [ 7884.061044] [ 7884.061047] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7884.061053] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7884.061058] do_raw_spin_lock+0x177/0x190 [ 7884.061063] _raw_spin_lock+0x21/0x30 [ 7884.061069] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7884.061335] obd_mod_free+0x46/0x140 [ptlrpc] [ 7884.061532] mdc_commit_open+0x91/0xb0 [mdc] [ 7884.061567] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7884.061813] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7884.062034] after_reply+0xe2c/0x1780 [ptlrpc] [ 7884.062227] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7884.062431] ? __next_timer_interrupt+0x160/0x160 [ 7884.062441] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7884.062643] ? do_wait_intr+0xf0/0xf0 [ 7884.062652] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7884.062866] kthread+0x1d1/0x200 [ 7884.062871] ? set_kthread_struct+0x70/0x70 [ 7884.062875] ret_from_fork+0x1f/0x30 [ 7884.094923] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7884.097993] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7884.098068] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7884.098076] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7884.098078] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7884.098093] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7884.098097] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7884.098101] RAX: 0000000000002229 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7884.098104] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 7884.098107] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7884.098109] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 7884.098112] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 7884.098115] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 7884.098118] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7884.098121] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 7884.098129] Call Trace: [ 7884.101305] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7884.101315] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7884.101330] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7884.119393] [ 7884.124404] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 [ 7884.127129] ? show_regs.cold.9+0x22/0x2f [ 7884.127149] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7884.127154] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7884.127158] ? watchdog+0x40/0x40 [ 7884.127161] ? hrtimer_interrupt+0x128/0x2c0 [ 7884.127165] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7884.127172] ? apic_timer_interrupt+0xf/0x20 [ 7884.127176] [ 7884.127180] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7884.127189] do_raw_spin_lock+0x177/0x190 [ 7884.127194] _raw_spin_lock+0x21/0x30 [ 7884.127198] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7884.127453] ? do_raw_spin_unlock+0x75/0x190 [ 7884.127458] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7884.127658] ? do_wait_intr+0xf0/0xf0 [ 7884.127665] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7884.130262] ORIG_RAX: ffffffffffffff13 [ 7884.136588] mdc_close+0x53b/0xd70 [mdc] [ 7884.136638] lmv_close+0x18d/0x490 [lmv] [ 7884.136667] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7884.140964] RAX: 0000000000005985 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7884.143674] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7884.143817] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7884.146822] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 7884.146831] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 7884.149140] ll_md_close+0x274/0x800 [lustre] [ 7884.152181] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 7884.154520] ll_file_release+0x128/0xc50 [lustre] [ 7884.154691] __fput+0x10d/0x440 [ 7884.154701] ____fput+0x12/0x20 [ 7884.154715] task_work_run+0xc6/0x110 [ 7884.154724] exit_to_usermode_loop+0x1dd/0x1f0 [ 7884.154732] do_syscall_64+0x3d6/0x440 [ 7884.154737] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7884.154747] RIP: 0033:0x7f151bee3cd8 [ 7884.154754] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7884.154758] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7884.154763] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 7884.154765] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 7884.154767] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7884.154769] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7884.154771] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7884.376308] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 7884.378823] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 7884.381394] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7884.383796] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 7884.386371] Call Trace: [ 7884.387252] [ 7884.387942] ? show_regs.cold.9+0x22/0x2f [ 7884.389147] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7884.390543] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7884.392312] ? watchdog+0x40/0x40 [ 7884.393249] ? hrtimer_interrupt+0x128/0x2c0 [ 7884.394852] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7884.396251] ? apic_timer_interrupt+0xf/0x20 [ 7884.397468] [ 7884.398259] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7884.399914] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7884.401392] do_raw_spin_lock+0x177/0x190 [ 7884.402507] _raw_spin_lock+0x21/0x30 [ 7884.403913] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7884.405833] ? do_raw_spin_unlock+0x75/0x190 [ 7884.407018] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7884.408965] ? do_wait_intr+0xf0/0xf0 [ 7884.410190] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7884.412377] mdc_reint+0x2d/0x110 [mdc] [ 7884.413829] mdc_unlink+0x658/0x870 [mdc] [ 7884.415124] lmv_unlink+0x34d/0xbd0 [lmv] [ 7884.416542] ll_unlink+0x24d/0x7f0 [lustre] [ 7884.418037] vfs_unlink+0x202/0x330 [ 7884.419168] do_unlinkat+0x2fa/0x460 [ 7884.420419] __x64_sys_unlink+0x37/0x50 [ 7884.421556] do_syscall_64+0xc1/0x440 [ 7884.422930] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7884.424484] RIP: 0033:0x7f151bdee10b [ 7884.426010] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7884.433390] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7884.435897] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 7884.438542] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 7884.440868] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7884.443645] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7884.446399] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7888.875253] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5958s! [ 7888.885761] Showing busy workqueues and worker pools: [ 7888.890160] workqueue events: flags=0x0 [ 7888.895010] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7888.902018] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 7888.914713] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7888.929801] pending: kfree_rcu_monitor [ 7888.938832] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7888.949312] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 7888.963079] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7888.970640] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 7888.973579] workqueue events_long: flags=0x0 [ 7888.986916] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7888.992460] pending: lu_objects_flush [obdclass] [ 7888.994504] workqueue events_unbound: flags=0x2 [ 7889.001972] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7889.006819] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7889.018064] workqueue events_power_efficient: flags=0x80 [ 7889.028019] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7889.038498] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 7889.050884] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7889.053866] pending: neigh_periodic_work [ 7889.060637] workqueue mm_percpu_wq: flags=0x8 [ 7889.063057] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7889.071342] pending: vmstat_update, lru_add_drain_per_cpu [ 7889.077476] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7889.082456] pending: vmstat_update [ 7889.085062] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7889.088953] pending: lru_add_drain_per_cpu [ 7889.094782] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7889.099644] pending: lru_add_drain_per_cpu BAR(48) [ 7889.102282] workqueue writeback: flags=0x4a [ 7889.107063] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 7889.109472] pending: wb_workfn [ 7889.110587] workqueue xprtiod: flags=0xa [ 7889.120414] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7889.123978] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7889.133415] workqueue ipv6_addrconf: flags=0x40008 [ 7889.137220] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7889.145944] pending: addrconf_verify_work [ 7889.149613] workqueue knbd0-recv: flags=0x1a [ 7889.151030] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7889.158338] in-flight: 658:recv_work [ 7889.161755] workqueue ptlrpc_pinger: flags=0x2 [ 7889.163922] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7889.166721] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7889.169314] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5958s workers=4 idle: 32984 37 8 [ 7889.172964] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7873s workers=2 idle: 661 [ 7900.056624] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7900.063877] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7900.110040] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7900.114297] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7900.117967] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7900.120835] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7900.123264] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7900.130985] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7900.134106] RAX: 0000000000005df3 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7900.136974] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 7900.139836] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 7900.142768] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 7900.145575] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 7900.147845] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 7900.150643] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7900.152920] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7900.155750] Call Trace: [ 7900.156872] [ 7900.157757] ? show_regs.cold.9+0x22/0x2f [ 7900.159286] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7900.161225] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7900.163141] ? watchdog+0x40/0x40 [ 7900.164675] ? hrtimer_interrupt+0x128/0x2c0 [ 7900.168242] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7900.170270] ? apic_timer_interrupt+0xf/0x20 [ 7900.172031] [ 7900.172938] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7900.175590] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7900.177698] do_raw_spin_lock+0x177/0x190 [ 7900.179243] _raw_spin_lock+0x21/0x30 [ 7900.180902] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7900.183135] process_one_work+0x2c8/0x700 [ 7900.184871] worker_thread+0x296/0x6e0 [ 7900.186299] ? rescuer_thread+0x570/0x570 [ 7900.188033] kthread+0x1d1/0x200 [ 7900.189593] ? set_kthread_struct+0x70/0x70 [ 7900.191252] ret_from_fork+0x1f/0x30 [ 7912.057068] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 7912.058588] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 7912.063487] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7912.063565] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7912.063573] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7912.063577] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7912.063592] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7912.063597] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7912.063601] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7912.063604] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 7912.063606] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7912.063609] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 7912.063611] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 7912.063615] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 7912.063618] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7912.063620] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 7912.063630] Call Trace: [ 7912.063635] [ 7912.063642] ? show_regs.cold.9+0x22/0x2f [ 7912.063652] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7912.063659] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7912.063664] ? watchdog+0x40/0x40 [ 7912.063668] ? hrtimer_interrupt+0x128/0x2c0 [ 7912.063673] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7912.063680] ? apic_timer_interrupt+0xf/0x20 [ 7912.063685] [ 7912.063689] ? native_safe_halt+0x2e/0x50 [ 7912.063692] kvm_wait+0xcd/0xe0 [ 7912.063699] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7912.063708] do_raw_spin_lock+0x177/0x190 [ 7912.063712] _raw_spin_lock+0x21/0x30 [ 7912.063717] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7912.064478] ? do_raw_spin_unlock+0x75/0x190 [ 7912.064663] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7912.065839] ? do_wait_intr+0xf0/0xf0 [ 7912.066172] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7912.076446] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7912.076553] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7912.076561] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7912.076568] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7912.076587] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7912.076592] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7912.076600] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 7912.076602] RDX: 0000000000007f8f RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 7912.076605] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 7912.076612] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 7912.076615] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7912.076618] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 7912.076621] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7912.076623] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7912.077217] Call Trace: [ 7912.077227] [ 7912.077233] ? show_regs.cold.9+0x22/0x2f [ 7912.077246] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7912.077508] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7912.077512] ? watchdog+0x40/0x40 [ 7912.077516] ? hrtimer_interrupt+0x128/0x2c0 [ 7912.077670] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7912.077771] ? apic_timer_interrupt+0xf/0x20 [ 7912.077778] [ 7912.077782] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7912.077788] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7912.077796] do_raw_spin_lock+0x177/0x190 [ 7912.077805] _raw_spin_lock+0x21/0x30 [ 7912.077835] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7912.078723] obd_mod_free+0x46/0x140 [ptlrpc] [ 7912.144724] mdc_close+0x53b/0xd70 [mdc] [ 7912.144780] lmv_close+0x18d/0x490 [lmv] [ 7912.144812] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7912.152402] mdc_commit_open+0x91/0xb0 [mdc] [ 7912.152477] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7912.152912] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7912.153593] after_reply+0xe2c/0x1780 [ptlrpc] [ 7912.153817] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7912.154652] ? __next_timer_interrupt+0x160/0x160 [ 7912.154664] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7912.155067] ? do_wait_intr+0xf0/0xf0 [ 7912.155078] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7912.155505] kthread+0x1d1/0x200 [ 7912.155512] ? set_kthread_struct+0x70/0x70 [ 7912.155516] ret_from_fork+0x1f/0x30 [ 7912.718284] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7912.724273] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7912.728214] ll_md_close+0x274/0x800 [lustre] [ 7912.730794] ll_file_release+0x128/0xc50 [lustre] [ 7912.740112] __fput+0x10d/0x440 [ 7912.744306] ____fput+0x12/0x20 [ 7912.751289] task_work_run+0xc6/0x110 [ 7912.755759] exit_to_usermode_loop+0x1dd/0x1f0 [ 7912.759064] do_syscall_64+0x3d6/0x440 [ 7912.760315] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7912.770255] RIP: 0033:0x7f151bee3cd8 [ 7912.776903] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7912.793842] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7912.798560] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 7912.803109] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 7912.810903] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7912.816887] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7912.822158] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7916.053526] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 7916.062537] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7916.103214] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7916.123902] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7916.131849] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7916.134233] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7916.147358] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7916.155087] RAX: 000000000000638d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7916.166633] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 7916.170138] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 7916.175098] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 7916.184634] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 7916.189056] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 7916.193565] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7916.200260] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 7916.205089] Call Trace: [ 7916.208940] [ 7916.209959] ? show_regs.cold.9+0x22/0x2f [ 7916.213158] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7916.215864] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7916.218707] ? watchdog+0x40/0x40 [ 7916.221092] ? hrtimer_interrupt+0x128/0x2c0 [ 7916.223373] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7916.228954] ? apic_timer_interrupt+0xf/0x20 [ 7916.230497] [ 7916.231302] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7916.243079] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7916.252435] do_raw_spin_lock+0x177/0x190 [ 7916.253850] _raw_spin_lock+0x21/0x30 [ 7916.257714] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7916.261091] ? do_raw_spin_unlock+0x75/0x190 [ 7916.263436] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7916.269007] ? do_wait_intr+0xf0/0xf0 [ 7916.272617] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7916.276204] mdc_reint+0x2d/0x110 [mdc] [ 7916.283515] mdc_unlink+0x658/0x870 [mdc] [ 7916.287421] lmv_unlink+0x34d/0xbd0 [lmv] [ 7916.288810] ll_unlink+0x24d/0x7f0 [lustre] [ 7916.290962] vfs_unlink+0x202/0x330 [ 7916.296805] do_unlinkat+0x2fa/0x460 [ 7916.304186] __x64_sys_unlink+0x37/0x50 [ 7916.306941] do_syscall_64+0xc1/0x440 [ 7916.312759] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7916.317602] RIP: 0033:0x7f151bdee10b [ 7916.319201] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7916.342125] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7916.353428] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 7916.357051] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 7916.367632] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7916.375221] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7916.379442] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7919.584042] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5988s! [ 7919.588773] Showing busy workqueues and worker pools: [ 7919.594262] workqueue events: flags=0x0 [ 7919.595629] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7919.600096] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 7919.609995] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7919.614225] pending: kfree_rcu_monitor [ 7919.618447] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7919.621628] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 7919.634078] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7919.639817] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 7919.645795] workqueue events_long: flags=0x0 [ 7919.649263] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7919.654790] pending: lu_objects_flush [obdclass] [ 7919.660978] workqueue events_unbound: flags=0x2 [ 7919.663603] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7919.666607] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7919.670857] workqueue events_power_efficient: flags=0x80 [ 7919.673583] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7919.678794] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 7919.681994] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7919.689026] pending: neigh_periodic_work [ 7919.694622] workqueue mm_percpu_wq: flags=0x8 [ 7919.699919] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7919.709381] pending: vmstat_update, lru_add_drain_per_cpu [ 7919.712087] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7919.715003] pending: vmstat_update [ 7919.720325] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7919.728538] pending: lru_add_drain_per_cpu [ 7919.731042] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7919.738925] pending: lru_add_drain_per_cpu BAR(48) [ 7919.742960] workqueue writeback: flags=0x4a [ 7919.747964] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 7919.754067] pending: wb_workfn [ 7919.756098] workqueue xprtiod: flags=0xa [ 7919.762371] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7919.767281] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7919.770211] workqueue ipv6_addrconf: flags=0x40008 [ 7919.776461] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7919.786631] pending: addrconf_verify_work [ 7919.790891] workqueue knbd0-recv: flags=0x1a [ 7919.792694] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7919.796718] in-flight: 658:recv_work [ 7919.798889] workqueue ptlrpc_pinger: flags=0x2 [ 7919.801122] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7919.805413] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7919.809481] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5989s workers=4 idle: 32984 37 8 [ 7919.813245] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7904s workers=2 idle: 661 [ 7924.777931] rcu: INFO: rcu_sched self-detected stall on CPU [ 7924.787294] rcu: 1-...!: (3404464 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 7924.793043] (t=6000130 jiffies g=249657 q=4073) [ 7924.801407] rcu: rcu_sched kthread starved for 5995082 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 7924.806868] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7924.814095] rcu: RCU grace-period kthread stack dump: [ 7924.816338] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7924.820647] Call Trace: [ 7924.821968] __schedule+0x351/0xcb0 [ 7924.824126] schedule+0xc0/0x180 [ 7924.825959] schedule_timeout+0xb4/0x190 [ 7924.828372] ? do_raw_spin_unlock+0x75/0x190 [ 7924.831045] ? __next_timer_interrupt+0x160/0x160 [ 7924.833261] ? prepare_to_swait_event+0xf3/0x140 [ 7924.835636] rcu_gp_kthread+0x8dd/0x15b0 [ 7924.837429] ? force_qs_rnp+0x2c0/0x2c0 [ 7924.839362] kthread+0x1d1/0x200 [ 7924.840977] ? set_kthread_struct+0x70/0x70 [ 7924.842792] ret_from_fork+0x1f/0x30 [ 7924.845126] rcu: Stack dump where RCU GP kthread last ran: [ 7924.848057] NMI backtrace for cpu 1 [ 7924.849782] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7924.855028] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7924.864151] Call Trace: [ 7924.866548] [ 7924.868604] ? dump_stack+0xbb/0x10e [ 7924.872390] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7924.874994] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7924.877731] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7924.879582] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7924.882335] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7924.884569] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 7924.887047] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 7924.890674] ? update_process_times+0x6d/0xc0 [ 7924.896400] ? tick_sched_handle.isra.18+0x27/0x90 [ 7924.900460] ? tick_sched_timer+0x53/0xb0 [ 7924.904869] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7924.907109] ? tick_sched_handle.isra.18+0x90/0x90 [ 7924.909615] ? hrtimer_interrupt+0x128/0x2c0 [ 7924.911641] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7924.913730] ? apic_timer_interrupt+0xf/0x20 [ 7924.915740] [ 7924.916700] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7924.919278] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7924.921873] ? do_raw_spin_lock+0x177/0x190 [ 7924.923750] ? _raw_spin_lock+0x21/0x30 [ 7924.926015] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7924.928783] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7924.932322] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7924.934167] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7924.937463] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7924.940662] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7924.943926] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7924.947709] ? __next_timer_interrupt+0x160/0x160 [ 7924.949870] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7924.953037] ? do_wait_intr+0xf0/0xf0 [ 7924.958259] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7924.964253] ? kthread+0x1d1/0x200 [ 7924.967169] ? set_kthread_struct+0x70/0x70 [ 7924.969985] ? ret_from_fork+0x1f/0x30 [ 7924.971758] NMI backtrace for cpu 1 [ 7924.973397] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7924.978771] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7924.983120] Call Trace: [ 7924.984395] [ 7924.985482] ? dump_stack+0xbb/0x10e [ 7924.987805] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7924.991544] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7924.995421] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7924.997686] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7925.000366] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7925.002503] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 7925.004488] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 7925.006757] ? update_process_times+0x6d/0xc0 [ 7925.008759] ? tick_sched_handle.isra.18+0x27/0x90 [ 7925.010983] ? tick_sched_timer+0x53/0xb0 [ 7925.012900] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7925.014762] ? tick_sched_handle.isra.18+0x90/0x90 [ 7925.018864] ? hrtimer_interrupt+0x128/0x2c0 [ 7925.020609] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7925.022778] ? apic_timer_interrupt+0xf/0x20 [ 7925.024806] [ 7925.025720] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7925.028343] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7925.030463] ? do_raw_spin_lock+0x177/0x190 [ 7925.032336] ? _raw_spin_lock+0x21/0x30 [ 7925.034690] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7925.038389] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7925.040552] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7925.042251] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7925.044548] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7925.046958] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7925.049474] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7925.051913] ? __next_timer_interrupt+0x160/0x160 [ 7925.055324] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7925.058034] ? do_wait_intr+0xf0/0xf0 [ 7925.059585] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7925.063889] ? kthread+0x1d1/0x200 [ 7925.068709] ? set_kthread_struct+0x70/0x70 [ 7925.070136] ? ret_from_fork+0x1f/0x30 [ 7928.057757] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7928.061961] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7928.131397] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7928.146809] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7928.153611] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7928.162430] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7928.170034] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7928.187220] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 7928.191132] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7928.195580] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 7928.198574] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 7928.210836] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 7928.216952] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 7928.224874] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 7928.234952] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7928.241886] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7928.248733] Call Trace: [ 7928.250157] [ 7928.252727] ? show_regs.cold.9+0x22/0x2f [ 7928.257591] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7928.261908] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7928.267719] ? watchdog+0x40/0x40 [ 7928.271058] ? hrtimer_interrupt+0x128/0x2c0 [ 7928.274441] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7928.279897] ? apic_timer_interrupt+0xf/0x20 [ 7928.283255] [ 7928.286576] ? native_safe_halt+0x2e/0x50 [ 7928.290908] kvm_wait+0xcd/0xe0 [ 7928.294161] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7928.299047] do_raw_spin_lock+0x177/0x190 [ 7928.302041] _raw_spin_lock+0x21/0x30 [ 7928.319307] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7928.321389] process_one_work+0x2c8/0x700 [ 7928.328013] worker_thread+0x296/0x6e0 [ 7928.329444] ? rescuer_thread+0x570/0x570 [ 7928.338723] kthread+0x1d1/0x200 [ 7928.341906] ? set_kthread_struct+0x70/0x70 [ 7928.350640] ret_from_fork+0x1f/0x30 [ 7940.058440] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 7940.066672] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7940.138280] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7940.149372] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7940.156865] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7940.165055] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7940.192851] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7940.200370] RAX: 0000000000007984 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7940.203765] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 7940.209669] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7940.216376] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 7940.221673] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 7940.224334] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 7940.239617] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7940.249681] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 7940.260587] Call Trace: [ 7940.261538] [ 7940.262267] ? show_regs.cold.9+0x22/0x2f [ 7940.263806] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7940.273166] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7940.276711] ? watchdog+0x40/0x40 [ 7940.278450] ? hrtimer_interrupt+0x128/0x2c0 [ 7940.280607] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7940.283183] ? apic_timer_interrupt+0xf/0x20 [ 7940.285285] [ 7940.286714] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7940.289518] do_raw_spin_lock+0x177/0x190 [ 7940.291355] _raw_spin_lock+0x21/0x30 [ 7940.292541] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7940.295130] ? do_raw_spin_unlock+0x75/0x190 [ 7940.296603] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7940.299481] ? do_wait_intr+0xf0/0xf0 [ 7940.301199] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7940.303362] mdc_close+0x53b/0xd70 [mdc] [ 7940.304981] lmv_close+0x18d/0x490 [lmv] [ 7940.306597] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7940.308670] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7940.311389] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7940.313391] ll_md_close+0x274/0x800 [lustre] [ 7940.315169] ll_file_release+0x128/0xc50 [lustre] [ 7940.317260] __fput+0x10d/0x440 [ 7940.318318] ____fput+0x12/0x20 [ 7940.319507] task_work_run+0xc6/0x110 [ 7940.320903] exit_to_usermode_loop+0x1dd/0x1f0 [ 7940.322788] do_syscall_64+0x3d6/0x440 [ 7940.324006] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7940.326258] RIP: 0033:0x7f151bee3cd8 [ 7940.327652] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7940.338591] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7940.344015] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 7940.346927] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 7940.356980] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7940.361084] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7940.364529] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7944.057101] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 7944.061685] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7944.109001] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7944.132022] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7944.144391] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7944.153630] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7944.183038] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7944.190939] RAX: 0000000000006883 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7944.199466] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 7944.207113] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 7944.211771] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 7944.216100] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 7944.224839] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 7944.237618] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7944.242951] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 7944.256973] Call Trace: [ 7944.261449] [ 7944.264567] ? show_regs.cold.9+0x22/0x2f [ 7944.267428] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7944.274167] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7944.275778] ? watchdog+0x40/0x40 [ 7944.284475] ? hrtimer_interrupt+0x128/0x2c0 [ 7944.287518] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7944.290719] ? apic_timer_interrupt+0xf/0x20 [ 7944.297336] [ 7944.299867] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7944.304672] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7944.307674] do_raw_spin_lock+0x177/0x190 [ 7944.311968] _raw_spin_lock+0x21/0x30 [ 7944.313846] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7944.322334] ? do_raw_spin_unlock+0x75/0x190 [ 7944.325424] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7944.328640] ? do_wait_intr+0xf0/0xf0 [ 7944.330444] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7944.340017] mdc_reint+0x2d/0x110 [mdc] [ 7944.346597] mdc_unlink+0x658/0x870 [mdc] [ 7944.348699] lmv_unlink+0x34d/0xbd0 [lmv] [ 7944.352590] ll_unlink+0x24d/0x7f0 [lustre] [ 7944.356367] vfs_unlink+0x202/0x330 [ 7944.359428] do_unlinkat+0x2fa/0x460 [ 7944.364717] __x64_sys_unlink+0x37/0x50 [ 7944.366757] do_syscall_64+0xc1/0x440 [ 7944.368509] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7944.373612] RIP: 0033:0x7f151bdee10b [ 7944.377597] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7944.404354] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7944.415377] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 7944.428968] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 7944.439777] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7944.450600] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7944.460838] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7950.312223] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6019s! [ 7950.327683] Showing busy workqueues and worker pools: [ 7950.332840] workqueue events: flags=0x0 [ 7950.335991] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7950.343759] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 7950.353895] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7950.360171] pending: kfree_rcu_monitor [ 7950.367472] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7950.369616] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 7950.378884] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7950.388801] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 7950.400824] workqueue events_long: flags=0x0 [ 7950.408176] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7950.422827] pending: lu_objects_flush [obdclass] [ 7950.435159] workqueue events_unbound: flags=0x2 [ 7950.443199] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7950.461726] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7950.493487] workqueue events_power_efficient: flags=0x80 [ 7950.513376] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7950.532767] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 7950.549515] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7950.564478] pending: neigh_periodic_work [ 7950.568692] workqueue mm_percpu_wq: flags=0x8 [ 7950.576348] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7950.587048] pending: vmstat_update, lru_add_drain_per_cpu [ 7950.593531] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7950.604002] pending: vmstat_update [ 7950.605479] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7950.621622] pending: lru_add_drain_per_cpu [ 7950.635382] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7950.647768] pending: lru_add_drain_per_cpu BAR(48) [ 7950.656497] workqueue writeback: flags=0x4a [ 7950.660430] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 7950.665197] pending: wb_workfn [ 7950.674076] workqueue xprtiod: flags=0xa [ 7950.680693] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7950.686737] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7950.697906] workqueue ipv6_addrconf: flags=0x40008 [ 7950.706206] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7950.721940] pending: addrconf_verify_work [ 7950.725550] workqueue knbd0-recv: flags=0x1a [ 7950.737516] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7950.742113] in-flight: 658:recv_work [ 7950.745901] workqueue ptlrpc_pinger: flags=0x2 [ 7950.756667] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7950.763174] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7950.772515] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6020s workers=4 idle: 32984 37 8 [ 7950.783420] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7935s workers=2 idle: 661 [ 7952.057090] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 7952.062467] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7952.098451] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7952.122921] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7952.140556] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7952.142683] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7952.188747] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7952.191939] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 7952.196652] RDX: 00000000000050b9 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 7952.206316] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 7952.213752] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 7952.220433] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7952.236362] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 7952.241419] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7952.245157] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7952.250400] Call Trace: [ 7952.253172] [ 7952.258265] ? show_regs.cold.9+0x22/0x2f [ 7952.264192] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7952.272854] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7952.275433] ? watchdog+0x40/0x40 [ 7952.281917] ? hrtimer_interrupt+0x128/0x2c0 [ 7952.283391] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7952.284973] ? apic_timer_interrupt+0xf/0x20 [ 7952.289321] [ 7952.291074] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7952.296349] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7952.300140] do_raw_spin_lock+0x177/0x190 [ 7952.302342] _raw_spin_lock+0x21/0x30 [ 7952.304350] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7952.307897] obd_mod_free+0x46/0x140 [ptlrpc] [ 7952.312450] mdc_commit_open+0x91/0xb0 [mdc] [ 7952.315330] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7952.326447] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7952.340369] after_reply+0xe2c/0x1780 [ptlrpc] [ 7952.350514] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7952.356241] ? __next_timer_interrupt+0x160/0x160 [ 7952.357871] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7952.374291] ? do_wait_intr+0xf0/0xf0 [ 7952.379480] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7952.390302] kthread+0x1d1/0x200 [ 7952.395249] ? set_kthread_struct+0x70/0x70 [ 7952.403703] ret_from_fork+0x1f/0x30 [ 7956.056961] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7956.066318] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7956.142839] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7956.153634] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7956.180985] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7956.194783] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7956.207488] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7956.229167] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7956.233261] RAX: 0000000000004746 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7956.249744] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 7956.254460] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 7956.264862] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 7956.275700] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 7956.285255] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 7956.288610] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7956.297691] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7956.303930] Call Trace: [ 7956.304544] [ 7956.305066] ? show_regs.cold.9+0x22/0x2f [ 7956.314509] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7956.320351] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7956.331293] ? watchdog+0x40/0x40 [ 7956.334705] ? hrtimer_interrupt+0x128/0x2c0 [ 7956.339664] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7956.344641] ? apic_timer_interrupt+0xf/0x20 [ 7956.348747] [ 7956.351747] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7956.355574] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7956.357713] do_raw_spin_lock+0x177/0x190 [ 7956.363652] _raw_spin_lock+0x21/0x30 [ 7956.368990] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7956.373560] process_one_work+0x2c8/0x700 [ 7956.379559] worker_thread+0x296/0x6e0 [ 7956.380870] ? rescuer_thread+0x570/0x570 [ 7956.387905] kthread+0x1d1/0x200 [ 7956.389083] ? set_kthread_struct+0x70/0x70 [ 7956.390512] ret_from_fork+0x1f/0x30 [ 7968.060136] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 7968.063268] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7968.082876] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7968.097934] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7968.106558] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7968.119540] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7968.137858] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7968.140619] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7968.148653] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 7968.159684] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7968.163154] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 7968.169082] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 7968.172710] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 7968.186255] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7968.196229] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 7968.207544] Call Trace: [ 7968.208568] [ 7968.210707] ? show_regs.cold.9+0x22/0x2f [ 7968.216907] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7968.225186] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7968.228556] ? watchdog+0x40/0x40 [ 7968.236984] ? hrtimer_interrupt+0x128/0x2c0 [ 7968.242344] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7968.245938] ? apic_timer_interrupt+0xf/0x20 [ 7968.252217] [ 7968.254653] ? native_safe_halt+0x2e/0x50 [ 7968.256828] kvm_wait+0xcd/0xe0 [ 7968.263329] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7968.269403] do_raw_spin_lock+0x177/0x190 [ 7968.271136] _raw_spin_lock+0x21/0x30 [ 7968.272877] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7968.278964] ? do_raw_spin_unlock+0x75/0x190 [ 7968.281844] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7968.285194] ? do_wait_intr+0xf0/0xf0 [ 7968.287919] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7968.290611] mdc_close+0x53b/0xd70 [mdc] [ 7968.293310] lmv_close+0x18d/0x490 [lmv] [ 7968.299796] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7968.302328] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7968.306328] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7968.310338] ll_md_close+0x274/0x800 [lustre] [ 7968.314714] ll_file_release+0x128/0xc50 [lustre] [ 7968.318471] __fput+0x10d/0x440 [ 7968.320216] ____fput+0x12/0x20 [ 7968.322708] task_work_run+0xc6/0x110 [ 7968.324168] exit_to_usermode_loop+0x1dd/0x1f0 [ 7968.329895] do_syscall_64+0x3d6/0x440 [ 7968.331633] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7968.334499] RIP: 0033:0x7f151bee3cd8 [ 7968.336180] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7968.346523] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7968.350626] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 7968.354015] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 7968.356967] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7968.364500] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7968.372194] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7976.055431] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 7976.068708] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7976.123339] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7976.139574] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7976.153396] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7976.159295] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7976.183182] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 7976.189988] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7976.200571] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 7976.212392] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 7976.229283] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 7976.242199] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 7976.253713] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 7976.268659] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7976.276144] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 7976.283657] Call Trace: [ 7976.285127] [ 7976.286149] ? show_regs.cold.9+0x22/0x2f [ 7976.291072] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7976.297308] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7976.300496] ? watchdog+0x40/0x40 [ 7976.304198] ? hrtimer_interrupt+0x128/0x2c0 [ 7976.309199] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7976.314337] ? apic_timer_interrupt+0xf/0x20 [ 7976.318212] [ 7976.319429] ? native_safe_halt+0x2e/0x50 [ 7976.321348] kvm_wait+0xcd/0xe0 [ 7976.322424] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7976.324229] do_raw_spin_lock+0x177/0x190 [ 7976.325689] _raw_spin_lock+0x21/0x30 [ 7976.329456] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7976.336295] ? do_raw_spin_unlock+0x75/0x190 [ 7976.344118] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7976.350491] ? do_wait_intr+0xf0/0xf0 [ 7976.358875] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7976.365036] mdc_reint+0x2d/0x110 [mdc] [ 7976.374939] mdc_unlink+0x658/0x870 [mdc] [ 7976.382361] lmv_unlink+0x34d/0xbd0 [lmv] [ 7976.387563] ll_unlink+0x24d/0x7f0 [lustre] [ 7976.392047] vfs_unlink+0x202/0x330 [ 7976.393349] do_unlinkat+0x2fa/0x460 [ 7976.394395] __x64_sys_unlink+0x37/0x50 [ 7976.395433] do_syscall_64+0xc1/0x440 [ 7976.396417] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7976.398121] RIP: 0033:0x7f151bdee10b [ 7976.399631] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7976.423023] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7976.440387] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 7976.457277] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 7976.459792] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7976.479934] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7976.485027] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7980.057478] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 7980.065520] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7980.123806] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7980.135833] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7980.145605] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7980.150422] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7980.185356] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7980.200444] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 7980.202730] RDX: 0000000000002a6c RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 7980.218882] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 7980.227856] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 7980.237323] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7980.242377] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 7980.248995] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7980.254714] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7980.259563] Call Trace: [ 7980.260733] [ 7980.262073] ? show_regs.cold.9+0x22/0x2f [ 7980.264066] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7980.266610] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7980.272359] ? watchdog+0x40/0x40 [ 7980.274348] ? hrtimer_interrupt+0x128/0x2c0 [ 7980.280454] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7980.283631] ? apic_timer_interrupt+0xf/0x20 [ 7980.289440] [ 7980.292884] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7980.299019] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7980.310343] do_raw_spin_lock+0x177/0x190 [ 7980.313893] _raw_spin_lock+0x21/0x30 [ 7980.319807] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7980.321739] obd_mod_free+0x46/0x140 [ptlrpc] [ 7980.327076] mdc_commit_open+0x91/0xb0 [mdc] [ 7980.328638] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7980.340218] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7980.342729] after_reply+0xe2c/0x1780 [ptlrpc] [ 7980.344747] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7980.346828] ? __next_timer_interrupt+0x160/0x160 [ 7980.348609] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7980.350334] ? do_wait_intr+0xf0/0xf0 [ 7980.351551] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7980.353456] kthread+0x1d1/0x200 [ 7980.354977] ? set_kthread_struct+0x70/0x70 [ 7980.356443] ret_from_fork+0x1f/0x30 [ 7981.028025] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6050s! [ 7981.032902] Showing busy workqueues and worker pools: [ 7981.035641] workqueue events: flags=0x0 [ 7981.037142] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7981.039491] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 7981.044124] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7981.046624] pending: kfree_rcu_monitor [ 7981.054229] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7981.061969] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 7981.067435] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7981.071075] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 7981.074811] workqueue events_long: flags=0x0 [ 7981.076962] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7981.080348] pending: lu_objects_flush [obdclass] [ 7981.083265] workqueue events_unbound: flags=0x2 [ 7981.085506] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 7981.088153] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 7981.093593] workqueue events_power_efficient: flags=0x80 [ 7981.101224] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7981.116606] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 7981.124549] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7981.133360] pending: neigh_periodic_work [ 7981.137163] workqueue mm_percpu_wq: flags=0x8 [ 7981.141634] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7981.150775] pending: vmstat_update, lru_add_drain_per_cpu [ 7981.157136] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7981.167549] pending: vmstat_update [ 7981.169863] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7981.175825] pending: lru_add_drain_per_cpu [ 7981.180504] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7981.195450] pending: lru_add_drain_per_cpu BAR(48) [ 7981.203517] workqueue writeback: flags=0x4a [ 7981.212637] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 7981.220459] pending: wb_workfn [ 7981.229568] workqueue xprtiod: flags=0xa [ 7981.231897] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 7981.235503] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 7981.240983] workqueue ipv6_addrconf: flags=0x40008 [ 7981.242728] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7981.254269] pending: addrconf_verify_work [ 7981.257525] workqueue knbd0-recv: flags=0x1a [ 7981.261399] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7981.264895] in-flight: 658:recv_work [ 7981.266093] workqueue ptlrpc_pinger: flags=0x2 [ 7981.269239] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7981.272278] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 7981.279341] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6050s workers=4 idle: 32984 37 8 [ 7981.285567] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7966s workers=2 idle: 661 [ 7984.056065] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 7984.061017] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7984.104384] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7984.110536] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7984.120969] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7984.124355] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7984.127560] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7984.153486] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 7984.165493] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7984.178266] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 7984.187885] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 7984.201616] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 7984.207059] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 7984.210632] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 7984.216229] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7984.223218] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 7984.227700] Call Trace: [ 7984.228658] [ 7984.229352] ? show_regs.cold.9+0x22/0x2f [ 7984.235197] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7984.237042] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7984.245326] ? watchdog+0x40/0x40 [ 7984.255379] ? hrtimer_interrupt+0x128/0x2c0 [ 7984.261293] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7984.264456] ? apic_timer_interrupt+0xf/0x20 [ 7984.276488] [ 7984.278297] ? native_safe_halt+0x2e/0x50 [ 7984.281387] kvm_wait+0xcd/0xe0 [ 7984.284206] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7984.286081] do_raw_spin_lock+0x177/0x190 [ 7984.295169] _raw_spin_lock+0x21/0x30 [ 7984.303070] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7984.316173] process_one_work+0x2c8/0x700 [ 7984.323124] worker_thread+0x296/0x6e0 [ 7984.326209] ? rescuer_thread+0x570/0x570 [ 7984.327566] kthread+0x1d1/0x200 [ 7984.336438] ? set_kthread_struct+0x70/0x70 [ 7984.344452] ret_from_fork+0x1f/0x30 [ 7996.057438] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 7996.063632] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7996.085757] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7996.094291] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7996.098576] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7996.101271] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7996.109170] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7996.113025] RAX: 0000000000006476 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 7996.116605] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 7996.119736] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7996.123340] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 7996.127128] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 7996.131708] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 7996.135910] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7996.139214] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 7996.142758] Call Trace: [ 7996.143644] [ 7996.144479] ? show_regs.cold.9+0x22/0x2f [ 7996.146397] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7996.149719] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7996.153036] ? watchdog+0x40/0x40 [ 7996.154468] ? hrtimer_interrupt+0x128/0x2c0 [ 7996.157079] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7996.159220] ? apic_timer_interrupt+0xf/0x20 [ 7996.161595] [ 7996.162726] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7996.165275] do_raw_spin_lock+0x177/0x190 [ 7996.167373] _raw_spin_lock+0x21/0x30 [ 7996.169574] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7996.172389] ? do_raw_spin_unlock+0x75/0x190 [ 7996.174002] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7996.176739] ? do_wait_intr+0xf0/0xf0 [ 7996.178431] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7996.181042] mdc_close+0x53b/0xd70 [mdc] [ 7996.182666] lmv_close+0x18d/0x490 [lmv] [ 7996.184636] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7996.188216] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 7996.191566] ll_md_real_close+0xa5/0x2a0 [lustre] [ 7996.194092] ll_md_close+0x274/0x800 [lustre] [ 7996.197491] ll_file_release+0x128/0xc50 [lustre] [ 7996.198991] __fput+0x10d/0x440 [ 7996.200659] ____fput+0x12/0x20 [ 7996.202596] task_work_run+0xc6/0x110 [ 7996.205431] exit_to_usermode_loop+0x1dd/0x1f0 [ 7996.207566] do_syscall_64+0x3d6/0x440 [ 7996.209209] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7996.211399] RIP: 0033:0x7f151bee3cd8 [ 7996.212607] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 7996.224287] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 7996.228115] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 7996.231219] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 7996.234384] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 7996.237393] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 7996.241181] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8008.052148] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 8008.055971] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 8008.062219] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8008.062300] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8008.062308] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8008.062311] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8008.062326] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8008.062499] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 8008.062513] RAX: 0000000000001f0a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8008.062515] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 8008.062517] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 8008.062519] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 8008.062522] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 8008.062525] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 8008.062529] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8008.062531] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 8008.062539] Call Trace: [ 8008.062545] [ 8008.062550] ? show_regs.cold.9+0x22/0x2f [ 8008.062561] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8008.062567] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8008.062570] ? watchdog+0x40/0x40 [ 8008.062575] ? hrtimer_interrupt+0x128/0x2c0 [ 8008.062579] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8008.062587] ? apic_timer_interrupt+0xf/0x20 [ 8008.062592] [ 8008.062594] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8008.062599] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8008.062604] do_raw_spin_lock+0x177/0x190 [ 8008.062616] _raw_spin_lock+0x21/0x30 [ 8008.062627] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8008.063853] ? do_raw_spin_unlock+0x75/0x190 [ 8008.063862] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8008.064078] ? do_wait_intr+0xf0/0xf0 [ 8008.064087] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8008.064298] mdc_reint+0x2d/0x110 [mdc] [ 8008.075121] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8008.075201] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8008.075209] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8008.075212] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8008.075227] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8008.075232] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8008.075236] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 8008.075238] RDX: 0000000000007f82 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 8008.075241] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 8008.075243] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 8008.075246] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8008.075249] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 8008.075252] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8008.075255] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8008.075264] Call Trace: [ 8008.075269] [ 8008.075275] ? show_regs.cold.9+0x22/0x2f [ 8008.075284] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8008.075291] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8008.075295] ? watchdog+0x40/0x40 [ 8008.075299] ? hrtimer_interrupt+0x128/0x2c0 [ 8008.075304] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8008.075311] ? apic_timer_interrupt+0xf/0x20 [ 8008.075316] [ 8008.075319] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8008.075324] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8008.075329] do_raw_spin_lock+0x177/0x190 [ 8008.075334] _raw_spin_lock+0x21/0x30 [ 8008.075339] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8008.075649] obd_mod_free+0x46/0x140 [ptlrpc] [ 8008.101785] mdc_unlink+0x658/0x870 [mdc] [ 8008.101838] lmv_unlink+0x34d/0xbd0 [lmv] [ 8008.101872] ll_unlink+0x24d/0x7f0 [lustre] [ 8008.113223] mdc_commit_open+0x91/0xb0 [mdc] [ 8008.113282] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8008.120222] vfs_unlink+0x202/0x330 [ 8008.120244] do_unlinkat+0x2fa/0x460 [ 8008.120250] __x64_sys_unlink+0x37/0x50 [ 8008.120254] do_syscall_64+0xc1/0x440 [ 8008.120261] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8008.120272] RIP: 0033:0x7f151bdee10b [ 8008.120281] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8008.120284] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8008.120290] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 8008.120292] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 8008.120294] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8008.120296] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8008.120299] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8008.435915] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8008.438834] after_reply+0xe2c/0x1780 [ptlrpc] [ 8008.441512] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8008.444176] ? __next_timer_interrupt+0x160/0x160 [ 8008.446578] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8008.448867] ? do_wait_intr+0xf0/0xf0 [ 8008.450599] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8008.455569] kthread+0x1d1/0x200 [ 8008.456692] ? set_kthread_struct+0x70/0x70 [ 8008.459099] ret_from_fork+0x1f/0x30 [ 8011.746378] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6081s! [ 8011.754906] Showing busy workqueues and worker pools: [ 8011.759768] workqueue events: flags=0x0 [ 8011.766562] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8011.769117] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 8011.777272] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8011.781815] pending: kfree_rcu_monitor [ 8011.784583] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8011.787127] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 8011.793264] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8011.796826] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 8011.800508] workqueue events_long: flags=0x0 [ 8011.802617] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8011.806179] pending: lu_objects_flush [obdclass] [ 8011.811856] workqueue events_unbound: flags=0x2 [ 8011.814391] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8011.818773] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8011.826543] workqueue events_power_efficient: flags=0x80 [ 8011.833491] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8011.842681] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 8011.849149] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8011.854272] pending: neigh_periodic_work [ 8011.856657] workqueue mm_percpu_wq: flags=0x8 [ 8011.859244] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8011.863032] pending: vmstat_update, lru_add_drain_per_cpu [ 8011.866163] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8011.869534] pending: vmstat_update [ 8011.873200] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8011.876312] pending: lru_add_drain_per_cpu [ 8011.878162] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8011.881674] pending: lru_add_drain_per_cpu BAR(48) [ 8011.884195] workqueue writeback: flags=0x4a [ 8011.886699] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 8011.890771] pending: wb_workfn [ 8011.892638] workqueue xprtiod: flags=0xa [ 8011.894946] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8011.898277] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8011.901522] workqueue ipv6_addrconf: flags=0x40008 [ 8011.903457] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8011.906743] pending: addrconf_verify_work [ 8011.909168] workqueue knbd0-recv: flags=0x1a [ 8011.911633] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8011.917801] in-flight: 658:recv_work [ 8011.919977] workqueue ptlrpc_pinger: flags=0x2 [ 8011.921623] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8011.924943] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8011.930137] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6081s workers=4 idle: 32984 37 8 [ 8011.939292] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7996s workers=2 idle: 661 [ 8012.056010] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8012.063955] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8012.107136] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8012.115845] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8012.122412] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8012.127396] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30f/0x6d0 [ 8012.132450] Code: c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 <0f> 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 [ 8012.140323] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8012.146701] RAX: 000000000000694e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8012.150178] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 8012.154070] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 8012.158137] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 8012.161283] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 8012.166398] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 8012.171257] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8012.179002] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8012.184323] Call Trace: [ 8012.185069] [ 8012.186454] ? show_regs.cold.9+0x22/0x2f [ 8012.188179] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8012.190450] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8012.192188] ? watchdog+0x40/0x40 [ 8012.193303] ? hrtimer_interrupt+0x128/0x2c0 [ 8012.194870] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8012.196912] ? apic_timer_interrupt+0xf/0x20 [ 8012.198778] [ 8012.199679] ? __pv_queued_spin_lock_slowpath+0x30f/0x6d0 [ 8012.201760] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8012.204110] do_raw_spin_lock+0x177/0x190 [ 8012.207822] _raw_spin_lock+0x21/0x30 [ 8012.211086] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8012.213962] process_one_work+0x2c8/0x700 [ 8012.215485] worker_thread+0x296/0x6e0 [ 8012.216905] ? rescuer_thread+0x570/0x570 [ 8012.218109] kthread+0x1d1/0x200 [ 8012.218946] ? set_kthread_struct+0x70/0x70 [ 8012.220178] ret_from_fork+0x1f/0x30 [ 8024.057527] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 8024.063630] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8024.096312] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8024.113842] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8024.119698] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8024.123451] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8024.137262] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8024.143795] RAX: 000000000000538c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8024.149434] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 8024.154468] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8024.159677] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 8024.161823] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 8024.165094] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 8024.169758] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8024.172667] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 8024.184091] Call Trace: [ 8024.189379] [ 8024.190134] ? show_regs.cold.9+0x22/0x2f [ 8024.191686] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8024.201870] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8024.209896] ? watchdog+0x40/0x40 [ 8024.213780] ? hrtimer_interrupt+0x128/0x2c0 [ 8024.219966] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8024.224201] ? apic_timer_interrupt+0xf/0x20 [ 8024.230820] [ 8024.233440] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8024.243634] do_raw_spin_lock+0x177/0x190 [ 8024.251796] _raw_spin_lock+0x21/0x30 [ 8024.253617] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8024.255665] ? do_raw_spin_unlock+0x75/0x190 [ 8024.262530] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8024.265234] ? do_wait_intr+0xf0/0xf0 [ 8024.268010] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8024.273847] mdc_close+0x53b/0xd70 [mdc] [ 8024.281068] lmv_close+0x18d/0x490 [lmv] [ 8024.286059] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8024.293978] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8024.299078] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8024.305113] ll_md_close+0x274/0x800 [lustre] [ 8024.315020] ll_file_release+0x128/0xc50 [lustre] [ 8024.319872] __fput+0x10d/0x440 [ 8024.321393] ____fput+0x12/0x20 [ 8024.323627] task_work_run+0xc6/0x110 [ 8024.325906] exit_to_usermode_loop+0x1dd/0x1f0 [ 8024.328062] do_syscall_64+0x3d6/0x440 [ 8024.329988] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8024.332351] RIP: 0033:0x7f151bee3cd8 [ 8024.334872] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8024.354381] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8024.357557] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 8024.362960] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 8024.367897] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8024.370740] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8024.379077] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8036.052025] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 8036.055445] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8036.059609] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 8036.091450] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8036.091463] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8036.091465] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8036.102013] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8036.102103] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8036.102111] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8036.102114] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8036.102129] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8036.102133] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8036.102139] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 8036.102141] RDX: 00000000000052f6 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 8036.102143] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 8036.102146] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 8036.102148] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8036.102152] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 8036.102155] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8036.102158] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8036.102166] Call Trace: [ 8036.102172] [ 8036.102177] ? show_regs.cold.9+0x22/0x2f [ 8036.102187] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8036.102194] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8036.102198] ? watchdog+0x40/0x40 [ 8036.102202] ? hrtimer_interrupt+0x128/0x2c0 [ 8036.102206] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8036.102214] ? apic_timer_interrupt+0xf/0x20 [ 8036.102218] [ 8036.102221] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8036.102226] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8036.102245] do_raw_spin_lock+0x177/0x190 [ 8036.102249] _raw_spin_lock+0x21/0x30 [ 8036.102256] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8036.119047] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8036.119061] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8036.119069] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8036.119071] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 8036.119073] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 8036.119075] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 8036.119077] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 8036.119081] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 8036.119084] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8036.119086] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 8036.119418] Call Trace: [ 8036.119426] [ 8036.119431] ? show_regs.cold.9+0x22/0x2f [ 8036.119446] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8036.119453] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8036.119459] ? watchdog+0x40/0x40 [ 8036.119463] ? hrtimer_interrupt+0x128/0x2c0 [ 8036.119467] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8036.119476] ? apic_timer_interrupt+0xf/0x20 [ 8036.119480] [ 8036.119483] ? native_safe_halt+0x2e/0x50 [ 8036.119487] kvm_wait+0xcd/0xe0 [ 8036.119495] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8036.119502] do_raw_spin_lock+0x177/0x190 [ 8036.119506] _raw_spin_lock+0x21/0x30 [ 8036.119513] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8036.123752] obd_mod_free+0x46/0x140 [ptlrpc] [ 8036.124022] mdc_commit_open+0x91/0xb0 [mdc] [ 8036.124057] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8036.179808] ? do_raw_spin_unlock+0x75/0x190 [ 8036.179828] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8036.180092] ? do_wait_intr+0xf0/0xf0 [ 8036.180099] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8036.198164] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8036.198483] after_reply+0xe2c/0x1780 [ptlrpc] [ 8036.198688] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8036.198899] ? __next_timer_interrupt+0x160/0x160 [ 8036.198909] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8036.199117] ? do_wait_intr+0xf0/0xf0 [ 8036.199126] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8036.199350] kthread+0x1d1/0x200 [ 8036.199356] ? set_kthread_struct+0x70/0x70 [ 8036.199360] ret_from_fork+0x1f/0x30 [ 8036.563677] mdc_reint+0x2d/0x110 [mdc] [ 8036.566407] mdc_unlink+0x658/0x870 [mdc] [ 8036.567912] lmv_unlink+0x34d/0xbd0 [lmv] [ 8036.569758] ll_unlink+0x24d/0x7f0 [lustre] [ 8036.571704] vfs_unlink+0x202/0x330 [ 8036.573217] do_unlinkat+0x2fa/0x460 [ 8036.574680] __x64_sys_unlink+0x37/0x50 [ 8036.576311] do_syscall_64+0xc1/0x440 [ 8036.577977] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8036.580890] RIP: 0033:0x7f151bdee10b [ 8036.584811] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8036.610404] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8036.617557] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 8036.626595] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 8036.632808] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8036.643599] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8036.650538] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8040.056188] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8040.059800] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8040.083262] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8040.087545] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8040.091277] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8040.094633] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8040.097663] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8040.106637] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8040.110532] RAX: 0000000000005a1d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8040.113608] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 8040.117331] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 8040.120831] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 8040.123974] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 8040.128267] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 8040.131476] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8040.134045] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8040.137030] Call Trace: [ 8040.137984] [ 8040.139047] ? show_regs.cold.9+0x22/0x2f [ 8040.141061] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8040.143482] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8040.146251] ? watchdog+0x40/0x40 [ 8040.147705] ? hrtimer_interrupt+0x128/0x2c0 [ 8040.149467] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8040.151756] ? apic_timer_interrupt+0xf/0x20 [ 8040.154494] [ 8040.155598] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8040.157989] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8040.160630] do_raw_spin_lock+0x177/0x190 [ 8040.162709] _raw_spin_lock+0x21/0x30 [ 8040.164480] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8040.166971] process_one_work+0x2c8/0x700 [ 8040.169403] worker_thread+0x296/0x6e0 [ 8040.171300] ? rescuer_thread+0x570/0x570 [ 8040.173474] kthread+0x1d1/0x200 [ 8040.174933] ? set_kthread_struct+0x70/0x70 [ 8040.177918] ret_from_fork+0x1f/0x30 [ 8041.953350] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6111s! [ 8041.973314] Showing busy workqueues and worker pools: [ 8041.978064] workqueue events: flags=0x0 [ 8041.979238] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8041.982250] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 8041.986546] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8041.992824] pending: kfree_rcu_monitor [ 8041.997053] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8042.008766] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 8042.019705] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8042.024581] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 8042.029673] workqueue events_long: flags=0x0 [ 8042.033204] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8042.036833] pending: lu_objects_flush [obdclass] [ 8042.038997] workqueue events_unbound: flags=0x2 [ 8042.040892] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8042.043462] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8042.049742] workqueue events_power_efficient: flags=0x80 [ 8042.052449] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8042.055350] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 8042.058231] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8042.060784] pending: neigh_periodic_work [ 8042.065293] workqueue mm_percpu_wq: flags=0x8 [ 8042.067038] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8042.071957] pending: vmstat_update, lru_add_drain_per_cpu [ 8042.074579] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8042.079908] pending: vmstat_update [ 8042.085117] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8042.090701] pending: lru_add_drain_per_cpu [ 8042.098606] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8042.109313] pending: lru_add_drain_per_cpu BAR(48) [ 8042.111729] workqueue writeback: flags=0x4a [ 8042.113866] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 8042.117568] pending: wb_workfn [ 8042.119432] workqueue xprtiod: flags=0xa [ 8042.121635] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8042.127596] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8042.136440] workqueue ipv6_addrconf: flags=0x40008 [ 8042.141019] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8042.148658] pending: addrconf_verify_work [ 8042.151466] workqueue knbd0-recv: flags=0x1a [ 8042.159416] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8042.164143] in-flight: 658:recv_work [ 8042.169033] workqueue ptlrpc_pinger: flags=0x2 [ 8042.173016] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8042.183625] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8042.197469] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6111s workers=4 idle: 32984 37 8 [ 8042.201870] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8026s workers=2 idle: 661 [ 8052.057498] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 8052.063124] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8052.078023] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8052.082212] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8052.085898] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8052.087753] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8052.094315] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8052.097098] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8052.099871] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 8052.102308] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8052.105474] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 8052.108379] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 8052.111021] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 8052.113919] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8052.116205] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 8052.118580] Call Trace: [ 8052.119774] [ 8052.120550] ? show_regs.cold.9+0x22/0x2f [ 8052.122209] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8052.123953] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8052.125483] ? watchdog+0x40/0x40 [ 8052.126862] ? hrtimer_interrupt+0x128/0x2c0 [ 8052.128162] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8052.129760] ? apic_timer_interrupt+0xf/0x20 [ 8052.131477] [ 8052.132175] ? native_safe_halt+0x2e/0x50 [ 8052.134155] kvm_wait+0xcd/0xe0 [ 8052.135178] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8052.136915] do_raw_spin_lock+0x177/0x190 [ 8052.138535] _raw_spin_lock+0x21/0x30 [ 8052.139842] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8052.142634] ? do_raw_spin_unlock+0x75/0x190 [ 8052.144417] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8052.146475] ? do_wait_intr+0xf0/0xf0 [ 8052.148051] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8052.150287] mdc_close+0x53b/0xd70 [mdc] [ 8052.151767] lmv_close+0x18d/0x490 [lmv] [ 8052.153365] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8052.155354] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8052.157806] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8052.159542] ll_md_close+0x274/0x800 [lustre] [ 8052.161387] ll_file_release+0x128/0xc50 [lustre] [ 8052.163088] __fput+0x10d/0x440 [ 8052.164417] ____fput+0x12/0x20 [ 8052.165416] task_work_run+0xc6/0x110 [ 8052.166660] exit_to_usermode_loop+0x1dd/0x1f0 [ 8052.168347] do_syscall_64+0x3d6/0x440 [ 8052.169613] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8052.171266] RIP: 0033:0x7f151bee3cd8 [ 8052.172710] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8052.178800] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8052.182646] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 8052.185479] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 8052.188263] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8052.190785] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8052.193510] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8064.055277] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 8064.067400] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8064.119565] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8064.129605] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8064.144526] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8064.152551] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8064.169470] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8064.173802] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 8064.184206] RDX: 0000000000007e83 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 8064.192727] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 8064.202887] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 8064.216162] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8064.233301] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 8064.243673] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8064.251536] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8064.259294] Call Trace: [ 8064.261043] [ 8064.262023] ? show_regs.cold.9+0x22/0x2f [ 8064.264027] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8064.266289] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8064.269182] ? watchdog+0x40/0x40 [ 8064.270991] ? hrtimer_interrupt+0x128/0x2c0 [ 8064.273039] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8064.275407] ? apic_timer_interrupt+0xf/0x20 [ 8064.278319] [ 8064.279264] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8064.282429] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8064.288455] do_raw_spin_lock+0x177/0x190 [ 8064.292425] _raw_spin_lock+0x21/0x30 [ 8064.293737] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8064.298411] obd_mod_free+0x46/0x140 [ptlrpc] [ 8064.300684] mdc_commit_open+0x91/0xb0 [mdc] [ 8064.302588] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8064.305013] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8064.309346] after_reply+0xe2c/0x1780 [ptlrpc] [ 8064.314723] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8064.318163] ? __next_timer_interrupt+0x160/0x160 [ 8064.320526] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8064.323010] ? do_wait_intr+0xf0/0xf0 [ 8064.324815] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8064.328397] kthread+0x1d1/0x200 [ 8064.330153] ? set_kthread_struct+0x70/0x70 [ 8064.332058] ret_from_fork+0x1f/0x30 [ 8068.052630] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 8068.055259] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8068.057012] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8068.077301] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8068.084620] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8068.084702] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8068.084710] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8068.084717] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8068.092584] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8068.138583] [ 8068.138593] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8068.138612] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8068.138616] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8068.138621] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8068.138623] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 8068.138625] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 8068.138628] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 8068.138630] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 8068.138633] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 8068.138636] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8068.138638] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8068.138647] Call Trace: [ 8068.138653] [ 8068.138659] ? show_regs.cold.9+0x22/0x2f [ 8068.138675] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8068.138682] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8068.138686] ? watchdog+0x40/0x40 [ 8068.138690] ? hrtimer_interrupt+0x128/0x2c0 [ 8068.138694] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8068.138701] ? apic_timer_interrupt+0xf/0x20 [ 8068.138706] [ 8068.138709] ? native_safe_halt+0x2e/0x50 [ 8068.138714] kvm_wait+0xcd/0xe0 [ 8068.138721] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8068.138731] do_raw_spin_lock+0x177/0x190 [ 8068.138735] _raw_spin_lock+0x21/0x30 [ 8068.138739] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8068.139101] process_one_work+0x2c8/0x700 [ 8068.139108] worker_thread+0x296/0x6e0 [ 8068.139112] ? rescuer_thread+0x570/0x570 [ 8068.139116] kthread+0x1d1/0x200 [ 8068.139121] ? set_kthread_struct+0x70/0x70 [ 8068.139124] ret_from_fork+0x1f/0x30 [ 8068.340527] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8068.347960] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8068.366268] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 8068.376114] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8068.382397] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 8068.386094] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 8068.393973] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 8068.397176] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 8068.402571] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 8068.409049] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8068.412436] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 8068.415696] Call Trace: [ 8068.417759] [ 8068.418691] ? show_regs.cold.9+0x22/0x2f [ 8068.420187] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8068.423430] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8068.432945] ? watchdog+0x40/0x40 [ 8068.434165] ? hrtimer_interrupt+0x128/0x2c0 [ 8068.435639] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8068.441772] ? apic_timer_interrupt+0xf/0x20 [ 8068.445173] [ 8068.446634] ? native_safe_halt+0x2e/0x50 [ 8068.449341] kvm_wait+0xcd/0xe0 [ 8068.452258] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8068.455757] do_raw_spin_lock+0x177/0x190 [ 8068.457288] _raw_spin_lock+0x21/0x30 [ 8068.458526] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8068.463431] ? do_raw_spin_unlock+0x75/0x190 [ 8068.466307] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8068.469531] ? do_wait_intr+0xf0/0xf0 [ 8068.472578] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8068.475755] mdc_reint+0x2d/0x110 [mdc] [ 8068.477769] mdc_unlink+0x658/0x870 [mdc] [ 8068.480028] lmv_unlink+0x34d/0xbd0 [lmv] [ 8068.481983] ll_unlink+0x24d/0x7f0 [lustre] [ 8068.484476] vfs_unlink+0x202/0x330 [ 8068.486349] do_unlinkat+0x2fa/0x460 [ 8068.488045] __x64_sys_unlink+0x37/0x50 [ 8068.490036] do_syscall_64+0xc1/0x440 [ 8068.491613] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8068.494321] RIP: 0033:0x7f151bdee10b [ 8068.496110] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8068.504708] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8068.511439] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 8068.516600] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 8068.523537] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8068.532045] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8068.542751] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8072.673047] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6142s! [ 8072.687570] Showing busy workqueues and worker pools: [ 8072.690464] workqueue events: flags=0x0 [ 8072.692482] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8072.699609] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 8072.707265] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8072.715956] pending: kfree_rcu_monitor [ 8072.718519] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8072.721849] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 8072.730624] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8072.734278] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 8072.743150] workqueue events_long: flags=0x0 [ 8072.745662] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8072.750497] pending: lu_objects_flush [obdclass] [ 8072.754669] workqueue events_unbound: flags=0x2 [ 8072.756598] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8072.759668] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8072.767473] workqueue events_power_efficient: flags=0x80 [ 8072.769731] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8072.776252] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 8072.780670] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8072.783707] pending: neigh_periodic_work [ 8072.786370] workqueue mm_percpu_wq: flags=0x8 [ 8072.795972] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8072.801253] pending: vmstat_update, lru_add_drain_per_cpu [ 8072.806728] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8072.811350] pending: vmstat_update [ 8072.813009] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8072.816600] pending: lru_add_drain_per_cpu [ 8072.818447] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8072.821930] pending: lru_add_drain_per_cpu BAR(48) [ 8072.824423] workqueue writeback: flags=0x4a [ 8072.825886] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 8072.828428] pending: wb_workfn [ 8072.829966] workqueue xprtiod: flags=0xa [ 8072.831616] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8072.833886] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8072.836961] workqueue ipv6_addrconf: flags=0x40008 [ 8072.838468] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8072.840902] pending: addrconf_verify_work [ 8072.844019] workqueue knbd0-recv: flags=0x1a [ 8072.845673] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8072.848926] in-flight: 658:recv_work [ 8072.850063] workqueue ptlrpc_pinger: flags=0x2 [ 8072.851400] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8072.853657] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8072.856608] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6142s workers=4 idle: 32984 37 8 [ 8072.859732] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8057s workers=2 idle: 661 [ 8080.057278] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 8080.059854] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8080.093928] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8080.108264] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8080.116497] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8080.120299] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8080.138175] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8080.142209] RAX: 0000000000006cce RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8080.149381] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 8080.153849] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8080.162509] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 8080.172017] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 8080.177371] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 8080.181870] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8080.187897] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 8080.195015] Call Trace: [ 8080.196112] [ 8080.196569] ? show_regs.cold.9+0x22/0x2f [ 8080.200289] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8080.204408] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8080.208328] ? watchdog+0x40/0x40 [ 8080.211531] ? hrtimer_interrupt+0x128/0x2c0 [ 8080.213236] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8080.214857] ? apic_timer_interrupt+0xf/0x20 [ 8080.216174] [ 8080.217009] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8080.220209] do_raw_spin_lock+0x177/0x190 [ 8080.221453] _raw_spin_lock+0x21/0x30 [ 8080.222885] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8080.227046] ? do_raw_spin_unlock+0x75/0x190 [ 8080.231215] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8080.234523] ? do_wait_intr+0xf0/0xf0 [ 8080.235605] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8080.237015] mdc_close+0x53b/0xd70 [mdc] [ 8080.239836] lmv_close+0x18d/0x490 [lmv] [ 8080.243449] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8080.245964] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8080.248431] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8080.250032] ll_md_close+0x274/0x800 [lustre] [ 8080.251783] ll_file_release+0x128/0xc50 [lustre] [ 8080.254069] __fput+0x10d/0x440 [ 8080.255521] ____fput+0x12/0x20 [ 8080.257329] task_work_run+0xc6/0x110 [ 8080.261831] exit_to_usermode_loop+0x1dd/0x1f0 [ 8080.264648] do_syscall_64+0x3d6/0x440 [ 8080.267421] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8080.272266] RIP: 0033:0x7f151bee3cd8 [ 8080.276971] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8080.287230] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8080.302458] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 8080.316793] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 8080.319402] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8080.339449] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8080.347256] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8092.057808] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 8092.069327] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8092.102771] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8092.114391] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8092.120095] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8092.124283] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8092.142814] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8092.146488] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 8092.151091] RDX: 0000000000006ca8 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 8092.158555] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 8092.169799] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 8092.178566] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8092.185409] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 8092.191821] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8092.194908] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8092.199488] Call Trace: [ 8092.200940] [ 8092.201994] ? show_regs.cold.9+0x22/0x2f [ 8092.205214] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8092.208112] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8092.210331] ? watchdog+0x40/0x40 [ 8092.212295] ? hrtimer_interrupt+0x128/0x2c0 [ 8092.214779] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8092.220375] ? apic_timer_interrupt+0xf/0x20 [ 8092.224087] [ 8092.227836] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8092.231349] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8092.235143] do_raw_spin_lock+0x177/0x190 [ 8092.237542] _raw_spin_lock+0x21/0x30 [ 8092.239961] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8092.242980] obd_mod_free+0x46/0x140 [ptlrpc] [ 8092.246052] mdc_commit_open+0x91/0xb0 [mdc] [ 8092.248128] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8092.251052] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8092.254483] after_reply+0xe2c/0x1780 [ptlrpc] [ 8092.257386] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8092.260371] ? __next_timer_interrupt+0x160/0x160 [ 8092.262776] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8092.265314] ? do_wait_intr+0xf0/0xf0 [ 8092.267077] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8092.270099] kthread+0x1d1/0x200 [ 8092.271712] ? set_kthread_struct+0x70/0x70 [ 8092.273751] ret_from_fork+0x1f/0x30 [ 8096.057280] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 8096.060395] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8096.083156] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8096.087717] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8096.091003] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8096.093871] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8096.095470] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8096.110880] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8096.114378] RAX: 0000000000006f87 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8096.117792] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 8096.119989] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 8096.122232] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 8096.124759] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 8096.126671] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 8096.129160] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8096.131023] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8096.133074] Call Trace: [ 8096.133811] [ 8096.134282] ? show_regs.cold.9+0x22/0x2f [ 8096.135483] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8096.136802] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8096.138214] ? watchdog+0x40/0x40 [ 8096.139266] ? hrtimer_interrupt+0x128/0x2c0 [ 8096.140635] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8096.142465] ? apic_timer_interrupt+0xf/0x20 [ 8096.143939] [ 8096.144587] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8096.146316] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8096.148030] do_raw_spin_lock+0x177/0x190 [ 8096.149271] _raw_spin_lock+0x21/0x30 [ 8096.151256] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8096.154036] process_one_work+0x2c8/0x700 [ 8096.155162] worker_thread+0x296/0x6e0 [ 8096.156091] ? rescuer_thread+0x570/0x570 [ 8096.158339] kthread+0x1d1/0x200 [ 8096.160049] ? set_kthread_struct+0x70/0x70 [ 8096.162251] ret_from_fork+0x1f/0x30 [ 8100.052820] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 8100.056049] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8100.127214] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8100.145098] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8100.148344] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8100.157269] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8100.190102] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8100.194352] RAX: 0000000000007206 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8100.203066] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 8100.215359] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 8100.222720] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 8100.231454] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 8100.239719] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 8100.248941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8100.253373] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 8100.268094] Call Trace: [ 8100.275193] [ 8100.275955] ? show_regs.cold.9+0x22/0x2f [ 8100.277459] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8100.286940] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8100.289493] ? watchdog+0x40/0x40 [ 8100.290766] ? hrtimer_interrupt+0x128/0x2c0 [ 8100.295963] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8100.302060] ? apic_timer_interrupt+0xf/0x20 [ 8100.303585] [ 8100.304704] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8100.306932] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8100.309773] do_raw_spin_lock+0x177/0x190 [ 8100.312407] _raw_spin_lock+0x21/0x30 [ 8100.313780] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8100.319729] ? do_raw_spin_unlock+0x75/0x190 [ 8100.334156] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8100.338000] ? do_wait_intr+0xf0/0xf0 [ 8100.342195] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8100.344988] mdc_reint+0x2d/0x110 [mdc] [ 8100.352647] mdc_unlink+0x658/0x870 [mdc] [ 8100.354924] lmv_unlink+0x34d/0xbd0 [lmv] [ 8100.357412] ll_unlink+0x24d/0x7f0 [lustre] [ 8100.360308] vfs_unlink+0x202/0x330 [ 8100.362182] do_unlinkat+0x2fa/0x460 [ 8100.364120] __x64_sys_unlink+0x37/0x50 [ 8100.368670] do_syscall_64+0xc1/0x440 [ 8100.374187] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8100.377378] RIP: 0033:0x7f151bdee10b [ 8100.379592] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8100.395523] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8100.399460] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 8100.405415] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 8100.411336] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8100.417109] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8100.429466] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8102.880288] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6172s! [ 8102.882798] Showing busy workqueues and worker pools: [ 8102.884646] workqueue events: flags=0x0 [ 8102.886008] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8102.891986] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 8102.898681] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8102.902863] pending: kfree_rcu_monitor [ 8102.904796] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8102.910313] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 8102.918668] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8102.924938] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 8102.931405] workqueue events_long: flags=0x0 [ 8102.934814] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8102.939735] pending: lu_objects_flush [obdclass] [ 8102.943475] workqueue events_unbound: flags=0x2 [ 8102.946913] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8102.951802] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8102.958723] workqueue events_power_efficient: flags=0x80 [ 8102.961691] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8102.965886] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 8102.969825] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8102.972964] pending: neigh_periodic_work [ 8102.975773] workqueue mm_percpu_wq: flags=0x8 [ 8102.978025] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8102.982007] pending: vmstat_update, lru_add_drain_per_cpu [ 8102.985297] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8102.989369] pending: vmstat_update [ 8102.991408] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8102.995013] pending: lru_add_drain_per_cpu [ 8102.997679] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8103.002651] pending: lru_add_drain_per_cpu BAR(48) [ 8103.005511] workqueue writeback: flags=0x4a [ 8103.008507] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 8103.011967] pending: wb_workfn [ 8103.014324] workqueue xprtiod: flags=0xa [ 8103.016350] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8103.019847] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8103.025848] workqueue ipv6_addrconf: flags=0x40008 [ 8103.028422] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8103.031986] pending: addrconf_verify_work [ 8103.034964] workqueue knbd0-recv: flags=0x1a [ 8103.037168] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8103.040989] in-flight: 658:recv_work [ 8103.043155] workqueue ptlrpc_pinger: flags=0x2 [ 8103.045902] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8103.049503] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8103.053685] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6172s workers=4 idle: 32984 37 8 [ 8103.058360] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8087s workers=2 idle: 661 [ 8104.780028] rcu: INFO: rcu_sched self-detected stall on CPU [ 8104.782658] rcu: 1-...!: (3500610 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 8104.786774] (t=6180133 jiffies g=249657 q=4120) [ 8104.788530] rcu: rcu_sched kthread starved for 6175085 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 8104.793181] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8104.797296] rcu: RCU grace-period kthread stack dump: [ 8104.799758] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8104.803955] Call Trace: [ 8104.805078] __schedule+0x351/0xcb0 [ 8104.806703] schedule+0xc0/0x180 [ 8104.808010] schedule_timeout+0xb4/0x190 [ 8104.809719] ? do_raw_spin_unlock+0x75/0x190 [ 8104.811652] ? __next_timer_interrupt+0x160/0x160 [ 8104.813737] ? prepare_to_swait_event+0xf3/0x140 [ 8104.815740] rcu_gp_kthread+0x8dd/0x15b0 [ 8104.817486] ? force_qs_rnp+0x2c0/0x2c0 [ 8104.819081] kthread+0x1d1/0x200 [ 8104.820622] ? set_kthread_struct+0x70/0x70 [ 8104.822698] ret_from_fork+0x1f/0x30 [ 8104.824250] rcu: Stack dump where RCU GP kthread last ran: [ 8104.826709] Sending NMI from CPU 1 to CPUs 2: [ 8104.828865] NMI backtrace for cpu 2 skipped: idling at native_safe_halt+0x2e/0x50 [ 8104.829850] NMI backtrace for cpu 1 [ 8104.845949] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8104.852933] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8104.858058] Call Trace: [ 8104.859597] [ 8104.860764] ? dump_stack+0xbb/0x10e [ 8104.862498] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8104.864942] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8104.867013] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8104.869078] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8104.871343] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8104.873524] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 8104.875324] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 8104.877992] ? update_process_times+0x6d/0xc0 [ 8104.880331] ? tick_sched_handle.isra.18+0x27/0x90 [ 8104.882924] ? tick_sched_timer+0x53/0xb0 [ 8104.884954] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8104.887540] ? tick_sched_handle.isra.18+0x90/0x90 [ 8104.889938] ? hrtimer_interrupt+0x128/0x2c0 [ 8104.892301] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8104.894947] ? apic_timer_interrupt+0xf/0x20 [ 8104.897007] [ 8104.898285] ? native_safe_halt+0x2e/0x50 [ 8104.900942] ? kvm_wait+0xcd/0xe0 [ 8104.902965] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8104.906320] ? do_raw_spin_lock+0x177/0x190 [ 8104.909372] ? _raw_spin_lock+0x21/0x30 [ 8104.911547] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8104.914315] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 8104.916937] ? mdc_commit_open+0x91/0xb0 [mdc] [ 8104.919033] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8104.922315] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8104.925390] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 8104.928280] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8104.931195] ? __next_timer_interrupt+0x160/0x160 [ 8104.933021] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8104.934829] ? do_wait_intr+0xf0/0xf0 [ 8104.938006] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8104.941512] ? kthread+0x1d1/0x200 [ 8104.942947] ? set_kthread_struct+0x70/0x70 [ 8104.944965] ? ret_from_fork+0x1f/0x30 [ 8108.059723] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 8108.066172] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8108.107386] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8108.111482] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8108.116580] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8108.118092] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8108.125253] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8108.128099] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8108.131338] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 8108.134942] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8108.140030] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 8108.144246] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 8108.154698] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 8108.160730] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8108.166343] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 8108.170466] Call Trace: [ 8108.171786] [ 8108.172470] ? show_regs.cold.9+0x22/0x2f [ 8108.173523] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8108.174747] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8108.177236] ? watchdog+0x40/0x40 [ 8108.181564] ? hrtimer_interrupt+0x128/0x2c0 [ 8108.183775] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8108.186345] ? apic_timer_interrupt+0xf/0x20 [ 8108.189098] [ 8108.189797] ? native_safe_halt+0x2e/0x50 [ 8108.191022] kvm_wait+0xcd/0xe0 [ 8108.193317] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8108.198755] do_raw_spin_lock+0x177/0x190 [ 8108.201746] _raw_spin_lock+0x21/0x30 [ 8108.204696] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8108.210868] ? do_raw_spin_unlock+0x75/0x190 [ 8108.213074] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8108.215387] ? do_wait_intr+0xf0/0xf0 [ 8108.216278] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8108.219234] mdc_close+0x53b/0xd70 [mdc] [ 8108.221752] lmv_close+0x18d/0x490 [lmv] [ 8108.225052] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8108.228003] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8108.230915] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8108.232674] ll_md_close+0x274/0x800 [lustre] [ 8108.235086] ll_file_release+0x128/0xc50 [lustre] [ 8108.236733] __fput+0x10d/0x440 [ 8108.238289] ____fput+0x12/0x20 [ 8108.240232] task_work_run+0xc6/0x110 [ 8108.241881] exit_to_usermode_loop+0x1dd/0x1f0 [ 8108.243578] do_syscall_64+0x3d6/0x440 [ 8108.245340] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8108.248691] RIP: 0033:0x7f151bee3cd8 [ 8108.251136] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8108.266053] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8108.270728] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 8108.276790] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 8108.279579] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8108.283501] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8108.286626] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8128.052020] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 8128.067671] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8128.102703] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8128.112864] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8128.117032] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8128.119096] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8128.133060] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8128.137045] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8128.140606] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 8128.144117] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 8128.149675] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 8128.152871] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 8128.160926] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 8128.165580] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8128.170529] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 8128.174016] Call Trace: [ 8128.175022] [ 8128.175911] ? show_regs.cold.9+0x22/0x2f [ 8128.178922] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8128.181915] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8128.183954] ? watchdog+0x40/0x40 [ 8128.185721] ? hrtimer_interrupt+0x128/0x2c0 [ 8128.187921] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8128.190085] ? apic_timer_interrupt+0xf/0x20 [ 8128.191879] [ 8128.193180] ? native_safe_halt+0x2e/0x50 [ 8128.195314] kvm_wait+0xcd/0xe0 [ 8128.197549] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8128.201584] do_raw_spin_lock+0x177/0x190 [ 8128.203473] _raw_spin_lock+0x21/0x30 [ 8128.205017] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8128.208067] ? do_raw_spin_unlock+0x75/0x190 [ 8128.210216] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8128.212163] ? do_wait_intr+0xf0/0xf0 [ 8128.214818] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8128.218969] mdc_reint+0x2d/0x110 [mdc] [ 8128.221471] mdc_unlink+0x658/0x870 [mdc] [ 8128.223141] lmv_unlink+0x34d/0xbd0 [lmv] [ 8128.229785] ll_unlink+0x24d/0x7f0 [lustre] [ 8128.231244] vfs_unlink+0x202/0x330 [ 8128.233655] do_unlinkat+0x2fa/0x460 [ 8128.235612] __x64_sys_unlink+0x37/0x50 [ 8128.237607] do_syscall_64+0xc1/0x440 [ 8128.239630] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8128.244173] RIP: 0033:0x7f151bdee10b [ 8128.246482] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8128.266195] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8128.270593] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 8128.277313] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 8128.282699] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8128.286652] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8128.293122] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8132.055449] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 8132.057189] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8132.060633] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8132.060785] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8132.060793] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8132.060796] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8132.060811] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8132.060880] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8132.060925] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 8132.060928] RDX: 000000000000433c RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 8132.060931] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 8132.060933] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 8132.060936] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8132.060939] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 8132.060942] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8132.060945] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8132.060953] Call Trace: [ 8132.060958] [ 8132.060964] ? show_regs.cold.9+0x22/0x2f [ 8132.063695] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8132.063770] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8132.063778] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8132.063784] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8132.095983] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8132.101666] [ 8132.101675] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8132.101691] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8132.101696] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8132.101705] RAX: 0000000000005c34 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8132.101781] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 8132.101784] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 8132.101787] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 8132.101789] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 8132.101793] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 8132.101796] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8132.101798] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8132.101807] Call Trace: [ 8132.101812] [ 8132.101817] ? show_regs.cold.9+0x22/0x2f [ 8132.101826] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8132.101833] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8132.101837] ? watchdog+0x40/0x40 [ 8132.101914] ? hrtimer_interrupt+0x128/0x2c0 [ 8132.101919] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8132.101927] ? apic_timer_interrupt+0xf/0x20 [ 8132.101932] [ 8132.101935] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8132.101939] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8132.101947] do_raw_spin_lock+0x177/0x190 [ 8132.101952] _raw_spin_lock+0x21/0x30 [ 8132.101958] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8132.102567] process_one_work+0x2c8/0x700 [ 8132.102575] worker_thread+0x296/0x6e0 [ 8132.102579] ? rescuer_thread+0x570/0x570 [ 8132.102583] kthread+0x1d1/0x200 [ 8132.102588] ? set_kthread_struct+0x70/0x70 [ 8132.102592] ret_from_fork+0x1f/0x30 [ 8132.489992] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8132.493063] ? watchdog+0x40/0x40 [ 8132.495603] ? hrtimer_interrupt+0x128/0x2c0 [ 8132.500660] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8132.504438] ? apic_timer_interrupt+0xf/0x20 [ 8132.508008] [ 8132.509446] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8132.514665] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8132.520029] do_raw_spin_lock+0x177/0x190 [ 8132.522050] _raw_spin_lock+0x21/0x30 [ 8132.527194] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8132.531209] obd_mod_free+0x46/0x140 [ptlrpc] [ 8132.533475] mdc_commit_open+0x91/0xb0 [mdc] [ 8132.538623] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8132.542109] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8132.550501] after_reply+0xe2c/0x1780 [ptlrpc] [ 8132.555046] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8132.557029] ? __next_timer_interrupt+0x160/0x160 [ 8132.562550] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8132.569058] ? do_wait_intr+0xf0/0xf0 [ 8132.571608] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8132.576151] kthread+0x1d1/0x200 [ 8132.579509] ? set_kthread_struct+0x70/0x70 [ 8132.582153] ret_from_fork+0x1f/0x30 [ 8133.096650] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6202s! [ 8133.114595] Showing busy workqueues and worker pools: [ 8133.124196] workqueue events: flags=0x0 [ 8133.134961] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8133.141912] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 8133.158866] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8133.165406] pending: kfree_rcu_monitor [ 8133.167953] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8133.171418] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 8133.177643] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8133.181094] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 8133.184641] workqueue events_long: flags=0x0 [ 8133.186707] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8133.189624] pending: lu_objects_flush [obdclass] [ 8133.192050] workqueue events_unbound: flags=0x2 [ 8133.194015] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8133.196947] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8133.201459] workqueue events_power_efficient: flags=0x80 [ 8133.203691] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8133.206681] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 8133.209898] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8133.212788] pending: neigh_periodic_work [ 8133.214384] workqueue mm_percpu_wq: flags=0x8 [ 8133.216131] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8133.218683] pending: vmstat_update, lru_add_drain_per_cpu [ 8133.221377] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8133.224608] pending: vmstat_update [ 8133.226311] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8133.228862] pending: lru_add_drain_per_cpu [ 8133.230394] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8133.232784] pending: lru_add_drain_per_cpu BAR(48) [ 8133.234914] workqueue writeback: flags=0x4a [ 8133.236517] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 8133.238993] pending: wb_workfn [ 8133.240084] workqueue xprtiod: flags=0xa [ 8133.241608] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8133.244103] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8133.247587] workqueue ipv6_addrconf: flags=0x40008 [ 8133.249276] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8133.251812] pending: addrconf_verify_work [ 8133.253855] workqueue knbd0-recv: flags=0x1a [ 8133.255564] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8133.258130] in-flight: 658:recv_work [ 8133.259843] workqueue ptlrpc_pinger: flags=0x2 [ 8133.261767] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8133.264276] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8133.266864] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6202s workers=4 idle: 32984 37 8 [ 8133.275341] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8118s workers=2 idle: 661 [ 8136.058039] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 8136.069669] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8136.092487] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8136.097509] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8136.101640] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8136.104517] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8136.111749] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8136.114969] RAX: 0000000000007602 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8136.118691] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 8136.121394] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8136.123907] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 8136.126223] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 8136.129485] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 8136.131498] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8136.133055] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 8136.134956] Call Trace: [ 8136.135791] [ 8136.136340] ? show_regs.cold.9+0x22/0x2f [ 8136.137433] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8136.138854] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8136.140004] ? watchdog+0x40/0x40 [ 8136.140858] ? hrtimer_interrupt+0x128/0x2c0 [ 8136.142001] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8136.143573] ? apic_timer_interrupt+0xf/0x20 [ 8136.144834] [ 8136.145383] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8136.146844] do_raw_spin_lock+0x177/0x190 [ 8136.147857] _raw_spin_lock+0x21/0x30 [ 8136.148876] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8136.151415] ? do_raw_spin_unlock+0x75/0x190 [ 8136.153548] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8136.155086] ? do_wait_intr+0xf0/0xf0 [ 8136.157169] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8136.158825] mdc_close+0x53b/0xd70 [mdc] [ 8136.160065] lmv_close+0x18d/0x490 [lmv] [ 8136.162292] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8136.166555] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8136.168912] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8136.174325] ll_md_close+0x274/0x800 [lustre] [ 8136.175744] ll_file_release+0x128/0xc50 [lustre] [ 8136.180041] __fput+0x10d/0x440 [ 8136.181742] ____fput+0x12/0x20 [ 8136.184389] task_work_run+0xc6/0x110 [ 8136.186004] exit_to_usermode_loop+0x1dd/0x1f0 [ 8136.187967] do_syscall_64+0x3d6/0x440 [ 8136.191344] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8136.196180] RIP: 0033:0x7f151bee3cd8 [ 8136.197064] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8136.206120] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8136.211117] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 8136.214280] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 8136.217344] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8136.220066] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8136.224772] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8160.054878] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 8160.055197] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 8160.059502] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8160.059516] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8160.059590] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8160.059596] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8160.059602] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8160.059860] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8160.059885] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8160.059890] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8160.059894] RAX: 0000000000004708 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8160.059897] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 8160.059899] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 8160.059901] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 8160.059904] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 8160.059907] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 8160.059910] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8160.059913] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8160.059921] Call Trace: [ 8160.059927] [ 8160.059932] ? show_regs.cold.9+0x22/0x2f [ 8160.059942] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8160.059948] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8160.059953] ? watchdog+0x40/0x40 [ 8160.059957] ? hrtimer_interrupt+0x128/0x2c0 [ 8160.059961] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8160.059970] ? apic_timer_interrupt+0xf/0x20 [ 8160.059975] [ 8160.059977] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8160.059982] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8160.059987] do_raw_spin_lock+0x177/0x190 [ 8160.059992] _raw_spin_lock+0x21/0x30 [ 8160.059998] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8160.060235] process_one_work+0x2c8/0x700 [ 8160.060242] worker_thread+0x296/0x6e0 [ 8160.060247] ? rescuer_thread+0x570/0x570 [ 8160.060250] kthread+0x1d1/0x200 [ 8160.060255] ? set_kthread_struct+0x70/0x70 [ 8160.060259] ret_from_fork+0x1f/0x30 [ 8160.071120] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8160.071297] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8160.071304] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8160.071307] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8160.071322] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8160.071327] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8160.071331] RAX: 0000000000007b09 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8160.071334] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 8160.071336] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 8160.071338] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 8160.071341] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 8160.071345] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 8160.071348] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8160.071350] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 8160.071361] Call Trace: [ 8160.071366] [ 8160.071371] ? show_regs.cold.9+0x22/0x2f [ 8160.071381] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8160.071388] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8160.071392] ? watchdog+0x40/0x40 [ 8160.071396] ? hrtimer_interrupt+0x128/0x2c0 [ 8160.071400] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8160.071409] ? apic_timer_interrupt+0xf/0x20 [ 8160.071414] [ 8160.071417] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8160.071422] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8160.071427] do_raw_spin_lock+0x177/0x190 [ 8160.071432] _raw_spin_lock+0x21/0x30 [ 8160.071438] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8160.071710] ? do_raw_spin_unlock+0x75/0x190 [ 8160.071716] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8160.072986] ? do_wait_intr+0xf0/0xf0 [ 8160.072995] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8160.081987] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8160.082087] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8160.082095] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8160.082098] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8160.082113] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8160.082116] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8160.082121] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 8160.082123] RDX: 000000000000673b RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 8160.082125] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 8160.082127] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 8160.082130] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8160.082203] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 8160.082207] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8160.082210] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8160.082218] Call Trace: [ 8160.082223] [ 8160.082228] ? show_regs.cold.9+0x22/0x2f [ 8160.082238] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8160.082245] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8160.082249] ? watchdog+0x40/0x40 [ 8160.082253] ? hrtimer_interrupt+0x128/0x2c0 [ 8160.082257] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8160.082265] ? apic_timer_interrupt+0xf/0x20 [ 8160.082269] [ 8160.082272] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8160.082277] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8160.082282] do_raw_spin_lock+0x177/0x190 [ 8160.082287] _raw_spin_lock+0x21/0x30 [ 8160.082292] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8160.082560] obd_mod_free+0x46/0x140 [ptlrpc] [ 8160.083431] mdc_commit_open+0x91/0xb0 [mdc] [ 8160.084053] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8160.084765] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8160.088381] after_reply+0xe2c/0x1780 [ptlrpc] [ 8160.089086] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8160.089603] ? __next_timer_interrupt+0x160/0x160 [ 8160.089614] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8160.090118] ? do_wait_intr+0xf0/0xf0 [ 8160.090128] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8160.090611] kthread+0x1d1/0x200 [ 8160.090617] ? set_kthread_struct+0x70/0x70 [ 8160.090621] ret_from_fork+0x1f/0x30 [ 8161.073543] mdc_reint+0x2d/0x110 [mdc] [ 8161.079129] mdc_unlink+0x658/0x870 [mdc] [ 8161.083596] lmv_unlink+0x34d/0xbd0 [lmv] [ 8161.085152] ll_unlink+0x24d/0x7f0 [lustre] [ 8161.091198] vfs_unlink+0x202/0x330 [ 8161.095345] do_unlinkat+0x2fa/0x460 [ 8161.102547] __x64_sys_unlink+0x37/0x50 [ 8161.110873] do_syscall_64+0xc1/0x440 [ 8161.116740] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8161.119567] RIP: 0033:0x7f151bdee10b [ 8161.130184] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8161.153033] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8161.160447] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 8161.168560] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 8161.177417] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8161.183872] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8161.193755] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8163.297031] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6232s! [ 8163.300202] Showing busy workqueues and worker pools: [ 8163.301741] workqueue events: flags=0x0 [ 8163.304364] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8163.308689] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 8163.313572] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8163.316704] pending: kfree_rcu_monitor [ 8163.318360] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8163.321222] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 8163.325676] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8163.329503] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 8163.333640] workqueue events_long: flags=0x0 [ 8163.337154] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8163.341113] pending: lu_objects_flush [obdclass] [ 8163.343663] workqueue events_unbound: flags=0x2 [ 8163.345731] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8163.349522] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8163.355514] workqueue events_power_efficient: flags=0x80 [ 8163.365628] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8163.374516] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 8163.379496] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8163.382635] pending: neigh_periodic_work [ 8163.384732] workqueue mm_percpu_wq: flags=0x8 [ 8163.386698] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8163.390671] pending: vmstat_update, lru_add_drain_per_cpu [ 8163.394020] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8163.397610] pending: vmstat_update [ 8163.399815] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8163.402899] pending: lru_add_drain_per_cpu [ 8163.405800] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8163.410080] pending: lru_add_drain_per_cpu BAR(48) [ 8163.413560] workqueue writeback: flags=0x4a [ 8163.418621] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 8163.425538] pending: wb_workfn [ 8163.426848] workqueue xprtiod: flags=0xa [ 8163.437046] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8163.446365] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8163.453365] workqueue ipv6_addrconf: flags=0x40008 [ 8163.456578] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8163.461130] pending: addrconf_verify_work [ 8163.466268] workqueue knbd0-recv: flags=0x1a [ 8163.470435] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8163.476021] in-flight: 658:recv_work [ 8163.483016] workqueue ptlrpc_pinger: flags=0x2 [ 8163.488734] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8163.496132] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8163.501322] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6232s workers=4 idle: 32984 37 8 [ 8163.506099] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8148s workers=2 idle: 661 [ 8164.061303] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 8164.076598] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8164.141518] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8164.149203] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8164.156458] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8164.170456] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8164.204055] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8164.209622] RAX: 0000000000005987 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8164.215649] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 8164.221465] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8164.227709] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 8164.234168] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 8164.242422] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 8164.248446] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8164.250542] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 8164.261231] Call Trace: [ 8164.264146] [ 8164.266496] ? show_regs.cold.9+0x22/0x2f [ 8164.270432] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8164.272264] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8164.275816] ? watchdog+0x40/0x40 [ 8164.279076] ? hrtimer_interrupt+0x128/0x2c0 [ 8164.282410] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8164.286372] ? apic_timer_interrupt+0xf/0x20 [ 8164.293652] [ 8164.295637] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8164.303122] do_raw_spin_lock+0x177/0x190 [ 8164.307590] _raw_spin_lock+0x21/0x30 [ 8164.311735] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8164.316241] ? do_raw_spin_unlock+0x75/0x190 [ 8164.319310] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8164.328500] ? do_wait_intr+0xf0/0xf0 [ 8164.331642] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8164.339094] mdc_close+0x53b/0xd70 [mdc] [ 8164.343042] lmv_close+0x18d/0x490 [lmv] [ 8164.346164] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8164.353538] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8164.358824] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8164.360407] ll_md_close+0x274/0x800 [lustre] [ 8164.364533] ll_file_release+0x128/0xc50 [lustre] [ 8164.368178] __fput+0x10d/0x440 [ 8164.370589] ____fput+0x12/0x20 [ 8164.373426] task_work_run+0xc6/0x110 [ 8164.375334] exit_to_usermode_loop+0x1dd/0x1f0 [ 8164.377661] do_syscall_64+0x3d6/0x440 [ 8164.383712] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8164.392671] RIP: 0033:0x7f151bee3cd8 [ 8164.396432] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8164.411505] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8164.422262] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 8164.429498] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 8164.433510] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8164.437844] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8164.441834] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8188.055335] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 8188.055772] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 8188.056009] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8188.056017] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8188.056086] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8188.056092] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8188.056098] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8188.056401] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8188.056413] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8188.056417] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8188.056422] RAX: 0000000000000757 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8188.056424] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 8188.056426] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 8188.056717] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 8188.056724] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 8188.056728] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 8188.056731] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8188.056733] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8188.056743] Call Trace: [ 8188.056748] [ 8188.056763] ? show_regs.cold.9+0x22/0x2f [ 8188.056774] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8188.056783] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8188.056787] ? watchdog+0x40/0x40 [ 8188.056791] ? hrtimer_interrupt+0x128/0x2c0 [ 8188.056795] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8188.056805] ? apic_timer_interrupt+0xf/0x20 [ 8188.056810] [ 8188.056813] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8188.056818] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8188.056824] do_raw_spin_lock+0x177/0x190 [ 8188.056829] _raw_spin_lock+0x21/0x30 [ 8188.056834] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8188.057059] process_one_work+0x2c8/0x700 [ 8188.057066] worker_thread+0x296/0x6e0 [ 8188.057070] ? rescuer_thread+0x570/0x570 [ 8188.057074] kthread+0x1d1/0x200 [ 8188.057078] ? set_kthread_struct+0x70/0x70 [ 8188.057081] ret_from_fork+0x1f/0x30 [ 8188.068208] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8188.068297] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8188.068305] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8188.068309] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8188.068324] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8188.068328] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8188.068333] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 8188.068336] RDX: 00000000000045a6 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 8188.068338] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 8188.068340] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 8188.068343] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8188.068346] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 8188.068349] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8188.068352] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8188.068361] Call Trace: [ 8188.068367] [ 8188.068373] ? show_regs.cold.9+0x22/0x2f [ 8188.068383] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8188.068389] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8188.068394] ? watchdog+0x40/0x40 [ 8188.068398] ? hrtimer_interrupt+0x128/0x2c0 [ 8188.068402] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8188.068411] ? apic_timer_interrupt+0xf/0x20 [ 8188.068415] [ 8188.068418] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8188.068423] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8188.068428] do_raw_spin_lock+0x177/0x190 [ 8188.068432] _raw_spin_lock+0x21/0x30 [ 8188.068439] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8188.068801] obd_mod_free+0x46/0x140 [ptlrpc] [ 8188.069003] mdc_commit_open+0x91/0xb0 [mdc] [ 8188.069041] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8188.069648] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8188.069887] after_reply+0xe2c/0x1780 [ptlrpc] [ 8188.070100] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8188.071687] ? __next_timer_interrupt+0x160/0x160 [ 8188.071699] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8188.073122] ? do_wait_intr+0xf0/0xf0 [ 8188.078652] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8188.078731] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8188.078740] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8188.078744] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8188.078906] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8188.078910] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8188.078915] RAX: 0000000000007e85 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8188.078917] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 8188.078920] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 8188.078922] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 8188.078924] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 8188.078928] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 8188.078931] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8188.078933] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 8188.078942] Call Trace: [ 8188.078947] [ 8188.078953] ? show_regs.cold.9+0x22/0x2f [ 8188.078962] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8188.078968] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8188.078972] ? watchdog+0x40/0x40 [ 8188.078977] ? hrtimer_interrupt+0x128/0x2c0 [ 8188.078981] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8188.078989] ? apic_timer_interrupt+0xf/0x20 [ 8188.078994] [ 8188.078997] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8188.079002] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8188.079007] do_raw_spin_lock+0x177/0x190 [ 8188.079012] _raw_spin_lock+0x21/0x30 [ 8188.079018] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8188.079317] ? do_raw_spin_unlock+0x75/0x190 [ 8188.079323] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8188.079559] ? do_wait_intr+0xf0/0xf0 [ 8188.079566] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8188.080021] mdc_reint+0x2d/0x110 [mdc] [ 8188.080062] mdc_unlink+0x658/0x870 [mdc] [ 8188.080093] lmv_unlink+0x34d/0xbd0 [lmv] [ 8188.080127] ll_unlink+0x24d/0x7f0 [lustre] [ 8188.080310] vfs_unlink+0x202/0x330 [ 8188.080318] do_unlinkat+0x2fa/0x460 [ 8188.080324] __x64_sys_unlink+0x37/0x50 [ 8188.080327] do_syscall_64+0xc1/0x440 [ 8188.080334] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8188.080341] RIP: 0033:0x7f151bdee10b [ 8188.080348] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8188.080352] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8188.080357] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 8188.080359] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 8188.080362] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8188.080364] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8188.080367] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8188.638378] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8188.640921] kthread+0x1d1/0x200 [ 8188.642324] ? set_kthread_struct+0x70/0x70 [ 8188.643441] ret_from_fork+0x1f/0x30 [ 8192.057090] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 8192.061000] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8192.088883] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8192.096234] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8192.099940] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8192.103109] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8192.113208] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8192.117692] RAX: 000000000000782c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8192.121371] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 8192.124941] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8192.128410] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 8192.131909] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 8192.135643] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 8192.139456] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8192.142571] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 8192.146111] Call Trace: [ 8192.147119] [ 8192.148234] ? show_regs.cold.9+0x22/0x2f [ 8192.150331] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8192.152488] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8192.154840] ? watchdog+0x40/0x40 [ 8192.156738] ? hrtimer_interrupt+0x128/0x2c0 [ 8192.158611] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8192.160998] ? apic_timer_interrupt+0xf/0x20 [ 8192.162631] [ 8192.164868] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8192.167575] do_raw_spin_lock+0x177/0x190 [ 8192.169393] _raw_spin_lock+0x21/0x30 [ 8192.170914] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8192.174049] ? do_raw_spin_unlock+0x75/0x190 [ 8192.176259] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8192.178905] ? do_wait_intr+0xf0/0xf0 [ 8192.180662] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8192.183889] mdc_close+0x53b/0xd70 [mdc] [ 8192.186014] lmv_close+0x18d/0x490 [lmv] [ 8192.188079] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8192.191306] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8192.194275] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8192.197206] ll_md_close+0x274/0x800 [lustre] [ 8192.199489] ll_file_release+0x128/0xc50 [lustre] [ 8192.201938] __fput+0x10d/0x440 [ 8192.203554] ____fput+0x12/0x20 [ 8192.205022] task_work_run+0xc6/0x110 [ 8192.206403] exit_to_usermode_loop+0x1dd/0x1f0 [ 8192.208565] do_syscall_64+0x3d6/0x440 [ 8192.210487] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8192.212931] RIP: 0033:0x7f151bee3cd8 [ 8192.215120] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8192.223471] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8192.227012] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 8192.230506] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 8192.233937] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8192.237442] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8192.241392] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8194.018033] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6263s! [ 8194.023678] Showing busy workqueues and worker pools: [ 8194.026649] workqueue events: flags=0x0 [ 8194.029143] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8194.033550] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 8194.039063] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8194.042858] pending: kfree_rcu_monitor [ 8194.045040] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8194.048423] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 8194.054151] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8194.059141] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 8194.064013] workqueue events_long: flags=0x0 [ 8194.067125] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8194.071737] pending: lu_objects_flush [obdclass] [ 8194.076995] workqueue events_unbound: flags=0x2 [ 8194.084651] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8194.090663] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8194.098107] workqueue events_power_efficient: flags=0x80 [ 8194.101067] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8194.115786] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 8194.118688] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8194.122537] pending: neigh_periodic_work [ 8194.124680] workqueue mm_percpu_wq: flags=0x8 [ 8194.126271] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8194.129803] pending: vmstat_update, lru_add_drain_per_cpu [ 8194.132275] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8194.136112] pending: vmstat_update [ 8194.138419] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8194.142922] pending: lru_add_drain_per_cpu [ 8194.144947] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8194.147988] pending: lru_add_drain_per_cpu BAR(48) [ 8194.152412] workqueue writeback: flags=0x4a [ 8194.154535] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 8194.157509] pending: wb_workfn [ 8194.159116] workqueue xprtiod: flags=0xa [ 8194.160821] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8194.163831] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8194.167677] workqueue ipv6_addrconf: flags=0x40008 [ 8194.171336] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8194.174881] pending: addrconf_verify_work [ 8194.176753] workqueue knbd0-recv: flags=0x1a [ 8194.178951] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8194.185689] in-flight: 658:recv_work [ 8194.187834] workqueue ptlrpc_pinger: flags=0x2 [ 8194.189825] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8194.194918] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8194.197957] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6263s workers=4 idle: 32984 37 8 [ 8194.204938] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8178s workers=2 idle: 661 [ 8216.057643] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8216.064529] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 8216.068698] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8216.068778] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8216.068786] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8216.068793] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8216.071185] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8216.073756] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8216.073771] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8216.073777] RAX: 0000000000005b1e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8216.073780] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 8216.073783] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 8216.073785] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 8216.073787] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 8216.073790] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 8216.073792] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8216.073794] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8216.073804] Call Trace: [ 8216.073809] [ 8216.073816] ? show_regs.cold.9+0x22/0x2f [ 8216.073829] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8216.073835] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8216.073840] ? watchdog+0x40/0x40 [ 8216.073845] ? hrtimer_interrupt+0x128/0x2c0 [ 8216.073849] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8216.074429] ? apic_timer_interrupt+0xf/0x20 [ 8216.074439] [ 8216.074443] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8216.074450] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8216.074458] do_raw_spin_lock+0x177/0x190 [ 8216.074463] _raw_spin_lock+0x21/0x30 [ 8216.074468] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8216.074757] process_one_work+0x2c8/0x700 [ 8216.074765] worker_thread+0x296/0x6e0 [ 8216.074769] ? rescuer_thread+0x570/0x570 [ 8216.074774] kthread+0x1d1/0x200 [ 8216.074778] ? set_kthread_struct+0x70/0x70 [ 8216.074781] ret_from_fork+0x1f/0x30 [ 8216.443542] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8216.508910] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8216.521827] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8216.527212] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8216.533456] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8216.546046] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8216.550637] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 8216.559721] RDX: 000000000000796e RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 8216.563597] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 8216.567729] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 8216.576488] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8216.581136] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 8216.592470] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8216.598610] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8216.614836] Call Trace: [ 8216.621660] [ 8216.622646] ? show_regs.cold.9+0x22/0x2f [ 8216.623808] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8216.625256] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8216.641848] ? watchdog+0x40/0x40 [ 8216.642969] ? hrtimer_interrupt+0x128/0x2c0 [ 8216.649828] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8216.654902] ? apic_timer_interrupt+0xf/0x20 [ 8216.658790] [ 8216.662573] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8216.668321] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8216.678168] do_raw_spin_lock+0x177/0x190 [ 8216.679191] _raw_spin_lock+0x21/0x30 [ 8216.681543] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8216.685077] obd_mod_free+0x46/0x140 [ptlrpc] [ 8216.689289] mdc_commit_open+0x91/0xb0 [mdc] [ 8216.693512] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8216.701592] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8216.704854] after_reply+0xe2c/0x1780 [ptlrpc] [ 8216.712403] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8216.720177] ? __next_timer_interrupt+0x160/0x160 [ 8216.724944] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8216.734957] ? do_wait_intr+0xf0/0xf0 [ 8216.736400] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8216.753049] kthread+0x1d1/0x200 [ 8216.754276] ? set_kthread_struct+0x70/0x70 [ 8216.758669] ret_from_fork+0x1f/0x30 [ 8220.056107] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 8220.059816] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8220.062279] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 8220.083862] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8220.083877] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8220.083880] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8220.083894] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8220.083899] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8220.083904] RAX: 0000000000007a8b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8220.083906] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 8220.083909] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 8220.083911] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 8220.083914] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 8220.083917] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 8220.083921] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8220.083923] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 8220.083932] Call Trace: [ 8220.083938] [ 8220.083944] ? show_regs.cold.9+0x22/0x2f [ 8220.083956] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8220.083962] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8220.083967] ? watchdog+0x40/0x40 [ 8220.083971] ? hrtimer_interrupt+0x128/0x2c0 [ 8220.083975] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8220.083983] ? apic_timer_interrupt+0xf/0x20 [ 8220.083988] [ 8220.083991] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8220.083996] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8220.084001] do_raw_spin_lock+0x177/0x190 [ 8220.084005] _raw_spin_lock+0x21/0x30 [ 8220.084012] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8220.084295] ? do_raw_spin_unlock+0x75/0x190 [ 8220.084302] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8220.084537] ? do_wait_intr+0xf0/0xf0 [ 8220.084544] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8220.084765] mdc_reint+0x2d/0x110 [mdc] [ 8220.084807] mdc_unlink+0x658/0x870 [mdc] [ 8220.084836] lmv_unlink+0x34d/0xbd0 [lmv] [ 8220.084866] ll_unlink+0x24d/0x7f0 [lustre] [ 8220.084988] vfs_unlink+0x202/0x330 [ 8220.084997] do_unlinkat+0x2fa/0x460 [ 8220.085002] __x64_sys_unlink+0x37/0x50 [ 8220.085005] do_syscall_64+0xc1/0x440 [ 8220.085012] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8220.085019] RIP: 0033:0x7f151bdee10b [ 8220.085026] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8220.085031] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8220.085035] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 8220.085038] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 8220.085040] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8220.085042] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8220.085044] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8220.314426] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8220.365285] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8220.371760] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8220.378665] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8220.386385] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8220.405026] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8220.412156] RAX: 000000000000350a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8220.418512] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 8220.425510] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8220.432084] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 8220.436854] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 8220.440024] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 8220.443903] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8220.447260] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 8220.454131] Call Trace: [ 8220.457629] [ 8220.459622] ? show_regs.cold.9+0x22/0x2f [ 8220.462609] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8220.465092] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8220.470863] ? watchdog+0x40/0x40 [ 8220.473574] ? hrtimer_interrupt+0x128/0x2c0 [ 8220.478270] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8220.482042] ? apic_timer_interrupt+0xf/0x20 [ 8220.484265] [ 8220.487101] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8220.494905] do_raw_spin_lock+0x177/0x190 [ 8220.497949] _raw_spin_lock+0x21/0x30 [ 8220.500337] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8220.503366] ? do_raw_spin_unlock+0x75/0x190 [ 8220.507758] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8220.510379] ? do_wait_intr+0xf0/0xf0 [ 8220.512529] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8220.515529] mdc_close+0x53b/0xd70 [mdc] [ 8220.517938] lmv_close+0x18d/0x490 [lmv] [ 8220.520027] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8220.522648] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8220.525502] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8220.528097] ll_md_close+0x274/0x800 [lustre] [ 8220.530588] ll_file_release+0x128/0xc50 [lustre] [ 8220.532700] __fput+0x10d/0x440 [ 8220.534179] ____fput+0x12/0x20 [ 8220.537273] task_work_run+0xc6/0x110 [ 8220.541038] exit_to_usermode_loop+0x1dd/0x1f0 [ 8220.547193] do_syscall_64+0x3d6/0x440 [ 8220.556219] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8220.560198] RIP: 0033:0x7f151bee3cd8 [ 8220.561578] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8220.572753] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8220.580799] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 8220.585996] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 8220.589977] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8220.594585] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8220.598863] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8224.228184] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6293s! [ 8224.238486] Showing busy workqueues and worker pools: [ 8224.241622] workqueue events: flags=0x0 [ 8224.244343] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8224.250393] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 8224.258737] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8224.274579] pending: kfree_rcu_monitor [ 8224.283542] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8224.288472] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 8224.305749] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8224.318456] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 8224.325492] workqueue events_long: flags=0x0 [ 8224.335973] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8224.345604] pending: lu_objects_flush [obdclass] [ 8224.347671] workqueue events_unbound: flags=0x2 [ 8224.352511] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8224.361088] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8224.373096] workqueue events_power_efficient: flags=0x80 [ 8224.379435] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8224.383879] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 8224.391198] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8224.399083] pending: neigh_periodic_work [ 8224.407265] workqueue mm_percpu_wq: flags=0x8 [ 8224.412395] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8224.420491] pending: vmstat_update, lru_add_drain_per_cpu [ 8224.427626] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8224.433747] pending: vmstat_update [ 8224.434966] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8224.443891] pending: lru_add_drain_per_cpu [ 8224.450993] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8224.460152] pending: lru_add_drain_per_cpu BAR(48) [ 8224.466910] workqueue writeback: flags=0x4a [ 8224.472073] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 8224.479206] pending: wb_workfn [ 8224.483543] workqueue xprtiod: flags=0xa [ 8224.488485] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8224.499199] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8224.514352] workqueue ipv6_addrconf: flags=0x40008 [ 8224.517120] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8224.532336] pending: addrconf_verify_work [ 8224.542388] workqueue knbd0-recv: flags=0x1a [ 8224.548184] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8224.555537] in-flight: 658:recv_work [ 8224.559419] workqueue ptlrpc_pinger: flags=0x2 [ 8224.564530] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8224.573534] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8224.577970] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6293s workers=4 idle: 32984 37 8 [ 8224.587788] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8209s workers=2 idle: 661 [ 8244.057394] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8244.058855] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 8244.065859] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8244.066034] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8244.066041] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8244.066055] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8244.066350] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8244.066374] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8244.066378] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8244.066382] RAX: 0000000000006f08 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8244.066384] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 8244.066386] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 8244.066388] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 8244.066391] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 8244.066394] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 8244.066397] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8244.066399] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8244.066407] Call Trace: [ 8244.066422] [ 8244.066428] ? show_regs.cold.9+0x22/0x2f [ 8244.066437] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8244.066444] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8244.066448] ? watchdog+0x40/0x40 [ 8244.066452] ? hrtimer_interrupt+0x128/0x2c0 [ 8244.066456] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8244.066465] ? apic_timer_interrupt+0xf/0x20 [ 8244.066470] [ 8244.066473] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8244.066478] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8244.066483] do_raw_spin_lock+0x177/0x190 [ 8244.066488] _raw_spin_lock+0x21/0x30 [ 8244.066492] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8244.066719] process_one_work+0x2c8/0x700 [ 8244.066728] worker_thread+0x296/0x6e0 [ 8244.066733] ? rescuer_thread+0x570/0x570 [ 8244.066737] kthread+0x1d1/0x200 [ 8244.066741] ? set_kthread_struct+0x70/0x70 [ 8244.066745] ret_from_fork+0x1f/0x30 [ 8244.369578] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8244.396679] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8244.406593] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8244.418600] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8244.426429] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8244.449684] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8244.455599] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 8244.462123] RDX: 0000000000006cdd RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 8244.466627] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 8244.477860] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 8244.482240] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8244.491933] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 8244.501409] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8244.503519] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8244.508690] Call Trace: [ 8244.511389] [ 8244.512240] ? show_regs.cold.9+0x22/0x2f [ 8244.514890] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8244.516650] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8244.518878] ? watchdog+0x40/0x40 [ 8244.521710] ? hrtimer_interrupt+0x128/0x2c0 [ 8244.524958] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8244.530950] ? apic_timer_interrupt+0xf/0x20 [ 8244.532687] [ 8244.533694] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8244.537128] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8244.545387] do_raw_spin_lock+0x177/0x190 [ 8244.546617] _raw_spin_lock+0x21/0x30 [ 8244.547743] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8244.553545] obd_mod_free+0x46/0x140 [ptlrpc] [ 8244.564533] mdc_commit_open+0x91/0xb0 [mdc] [ 8244.567819] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8244.570831] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8244.575333] after_reply+0xe2c/0x1780 [ptlrpc] [ 8244.578124] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8244.588440] ? __next_timer_interrupt+0x160/0x160 [ 8244.589981] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8244.594096] ? do_wait_intr+0xf0/0xf0 [ 8244.595924] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8244.600610] kthread+0x1d1/0x200 [ 8244.602538] ? set_kthread_struct+0x70/0x70 [ 8244.606312] ret_from_fork+0x1f/0x30 [ 8248.057030] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 8248.061309] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8248.081804] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8248.086725] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8248.090986] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8248.093365] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8248.101288] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8248.104830] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8248.108766] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 8248.112466] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8248.116212] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 8248.120776] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 8248.124500] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 8248.128017] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8248.130674] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 8248.134139] Call Trace: [ 8248.135027] [ 8248.135570] ? show_regs.cold.9+0x22/0x2f [ 8248.137224] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8248.139423] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8248.141831] ? watchdog+0x40/0x40 [ 8248.143596] ? hrtimer_interrupt+0x128/0x2c0 [ 8248.145584] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8248.147757] ? apic_timer_interrupt+0xf/0x20 [ 8248.149948] [ 8248.151763] ? native_safe_halt+0x2e/0x50 [ 8248.154197] kvm_wait+0xcd/0xe0 [ 8248.155501] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8248.158117] do_raw_spin_lock+0x177/0x190 [ 8248.163341] _raw_spin_lock+0x21/0x30 [ 8248.164431] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8248.169037] ? do_raw_spin_unlock+0x75/0x190 [ 8248.170269] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8248.172064] ? do_wait_intr+0xf0/0xf0 [ 8248.173598] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8248.175948] mdc_close+0x53b/0xd70 [mdc] [ 8248.177467] lmv_close+0x18d/0x490 [lmv] [ 8248.178937] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8248.181653] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8248.185303] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8248.187582] ll_md_close+0x274/0x800 [lustre] [ 8248.190015] ll_file_release+0x128/0xc50 [lustre] [ 8248.192756] __fput+0x10d/0x440 [ 8248.194372] ____fput+0x12/0x20 [ 8248.195732] task_work_run+0xc6/0x110 [ 8248.197131] exit_to_usermode_loop+0x1dd/0x1f0 [ 8248.198563] do_syscall_64+0x3d6/0x440 [ 8248.199902] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8248.201909] RIP: 0033:0x7f151bee3cd8 [ 8248.203548] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8248.215317] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8248.222752] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 8248.225274] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 8248.228244] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8248.231571] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8248.234646] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8252.053588] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 8252.058376] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8252.096656] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8252.103266] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8252.108466] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8252.111411] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8252.124195] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8252.129876] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8252.134437] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 8252.143599] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 8252.146659] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 8252.149342] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 8252.152342] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 8252.157081] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8252.160626] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 8252.166073] Call Trace: [ 8252.167626] [ 8252.168596] ? show_regs.cold.9+0x22/0x2f [ 8252.170294] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8252.172594] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8252.174625] ? watchdog+0x40/0x40 [ 8252.176188] ? hrtimer_interrupt+0x128/0x2c0 [ 8252.177613] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8252.179479] ? apic_timer_interrupt+0xf/0x20 [ 8252.181677] [ 8252.182740] ? native_safe_halt+0x2e/0x50 [ 8252.184612] kvm_wait+0xcd/0xe0 [ 8252.186209] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8252.188935] do_raw_spin_lock+0x177/0x190 [ 8252.193487] _raw_spin_lock+0x21/0x30 [ 8252.196998] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8252.201574] ? do_raw_spin_unlock+0x75/0x190 [ 8252.206427] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8252.209201] ? do_wait_intr+0xf0/0xf0 [ 8252.214279] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8252.217390] mdc_reint+0x2d/0x110 [mdc] [ 8252.225233] mdc_unlink+0x658/0x870 [mdc] [ 8252.226763] lmv_unlink+0x34d/0xbd0 [lmv] [ 8252.231651] ll_unlink+0x24d/0x7f0 [lustre] [ 8252.234074] vfs_unlink+0x202/0x330 [ 8252.236224] do_unlinkat+0x2fa/0x460 [ 8252.238543] __x64_sys_unlink+0x37/0x50 [ 8252.244425] do_syscall_64+0xc1/0x440 [ 8252.251293] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8252.255557] RIP: 0033:0x7f151bdee10b [ 8252.258869] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8252.278653] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8252.287769] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 8252.295969] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 8252.304489] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8252.307411] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8252.312808] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8254.946715] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6324s! [ 8254.957131] Showing busy workqueues and worker pools: [ 8254.962080] workqueue events: flags=0x0 [ 8254.964599] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8254.968475] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 8254.977074] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8254.987231] pending: kfree_rcu_monitor [ 8254.988819] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8254.994280] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 8255.005329] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8255.014306] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 8255.016334] workqueue events_long: flags=0x0 [ 8255.021183] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8255.029861] pending: lu_objects_flush [obdclass] [ 8255.036819] workqueue events_unbound: flags=0x2 [ 8255.041239] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8255.045553] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8255.052159] workqueue events_power_efficient: flags=0x80 [ 8255.056192] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8255.061947] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 8255.075122] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8255.081426] pending: neigh_periodic_work [ 8255.087926] workqueue mm_percpu_wq: flags=0x8 [ 8255.092227] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8255.097405] pending: vmstat_update, lru_add_drain_per_cpu [ 8255.100101] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8255.104263] pending: vmstat_update [ 8255.112271] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8255.117846] pending: lru_add_drain_per_cpu [ 8255.121052] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8255.130051] pending: lru_add_drain_per_cpu BAR(48) [ 8255.131540] workqueue writeback: flags=0x4a [ 8255.132686] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 8255.142938] pending: wb_workfn [ 8255.144941] workqueue xprtiod: flags=0xa [ 8255.147669] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8255.154763] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8255.158242] workqueue ipv6_addrconf: flags=0x40008 [ 8255.162331] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8255.166763] pending: addrconf_verify_work [ 8255.169948] workqueue knbd0-recv: flags=0x1a [ 8255.172434] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8255.176309] in-flight: 658:recv_work [ 8255.178282] workqueue ptlrpc_pinger: flags=0x2 [ 8255.182324] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8255.188729] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8255.196134] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6324s workers=4 idle: 32984 37 8 [ 8255.201052] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8239s workers=2 idle: 661 [ 8272.056274] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 8272.057465] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8272.058232] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8272.058305] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8272.058313] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8272.058315] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8272.058330] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8272.058334] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8272.058339] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 8272.058342] RDX: 00000000000040a4 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 8272.058344] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 8272.058347] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 8272.058349] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8272.058352] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 8272.058356] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8272.058358] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8272.058367] Call Trace: [ 8272.058373] [ 8272.058380] ? show_regs.cold.9+0x22/0x2f [ 8272.058389] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8272.058395] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8272.058400] ? watchdog+0x40/0x40 [ 8272.058405] ? hrtimer_interrupt+0x128/0x2c0 [ 8272.058409] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8272.058419] ? apic_timer_interrupt+0xf/0x20 [ 8272.058424] [ 8272.058427] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8272.058433] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8272.058437] do_raw_spin_lock+0x177/0x190 [ 8272.058441] _raw_spin_lock+0x21/0x30 [ 8272.058448] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8272.060317] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8272.060366] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8272.060372] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8272.060377] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8272.080309] obd_mod_free+0x46/0x140 [ptlrpc] [ 8272.086161] [ 8272.086169] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8272.086188] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8272.086192] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8272.086198] RAX: 0000000000007c2e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8272.086200] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 8272.086202] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 8272.086205] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 8272.086207] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 8272.086211] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 8272.086214] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8272.086216] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8272.086225] Call Trace: [ 8272.086230] [ 8272.086235] ? show_regs.cold.9+0x22/0x2f [ 8272.086245] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8272.086251] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8272.086256] ? watchdog+0x40/0x40 [ 8272.086261] ? hrtimer_interrupt+0x128/0x2c0 [ 8272.086265] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8272.086274] ? apic_timer_interrupt+0xf/0x20 [ 8272.086279] [ 8272.086282] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8272.086287] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8272.086294] do_raw_spin_lock+0x177/0x190 [ 8272.086299] _raw_spin_lock+0x21/0x30 [ 8272.086305] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8272.091179] mdc_commit_open+0x91/0xb0 [mdc] [ 8272.091403] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8272.092040] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8272.092410] after_reply+0xe2c/0x1780 [ptlrpc] [ 8272.092608] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8272.093527] ? __next_timer_interrupt+0x160/0x160 [ 8272.093538] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8272.094680] ? do_wait_intr+0xf0/0xf0 [ 8272.094696] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8272.095014] kthread+0x1d1/0x200 [ 8272.095022] ? set_kthread_struct+0x70/0x70 [ 8272.095026] ret_from_fork+0x1f/0x30 [ 8272.410550] process_one_work+0x2c8/0x700 [ 8272.412382] worker_thread+0x296/0x6e0 [ 8272.413607] ? rescuer_thread+0x570/0x570 [ 8272.415127] kthread+0x1d1/0x200 [ 8272.416499] ? set_kthread_struct+0x70/0x70 [ 8272.419685] ret_from_fork+0x1f/0x30 [ 8276.057008] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 8276.063728] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8276.106134] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8276.127237] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8276.136707] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8276.140956] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8276.166550] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8276.174668] RAX: 0000000000002bf8 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8276.184257] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 8276.190896] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8276.200856] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 8276.211058] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 8276.222533] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 8276.228120] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8276.234745] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 8276.242315] Call Trace: [ 8276.243360] [ 8276.244084] ? show_regs.cold.9+0x22/0x2f [ 8276.247881] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8276.250529] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8276.255983] ? watchdog+0x40/0x40 [ 8276.258821] ? hrtimer_interrupt+0x128/0x2c0 [ 8276.264869] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8276.266613] ? apic_timer_interrupt+0xf/0x20 [ 8276.271644] [ 8276.273303] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8276.279612] do_raw_spin_lock+0x177/0x190 [ 8276.285088] _raw_spin_lock+0x21/0x30 [ 8276.286413] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8276.296694] ? do_raw_spin_unlock+0x75/0x190 [ 8276.307117] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8276.311734] ? do_wait_intr+0xf0/0xf0 [ 8276.313665] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8276.317331] mdc_close+0x53b/0xd70 [mdc] [ 8276.319567] lmv_close+0x18d/0x490 [lmv] [ 8276.321952] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8276.327688] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8276.330824] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8276.333376] ll_md_close+0x274/0x800 [lustre] [ 8276.334669] ll_file_release+0x128/0xc50 [lustre] [ 8276.336890] __fput+0x10d/0x440 [ 8276.338461] ____fput+0x12/0x20 [ 8276.339923] task_work_run+0xc6/0x110 [ 8276.341651] exit_to_usermode_loop+0x1dd/0x1f0 [ 8276.343878] do_syscall_64+0x3d6/0x440 [ 8276.345617] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8276.352248] RIP: 0033:0x7f151bee3cd8 [ 8276.353935] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8276.366289] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8276.368702] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 8276.371949] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 8276.375632] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8276.380701] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8276.386432] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8280.052972] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 8280.068064] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8280.107086] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8280.110619] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8280.114366] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8280.116518] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8280.128251] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8280.132324] RAX: 0000000000001752 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8280.136185] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 8280.138302] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 8280.140386] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 8280.143132] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 8280.147477] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 8280.155709] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8280.157928] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 8280.160723] Call Trace: [ 8280.162479] [ 8280.165315] ? show_regs.cold.9+0x22/0x2f [ 8280.169553] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8280.171339] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8280.175140] ? watchdog+0x40/0x40 [ 8280.176382] ? hrtimer_interrupt+0x128/0x2c0 [ 8280.177573] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8280.180569] ? apic_timer_interrupt+0xf/0x20 [ 8280.185335] [ 8280.186039] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8280.189600] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8280.192183] do_raw_spin_lock+0x177/0x190 [ 8280.196298] _raw_spin_lock+0x21/0x30 [ 8280.197157] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8280.198583] ? do_raw_spin_unlock+0x75/0x190 [ 8280.201644] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8280.203014] ? do_wait_intr+0xf0/0xf0 [ 8280.203825] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8280.209351] mdc_reint+0x2d/0x110 [mdc] [ 8280.210384] mdc_unlink+0x658/0x870 [mdc] [ 8280.212506] lmv_unlink+0x34d/0xbd0 [lmv] [ 8280.213609] ll_unlink+0x24d/0x7f0 [lustre] [ 8280.215132] vfs_unlink+0x202/0x330 [ 8280.216254] do_unlinkat+0x2fa/0x460 [ 8280.217299] __x64_sys_unlink+0x37/0x50 [ 8280.218584] do_syscall_64+0xc1/0x440 [ 8280.219811] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8280.221472] RIP: 0033:0x7f151bdee10b [ 8280.222612] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8280.228562] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8280.231575] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 8280.238063] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 8280.240528] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8280.245028] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8280.247298] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8284.783025] rcu: INFO: rcu_sched self-detected stall on CPU [ 8284.788072] rcu: 1-...!: (3612398 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 8284.799274] (t=6360136 jiffies g=249657 q=4177) [ 8284.809146] rcu: rcu_sched kthread starved for 6355088 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 8284.816714] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8284.841873] rcu: RCU grace-period kthread stack dump: [ 8284.848120] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8284.870846] Call Trace: [ 8284.872626] __schedule+0x351/0xcb0 [ 8284.876935] schedule+0xc0/0x180 [ 8284.879607] schedule_timeout+0xb4/0x190 [ 8284.884276] ? do_raw_spin_unlock+0x75/0x190 [ 8284.894203] ? __next_timer_interrupt+0x160/0x160 [ 8284.901055] ? prepare_to_swait_event+0xf3/0x140 [ 8284.908432] rcu_gp_kthread+0x8dd/0x15b0 [ 8284.909962] ? force_qs_rnp+0x2c0/0x2c0 [ 8284.911363] kthread+0x1d1/0x200 [ 8284.917776] ? set_kthread_struct+0x70/0x70 [ 8284.924510] ret_from_fork+0x1f/0x30 [ 8284.929010] rcu: Stack dump where RCU GP kthread last ran: [ 8284.934705] Sending NMI from CPU 1 to CPUs 0: [ 8284.941565] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [ 8284.944502] NMI backtrace for cpu 1 [ 8284.948603] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8284.973743] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8284.991390] Call Trace: [ 8284.992660] [ 8284.995623] ? dump_stack+0xbb/0x10e [ 8285.001441] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8285.008435] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8285.011150] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8285.014597] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8285.018482] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8285.021192] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 8285.024196] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 8285.027386] ? update_process_times+0x6d/0xc0 [ 8285.030623] ? tick_sched_handle.isra.18+0x27/0x90 [ 8285.033732] ? tick_sched_timer+0x53/0xb0 [ 8285.037578] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8285.041287] ? tick_sched_handle.isra.18+0x90/0x90 [ 8285.044403] ? hrtimer_interrupt+0x128/0x2c0 [ 8285.047339] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8285.049582] ? apic_timer_interrupt+0xf/0x20 [ 8285.051382] [ 8285.053289] ? native_safe_halt+0x2e/0x50 [ 8285.055205] ? kvm_wait+0xcd/0xe0 [ 8285.056348] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8285.063147] ? do_raw_spin_lock+0x177/0x190 [ 8285.065897] ? _raw_spin_lock+0x21/0x30 [ 8285.068274] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8285.071634] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 8285.076595] ? mdc_commit_open+0x91/0xb0 [mdc] [ 8285.078979] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8285.083217] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8285.088254] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 8285.091771] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8285.095605] ? __next_timer_interrupt+0x160/0x160 [ 8285.098223] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8285.104565] ? do_wait_intr+0xf0/0xf0 [ 8285.109809] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8285.112976] ? kthread+0x1d1/0x200 [ 8285.115171] ? set_kthread_struct+0x70/0x70 [ 8285.117780] ? ret_from_fork+0x1f/0x30 [ 8285.664113] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6355s! [ 8285.679019] Showing busy workqueues and worker pools: [ 8285.681549] workqueue events: flags=0x0 [ 8285.686284] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8285.691247] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 8285.698544] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8285.704646] pending: kfree_rcu_monitor [ 8285.706560] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8285.713598] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 8285.722365] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8285.726259] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 8285.729986] workqueue events_long: flags=0x0 [ 8285.732109] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8285.735126] pending: lu_objects_flush [obdclass] [ 8285.739239] workqueue events_unbound: flags=0x2 [ 8285.742553] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8285.750438] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8285.756542] workqueue events_power_efficient: flags=0x80 [ 8285.759131] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8285.762263] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 8285.765567] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8285.770219] pending: neigh_periodic_work [ 8285.772478] workqueue mm_percpu_wq: flags=0x8 [ 8285.774284] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8285.777987] pending: vmstat_update, lru_add_drain_per_cpu [ 8285.780914] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8285.784658] pending: vmstat_update [ 8285.786345] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8285.789907] pending: lru_add_drain_per_cpu [ 8285.792543] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8285.798217] pending: lru_add_drain_per_cpu BAR(48) [ 8285.802145] workqueue writeback: flags=0x4a [ 8285.804436] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 8285.808297] pending: wb_workfn [ 8285.810380] workqueue xprtiod: flags=0xa [ 8285.812226] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8285.816364] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8285.822136] workqueue ipv6_addrconf: flags=0x40008 [ 8285.825930] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8285.831533] pending: addrconf_verify_work [ 8285.834578] workqueue knbd0-recv: flags=0x1a [ 8285.837946] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8285.842348] in-flight: 658:recv_work [ 8285.847848] workqueue ptlrpc_pinger: flags=0x2 [ 8285.851159] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8285.856318] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8285.858488] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6355s workers=4 idle: 32984 37 8 [ 8285.873310] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8270s workers=2 idle: 661 [ 8300.065025] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 8300.073468] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8300.113192] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8300.122787] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8300.127640] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8300.139491] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8300.143125] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8300.158152] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8300.167062] RAX: 0000000000005f85 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8300.172547] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 8300.185453] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 8300.193958] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 8300.206527] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 8300.220982] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 8300.238434] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8300.243317] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8300.245928] Call Trace: [ 8300.246573] [ 8300.247756] ? show_regs.cold.9+0x22/0x2f [ 8300.249239] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8300.251990] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8300.256040] ? watchdog+0x40/0x40 [ 8300.266059] ? hrtimer_interrupt+0x128/0x2c0 [ 8300.267752] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8300.272521] ? apic_timer_interrupt+0xf/0x20 [ 8300.274186] [ 8300.278937] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8300.283688] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8300.285745] do_raw_spin_lock+0x177/0x190 [ 8300.290088] _raw_spin_lock+0x21/0x30 [ 8300.294486] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8300.301615] process_one_work+0x2c8/0x700 [ 8300.303206] worker_thread+0x296/0x6e0 [ 8300.305763] ? rescuer_thread+0x570/0x570 [ 8300.307388] kthread+0x1d1/0x200 [ 8300.308610] ? set_kthread_struct+0x70/0x70 [ 8300.313879] ret_from_fork+0x1f/0x30 [ 8304.057021] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 8304.066942] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8304.091761] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8304.097444] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8304.101088] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8304.103150] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8304.110973] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8304.112816] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8304.115283] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 8304.118432] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8304.122301] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 8304.125450] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 8304.128193] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 8304.130878] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8304.132485] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 8304.134822] Call Trace: [ 8304.135388] [ 8304.136012] ? show_regs.cold.9+0x22/0x2f [ 8304.137179] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8304.138404] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8304.139914] ? watchdog+0x40/0x40 [ 8304.141142] ? hrtimer_interrupt+0x128/0x2c0 [ 8304.142222] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8304.143545] ? apic_timer_interrupt+0xf/0x20 [ 8304.145194] [ 8304.146138] ? native_safe_halt+0x2e/0x50 [ 8304.148220] kvm_wait+0xcd/0xe0 [ 8304.149783] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8304.152179] do_raw_spin_lock+0x177/0x190 [ 8304.154424] _raw_spin_lock+0x21/0x30 [ 8304.156208] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8304.158642] ? do_raw_spin_unlock+0x75/0x190 [ 8304.160484] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8304.162908] ? do_wait_intr+0xf0/0xf0 [ 8304.164155] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8304.165951] mdc_close+0x53b/0xd70 [mdc] [ 8304.167108] lmv_close+0x18d/0x490 [lmv] [ 8304.168941] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8304.170858] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8304.172788] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8304.174097] ll_md_close+0x274/0x800 [lustre] [ 8304.175298] ll_file_release+0x128/0xc50 [lustre] [ 8304.177309] __fput+0x10d/0x440 [ 8304.178193] ____fput+0x12/0x20 [ 8304.179103] task_work_run+0xc6/0x110 [ 8304.180616] exit_to_usermode_loop+0x1dd/0x1f0 [ 8304.182263] do_syscall_64+0x3d6/0x440 [ 8304.183845] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8304.184994] RIP: 0033:0x7f151bee3cd8 [ 8304.185791] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8304.189885] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8304.193074] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 8304.195387] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 8304.197777] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8304.199143] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8304.200868] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8312.052089] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 8312.056989] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8312.059665] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 8312.085869] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8312.085883] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8312.085886] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8312.085900] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8312.085904] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8312.085909] RAX: 000000000000570a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8312.085911] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 8312.085914] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 8312.085917] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 8312.085919] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 8312.085923] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 8312.085926] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8312.085928] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 8312.085937] Call Trace: [ 8312.085942] [ 8312.085947] ? show_regs.cold.9+0x22/0x2f [ 8312.085957] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8312.085963] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8312.085967] ? watchdog+0x40/0x40 [ 8312.085971] ? hrtimer_interrupt+0x128/0x2c0 [ 8312.085974] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8312.085983] ? apic_timer_interrupt+0xf/0x20 [ 8312.085988] [ 8312.085991] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8312.085996] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8312.086001] do_raw_spin_lock+0x177/0x190 [ 8312.086078] _raw_spin_lock+0x21/0x30 [ 8312.086087] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8312.086621] ? do_raw_spin_unlock+0x75/0x190 [ 8312.086627] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8312.086822] ? do_wait_intr+0xf0/0xf0 [ 8312.086831] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8312.087219] mdc_reint+0x2d/0x110 [mdc] [ 8312.087259] mdc_unlink+0x658/0x870 [mdc] [ 8312.087286] lmv_unlink+0x34d/0xbd0 [lmv] [ 8312.087390] ll_unlink+0x24d/0x7f0 [lustre] [ 8312.091588] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8312.091664] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8312.091671] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8312.091674] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8312.091688] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8312.091692] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8312.091696] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 8312.091699] RDX: 0000000000007dce RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 8312.091702] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 8312.091704] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 8312.091707] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8312.091710] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 8312.091713] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8312.091716] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8312.091725] Call Trace: [ 8312.091730] [ 8312.091735] ? show_regs.cold.9+0x22/0x2f [ 8312.091743] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8312.091751] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8312.091755] ? watchdog+0x40/0x40 [ 8312.091760] ? hrtimer_interrupt+0x128/0x2c0 [ 8312.091764] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8312.091771] ? apic_timer_interrupt+0xf/0x20 [ 8312.091775] [ 8312.091779] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8312.091783] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8312.091788] do_raw_spin_lock+0x177/0x190 [ 8312.091792] _raw_spin_lock+0x21/0x30 [ 8312.091798] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8312.092128] obd_mod_free+0x46/0x140 [ptlrpc] [ 8312.092514] mdc_commit_open+0x91/0xb0 [mdc] [ 8312.092566] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8312.092759] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8312.092958] after_reply+0xe2c/0x1780 [ptlrpc] [ 8312.098962] vfs_unlink+0x202/0x330 [ 8312.098981] do_unlinkat+0x2fa/0x460 [ 8312.098987] __x64_sys_unlink+0x37/0x50 [ 8312.098990] do_syscall_64+0xc1/0x440 [ 8312.098997] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8312.099131] RIP: 0033:0x7f151bdee10b [ 8312.099214] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8312.099219] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8312.099224] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 8312.099227] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 8312.099229] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8312.099231] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8312.099233] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8312.510187] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8312.513305] ? __next_timer_interrupt+0x160/0x160 [ 8312.515300] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8312.517967] ? do_wait_intr+0xf0/0xf0 [ 8312.519541] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8312.522656] kthread+0x1d1/0x200 [ 8312.524273] ? set_kthread_struct+0x70/0x70 [ 8312.526339] ret_from_fork+0x1f/0x30 [ 8316.395589] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6385s! [ 8316.402311] Showing busy workqueues and worker pools: [ 8316.411238] workqueue events: flags=0x0 [ 8316.414917] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8316.425277] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 8316.441149] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8316.452500] pending: kfree_rcu_monitor [ 8316.455063] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8316.462046] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 8316.472406] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8316.479613] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 8316.493879] workqueue events_long: flags=0x0 [ 8316.499319] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8316.505698] pending: lu_objects_flush [obdclass] [ 8316.511113] workqueue events_unbound: flags=0x2 [ 8316.514584] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8316.518742] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8316.524681] workqueue events_power_efficient: flags=0x80 [ 8316.527934] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8316.531758] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 8316.535536] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8316.538650] pending: neigh_periodic_work [ 8316.540664] workqueue mm_percpu_wq: flags=0x8 [ 8316.543179] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8316.545720] pending: vmstat_update, lru_add_drain_per_cpu [ 8316.553848] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8316.561169] pending: vmstat_update [ 8316.564786] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8316.577858] pending: lru_add_drain_per_cpu [ 8316.584465] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8316.591203] pending: lru_add_drain_per_cpu BAR(48) [ 8316.601058] workqueue writeback: flags=0x4a [ 8316.604355] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 8316.623824] pending: wb_workfn [ 8316.629189] workqueue xprtiod: flags=0xa [ 8316.637364] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8316.642945] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8316.656058] workqueue ipv6_addrconf: flags=0x40008 [ 8316.661095] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8316.668050] pending: addrconf_verify_work [ 8316.674114] workqueue knbd0-recv: flags=0x1a [ 8316.680972] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8316.688733] in-flight: 658:recv_work [ 8316.691869] workqueue ptlrpc_pinger: flags=0x2 [ 8316.696254] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8316.703937] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8316.711624] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6386s workers=4 idle: 32984 37 8 [ 8316.720671] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8301s workers=2 idle: 661 [ 8328.062485] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 8328.073017] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8328.138252] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8328.155894] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8328.178011] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8328.182726] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8328.185711] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8328.193975] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8328.197222] RAX: 0000000000007f86 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8328.201657] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 8328.204409] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 8328.206763] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 8328.208879] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 8328.211639] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 8328.214897] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8328.217083] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8328.219382] Call Trace: [ 8328.220317] [ 8328.220829] ? show_regs.cold.9+0x22/0x2f [ 8328.222203] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8328.223929] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8328.225747] ? watchdog+0x40/0x40 [ 8328.227012] ? hrtimer_interrupt+0x128/0x2c0 [ 8328.228735] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8328.230948] ? apic_timer_interrupt+0xf/0x20 [ 8328.232224] [ 8328.233142] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8328.234552] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8328.236067] do_raw_spin_lock+0x177/0x190 [ 8328.237511] _raw_spin_lock+0x21/0x30 [ 8328.238956] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8328.241005] process_one_work+0x2c8/0x700 [ 8328.242504] worker_thread+0x296/0x6e0 [ 8328.246481] ? rescuer_thread+0x570/0x570 [ 8328.248075] kthread+0x1d1/0x200 [ 8328.250625] ? set_kthread_struct+0x70/0x70 [ 8328.254815] ret_from_fork+0x1f/0x30 [ 8332.057019] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 8332.060955] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8332.107798] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8332.121552] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8332.126943] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8332.131083] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8332.153658] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 8332.169506] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8332.183968] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 8332.191217] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8332.198175] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 8332.207070] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 8332.216410] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 8332.225075] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8332.228024] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 8332.234183] Call Trace: [ 8332.235884] [ 8332.236876] ? show_regs.cold.9+0x22/0x2f [ 8332.238984] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8332.241451] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8332.243739] ? watchdog+0x40/0x40 [ 8332.245106] ? hrtimer_interrupt+0x128/0x2c0 [ 8332.247509] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8332.250870] ? apic_timer_interrupt+0xf/0x20 [ 8332.253925] [ 8332.254920] ? native_safe_halt+0x2e/0x50 [ 8332.257193] kvm_wait+0xcd/0xe0 [ 8332.258812] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8332.261504] do_raw_spin_lock+0x177/0x190 [ 8332.263990] _raw_spin_lock+0x21/0x30 [ 8332.265721] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8332.270782] ? do_raw_spin_unlock+0x75/0x190 [ 8332.273745] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8332.276256] ? do_wait_intr+0xf0/0xf0 [ 8332.278048] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8332.280871] mdc_close+0x53b/0xd70 [mdc] [ 8332.282330] lmv_close+0x18d/0x490 [lmv] [ 8332.285005] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8332.287923] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8332.292574] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8332.294805] ll_md_close+0x274/0x800 [lustre] [ 8332.296912] ll_file_release+0x128/0xc50 [lustre] [ 8332.298906] __fput+0x10d/0x440 [ 8332.300010] ____fput+0x12/0x20 [ 8332.301119] task_work_run+0xc6/0x110 [ 8332.302314] exit_to_usermode_loop+0x1dd/0x1f0 [ 8332.305172] do_syscall_64+0x3d6/0x440 [ 8332.307860] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8332.310175] RIP: 0033:0x7f151bee3cd8 [ 8332.311947] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8332.320503] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8332.323952] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 8332.327173] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 8332.332871] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8332.337453] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8332.340986] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8340.058822] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 8340.067618] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8340.161254] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8340.183307] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8340.196127] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8340.199598] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8340.205263] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8340.208130] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 8340.210578] RDX: 0000000000002840 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 8340.213118] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 8340.216362] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 8340.222074] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8340.225443] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 8340.232599] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8340.234541] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8340.236070] Call Trace: [ 8340.236884] [ 8340.237641] ? show_regs.cold.9+0x22/0x2f [ 8340.238774] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8340.240069] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8340.241508] ? watchdog+0x40/0x40 [ 8340.242141] ? hrtimer_interrupt+0x128/0x2c0 [ 8340.243317] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8340.245299] ? apic_timer_interrupt+0xf/0x20 [ 8340.247128] [ 8340.247953] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8340.250259] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8340.257751] do_raw_spin_lock+0x177/0x190 [ 8340.259498] _raw_spin_lock+0x21/0x30 [ 8340.261361] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8340.263810] obd_mod_free+0x46/0x140 [ptlrpc] [ 8340.265842] mdc_commit_open+0x91/0xb0 [mdc] [ 8340.267726] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8340.270737] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8340.280811] after_reply+0xe2c/0x1780 [ptlrpc] [ 8340.283035] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8340.286135] ? __next_timer_interrupt+0x160/0x160 [ 8340.288468] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8340.292502] ? do_wait_intr+0xf0/0xf0 [ 8340.293864] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8340.298010] kthread+0x1d1/0x200 [ 8340.302805] ? set_kthread_struct+0x70/0x70 [ 8340.304724] ret_from_fork+0x1f/0x30 [ 8344.052023] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 8344.055544] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8344.075532] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8344.080368] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8344.085807] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8344.088026] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8344.097821] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 8344.100527] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8344.103562] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 8344.107267] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 8344.110023] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 8344.112537] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 8344.115282] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 8344.118483] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8344.120899] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 8344.123550] Call Trace: [ 8344.124502] [ 8344.125292] ? show_regs.cold.9+0x22/0x2f [ 8344.126685] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8344.128577] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8344.130189] ? watchdog+0x40/0x40 [ 8344.131373] ? hrtimer_interrupt+0x128/0x2c0 [ 8344.132936] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8344.136618] ? apic_timer_interrupt+0xf/0x20 [ 8344.138170] [ 8344.138913] ? native_safe_halt+0x2e/0x50 [ 8344.140379] kvm_wait+0xcd/0xe0 [ 8344.141455] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8344.143428] do_raw_spin_lock+0x177/0x190 [ 8344.145371] _raw_spin_lock+0x21/0x30 [ 8344.146813] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8344.149740] ? do_raw_spin_unlock+0x75/0x190 [ 8344.151085] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8344.153166] ? do_wait_intr+0xf0/0xf0 [ 8344.154217] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8344.156446] mdc_reint+0x2d/0x110 [mdc] [ 8344.157615] mdc_unlink+0x658/0x870 [mdc] [ 8344.158937] lmv_unlink+0x34d/0xbd0 [lmv] [ 8344.160405] ll_unlink+0x24d/0x7f0 [lustre] [ 8344.161770] vfs_unlink+0x202/0x330 [ 8344.163324] do_unlinkat+0x2fa/0x460 [ 8344.164358] __x64_sys_unlink+0x37/0x50 [ 8344.165432] do_syscall_64+0xc1/0x440 [ 8344.166764] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8344.172342] RIP: 0033:0x7f151bdee10b [ 8344.175448] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8344.182991] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8344.186306] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 8344.189369] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 8344.192078] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8344.194947] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8344.197918] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8347.104687] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6416s! [ 8347.119857] Showing busy workqueues and worker pools: [ 8347.124722] workqueue events: flags=0x0 [ 8347.131869] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8347.142413] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 8347.151467] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8347.166627] pending: kfree_rcu_monitor [ 8347.172753] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8347.179725] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 8347.199236] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8347.209327] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 8347.217670] workqueue events_long: flags=0x0 [ 8347.222443] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8347.237427] pending: lu_objects_flush [obdclass] [ 8347.247105] workqueue events_unbound: flags=0x2 [ 8347.256580] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8347.267906] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8347.292417] workqueue events_power_efficient: flags=0x80 [ 8347.300539] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8347.309979] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 8347.320099] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8347.335279] pending: neigh_periodic_work [ 8347.340446] workqueue mm_percpu_wq: flags=0x8 [ 8347.344146] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8347.348423] pending: vmstat_update, lru_add_drain_per_cpu [ 8347.353394] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8347.358512] pending: vmstat_update [ 8347.361449] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8347.365983] pending: lru_add_drain_per_cpu [ 8347.370967] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8347.378440] pending: lru_add_drain_per_cpu BAR(48) [ 8347.381586] workqueue writeback: flags=0x4a [ 8347.383800] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 8347.389826] pending: wb_workfn [ 8347.392561] workqueue xprtiod: flags=0xa [ 8347.394823] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8347.402378] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8347.415500] workqueue ipv6_addrconf: flags=0x40008 [ 8347.425388] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8347.441906] pending: addrconf_verify_work [ 8347.447642] workqueue knbd0-recv: flags=0x1a [ 8347.456736] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8347.463155] in-flight: 658:recv_work [ 8347.475894] workqueue ptlrpc_pinger: flags=0x2 [ 8347.483890] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8347.491156] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8347.494292] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6416s workers=4 idle: 32984 37 8 [ 8347.505705] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8332s workers=2 idle: 661 [ 8356.064756] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 8356.079134] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8356.164895] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8356.175335] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8356.184265] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8356.189421] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8356.197427] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8356.211826] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 8356.222262] RAX: 0000000000007e9d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8356.230749] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 8356.249313] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 8356.260036] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 8356.272489] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 8356.284768] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 8356.292339] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8356.309239] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8356.314238] Call Trace: [ 8356.315659] [ 8356.317247] ? show_regs.cold.9+0x22/0x2f [ 8356.319572] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8356.322608] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8356.325085] ? watchdog+0x40/0x40 [ 8356.326353] ? hrtimer_interrupt+0x128/0x2c0 [ 8356.330766] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8356.332545] ? apic_timer_interrupt+0xf/0x20 [ 8356.341023] [ 8356.341866] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8356.346718] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8356.353092] do_raw_spin_lock+0x177/0x190 [ 8356.354656] _raw_spin_lock+0x21/0x30 [ 8356.356042] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8356.363249] process_one_work+0x2c8/0x700 [ 8356.367843] worker_thread+0x296/0x6e0 [ 8356.376318] ? rescuer_thread+0x570/0x570 [ 8356.389297] kthread+0x1d1/0x200 [ 8356.395505] ? set_kthread_struct+0x70/0x70 [ 8356.398057] ret_from_fork+0x1f/0x30 [ 8360.060348] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 8360.070232] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8360.120497] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8360.132167] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8360.140057] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8360.147214] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8360.168324] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8360.173915] RAX: 0000000000004789 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8360.188275] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 8360.191470] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8360.203362] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 8360.212006] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 8360.217463] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 8360.229324] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8360.231025] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 8360.238536] Call Trace: [ 8360.241422] [ 8360.244888] ? show_regs.cold.9+0x22/0x2f [ 8360.256747] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8360.260306] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8360.264577] ? watchdog+0x40/0x40 [ 8360.265803] ? hrtimer_interrupt+0x128/0x2c0 [ 8360.272140] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8360.276239] ? apic_timer_interrupt+0xf/0x20 [ 8360.280282] [ 8360.281817] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8360.286200] do_raw_spin_lock+0x177/0x190 [ 8360.292510] _raw_spin_lock+0x21/0x30 [ 8360.297910] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8360.302421] ? do_raw_spin_unlock+0x75/0x190 [ 8360.312612] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8360.322364] ? do_wait_intr+0xf0/0xf0 [ 8360.337261] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8360.353152] mdc_close+0x53b/0xd70 [mdc] [ 8360.356748] lmv_close+0x18d/0x490 [lmv] [ 8360.361657] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8360.375345] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8360.384575] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8360.396798] ll_md_close+0x274/0x800 [lustre] [ 8360.399219] ll_file_release+0x128/0xc50 [lustre] [ 8360.406319] __fput+0x10d/0x440 [ 8360.408686] ____fput+0x12/0x20 [ 8360.412507] task_work_run+0xc6/0x110 [ 8360.418417] exit_to_usermode_loop+0x1dd/0x1f0 [ 8360.425187] do_syscall_64+0x3d6/0x440 [ 8360.433398] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8360.437175] RIP: 0033:0x7f151bee3cd8 [ 8360.441688] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8360.464410] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8360.472150] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 8360.479032] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 8360.490792] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8360.498127] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8360.506159] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8368.055121] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 8368.059371] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8368.093491] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8368.100582] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8368.106534] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8368.108805] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8368.134882] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8368.139409] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 8368.148474] RDX: 0000000000002630 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 8368.156559] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 8368.165517] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 8368.170832] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8368.175324] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 8368.179990] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8368.183348] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8368.190698] Call Trace: [ 8368.193731] [ 8368.195052] ? show_regs.cold.9+0x22/0x2f [ 8368.197663] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8368.200616] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8368.202775] ? watchdog+0x40/0x40 [ 8368.203923] ? hrtimer_interrupt+0x128/0x2c0 [ 8368.206186] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8368.208329] ? apic_timer_interrupt+0xf/0x20 [ 8368.210130] [ 8368.211093] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8368.216185] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8368.217393] do_raw_spin_lock+0x177/0x190 [ 8368.218276] _raw_spin_lock+0x21/0x30 [ 8368.220382] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8368.223703] obd_mod_free+0x46/0x140 [ptlrpc] [ 8368.226798] mdc_commit_open+0x91/0xb0 [mdc] [ 8368.229235] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8368.233724] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8368.237255] after_reply+0xe2c/0x1780 [ptlrpc] [ 8368.241460] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8368.253026] ? __next_timer_interrupt+0x160/0x160 [ 8368.256214] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8368.260941] ? do_wait_intr+0xf0/0xf0 [ 8368.264126] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8368.280299] kthread+0x1d1/0x200 [ 8368.285202] ? set_kthread_struct+0x70/0x70 [ 8368.286653] ret_from_fork+0x1f/0x30 [ 8372.059539] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 8372.072907] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8372.128786] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8372.144611] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8372.147887] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8372.158623] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8372.187752] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8372.191288] RAX: 0000000000006908 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8372.200491] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 8372.206492] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 8372.216074] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 8372.219994] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 8372.222644] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 8372.230851] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8372.235365] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 8372.243532] Call Trace: [ 8372.249690] [ 8372.250719] ? show_regs.cold.9+0x22/0x2f [ 8372.254727] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8372.257606] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8372.264133] ? watchdog+0x40/0x40 [ 8372.265677] ? hrtimer_interrupt+0x128/0x2c0 [ 8372.269303] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8372.271027] ? apic_timer_interrupt+0xf/0x20 [ 8372.272795] [ 8372.274680] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8372.279176] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8372.291175] do_raw_spin_lock+0x177/0x190 [ 8372.297001] _raw_spin_lock+0x21/0x30 [ 8372.299273] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8372.307297] ? do_raw_spin_unlock+0x75/0x190 [ 8372.309739] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8372.319498] ? do_wait_intr+0xf0/0xf0 [ 8372.321542] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8372.328476] mdc_reint+0x2d/0x110 [mdc] [ 8372.331104] mdc_unlink+0x658/0x870 [mdc] [ 8372.337914] lmv_unlink+0x34d/0xbd0 [lmv] [ 8372.339481] ll_unlink+0x24d/0x7f0 [lustre] [ 8372.341418] vfs_unlink+0x202/0x330 [ 8372.342625] do_unlinkat+0x2fa/0x460 [ 8372.346596] __x64_sys_unlink+0x37/0x50 [ 8372.348609] do_syscall_64+0xc1/0x440 [ 8372.350506] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8372.352426] RIP: 0033:0x7f151bdee10b [ 8372.353791] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8372.362635] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8372.366961] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 8372.371319] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 8372.375060] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8372.378794] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8372.382582] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8377.826313] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6447s! [ 8377.832676] Showing busy workqueues and worker pools: [ 8377.836155] workqueue events: flags=0x0 [ 8377.838585] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8377.842560] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 8377.846309] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8377.853519] pending: kfree_rcu_monitor [ 8377.859674] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8377.868845] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 8377.880250] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8377.886445] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 8377.911411] workqueue events_long: flags=0x0 [ 8377.914280] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8377.922310] pending: lu_objects_flush [obdclass] [ 8377.930855] workqueue events_unbound: flags=0x2 [ 8377.940394] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8377.947417] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8377.971505] workqueue events_power_efficient: flags=0x80 [ 8377.978338] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8377.984623] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 8378.005670] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8378.015705] pending: neigh_periodic_work [ 8378.024440] workqueue mm_percpu_wq: flags=0x8 [ 8378.032467] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8378.041814] pending: vmstat_update, lru_add_drain_per_cpu [ 8378.048785] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8378.054375] pending: vmstat_update [ 8378.055719] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8378.058942] pending: lru_add_drain_per_cpu [ 8378.061239] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8378.064087] pending: lru_add_drain_per_cpu BAR(48) [ 8378.066721] workqueue writeback: flags=0x4a [ 8378.068434] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 8378.071355] pending: wb_workfn [ 8378.075404] workqueue xprtiod: flags=0xa [ 8378.076888] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8378.081704] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8378.087064] workqueue ipv6_addrconf: flags=0x40008 [ 8378.089107] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8378.092066] pending: addrconf_verify_work [ 8378.094252] workqueue knbd0-recv: flags=0x1a [ 8378.097881] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8378.106518] in-flight: 658:recv_work [ 8378.109223] workqueue ptlrpc_pinger: flags=0x2 [ 8378.115756] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8378.122055] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8378.126366] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6447s workers=4 idle: 32984 37 8 [ 8378.128858] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8362s workers=2 idle: 661 [ 8384.057154] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 8384.067446] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8384.143069] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8384.149921] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8384.155019] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8384.158934] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8384.161372] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8384.170388] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8384.175294] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8384.179920] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 8384.183839] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 8384.187390] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 8384.191397] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 8384.204164] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 8384.212327] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8384.217087] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8384.229300] Call Trace: [ 8384.231227] [ 8384.232389] ? show_regs.cold.9+0x22/0x2f [ 8384.234452] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8384.236961] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8384.238692] ? watchdog+0x40/0x40 [ 8384.248583] ? hrtimer_interrupt+0x128/0x2c0 [ 8384.253557] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8384.262973] ? apic_timer_interrupt+0xf/0x20 [ 8384.276043] [ 8384.278916] ? native_safe_halt+0x2e/0x50 [ 8384.280471] kvm_wait+0xcd/0xe0 [ 8384.289435] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8384.303476] do_raw_spin_lock+0x177/0x190 [ 8384.309606] _raw_spin_lock+0x21/0x30 [ 8384.314609] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8384.325442] process_one_work+0x2c8/0x700 [ 8384.336438] worker_thread+0x296/0x6e0 [ 8384.343326] ? rescuer_thread+0x570/0x570 [ 8384.344806] kthread+0x1d1/0x200 [ 8384.348333] ? set_kthread_struct+0x70/0x70 [ 8384.349883] ret_from_fork+0x1f/0x30 [ 8388.063020] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 8388.076731] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8388.131372] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8388.153796] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8388.173752] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8388.187338] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8388.227787] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8388.235406] RAX: 0000000000002e84 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8388.245346] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 8388.255187] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8388.258651] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 8388.275206] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 8388.288348] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 8388.296542] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8388.306916] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 8388.315786] Call Trace: [ 8388.317192] [ 8388.317982] ? show_regs.cold.9+0x22/0x2f [ 8388.326702] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8388.335037] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8388.341441] ? watchdog+0x40/0x40 [ 8388.354682] ? hrtimer_interrupt+0x128/0x2c0 [ 8388.358886] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8388.363504] ? apic_timer_interrupt+0xf/0x20 [ 8388.366494] [ 8388.370852] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8388.376866] do_raw_spin_lock+0x177/0x190 [ 8388.380728] _raw_spin_lock+0x21/0x30 [ 8388.384627] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8388.387640] ? do_raw_spin_unlock+0x75/0x190 [ 8388.389187] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8388.392985] ? do_wait_intr+0xf0/0xf0 [ 8388.395826] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8388.403343] mdc_close+0x53b/0xd70 [mdc] [ 8388.410050] lmv_close+0x18d/0x490 [lmv] [ 8388.412292] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8388.416348] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8388.419488] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8388.422620] ll_md_close+0x274/0x800 [lustre] [ 8388.425488] ll_file_release+0x128/0xc50 [lustre] [ 8388.434453] __fput+0x10d/0x440 [ 8388.437437] ____fput+0x12/0x20 [ 8388.438830] task_work_run+0xc6/0x110 [ 8388.447898] exit_to_usermode_loop+0x1dd/0x1f0 [ 8388.457039] do_syscall_64+0x3d6/0x440 [ 8388.458551] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8388.465991] RIP: 0033:0x7f151bee3cd8 [ 8388.468050] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8388.489189] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8388.497601] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 8388.501786] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 8388.506845] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8388.521976] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8388.529220] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8396.055111] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 8396.072775] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8396.133529] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8396.140484] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8396.147911] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8396.157035] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8396.168573] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8396.176110] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 8396.184328] RDX: 0000000000007235 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 8396.196136] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 8396.200041] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 8396.204226] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8396.208346] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 8396.213359] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8396.215703] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8396.223436] Call Trace: [ 8396.224464] [ 8396.227214] ? show_regs.cold.9+0x22/0x2f [ 8396.230959] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8396.233913] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8396.236361] ? watchdog+0x40/0x40 [ 8396.238451] ? hrtimer_interrupt+0x128/0x2c0 [ 8396.240783] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8396.244032] ? apic_timer_interrupt+0xf/0x20 [ 8396.249106] [ 8396.251264] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8396.258094] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8396.262436] do_raw_spin_lock+0x177/0x190 [ 8396.265011] _raw_spin_lock+0x21/0x30 [ 8396.267573] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8396.272554] obd_mod_free+0x46/0x140 [ptlrpc] [ 8396.278928] mdc_commit_open+0x91/0xb0 [mdc] [ 8396.283422] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8396.296057] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8396.302538] after_reply+0xe2c/0x1780 [ptlrpc] [ 8396.311335] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8396.317391] ? __next_timer_interrupt+0x160/0x160 [ 8396.322163] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8396.324965] ? do_wait_intr+0xf0/0xf0 [ 8396.331539] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8396.335386] kthread+0x1d1/0x200 [ 8396.337321] ? set_kthread_struct+0x70/0x70 [ 8396.339100] ret_from_fork+0x1f/0x30 [ 8404.053265] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 8404.069645] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8404.127096] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8404.134720] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8404.139004] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8404.141736] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8404.156180] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8404.167253] RAX: 000000000000798d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8404.172502] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 8404.176321] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 8404.180158] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 8404.183246] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 8404.188998] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 8404.199393] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8404.205854] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 8404.210846] Call Trace: [ 8404.212158] [ 8404.213846] ? show_regs.cold.9+0x22/0x2f [ 8404.215341] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8404.220784] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8404.222435] ? watchdog+0x40/0x40 [ 8404.225873] ? hrtimer_interrupt+0x128/0x2c0 [ 8404.228851] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8404.235160] ? apic_timer_interrupt+0xf/0x20 [ 8404.243125] [ 8404.244599] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8404.249020] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8404.253090] do_raw_spin_lock+0x177/0x190 [ 8404.260057] _raw_spin_lock+0x21/0x30 [ 8404.264912] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8404.274012] ? do_raw_spin_unlock+0x75/0x190 [ 8404.281277] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8404.283504] ? do_wait_intr+0xf0/0xf0 [ 8404.287224] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8404.291305] mdc_reint+0x2d/0x110 [mdc] [ 8404.293915] mdc_unlink+0x658/0x870 [mdc] [ 8404.297138] lmv_unlink+0x34d/0xbd0 [lmv] [ 8404.299639] ll_unlink+0x24d/0x7f0 [lustre] [ 8404.304645] vfs_unlink+0x202/0x330 [ 8404.307903] do_unlinkat+0x2fa/0x460 [ 8404.311135] __x64_sys_unlink+0x37/0x50 [ 8404.321195] do_syscall_64+0xc1/0x440 [ 8404.326891] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8404.335037] RIP: 0033:0x7f151bdee10b [ 8404.337164] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8404.357879] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8404.363878] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 8404.381097] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 8404.387410] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8404.404906] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8404.413109] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8408.545040] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6477s! [ 8408.548021] Showing busy workqueues and worker pools: [ 8408.552422] workqueue events: flags=0x0 [ 8408.553963] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8408.560095] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 8408.566284] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8408.576041] pending: kfree_rcu_monitor [ 8408.582386] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8408.586769] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 8408.594842] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8408.600892] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 8408.606121] workqueue events_long: flags=0x0 [ 8408.608111] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8408.616306] pending: lu_objects_flush [obdclass] [ 8408.618634] workqueue events_unbound: flags=0x2 [ 8408.620675] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8408.623249] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8408.629171] workqueue events_power_efficient: flags=0x80 [ 8408.631179] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8408.635840] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 8408.638839] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8408.641252] pending: neigh_periodic_work [ 8408.642588] workqueue mm_percpu_wq: flags=0x8 [ 8408.643926] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8408.647755] pending: vmstat_update, lru_add_drain_per_cpu [ 8408.650845] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8408.660312] pending: vmstat_update [ 8408.662224] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8408.667738] pending: lru_add_drain_per_cpu [ 8408.669562] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8408.674235] pending: lru_add_drain_per_cpu BAR(48) [ 8408.682586] workqueue writeback: flags=0x4a [ 8408.685213] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 8408.688107] pending: wb_workfn [ 8408.689850] workqueue xprtiod: flags=0xa [ 8408.692096] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8408.694717] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8408.701032] workqueue ipv6_addrconf: flags=0x40008 [ 8408.702987] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8408.710465] pending: addrconf_verify_work [ 8408.715755] workqueue knbd0-recv: flags=0x1a [ 8408.721596] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8408.728122] in-flight: 658:recv_work [ 8408.730535] workqueue ptlrpc_pinger: flags=0x2 [ 8408.734392] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8408.742993] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8408.746041] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6478s workers=4 idle: 32984 37 8 [ 8408.749860] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8393s workers=2 idle: 661 [ 8412.056016] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 8412.060356] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8412.100985] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8412.106230] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8412.111211] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8412.114458] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8412.118148] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8412.128174] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8412.132244] RAX: 0000000000004788 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8412.136063] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 8412.139866] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 8412.142885] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 8412.145312] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 8412.149230] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 8412.153445] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8412.156598] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8412.160830] Call Trace: [ 8412.161953] [ 8412.162978] ? show_regs.cold.9+0x22/0x2f [ 8412.165048] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8412.168031] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8412.170863] ? watchdog+0x40/0x40 [ 8412.172433] ? hrtimer_interrupt+0x128/0x2c0 [ 8412.174753] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8412.176977] ? apic_timer_interrupt+0xf/0x20 [ 8412.179142] [ 8412.180244] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8412.183548] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8412.186724] do_raw_spin_lock+0x177/0x190 [ 8412.188808] _raw_spin_lock+0x21/0x30 [ 8412.190836] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8412.193980] process_one_work+0x2c8/0x700 [ 8412.196108] worker_thread+0x296/0x6e0 [ 8412.198190] ? rescuer_thread+0x570/0x570 [ 8412.200814] kthread+0x1d1/0x200 [ 8412.202826] ? set_kthread_struct+0x70/0x70 [ 8412.205244] ret_from_fork+0x1f/0x30 [ 8416.057422] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 8416.072639] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8416.120855] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8416.131641] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8416.149296] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8416.160730] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8416.196911] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8416.211310] RAX: 0000000000003183 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8416.222227] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 8416.233567] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8416.236146] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 8416.245400] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 8416.253849] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 8416.264793] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8416.269480] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 8416.280369] Call Trace: [ 8416.284695] [ 8416.286344] ? show_regs.cold.9+0x22/0x2f [ 8416.290871] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8416.295178] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8416.298262] ? watchdog+0x40/0x40 [ 8416.299465] ? hrtimer_interrupt+0x128/0x2c0 [ 8416.306825] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8416.312091] ? apic_timer_interrupt+0xf/0x20 [ 8416.317062] [ 8416.318827] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8416.325326] do_raw_spin_lock+0x177/0x190 [ 8416.329225] _raw_spin_lock+0x21/0x30 [ 8416.333214] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8416.341751] ? do_raw_spin_unlock+0x75/0x190 [ 8416.343387] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8416.355799] ? do_wait_intr+0xf0/0xf0 [ 8416.360673] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8416.371399] mdc_close+0x53b/0xd70 [mdc] [ 8416.374291] lmv_close+0x18d/0x490 [lmv] [ 8416.379461] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8416.386263] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8416.397785] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8416.406292] ll_md_close+0x274/0x800 [lustre] [ 8416.418995] ll_file_release+0x128/0xc50 [lustre] [ 8416.425293] __fput+0x10d/0x440 [ 8416.432823] ____fput+0x12/0x20 [ 8416.440379] task_work_run+0xc6/0x110 [ 8416.442084] exit_to_usermode_loop+0x1dd/0x1f0 [ 8416.444744] do_syscall_64+0x3d6/0x440 [ 8416.449627] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8416.456071] RIP: 0033:0x7f151bee3cd8 [ 8416.458665] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8416.471928] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8416.476292] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 8416.484163] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 8416.489826] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8416.494660] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8416.497989] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8424.057260] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 8424.059498] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8424.098080] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8424.105561] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8424.117899] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8424.126610] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8424.140975] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8424.146652] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 8424.154057] RDX: 0000000000004218 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 8424.161094] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 8424.164753] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 8424.172926] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8424.178875] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 8424.183329] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8424.184927] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8424.193564] Call Trace: [ 8424.197595] [ 8424.199428] ? show_regs.cold.9+0x22/0x2f [ 8424.201781] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8424.205067] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8424.206863] ? watchdog+0x40/0x40 [ 8424.210581] ? hrtimer_interrupt+0x128/0x2c0 [ 8424.215714] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8424.217214] ? apic_timer_interrupt+0xf/0x20 [ 8424.221552] [ 8424.222679] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8424.227714] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8424.232284] do_raw_spin_lock+0x177/0x190 [ 8424.234965] _raw_spin_lock+0x21/0x30 [ 8424.236240] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8424.238708] obd_mod_free+0x46/0x140 [ptlrpc] [ 8424.241455] mdc_commit_open+0x91/0xb0 [mdc] [ 8424.243974] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8424.247120] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8424.252913] after_reply+0xe2c/0x1780 [ptlrpc] [ 8424.256577] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8424.264643] ? __next_timer_interrupt+0x160/0x160 [ 8424.266208] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8424.270033] ? do_wait_intr+0xf0/0xf0 [ 8424.271898] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8424.275139] kthread+0x1d1/0x200 [ 8424.277026] ? set_kthread_struct+0x70/0x70 [ 8424.280428] ret_from_fork+0x1f/0x30 [ 8436.052437] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 8436.057768] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8436.080618] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8436.085741] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8436.089880] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8436.093144] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8436.101335] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8436.104854] RAX: 000000000000449b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8436.108551] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 8436.112132] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 8436.114853] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 8436.117744] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 8436.120472] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 8436.124162] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8436.126833] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 8436.130018] Call Trace: [ 8436.131328] [ 8436.132376] ? show_regs.cold.9+0x22/0x2f [ 8436.134169] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8436.136725] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8436.138307] ? watchdog+0x40/0x40 [ 8436.139832] ? hrtimer_interrupt+0x128/0x2c0 [ 8436.142465] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8436.144942] ? apic_timer_interrupt+0xf/0x20 [ 8436.146774] [ 8436.147945] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8436.150451] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8436.152821] do_raw_spin_lock+0x177/0x190 [ 8436.154869] _raw_spin_lock+0x21/0x30 [ 8436.156537] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8436.159220] ? do_raw_spin_unlock+0x75/0x190 [ 8436.161200] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8436.163989] ? do_wait_intr+0xf0/0xf0 [ 8436.165885] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8436.168553] mdc_reint+0x2d/0x110 [mdc] [ 8436.170466] mdc_unlink+0x658/0x870 [mdc] [ 8436.174250] lmv_unlink+0x34d/0xbd0 [lmv] [ 8436.176784] ll_unlink+0x24d/0x7f0 [lustre] [ 8436.180338] vfs_unlink+0x202/0x330 [ 8436.182263] do_unlinkat+0x2fa/0x460 [ 8436.184193] __x64_sys_unlink+0x37/0x50 [ 8436.186419] do_syscall_64+0xc1/0x440 [ 8436.188498] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8436.192116] RIP: 0033:0x7f151bdee10b [ 8436.194315] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8436.204121] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8436.209902] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 8436.214172] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 8436.218739] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8436.224947] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8436.230378] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8439.268125] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6508s! [ 8439.276085] Showing busy workqueues and worker pools: [ 8439.279015] workqueue events: flags=0x0 [ 8439.285780] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8439.292455] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 8439.297379] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8439.306104] pending: kfree_rcu_monitor [ 8439.311157] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8439.317250] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 8439.323113] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8439.327395] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 8439.331761] workqueue events_long: flags=0x0 [ 8439.337867] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8439.345255] pending: lu_objects_flush [obdclass] [ 8439.351503] workqueue events_unbound: flags=0x2 [ 8439.356349] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8439.360314] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8439.372760] workqueue events_power_efficient: flags=0x80 [ 8439.378750] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8439.384399] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 8439.388098] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8439.391730] pending: neigh_periodic_work [ 8439.394071] workqueue mm_percpu_wq: flags=0x8 [ 8439.396907] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8439.403497] pending: vmstat_update, lru_add_drain_per_cpu [ 8439.409439] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8439.413970] pending: vmstat_update [ 8439.418328] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8439.423727] pending: lru_add_drain_per_cpu [ 8439.426067] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8439.429401] pending: lru_add_drain_per_cpu BAR(48) [ 8439.431913] workqueue writeback: flags=0x4a [ 8439.433676] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 8439.436781] pending: wb_workfn [ 8439.438102] workqueue xprtiod: flags=0xa [ 8439.447855] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8439.449924] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8439.455460] workqueue ipv6_addrconf: flags=0x40008 [ 8439.456943] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8439.467397] pending: addrconf_verify_work [ 8439.474864] workqueue knbd0-recv: flags=0x1a [ 8439.480067] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8439.488489] in-flight: 658:recv_work [ 8439.490283] workqueue ptlrpc_pinger: flags=0x2 [ 8439.493090] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8439.495118] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8439.506126] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6508s workers=4 idle: 32984 37 8 [ 8439.516855] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8424s workers=2 idle: 661 [ 8440.056059] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8440.063350] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8440.115667] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8440.122617] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8440.127020] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8440.130565] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8440.139051] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8440.152040] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8440.156046] RAX: 0000000000007b8c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8440.159599] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 8440.162982] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 8440.172226] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 8440.178706] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 8440.189949] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 8440.198805] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8440.204883] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8440.214063] Call Trace: [ 8440.215026] [ 8440.218036] ? show_regs.cold.9+0x22/0x2f [ 8440.221622] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8440.226544] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8440.230352] ? watchdog+0x40/0x40 [ 8440.231682] ? hrtimer_interrupt+0x128/0x2c0 [ 8440.238853] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8440.242068] ? apic_timer_interrupt+0xf/0x20 [ 8440.247202] [ 8440.249001] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8440.254936] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8440.261224] do_raw_spin_lock+0x177/0x190 [ 8440.263670] _raw_spin_lock+0x21/0x30 [ 8440.266465] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8440.269635] process_one_work+0x2c8/0x700 [ 8440.275847] worker_thread+0x296/0x6e0 [ 8440.284910] ? rescuer_thread+0x570/0x570 [ 8440.289517] kthread+0x1d1/0x200 [ 8440.293600] ? set_kthread_struct+0x70/0x70 [ 8440.296700] ret_from_fork+0x1f/0x30 [ 8444.057027] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 8444.060097] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8444.078887] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8444.084627] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8444.088683] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8444.090741] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8444.098609] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 8444.101882] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8444.105131] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 8444.108289] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8444.111427] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 8444.114572] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 8444.117790] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 8444.121307] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8444.124031] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 8444.127928] Call Trace: [ 8444.129327] [ 8444.130381] ? show_regs.cold.9+0x22/0x2f [ 8444.132679] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8444.135240] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8444.137476] ? watchdog+0x40/0x40 [ 8444.139368] ? hrtimer_interrupt+0x128/0x2c0 [ 8444.142874] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8444.146643] ? apic_timer_interrupt+0xf/0x20 [ 8444.149432] [ 8444.150629] ? native_safe_halt+0x2e/0x50 [ 8444.153705] kvm_wait+0xcd/0xe0 [ 8444.157057] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8444.161409] do_raw_spin_lock+0x177/0x190 [ 8444.163708] _raw_spin_lock+0x21/0x30 [ 8444.165956] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8444.169231] ? do_raw_spin_unlock+0x75/0x190 [ 8444.171718] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8444.174755] ? do_wait_intr+0xf0/0xf0 [ 8444.179117] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8444.187009] mdc_close+0x53b/0xd70 [mdc] [ 8444.191696] lmv_close+0x18d/0x490 [lmv] [ 8444.194358] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8444.197304] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8444.203633] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8444.208496] ll_md_close+0x274/0x800 [lustre] [ 8444.211794] ll_file_release+0x128/0xc50 [lustre] [ 8444.215438] __fput+0x10d/0x440 [ 8444.219049] ____fput+0x12/0x20 [ 8444.221018] task_work_run+0xc6/0x110 [ 8444.222978] exit_to_usermode_loop+0x1dd/0x1f0 [ 8444.225025] do_syscall_64+0x3d6/0x440 [ 8444.227435] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8444.233024] RIP: 0033:0x7f151bee3cd8 [ 8444.235736] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8444.248902] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8444.257336] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 8444.262982] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 8444.268401] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8444.272594] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8444.280095] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8452.055987] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 8452.062034] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8452.109887] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8452.119114] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8452.129918] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8452.139419] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8452.170917] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8452.182365] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 8452.190681] RDX: 0000000000000961 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 8452.200493] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 8452.208390] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 8452.217317] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8452.223310] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 8452.228247] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8452.230935] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8452.234332] Call Trace: [ 8452.235343] [ 8452.236016] ? show_regs.cold.9+0x22/0x2f [ 8452.237517] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8452.248677] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8452.258611] ? watchdog+0x40/0x40 [ 8452.261871] ? hrtimer_interrupt+0x128/0x2c0 [ 8452.267801] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8452.273160] ? apic_timer_interrupt+0xf/0x20 [ 8452.275496] [ 8452.280655] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8452.284749] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8452.292469] do_raw_spin_lock+0x177/0x190 [ 8452.297941] _raw_spin_lock+0x21/0x30 [ 8452.301657] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8452.307944] obd_mod_free+0x46/0x140 [ptlrpc] [ 8452.314003] mdc_commit_open+0x91/0xb0 [mdc] [ 8452.320222] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8452.329548] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8452.334548] after_reply+0xe2c/0x1780 [ptlrpc] [ 8452.340693] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8452.344555] ? __next_timer_interrupt+0x160/0x160 [ 8452.351796] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8452.354950] ? do_wait_intr+0xf0/0xf0 [ 8452.359329] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8452.365522] kthread+0x1d1/0x200 [ 8452.367959] ? set_kthread_struct+0x70/0x70 [ 8452.371173] ret_from_fork+0x1f/0x30 [ 8464.054547] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 8464.065049] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8464.085612] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8464.089745] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8464.093001] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8464.095316] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8464.102241] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8464.105043] RAX: 0000000000004183 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8464.107536] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 8464.110291] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 8464.113089] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 8464.115525] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 8464.118205] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 8464.121215] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8464.123304] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 8464.126137] Call Trace: [ 8464.127194] [ 8464.128010] ? show_regs.cold.9+0x22/0x2f [ 8464.129491] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8464.131229] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8464.132899] ? watchdog+0x40/0x40 [ 8464.134191] ? hrtimer_interrupt+0x128/0x2c0 [ 8464.135774] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8464.145590] ? apic_timer_interrupt+0xf/0x20 [ 8464.147846] [ 8464.148665] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8464.153927] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8464.159427] do_raw_spin_lock+0x177/0x190 [ 8464.162187] _raw_spin_lock+0x21/0x30 [ 8464.165500] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8464.172728] ? do_raw_spin_unlock+0x75/0x190 [ 8464.178326] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8464.183631] ? do_wait_intr+0xf0/0xf0 [ 8464.189521] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8464.192434] mdc_reint+0x2d/0x110 [mdc] [ 8464.196771] mdc_unlink+0x658/0x870 [mdc] [ 8464.201846] lmv_unlink+0x34d/0xbd0 [lmv] [ 8464.205444] ll_unlink+0x24d/0x7f0 [lustre] [ 8464.210846] vfs_unlink+0x202/0x330 [ 8464.216926] do_unlinkat+0x2fa/0x460 [ 8464.218717] __x64_sys_unlink+0x37/0x50 [ 8464.221805] do_syscall_64+0xc1/0x440 [ 8464.225933] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8464.229288] RIP: 0033:0x7f151bdee10b [ 8464.232722] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8464.260409] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8464.268470] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 8464.274521] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 8464.284756] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8464.288997] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8464.298804] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8464.786430] rcu: INFO: rcu_sched self-detected stall on CPU [ 8464.789515] rcu: 1-...!: (3714413 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 8464.794114] (t=6540139 jiffies g=249657 q=4276) [ 8464.797864] rcu: rcu_sched kthread starved for 6535091 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 8464.805776] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8464.814103] rcu: RCU grace-period kthread stack dump: [ 8464.816093] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8464.822484] Call Trace: [ 8464.825286] __schedule+0x351/0xcb0 [ 8464.826679] schedule+0xc0/0x180 [ 8464.827931] schedule_timeout+0xb4/0x190 [ 8464.835934] ? do_raw_spin_unlock+0x75/0x190 [ 8464.839269] ? __next_timer_interrupt+0x160/0x160 [ 8464.842525] ? prepare_to_swait_event+0xf3/0x140 [ 8464.849118] rcu_gp_kthread+0x8dd/0x15b0 [ 8464.853823] ? force_qs_rnp+0x2c0/0x2c0 [ 8464.857167] kthread+0x1d1/0x200 [ 8464.860872] ? set_kthread_struct+0x70/0x70 [ 8464.863864] ret_from_fork+0x1f/0x30 [ 8464.864955] rcu: Stack dump where RCU GP kthread last ran: [ 8464.868128] Sending NMI from CPU 1 to CPUs 3: [ 8464.871803] NMI backtrace for cpu 3 skipped: idling at native_safe_halt+0x2e/0x50 [ 8464.872767] NMI backtrace for cpu 1 [ 8464.883327] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8464.894606] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8464.899747] Call Trace: [ 8464.900894] [ 8464.903090] ? dump_stack+0xbb/0x10e [ 8464.905578] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8464.908794] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8464.911323] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8464.919039] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8464.922003] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8464.924845] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 8464.930576] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 8464.934677] ? update_process_times+0x6d/0xc0 [ 8464.937296] ? tick_sched_handle.isra.18+0x27/0x90 [ 8464.938841] ? tick_sched_timer+0x53/0xb0 [ 8464.942305] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8464.944682] ? tick_sched_handle.isra.18+0x90/0x90 [ 8464.947920] ? hrtimer_interrupt+0x128/0x2c0 [ 8464.950400] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8464.953569] ? apic_timer_interrupt+0xf/0x20 [ 8464.957900] [ 8464.960228] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8464.963864] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8464.967148] ? do_raw_spin_lock+0x177/0x190 [ 8464.972181] ? _raw_spin_lock+0x21/0x30 [ 8464.974628] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8464.978550] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 8464.983247] ? mdc_commit_open+0x91/0xb0 [mdc] [ 8464.991728] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8464.995437] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8465.001436] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 8465.004335] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8465.007204] ? __next_timer_interrupt+0x160/0x160 [ 8465.009896] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8465.012221] ? do_wait_intr+0xf0/0xf0 [ 8465.019759] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8465.021945] ? kthread+0x1d1/0x200 [ 8465.026824] ? set_kthread_struct+0x70/0x70 [ 8465.028539] ? ret_from_fork+0x1f/0x30 [ 8468.060801] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8468.065561] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8468.105783] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8468.113275] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8468.121619] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8468.128541] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8468.130897] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8468.147414] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8468.152773] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8468.156575] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 8468.161404] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 8468.165174] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 8468.172436] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 8468.176491] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 8468.183375] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8468.186786] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8468.190479] Call Trace: [ 8468.191706] [ 8468.192985] ? show_regs.cold.9+0x22/0x2f [ 8468.195085] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8468.197197] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8468.199621] ? watchdog+0x40/0x40 [ 8468.201129] ? hrtimer_interrupt+0x128/0x2c0 [ 8468.203698] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8468.207543] ? apic_timer_interrupt+0xf/0x20 [ 8468.212682] [ 8468.214319] ? native_safe_halt+0x2e/0x50 [ 8468.219338] kvm_wait+0xcd/0xe0 [ 8468.221424] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8468.226466] do_raw_spin_lock+0x177/0x190 [ 8468.229132] _raw_spin_lock+0x21/0x30 [ 8468.231178] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8468.235476] process_one_work+0x2c8/0x700 [ 8468.242606] worker_thread+0x296/0x6e0 [ 8468.244685] ? rescuer_thread+0x570/0x570 [ 8468.247721] kthread+0x1d1/0x200 [ 8468.249476] ? set_kthread_struct+0x70/0x70 [ 8468.251391] ret_from_fork+0x1f/0x30 [ 8469.988483] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6539s! [ 8469.992202] Showing busy workqueues and worker pools: [ 8469.994606] workqueue events: flags=0x0 [ 8469.996190] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8470.000727] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 8470.012532] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8470.015659] pending: kfree_rcu_monitor [ 8470.017861] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8470.021431] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 8470.030548] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8470.040027] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 8470.043562] workqueue events_long: flags=0x0 [ 8470.046055] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8470.050727] pending: lu_objects_flush [obdclass] [ 8470.053640] workqueue events_unbound: flags=0x2 [ 8470.056015] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8470.062702] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8470.067959] workqueue events_power_efficient: flags=0x80 [ 8470.073704] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8470.076859] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 8470.080574] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8470.086106] pending: neigh_periodic_work [ 8470.087749] workqueue mm_percpu_wq: flags=0x8 [ 8470.089282] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8470.095731] pending: vmstat_update, lru_add_drain_per_cpu [ 8470.098595] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8470.102003] pending: vmstat_update [ 8470.103539] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8470.106573] pending: lru_add_drain_per_cpu [ 8470.108838] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8470.112033] pending: lru_add_drain_per_cpu BAR(48) [ 8470.115116] workqueue writeback: flags=0x4a [ 8470.117758] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 8470.120764] pending: wb_workfn [ 8470.122540] workqueue xprtiod: flags=0xa [ 8470.124454] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8470.128832] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8470.132731] workqueue ipv6_addrconf: flags=0x40008 [ 8470.134710] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8470.136856] pending: addrconf_verify_work [ 8470.142580] workqueue knbd0-recv: flags=0x1a [ 8470.146812] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8470.153038] in-flight: 658:recv_work [ 8470.155737] workqueue ptlrpc_pinger: flags=0x2 [ 8470.159495] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8470.163730] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8470.172354] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6539s workers=4 idle: 32984 37 8 [ 8470.177534] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8454s workers=2 idle: 661 [ 8492.055271] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 8492.057073] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 8492.065457] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) [ 8492.069473] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8492.069926] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8492.069934] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8492.069938] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8492.069953] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8492.069957] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8492.069962] RAX: 0000000000007488 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8492.069964] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 8492.069966] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8492.069968] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 8492.069971] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 8492.069974] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 8492.069977] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8492.069979] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 8492.069989] Call Trace: [ 8492.069994] [ 8492.070000] ? show_regs.cold.9+0x22/0x2f [ 8492.070011] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8492.070018] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8492.070023] ? watchdog+0x40/0x40 [ 8492.070028] ? hrtimer_interrupt+0x128/0x2c0 [ 8492.070032] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8492.071165] ? apic_timer_interrupt+0xf/0x20 [ 8492.071170] [ 8492.071174] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8492.071182] do_raw_spin_lock+0x177/0x190 [ 8492.071187] _raw_spin_lock+0x21/0x30 [ 8492.071192] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8492.071495] ? do_raw_spin_unlock+0x75/0x190 [ 8492.072412] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8492.073481] ? do_wait_intr+0xf0/0xf0 [ 8492.073491] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8492.080488] mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8492.080563] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8492.080570] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8492.080573] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8492.080588] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8492.080592] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8492.080597] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 8492.080599] RDX: 0000000000007e8a RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 8492.080602] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 8492.080605] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 8492.080608] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8492.080611] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 8492.080614] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8492.080616] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8492.080624] Call Trace: [ 8492.080629] [ 8492.080634] ? show_regs.cold.9+0x22/0x2f [ 8492.080642] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8492.080648] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8492.080652] ? watchdog+0x40/0x40 [ 8492.080656] ? hrtimer_interrupt+0x128/0x2c0 [ 8492.080660] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8492.080846] ? apic_timer_interrupt+0xf/0x20 [ 8492.080858] [ 8492.080861] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8492.080866] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8492.080871] do_raw_spin_lock+0x177/0x190 [ 8492.080876] _raw_spin_lock+0x21/0x30 [ 8492.080882] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8492.081391] obd_mod_free+0x46/0x140 [ptlrpc] [ 8492.081599] mdc_commit_open+0x91/0xb0 [mdc] [ 8492.081638] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8492.081965] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8492.087415] after_reply+0xe2c/0x1780 [ptlrpc] [ 8492.088174] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8492.088365] ? __next_timer_interrupt+0x160/0x160 [ 8492.088376] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8492.088587] ? do_wait_intr+0xf0/0xf0 [ 8492.125118] mdc_close+0x53b/0xd70 [mdc] [ 8492.131499] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8492.131778] kthread+0x1d1/0x200 [ 8492.131788] ? set_kthread_struct+0x70/0x70 [ 8492.131793] ret_from_fork+0x1f/0x30 [ 8492.601103] lmv_close+0x18d/0x490 [lmv] [ 8492.606984] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8492.610489] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8492.613436] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8492.618523] ll_md_close+0x274/0x800 [lustre] [ 8492.621973] ll_file_release+0x128/0xc50 [lustre] [ 8492.625369] __fput+0x10d/0x440 [ 8492.627617] ____fput+0x12/0x20 [ 8492.629133] task_work_run+0xc6/0x110 [ 8492.631554] exit_to_usermode_loop+0x1dd/0x1f0 [ 8492.633529] do_syscall_64+0x3d6/0x440 [ 8492.638645] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8492.640930] RIP: 0033:0x7f151bee3cd8 [ 8492.641924] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8492.651681] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8492.657132] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 8492.662211] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 8492.665660] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8492.671596] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8492.678636] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8496.052092] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 8496.056026] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8496.057339] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8496.057428] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8496.057436] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8496.057439] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8496.057454] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8496.057459] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8496.057463] RAX: 00000000000002e3 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8496.057466] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 8496.057469] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 8496.057471] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 8496.057474] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 8496.063000] Modules linked in: lustre(O) [ 8496.090309] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 8496.090320] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8496.090322] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 8496.090334] Call Trace: [ 8496.099803] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8496.099883] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8496.099891] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8496.099898] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8496.105340] [ 8496.105356] ? show_regs.cold.9+0x22/0x2f [ 8496.114594] [ 8496.114604] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8496.114624] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8496.121665] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8496.121687] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8496.121692] ? watchdog+0x40/0x40 [ 8496.121696] ? hrtimer_interrupt+0x128/0x2c0 [ 8496.121701] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8496.128975] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8496.128990] RAX: 00000000000058a0 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8496.128992] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 8496.128994] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 8496.129001] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 8496.129003] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 8496.129006] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 8496.129009] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8496.129012] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8496.135065] ? apic_timer_interrupt+0xf/0x20 [ 8496.135090] [ 8496.135095] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8496.135104] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8496.135110] do_raw_spin_lock+0x177/0x190 [ 8496.135117] _raw_spin_lock+0x21/0x30 [ 8496.135126] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8496.135980] ? do_raw_spin_unlock+0x75/0x190 [ 8496.135995] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8496.136518] ? do_wait_intr+0xf0/0xf0 [ 8496.136527] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8496.136943] mdc_reint+0x2d/0x110 [mdc] [ 8496.136983] mdc_unlink+0x658/0x870 [mdc] [ 8496.137012] lmv_unlink+0x34d/0xbd0 [lmv] [ 8496.145199] Call Trace: [ 8496.145214] [ 8496.145220] ? show_regs.cold.9+0x22/0x2f [ 8496.145237] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8496.145244] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8496.145249] ? watchdog+0x40/0x40 [ 8496.145253] ? hrtimer_interrupt+0x128/0x2c0 [ 8496.145257] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8496.145268] ? apic_timer_interrupt+0xf/0x20 [ 8496.145274] [ 8496.145277] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8496.145284] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8496.145292] do_raw_spin_lock+0x177/0x190 [ 8496.145303] _raw_spin_lock+0x21/0x30 [ 8496.145308] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8496.146848] process_one_work+0x2c8/0x700 [ 8496.146859] worker_thread+0x296/0x6e0 [ 8496.146864] ? rescuer_thread+0x570/0x570 [ 8496.146868] kthread+0x1d1/0x200 [ 8496.146873] ? set_kthread_struct+0x70/0x70 [ 8496.146877] ret_from_fork+0x1f/0x30 [ 8496.394688] ll_unlink+0x24d/0x7f0 [lustre] [ 8496.397556] vfs_unlink+0x202/0x330 [ 8496.401621] do_unlinkat+0x2fa/0x460 [ 8496.407155] __x64_sys_unlink+0x37/0x50 [ 8496.413891] do_syscall_64+0xc1/0x440 [ 8496.415951] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8496.419165] RIP: 0033:0x7f151bdee10b [ 8496.421142] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8496.430562] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8496.434573] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 8496.438236] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 8496.441884] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8496.445918] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8496.449745] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8500.193676] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6569s! [ 8500.197374] Showing busy workqueues and worker pools: [ 8500.199497] workqueue events: flags=0x0 [ 8500.201363] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8500.204338] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 8500.209564] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8500.212190] pending: kfree_rcu_monitor [ 8500.214063] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8500.217041] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 8500.220133] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8500.222974] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 8500.226229] workqueue events_long: flags=0x0 [ 8500.227838] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8500.230167] pending: lu_objects_flush [obdclass] [ 8500.232722] workqueue events_unbound: flags=0x2 [ 8500.234948] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8500.237935] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8500.242859] workqueue events_power_efficient: flags=0x80 [ 8500.244905] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8500.247628] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 8500.250811] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8500.254057] pending: neigh_periodic_work [ 8500.256233] workqueue mm_percpu_wq: flags=0x8 [ 8500.258066] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8500.260879] pending: vmstat_update, lru_add_drain_per_cpu [ 8500.263499] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8500.266473] pending: vmstat_update [ 8500.267984] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8500.270759] pending: lru_add_drain_per_cpu [ 8500.272986] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8500.275631] pending: lru_add_drain_per_cpu BAR(48) [ 8500.278059] workqueue writeback: flags=0x4a [ 8500.279882] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 8500.282504] pending: wb_workfn [ 8500.283897] workqueue xprtiod: flags=0xa [ 8500.285643] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8500.288324] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8500.291664] workqueue ipv6_addrconf: flags=0x40008 [ 8500.294016] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8500.297224] pending: addrconf_verify_work [ 8500.299564] workqueue knbd0-recv: flags=0x1a [ 8500.301536] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8500.304301] in-flight: 658:recv_work [ 8500.305557] workqueue ptlrpc_pinger: flags=0x2 [ 8500.308069] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8500.310720] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8500.313535] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6569s workers=4 idle: 32984 37 8 [ 8500.317297] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8485s workers=2 idle: 661 [ 8520.057747] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 8520.058224] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 8520.066240] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8520.066315] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8520.066323] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8520.066326] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 8520.066340] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 8520.066344] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8520.066348] RAX: 0000000000004680 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8520.066350] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 8520.066352] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8520.066354] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 8520.066357] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 8520.066360] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 8520.066363] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8520.066365] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 8520.066374] Call Trace: [ 8520.066380] [ 8520.066385] ? show_regs.cold.9+0x22/0x2f [ 8520.066395] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8520.066953] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8520.066957] ? watchdog+0x40/0x40 [ 8520.066961] ? hrtimer_interrupt+0x128/0x2c0 [ 8520.066966] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8520.066975] ? apic_timer_interrupt+0xf/0x20 [ 8520.066980] [ 8520.066984] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 8520.066991] do_raw_spin_lock+0x177/0x190 [ 8520.066996] _raw_spin_lock+0x21/0x30 [ 8520.067001] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8520.071043] Modules linked in: lustre(O) [ 8520.095569] ? do_raw_spin_unlock+0x75/0x190 [ 8520.095591] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8520.100546] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8520.100694] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8520.100703] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8520.100705] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8520.100725] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8520.100729] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8520.100738] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 8520.100740] RDX: 000000000000779d RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 8520.100743] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 8520.105414] ? do_wait_intr+0xf0/0xf0 [ 8520.108145] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 8520.108155] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8520.108158] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 8520.108161] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8520.108163] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8520.108170] Call Trace: [ 8520.108176] [ 8520.108181] ? show_regs.cold.9+0x22/0x2f [ 8520.108193] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8520.108200] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8520.108204] ? watchdog+0x40/0x40 [ 8520.108209] ? hrtimer_interrupt+0x128/0x2c0 [ 8520.108212] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8520.108222] ? apic_timer_interrupt+0xf/0x20 [ 8520.108227] [ 8520.108230] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8520.108237] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8520.108241] do_raw_spin_lock+0x177/0x190 [ 8520.108246] _raw_spin_lock+0x21/0x30 [ 8520.108253] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8520.108520] obd_mod_free+0x46/0x140 [ptlrpc] [ 8520.109015] mdc_commit_open+0x91/0xb0 [mdc] [ 8520.109054] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8520.109284] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8520.109490] after_reply+0xe2c/0x1780 [ptlrpc] [ 8520.109841] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8520.110162] ? __next_timer_interrupt+0x160/0x160 [ 8520.110175] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8520.110395] ? do_wait_intr+0xf0/0xf0 [ 8520.110405] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8520.110863] kthread+0x1d1/0x200 [ 8520.110869] ? set_kthread_struct+0x70/0x70 [ 8520.110873] ret_from_fork+0x1f/0x30 [ 8520.383092] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8520.385782] mdc_close+0x53b/0xd70 [mdc] [ 8520.387255] lmv_close+0x18d/0x490 [lmv] [ 8520.390940] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8520.394383] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8520.396552] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8520.398690] ll_md_close+0x274/0x800 [lustre] [ 8520.400304] ll_file_release+0x128/0xc50 [lustre] [ 8520.403230] __fput+0x10d/0x440 [ 8520.406155] ____fput+0x12/0x20 [ 8520.408239] task_work_run+0xc6/0x110 [ 8520.410098] exit_to_usermode_loop+0x1dd/0x1f0 [ 8520.411936] do_syscall_64+0x3d6/0x440 [ 8520.413881] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8520.418116] RIP: 0033:0x7f151bee3cd8 [ 8520.419883] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8520.433459] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8520.437580] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 8520.440137] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 8520.446233] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8520.450293] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8520.454085] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8524.058352] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8524.062635] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8524.149879] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8524.169058] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8524.176426] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8524.183299] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8524.186880] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8524.223907] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8524.238786] RAX: 0000000000003283 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8524.246117] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 8524.255246] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 8524.265475] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 8524.281842] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 8524.291475] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 8524.309022] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8524.319616] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8524.332061] Call Trace: [ 8524.337572] [ 8524.338363] ? show_regs.cold.9+0x22/0x2f [ 8524.340903] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8524.343604] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8524.344585] ? watchdog+0x40/0x40 [ 8524.345304] ? hrtimer_interrupt+0x128/0x2c0 [ 8524.352944] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8524.361780] ? apic_timer_interrupt+0xf/0x20 [ 8524.364632] [ 8524.367924] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8524.369928] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8524.382884] do_raw_spin_lock+0x177/0x190 [ 8524.386154] _raw_spin_lock+0x21/0x30 [ 8524.387478] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8524.393929] process_one_work+0x2c8/0x700 [ 8524.395371] worker_thread+0x296/0x6e0 [ 8524.401198] ? rescuer_thread+0x570/0x570 [ 8524.404467] kthread+0x1d1/0x200 [ 8524.415115] ? set_kthread_struct+0x70/0x70 [ 8524.419699] ret_from_fork+0x1f/0x30 [ 8528.052020] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 8528.056809] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8528.107852] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8528.122176] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8528.129917] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8528.133445] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8528.154776] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8528.162797] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8528.167765] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 8528.170513] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 8528.174443] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 8528.180285] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 8528.183185] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 8528.186206] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8528.188844] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 8528.192080] Call Trace: [ 8528.192911] [ 8528.193364] ? show_regs.cold.9+0x22/0x2f [ 8528.194950] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8528.200695] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8528.204888] ? watchdog+0x40/0x40 [ 8528.207901] ? hrtimer_interrupt+0x128/0x2c0 [ 8528.210313] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8528.214207] ? apic_timer_interrupt+0xf/0x20 [ 8528.216519] [ 8528.217114] ? native_safe_halt+0x2e/0x50 [ 8528.219555] kvm_wait+0xcd/0xe0 [ 8528.222537] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8528.225838] do_raw_spin_lock+0x177/0x190 [ 8528.229103] _raw_spin_lock+0x21/0x30 [ 8528.232212] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8528.240140] ? do_raw_spin_unlock+0x75/0x190 [ 8528.244912] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8528.250564] ? do_wait_intr+0xf0/0xf0 [ 8528.254228] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8528.262534] mdc_reint+0x2d/0x110 [mdc] [ 8528.265546] mdc_unlink+0x658/0x870 [mdc] [ 8528.268073] lmv_unlink+0x34d/0xbd0 [lmv] [ 8528.272313] ll_unlink+0x24d/0x7f0 [lustre] [ 8528.279445] vfs_unlink+0x202/0x330 [ 8528.282520] do_unlinkat+0x2fa/0x460 [ 8528.284535] __x64_sys_unlink+0x37/0x50 [ 8528.287591] do_syscall_64+0xc1/0x440 [ 8528.289793] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8528.294269] RIP: 0033:0x7f151bdee10b [ 8528.297814] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8528.312427] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8528.320700] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 8528.327227] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 8528.331649] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8528.338727] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8528.348431] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8530.413032] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6599s! [ 8530.420675] Showing busy workqueues and worker pools: [ 8530.424505] workqueue events: flags=0x0 [ 8530.427259] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8530.434407] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 8530.440732] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8530.443109] pending: kfree_rcu_monitor [ 8530.444603] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8530.449126] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 8530.455419] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8530.461419] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 8530.470552] workqueue events_long: flags=0x0 [ 8530.473707] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8530.482073] pending: lu_objects_flush [obdclass] [ 8530.487014] workqueue events_unbound: flags=0x2 [ 8530.489684] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8530.492340] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8530.496165] workqueue events_power_efficient: flags=0x80 [ 8530.498899] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8530.502340] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 8530.508229] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8530.512781] pending: neigh_periodic_work [ 8530.516133] workqueue mm_percpu_wq: flags=0x8 [ 8530.518957] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8530.521824] pending: vmstat_update, lru_add_drain_per_cpu [ 8530.524086] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8530.528838] pending: vmstat_update [ 8530.533447] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8530.541039] pending: lru_add_drain_per_cpu [ 8530.542901] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8530.545185] pending: lru_add_drain_per_cpu BAR(48) [ 8530.546797] workqueue writeback: flags=0x4a [ 8530.548519] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 8530.551719] pending: wb_workfn [ 8530.553197] workqueue xprtiod: flags=0xa [ 8530.555174] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8530.558606] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8530.563061] workqueue ipv6_addrconf: flags=0x40008 [ 8530.566465] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8530.570623] pending: addrconf_verify_work [ 8530.573216] workqueue knbd0-recv: flags=0x1a [ 8530.575930] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8530.579017] in-flight: 658:recv_work [ 8530.582054] workqueue ptlrpc_pinger: flags=0x2 [ 8530.585717] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8530.588023] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8530.595192] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6600s workers=4 idle: 32984 37 8 [ 8530.602450] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8515s workers=2 idle: 661 [ 8548.055060] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 8548.057434] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8548.058639] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 8548.074850] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8548.074864] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8548.074867] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8548.074880] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8548.074885] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8548.074890] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 8548.074892] RDX: 0000000000004655 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 8548.074894] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 8548.074896] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 8548.074899] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8548.074901] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 8548.074903] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8548.074906] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8548.074913] Call Trace: [ 8548.074963] [ 8548.074972] ? show_regs.cold.9+0x22/0x2f [ 8548.080113] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8548.080190] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8548.080198] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8548.080200] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8548.080215] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8548.080219] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8548.080223] RAX: 00000000000068af RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8548.080226] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 8548.080228] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8548.080230] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 8548.080233] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 8548.080236] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 8548.080239] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8548.080242] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 8548.080249] Call Trace: [ 8548.080255] [ 8548.080272] ? show_regs.cold.9+0x22/0x2f [ 8548.080282] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8548.080288] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8548.080293] ? watchdog+0x40/0x40 [ 8548.080297] ? hrtimer_interrupt+0x128/0x2c0 [ 8548.080301] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8548.080310] ? apic_timer_interrupt+0xf/0x20 [ 8548.080314] [ 8548.080318] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8548.080412] do_raw_spin_lock+0x177/0x190 [ 8548.080417] _raw_spin_lock+0x21/0x30 [ 8548.080421] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8548.080708] ? do_raw_spin_unlock+0x75/0x190 [ 8548.080777] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8548.086026] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8548.086047] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8548.086057] ? watchdog+0x40/0x40 [ 8548.086062] ? hrtimer_interrupt+0x128/0x2c0 [ 8548.086066] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8548.086073] ? apic_timer_interrupt+0xf/0x20 [ 8548.086076] [ 8548.086081] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8548.086085] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8548.086089] do_raw_spin_lock+0x177/0x190 [ 8548.086093] _raw_spin_lock+0x21/0x30 [ 8548.086099] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8548.086432] obd_mod_free+0x46/0x140 [ptlrpc] [ 8548.086685] mdc_commit_open+0x91/0xb0 [mdc] [ 8548.086728] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8548.086934] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8548.087207] after_reply+0xe2c/0x1780 [ptlrpc] [ 8548.096530] ? do_wait_intr+0xf0/0xf0 [ 8548.096553] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8548.100382] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8548.100693] ? __next_timer_interrupt+0x160/0x160 [ 8548.100704] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8548.100969] ? do_wait_intr+0xf0/0xf0 [ 8548.100990] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8548.120558] mdc_close+0x53b/0xd70 [mdc] [ 8548.123598] kthread+0x1d1/0x200 [ 8548.125769] lmv_close+0x18d/0x490 [lmv] [ 8548.125807] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8548.125939] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8548.126051] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8548.126112] ll_md_close+0x274/0x800 [lustre] [ 8548.126215] ll_file_release+0x128/0xc50 [lustre] [ 8548.126286] __fput+0x10d/0x440 [ 8548.126295] ____fput+0x12/0x20 [ 8548.126300] task_work_run+0xc6/0x110 [ 8548.126308] exit_to_usermode_loop+0x1dd/0x1f0 [ 8548.126317] do_syscall_64+0x3d6/0x440 [ 8548.126320] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8548.126327] RIP: 0033:0x7f151bee3cd8 [ 8548.126332] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8548.126335] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8548.126339] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 8548.126341] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 8548.126342] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8548.126344] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8548.126345] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8548.512404] ? set_kthread_struct+0x70/0x70 [ 8548.523484] ret_from_fork+0x1f/0x30 [ 8552.056063] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8552.059206] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8552.101460] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8552.109168] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8552.114774] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8552.123311] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8552.132551] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8552.160725] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8552.166839] RAX: 0000000000001f55 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8552.171796] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 8552.175794] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 8552.178476] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 8552.184520] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 8552.189994] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 8552.194062] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8552.198417] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8552.200685] Call Trace: [ 8552.201521] [ 8552.202253] ? show_regs.cold.9+0x22/0x2f [ 8552.203499] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8552.205420] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8552.209491] ? watchdog+0x40/0x40 [ 8552.210986] ? hrtimer_interrupt+0x128/0x2c0 [ 8552.212256] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8552.219130] ? apic_timer_interrupt+0xf/0x20 [ 8552.224198] [ 8552.224741] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8552.226620] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8552.228282] do_raw_spin_lock+0x177/0x190 [ 8552.229264] _raw_spin_lock+0x21/0x30 [ 8552.230129] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8552.233015] process_one_work+0x2c8/0x700 [ 8552.234957] worker_thread+0x296/0x6e0 [ 8552.236938] ? rescuer_thread+0x570/0x570 [ 8552.238724] kthread+0x1d1/0x200 [ 8552.239711] ? set_kthread_struct+0x70/0x70 [ 8552.241559] ret_from_fork+0x1f/0x30 [ 8556.052009] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 8556.056081] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8556.089815] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8556.098206] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8556.102752] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8556.104900] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8556.122005] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8556.123905] RAX: 0000000000007910 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8556.128852] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 8556.133298] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 8556.136739] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 8556.139645] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 8556.147716] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 8556.153612] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8556.156803] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 8556.160124] Call Trace: [ 8556.161169] [ 8556.161987] ? show_regs.cold.9+0x22/0x2f [ 8556.163693] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8556.165537] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8556.167458] ? watchdog+0x40/0x40 [ 8556.168735] ? hrtimer_interrupt+0x128/0x2c0 [ 8556.170465] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8556.173428] ? apic_timer_interrupt+0xf/0x20 [ 8556.176414] [ 8556.177373] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8556.181001] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8556.187316] do_raw_spin_lock+0x177/0x190 [ 8556.188863] _raw_spin_lock+0x21/0x30 [ 8556.191947] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8556.196480] ? do_raw_spin_unlock+0x75/0x190 [ 8556.199831] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8556.208614] ? do_wait_intr+0xf0/0xf0 [ 8556.215896] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8556.220855] mdc_reint+0x2d/0x110 [mdc] [ 8556.223519] mdc_unlink+0x658/0x870 [mdc] [ 8556.229846] lmv_unlink+0x34d/0xbd0 [lmv] [ 8556.232640] ll_unlink+0x24d/0x7f0 [lustre] [ 8556.238301] vfs_unlink+0x202/0x330 [ 8556.239780] do_unlinkat+0x2fa/0x460 [ 8556.248832] __x64_sys_unlink+0x37/0x50 [ 8556.250201] do_syscall_64+0xc1/0x440 [ 8556.251886] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8556.254828] RIP: 0033:0x7f151bdee10b [ 8556.257068] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8556.267645] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8556.271376] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 8556.283641] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 8556.287662] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8556.291199] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8556.297624] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8560.608063] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6630s! [ 8560.613902] Showing busy workqueues and worker pools: [ 8560.616338] workqueue events: flags=0x0 [ 8560.618722] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8560.622183] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 8560.627919] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8560.631320] pending: kfree_rcu_monitor [ 8560.633952] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8560.637537] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 8560.642226] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8560.645600] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 8560.649274] workqueue events_long: flags=0x0 [ 8560.651214] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8560.654756] pending: lu_objects_flush [obdclass] [ 8560.657337] workqueue events_unbound: flags=0x2 [ 8560.659698] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8560.662074] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8560.668356] workqueue events_power_efficient: flags=0x80 [ 8560.670893] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8560.674530] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 8560.677085] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8560.680470] pending: neigh_periodic_work [ 8560.682780] workqueue mm_percpu_wq: flags=0x8 [ 8560.685048] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8560.688730] pending: vmstat_update, lru_add_drain_per_cpu [ 8560.691635] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8560.695073] pending: vmstat_update [ 8560.696984] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8560.700480] pending: lru_add_drain_per_cpu [ 8560.702982] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8560.707022] pending: lru_add_drain_per_cpu BAR(48) [ 8560.709508] workqueue writeback: flags=0x4a [ 8560.711712] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 8560.715054] pending: wb_workfn [ 8560.716795] workqueue xprtiod: flags=0xa [ 8560.718613] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8560.721022] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8560.724133] workqueue ipv6_addrconf: flags=0x40008 [ 8560.726083] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8560.728780] pending: addrconf_verify_work [ 8560.730408] workqueue knbd0-recv: flags=0x1a [ 8560.732469] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8560.735974] in-flight: 658:recv_work [ 8560.738454] workqueue ptlrpc_pinger: flags=0x2 [ 8560.740777] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8560.743698] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8560.747984] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6630s workers=4 idle: 32984 37 8 [ 8560.752258] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8545s workers=2 idle: 661 [ 8576.056843] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 8576.059154] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 8576.063682] Modules linked in: [ 8576.073942] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8576.074037] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8576.074044] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8576.074047] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8576.074060] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8576.074064] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8576.074068] RAX: 000000000000150d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8576.074071] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 8576.074073] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8576.074075] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 8576.074077] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 8576.074080] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 8576.074082] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8576.074084] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 8576.074094] Call Trace: [ 8576.074099] [ 8576.074106] ? show_regs.cold.9+0x22/0x2f [ 8576.074116] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8576.074122] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8576.074127] ? watchdog+0x40/0x40 [ 8576.074132] ? hrtimer_interrupt+0x128/0x2c0 [ 8576.074135] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8576.074143] ? apic_timer_interrupt+0xf/0x20 [ 8576.074147] [ 8576.074150] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8576.074157] do_raw_spin_lock+0x177/0x190 [ 8576.074161] _raw_spin_lock+0x21/0x30 [ 8576.074166] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8576.083284] lustre(O) [ 8576.124559] ? do_raw_spin_unlock+0x75/0x190 [ 8576.124582] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8576.124838] ? do_wait_intr+0xf0/0xf0 [ 8576.124847] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8576.125162] mdc_close+0x53b/0xd70 [mdc] [ 8576.125202] lmv_close+0x18d/0x490 [lmv] [ 8576.125230] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8576.125392] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8576.132220] osp(O) [ 8576.136589] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8576.137008] ll_md_close+0x274/0x800 [lustre] [ 8576.137096] ll_file_release+0x128/0xc50 [lustre] [ 8576.137188] __fput+0x10d/0x440 [ 8576.137198] ____fput+0x12/0x20 [ 8576.137203] task_work_run+0xc6/0x110 [ 8576.137212] exit_to_usermode_loop+0x1dd/0x1f0 [ 8576.137220] do_syscall_64+0x3d6/0x440 [ 8576.137225] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8576.137234] RIP: 0033:0x7f151bee3cd8 [ 8576.137243] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8576.137247] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8576.137253] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 8576.137255] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 8576.137257] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8576.137259] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8576.137262] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8576.392552] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8576.446474] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8576.453408] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8576.464583] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8576.471703] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8576.489931] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8576.495497] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 8576.502309] RDX: 0000000000007a20 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 8576.506340] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 8576.509564] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 8576.513548] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8576.517594] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 8576.521691] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8576.525157] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8576.528646] Call Trace: [ 8576.530113] [ 8576.531036] ? show_regs.cold.9+0x22/0x2f [ 8576.533046] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8576.535080] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8576.537818] ? watchdog+0x40/0x40 [ 8576.540977] ? hrtimer_interrupt+0x128/0x2c0 [ 8576.546647] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8576.557527] ? apic_timer_interrupt+0xf/0x20 [ 8576.565190] [ 8576.565970] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8576.582180] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8576.588399] do_raw_spin_lock+0x177/0x190 [ 8576.596479] _raw_spin_lock+0x21/0x30 [ 8576.601932] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8576.612435] obd_mod_free+0x46/0x140 [ptlrpc] [ 8576.619499] mdc_commit_open+0x91/0xb0 [mdc] [ 8576.624298] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8576.634498] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8576.638703] after_reply+0xe2c/0x1780 [ptlrpc] [ 8576.649140] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8576.657082] ? __next_timer_interrupt+0x160/0x160 [ 8576.662277] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8576.670802] ? do_wait_intr+0xf0/0xf0 [ 8576.673727] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8576.679465] kthread+0x1d1/0x200 [ 8576.681810] ? set_kthread_struct+0x70/0x70 [ 8576.683350] ret_from_fork+0x1f/0x30 [ 8580.056407] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8580.064570] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8580.098231] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8580.104714] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8580.111940] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8580.116306] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8580.121523] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8580.136061] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8580.143172] RAX: 0000000000004a1c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8580.150719] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 8580.159601] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 8580.164657] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 8580.169492] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 8580.175660] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 8580.182405] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8580.185945] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8580.189348] Call Trace: [ 8580.190412] [ 8580.191296] ? show_regs.cold.9+0x22/0x2f [ 8580.193079] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8580.195107] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8580.197010] ? watchdog+0x40/0x40 [ 8580.198314] ? hrtimer_interrupt+0x128/0x2c0 [ 8580.200050] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8580.202598] ? apic_timer_interrupt+0xf/0x20 [ 8580.204497] [ 8580.205403] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8580.207544] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8580.210936] do_raw_spin_lock+0x177/0x190 [ 8580.213106] _raw_spin_lock+0x21/0x30 [ 8580.215196] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8580.221825] process_one_work+0x2c8/0x700 [ 8580.223141] worker_thread+0x296/0x6e0 [ 8580.224501] ? rescuer_thread+0x570/0x570 [ 8580.226718] kthread+0x1d1/0x200 [ 8580.229456] ? set_kthread_struct+0x70/0x70 [ 8580.232490] ret_from_fork+0x1f/0x30 [ 8588.052228] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 8588.063108] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8588.145022] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8588.158618] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8588.166337] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8588.171407] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8588.201792] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8588.206567] RAX: 0000000000001110 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8588.213942] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 8588.221092] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 8588.225538] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 8588.230181] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 8588.236036] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 8588.241951] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8588.248853] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 8588.251347] Call Trace: [ 8588.259095] [ 8588.260100] ? show_regs.cold.9+0x22/0x2f [ 8588.261720] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8588.268019] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8588.272801] ? watchdog+0x40/0x40 [ 8588.275952] ? hrtimer_interrupt+0x128/0x2c0 [ 8588.280044] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8588.285228] ? apic_timer_interrupt+0xf/0x20 [ 8588.289995] [ 8588.291149] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8588.297179] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8588.303087] do_raw_spin_lock+0x177/0x190 [ 8588.307259] _raw_spin_lock+0x21/0x30 [ 8588.311215] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8588.315180] ? do_raw_spin_unlock+0x75/0x190 [ 8588.317160] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8588.320422] ? do_wait_intr+0xf0/0xf0 [ 8588.324252] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8588.326694] mdc_reint+0x2d/0x110 [mdc] [ 8588.329175] mdc_unlink+0x658/0x870 [mdc] [ 8588.331915] lmv_unlink+0x34d/0xbd0 [lmv] [ 8588.334517] ll_unlink+0x24d/0x7f0 [lustre] [ 8588.341991] vfs_unlink+0x202/0x330 [ 8588.345167] do_unlinkat+0x2fa/0x460 [ 8588.348234] __x64_sys_unlink+0x37/0x50 [ 8588.355336] do_syscall_64+0xc1/0x440 [ 8588.360790] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8588.362531] RIP: 0033:0x7f151bdee10b [ 8588.367181] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8588.392655] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8588.394845] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 8588.407645] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 8588.417491] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8588.438011] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8588.448736] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8590.824260] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6660s! [ 8590.837036] Showing busy workqueues and worker pools: [ 8590.842020] workqueue events: flags=0x0 [ 8590.846884] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8590.853016] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 8590.862938] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8590.868077] pending: kfree_rcu_monitor [ 8590.871195] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8590.881427] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 8590.897192] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8590.911064] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 8590.913059] workqueue events_long: flags=0x0 [ 8590.930139] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8590.943471] pending: lu_objects_flush [obdclass] [ 8590.956287] workqueue events_unbound: flags=0x2 [ 8590.958440] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8590.963840] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8590.974152] workqueue events_power_efficient: flags=0x80 [ 8590.977866] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8590.988040] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 8590.996985] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8591.002098] pending: neigh_periodic_work [ 8591.004235] workqueue mm_percpu_wq: flags=0x8 [ 8591.008124] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8591.012459] pending: vmstat_update, lru_add_drain_per_cpu [ 8591.015419] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8591.021811] pending: vmstat_update [ 8591.024191] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8591.028255] pending: lru_add_drain_per_cpu [ 8591.031208] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8591.034871] pending: lru_add_drain_per_cpu BAR(48) [ 8591.037359] workqueue writeback: flags=0x4a [ 8591.039352] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 8591.044836] pending: wb_workfn [ 8591.046333] workqueue xprtiod: flags=0xa [ 8591.048962] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8591.056426] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8591.061365] workqueue ipv6_addrconf: flags=0x40008 [ 8591.070781] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8591.082382] pending: addrconf_verify_work [ 8591.088508] workqueue knbd0-recv: flags=0x1a [ 8591.097481] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8591.104254] in-flight: 658:recv_work [ 8591.112224] workqueue ptlrpc_pinger: flags=0x2 [ 8591.120502] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8591.135326] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8591.152893] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6660s workers=4 idle: 32984 37 8 [ 8591.163426] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8575s workers=2 idle: 661 [ 8604.056013] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 8604.057019] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 8604.061423] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8604.061601] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8604.061608] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8604.061611] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8604.061637] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8604.061641] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8604.061646] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 8604.061648] RDX: 00000000000008e4 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 8604.061651] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 8604.061653] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 8604.061656] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8604.061659] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 8604.061662] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8604.061665] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8604.061672] Call Trace: [ 8604.061685] [ 8604.061691] ? show_regs.cold.9+0x22/0x2f [ 8604.061885] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8604.066380] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8604.066459] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8604.066467] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8604.066470] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8604.066483] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8604.066488] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8604.066492] RAX: 0000000000007f84 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8604.066495] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 8604.066497] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8604.066499] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 8604.066664] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 8604.066668] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 8604.066671] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8604.066674] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 8604.066682] Call Trace: [ 8604.066687] [ 8604.066694] ? show_regs.cold.9+0x22/0x2f [ 8604.066705] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8604.066712] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8604.066726] ? watchdog+0x40/0x40 [ 8604.066732] ? hrtimer_interrupt+0x128/0x2c0 [ 8604.066736] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8604.066745] ? apic_timer_interrupt+0xf/0x20 [ 8604.066750] [ 8604.066752] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8604.066760] do_raw_spin_lock+0x177/0x190 [ 8604.066765] _raw_spin_lock+0x21/0x30 [ 8604.066770] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8604.067101] ? do_raw_spin_unlock+0x75/0x190 [ 8604.067107] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8604.067310] ? do_wait_intr+0xf0/0xf0 [ 8604.067319] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8604.067750] mdc_close+0x53b/0xd70 [mdc] [ 8604.067789] lmv_close+0x18d/0x490 [lmv] [ 8604.067815] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8604.067977] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8604.068086] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8604.068172] ll_md_close+0x274/0x800 [lustre] [ 8604.068261] ll_file_release+0x128/0xc50 [lustre] [ 8604.068348] __fput+0x10d/0x440 [ 8604.068357] ____fput+0x12/0x20 [ 8604.068361] task_work_run+0xc6/0x110 [ 8604.068369] exit_to_usermode_loop+0x1dd/0x1f0 [ 8604.068376] do_syscall_64+0x3d6/0x440 [ 8604.068381] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8604.068389] RIP: 0033:0x7f151bee3cd8 [ 8604.068436] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8604.068442] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8604.068448] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 8604.068450] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 8604.068452] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8604.068455] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8604.068457] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8604.762395] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8604.766412] ? watchdog+0x40/0x40 [ 8604.768350] ? hrtimer_interrupt+0x128/0x2c0 [ 8604.771386] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8604.773911] ? apic_timer_interrupt+0xf/0x20 [ 8604.776177] [ 8604.777071] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8604.781442] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8604.783006] do_raw_spin_lock+0x177/0x190 [ 8604.784361] _raw_spin_lock+0x21/0x30 [ 8604.785531] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8604.792213] obd_mod_free+0x46/0x140 [ptlrpc] [ 8604.802528] mdc_commit_open+0x91/0xb0 [mdc] [ 8604.803991] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8604.814951] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8604.820928] after_reply+0xe2c/0x1780 [ptlrpc] [ 8604.828905] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8604.832166] ? __next_timer_interrupt+0x160/0x160 [ 8604.840676] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8604.845494] ? do_wait_intr+0xf0/0xf0 [ 8604.850005] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8604.862948] kthread+0x1d1/0x200 [ 8604.863807] ? set_kthread_struct+0x70/0x70 [ 8604.865818] ret_from_fork+0x1f/0x30 [ 8608.068334] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8608.082034] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8608.123034] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8608.129602] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8608.137729] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8608.143276] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8608.148349] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8608.165869] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 8608.173770] RAX: 0000000000007921 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8608.179832] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 8608.185346] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 8608.195940] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 8608.201260] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 8608.210270] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 8608.216918] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8608.218941] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8608.224087] Call Trace: [ 8608.225745] [ 8608.226753] ? show_regs.cold.9+0x22/0x2f [ 8608.230255] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8608.233320] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8608.236746] ? watchdog+0x40/0x40 [ 8608.238255] ? hrtimer_interrupt+0x128/0x2c0 [ 8608.242630] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8608.245150] ? apic_timer_interrupt+0xf/0x20 [ 8608.249085] [ 8608.250363] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8608.254775] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8608.258733] do_raw_spin_lock+0x177/0x190 [ 8608.260641] _raw_spin_lock+0x21/0x30 [ 8608.262035] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8608.270708] process_one_work+0x2c8/0x700 [ 8608.272209] worker_thread+0x296/0x6e0 [ 8608.273616] ? rescuer_thread+0x570/0x570 [ 8608.282137] kthread+0x1d1/0x200 [ 8608.283320] ? set_kthread_struct+0x70/0x70 [ 8608.290028] ret_from_fork+0x1f/0x30 [ 8616.054200] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 8616.068531] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8616.178811] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8616.199453] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8616.220792] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8616.232934] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8616.271224] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8616.283271] RAX: 0000000000006c0d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8616.300106] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 8616.312141] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 8616.323240] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 8616.336539] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 8616.348055] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 8616.363095] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8616.368865] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 8616.376144] Call Trace: [ 8616.377007] [ 8616.377578] ? show_regs.cold.9+0x22/0x2f [ 8616.388789] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8616.392080] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8616.397005] ? watchdog+0x40/0x40 [ 8616.401698] ? hrtimer_interrupt+0x128/0x2c0 [ 8616.405086] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8616.409689] ? apic_timer_interrupt+0xf/0x20 [ 8616.414452] [ 8616.417487] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8616.421699] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8616.427712] do_raw_spin_lock+0x177/0x190 [ 8616.429807] _raw_spin_lock+0x21/0x30 [ 8616.436332] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8616.451093] ? do_raw_spin_unlock+0x75/0x190 [ 8616.457835] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8616.465223] ? do_wait_intr+0xf0/0xf0 [ 8616.470909] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8616.486755] mdc_reint+0x2d/0x110 [mdc] [ 8616.492134] mdc_unlink+0x658/0x870 [mdc] [ 8616.502033] lmv_unlink+0x34d/0xbd0 [lmv] [ 8616.506878] ll_unlink+0x24d/0x7f0 [lustre] [ 8616.511028] vfs_unlink+0x202/0x330 [ 8616.514979] do_unlinkat+0x2fa/0x460 [ 8616.518247] __x64_sys_unlink+0x37/0x50 [ 8616.520848] do_syscall_64+0xc1/0x440 [ 8616.523565] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8616.531195] RIP: 0033:0x7f151bdee10b [ 8616.533256] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8616.553618] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8616.561268] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 8616.565613] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 8616.572324] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8616.581424] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8616.597331] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8621.537565] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6690s! [ 8621.541942] Showing busy workqueues and worker pools: [ 8621.551308] workqueue events: flags=0x0 [ 8621.556278] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8621.564180] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 8621.585077] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8621.588371] pending: kfree_rcu_monitor [ 8621.594129] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8621.600577] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 8621.608657] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8621.617619] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 8621.626590] workqueue events_long: flags=0x0 [ 8621.628622] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8621.631135] pending: lu_objects_flush [obdclass] [ 8621.639498] workqueue events_unbound: flags=0x2 [ 8621.645163] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8621.648412] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8621.656226] workqueue events_power_efficient: flags=0x80 [ 8621.658154] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8621.664504] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 8621.667736] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8621.669366] pending: neigh_periodic_work [ 8621.670455] workqueue mm_percpu_wq: flags=0x8 [ 8621.672748] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8621.674398] pending: vmstat_update, lru_add_drain_per_cpu [ 8621.676986] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8621.681596] pending: vmstat_update [ 8621.685109] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8621.691617] pending: lru_add_drain_per_cpu [ 8621.694515] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8621.700728] pending: lru_add_drain_per_cpu BAR(48) [ 8621.704468] workqueue writeback: flags=0x4a [ 8621.706639] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 8621.713513] pending: wb_workfn [ 8621.714798] workqueue xprtiod: flags=0xa [ 8621.716184] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8621.731535] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8621.739022] workqueue ipv6_addrconf: flags=0x40008 [ 8621.751111] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8621.768750] pending: addrconf_verify_work [ 8621.770948] workqueue knbd0-recv: flags=0x1a [ 8621.774167] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8621.783466] in-flight: 658:recv_work [ 8621.786915] workqueue ptlrpc_pinger: flags=0x2 [ 8621.788149] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8621.792405] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8621.801647] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6691s workers=4 idle: 32984 37 8 [ 8621.815038] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8606s workers=2 idle: 661 [ 8632.056011] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 8632.057069] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 8632.062171] Modules linked in: [ 8632.066394] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8632.066477] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8632.066485] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8632.066488] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8632.066502] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8632.066506] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8632.066511] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8632.066513] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 8632.066515] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8632.066518] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 8632.066520] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 8632.066524] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 8632.066527] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8632.066529] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 8632.066538] Call Trace: [ 8632.066543] [ 8632.066549] ? show_regs.cold.9+0x22/0x2f [ 8632.066558] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8632.066564] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8632.066569] ? watchdog+0x40/0x40 [ 8632.066573] ? hrtimer_interrupt+0x128/0x2c0 [ 8632.066576] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8632.066584] ? apic_timer_interrupt+0xf/0x20 [ 8632.066588] [ 8632.066591] ? native_safe_halt+0x2e/0x50 [ 8632.066595] kvm_wait+0xcd/0xe0 [ 8632.066603] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8632.066611] do_raw_spin_lock+0x177/0x190 [ 8632.066615] _raw_spin_lock+0x21/0x30 [ 8632.066619] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8632.069287] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8632.069363] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8632.069372] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8632.139249] ? do_raw_spin_unlock+0x75/0x190 [ 8632.139271] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8632.139789] ? do_wait_intr+0xf0/0xf0 [ 8632.139797] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8632.140210] mdc_close+0x53b/0xd70 [mdc] [ 8632.140266] lmv_close+0x18d/0x490 [lmv] [ 8632.140294] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8632.140477] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8632.140760] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8632.140836] ll_md_close+0x274/0x800 [lustre] [ 8632.140994] ll_file_release+0x128/0xc50 [lustre] [ 8632.141075] __fput+0x10d/0x440 [ 8632.141086] ____fput+0x12/0x20 [ 8632.141091] task_work_run+0xc6/0x110 [ 8632.141098] exit_to_usermode_loop+0x1dd/0x1f0 [ 8632.141106] do_syscall_64+0x3d6/0x440 [ 8632.141111] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8632.141119] RIP: 0033:0x7f151bee3cd8 [ 8632.141129] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8632.148142] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8632.148165] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8632.148169] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8632.148176] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 8632.148178] RDX: 0000000000007c39 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 8632.148180] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 8632.148182] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 8632.148185] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8632.148189] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 8632.148191] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8632.148194] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8632.148202] Call Trace: [ 8632.148208] [ 8632.148213] ? show_regs.cold.9+0x22/0x2f [ 8632.148223] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8632.148229] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8632.148234] ? watchdog+0x40/0x40 [ 8632.148238] ? hrtimer_interrupt+0x128/0x2c0 [ 8632.148242] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8632.148251] ? apic_timer_interrupt+0xf/0x20 [ 8632.148255] [ 8632.148259] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8632.148264] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8632.148269] do_raw_spin_lock+0x177/0x190 [ 8632.148274] _raw_spin_lock+0x21/0x30 [ 8632.148281] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8632.149946] obd_mod_free+0x46/0x140 [ptlrpc] [ 8632.150148] mdc_commit_open+0x91/0xb0 [mdc] [ 8632.150187] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8632.165499] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8632.166153] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 8632.166156] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 8632.166159] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8632.166161] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8632.166164] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8632.642410] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8632.650557] after_reply+0xe2c/0x1780 [ptlrpc] [ 8632.662802] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8632.668824] ? __next_timer_interrupt+0x160/0x160 [ 8632.672585] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8632.677312] ? do_wait_intr+0xf0/0xf0 [ 8632.685430] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8632.690791] kthread+0x1d1/0x200 [ 8632.696514] ? set_kthread_struct+0x70/0x70 [ 8632.709517] ret_from_fork+0x1f/0x30 [ 8636.056112] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8636.059104] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8636.078427] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8636.083027] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8636.087072] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8636.089907] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8636.092652] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8636.102596] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8636.106360] RAX: 0000000000007a0a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8636.109672] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 8636.113231] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 8636.117556] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 8636.125396] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 8636.130531] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 8636.135155] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8636.141351] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8636.151550] Call Trace: [ 8636.156200] [ 8636.157643] ? show_regs.cold.9+0x22/0x2f [ 8636.160250] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8636.164115] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8636.173106] ? watchdog+0x40/0x40 [ 8636.175985] ? hrtimer_interrupt+0x128/0x2c0 [ 8636.177506] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8636.181466] ? apic_timer_interrupt+0xf/0x20 [ 8636.185730] [ 8636.186348] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8636.189572] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8636.194992] do_raw_spin_lock+0x177/0x190 [ 8636.196687] _raw_spin_lock+0x21/0x30 [ 8636.199951] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8636.202075] process_one_work+0x2c8/0x700 [ 8636.203748] worker_thread+0x296/0x6e0 [ 8636.204979] ? rescuer_thread+0x570/0x570 [ 8636.207245] kthread+0x1d1/0x200 [ 8636.208290] ? set_kthread_struct+0x70/0x70 [ 8636.210138] ret_from_fork+0x1f/0x30 [ 8644.789026] rcu: INFO: rcu_sched self-detected stall on CPU [ 8644.791498] rcu: 1-...!: (3812779 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 8644.795819] (t=6720142 jiffies g=249657 q=4310) [ 8644.798056] rcu: rcu_sched kthread starved for 6715094 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 8644.803428] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8644.807696] rcu: RCU grace-period kthread stack dump: [ 8644.810120] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8644.816071] Call Trace: [ 8644.817083] __schedule+0x351/0xcb0 [ 8644.818889] schedule+0xc0/0x180 [ 8644.819946] schedule_timeout+0xb4/0x190 [ 8644.821672] ? do_raw_spin_unlock+0x75/0x190 [ 8644.823247] ? __next_timer_interrupt+0x160/0x160 [ 8644.825291] ? prepare_to_swait_event+0xf3/0x140 [ 8644.827481] rcu_gp_kthread+0x8dd/0x15b0 [ 8644.829531] ? force_qs_rnp+0x2c0/0x2c0 [ 8644.831387] kthread+0x1d1/0x200 [ 8644.832919] ? set_kthread_struct+0x70/0x70 [ 8644.834799] ret_from_fork+0x1f/0x30 [ 8644.836721] rcu: Stack dump where RCU GP kthread last ran: [ 8644.839474] Sending NMI from CPU 1 to CPUs 0: [ 8644.841595] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [ 8644.842551] NMI backtrace for cpu 1 [ 8644.854155] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8644.859352] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8644.863530] Call Trace: [ 8644.864647] [ 8644.865799] ? dump_stack+0xbb/0x10e [ 8644.867504] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8644.869906] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8644.872448] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8644.874809] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8644.877397] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8644.879701] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 8644.881655] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 8644.884433] ? update_process_times+0x6d/0xc0 [ 8644.886470] ? tick_sched_handle.isra.18+0x27/0x90 [ 8644.888986] ? tick_sched_timer+0x53/0xb0 [ 8644.890999] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8644.893403] ? tick_sched_handle.isra.18+0x90/0x90 [ 8644.895905] ? hrtimer_interrupt+0x128/0x2c0 [ 8644.897802] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8644.900260] ? apic_timer_interrupt+0xf/0x20 [ 8644.902215] [ 8644.903105] ? native_safe_halt+0x2e/0x50 [ 8644.905084] ? kvm_wait+0xcd/0xe0 [ 8644.906750] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8644.910373] ? do_raw_spin_lock+0x177/0x190 [ 8644.912344] ? _raw_spin_lock+0x21/0x30 [ 8644.914094] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8644.916793] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 8644.919787] ? mdc_commit_open+0x91/0xb0 [mdc] [ 8644.922515] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8644.925341] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8644.928440] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 8644.930659] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8644.933741] ? __next_timer_interrupt+0x160/0x160 [ 8644.935727] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8644.937915] ? do_wait_intr+0xf0/0xf0 [ 8644.939921] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8644.942520] ? kthread+0x1d1/0x200 [ 8644.944257] ? set_kthread_struct+0x70/0x70 [ 8644.945888] ? ret_from_fork+0x1f/0x30 [ 8652.262093] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6721s! [ 8652.271864] Showing busy workqueues and worker pools: [ 8652.274486] workqueue events: flags=0x0 [ 8652.280926] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8652.284236] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 8652.291814] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8652.293584] pending: kfree_rcu_monitor [ 8652.305593] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8652.310920] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 8652.317850] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8652.323229] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 8652.333241] workqueue events_long: flags=0x0 [ 8652.337604] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8652.340265] pending: lu_objects_flush [obdclass] [ 8652.343040] workqueue events_unbound: flags=0x2 [ 8652.346835] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8652.351519] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8652.356176] workqueue events_power_efficient: flags=0x80 [ 8652.357232] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8652.359881] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 8652.365930] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8652.367797] pending: neigh_periodic_work [ 8652.369258] workqueue mm_percpu_wq: flags=0x8 [ 8652.370821] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8652.375852] pending: vmstat_update, lru_add_drain_per_cpu [ 8652.379429] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8652.387350] pending: vmstat_update [ 8652.389548] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8652.393088] pending: lru_add_drain_per_cpu [ 8652.398031] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8652.403620] pending: lru_add_drain_per_cpu BAR(48) [ 8652.406934] workqueue writeback: flags=0x4a [ 8652.409725] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 8652.412742] pending: wb_workfn [ 8652.415893] workqueue xprtiod: flags=0xa [ 8652.418689] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8652.423735] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8652.431332] workqueue ipv6_addrconf: flags=0x40008 [ 8652.434231] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8652.439706] pending: addrconf_verify_work [ 8652.441900] workqueue knbd0-recv: flags=0x1a [ 8652.449166] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8652.452580] in-flight: 658:recv_work [ 8652.455907] workqueue ptlrpc_pinger: flags=0x2 [ 8652.462562] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8652.466084] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8652.469805] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6721s workers=4 idle: 32984 37 8 [ 8652.475815] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8637s workers=2 idle: 661 [ 8660.057191] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 8660.061263] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8660.085706] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8660.094819] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8660.099032] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8660.101034] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8660.110534] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8660.114711] RAX: 0000000000007560 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8660.119796] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 8660.123210] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8660.127474] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 8660.130927] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 8660.134535] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 8660.139246] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8660.142315] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 8660.146569] Call Trace: [ 8660.148215] [ 8660.149126] ? show_regs.cold.9+0x22/0x2f [ 8660.150970] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8660.153106] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8660.155290] ? watchdog+0x40/0x40 [ 8660.157025] ? hrtimer_interrupt+0x128/0x2c0 [ 8660.161028] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8660.166127] ? apic_timer_interrupt+0xf/0x20 [ 8660.173746] [ 8660.175890] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8660.186610] do_raw_spin_lock+0x177/0x190 [ 8660.192153] _raw_spin_lock+0x21/0x30 [ 8660.199637] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8660.202825] ? do_raw_spin_unlock+0x75/0x190 [ 8660.207748] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8660.211655] ? do_wait_intr+0xf0/0xf0 [ 8660.215864] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8660.218989] mdc_close+0x53b/0xd70 [mdc] [ 8660.224688] lmv_close+0x18d/0x490 [lmv] [ 8660.226658] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8660.232451] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8660.239285] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8660.242342] ll_md_close+0x274/0x800 [lustre] [ 8660.250832] ll_file_release+0x128/0xc50 [lustre] [ 8660.261480] __fput+0x10d/0x440 [ 8660.266122] ____fput+0x12/0x20 [ 8660.274605] task_work_run+0xc6/0x110 [ 8660.280790] exit_to_usermode_loop+0x1dd/0x1f0 [ 8660.285302] do_syscall_64+0x3d6/0x440 [ 8660.297409] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8660.306701] RIP: 0033:0x7f151bee3cd8 [ 8660.313366] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8660.340088] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8660.352829] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 8660.360425] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 8660.366988] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8660.375781] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8660.384711] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8664.056009] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 8664.063761] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8664.106822] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8664.114966] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8664.119774] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8664.126916] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 8664.129063] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 8664.143811] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8664.147514] RAX: 00000000000027ef RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8664.154011] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 8664.164751] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 8664.167388] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 8664.177238] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 8664.183354] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 8664.189399] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8664.207014] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8664.214888] Call Trace: [ 8664.216247] [ 8664.217144] ? show_regs.cold.9+0x22/0x2f [ 8664.223074] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8664.225392] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8664.229096] ? watchdog+0x40/0x40 [ 8664.230411] ? hrtimer_interrupt+0x128/0x2c0 [ 8664.240450] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8664.243413] ? apic_timer_interrupt+0xf/0x20 [ 8664.247591] [ 8664.248764] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 8664.254121] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8664.256934] do_raw_spin_lock+0x177/0x190 [ 8664.259157] _raw_spin_lock+0x21/0x30 [ 8664.260633] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8664.264799] process_one_work+0x2c8/0x700 [ 8664.266941] worker_thread+0x296/0x6e0 [ 8664.268523] ? rescuer_thread+0x570/0x570 [ 8664.270930] kthread+0x1d1/0x200 [ 8664.272428] ? set_kthread_struct+0x70/0x70 [ 8664.278338] ret_from_fork+0x1f/0x30 [ 8672.055024] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 8672.058005] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8672.086280] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8672.090883] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8672.093961] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8672.097073] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8672.106367] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8672.108605] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 8672.110445] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 8672.112863] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 8672.115555] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 8672.117678] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8672.120251] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 8672.123319] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8672.125529] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8672.128284] Call Trace: [ 8672.129296] [ 8672.130085] ? show_regs.cold.9+0x22/0x2f [ 8672.131362] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8672.132983] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8672.135305] ? watchdog+0x40/0x40 [ 8672.136802] ? hrtimer_interrupt+0x128/0x2c0 [ 8672.138385] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8672.140266] ? apic_timer_interrupt+0xf/0x20 [ 8672.142369] [ 8672.143329] ? native_safe_halt+0x2e/0x50 [ 8672.145477] kvm_wait+0xcd/0xe0 [ 8672.148300] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8672.152280] do_raw_spin_lock+0x177/0x190 [ 8672.154585] _raw_spin_lock+0x21/0x30 [ 8672.156466] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8672.159838] obd_mod_free+0x46/0x140 [ptlrpc] [ 8672.163451] mdc_commit_open+0x91/0xb0 [mdc] [ 8672.164810] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8672.168589] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8672.174315] after_reply+0xe2c/0x1780 [ptlrpc] [ 8672.180104] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8672.187014] ? __next_timer_interrupt+0x160/0x160 [ 8672.190112] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8672.194392] ? do_wait_intr+0xf0/0xf0 [ 8672.199430] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8672.208207] kthread+0x1d1/0x200 [ 8672.213067] ? set_kthread_struct+0x70/0x70 [ 8672.215375] ret_from_fork+0x1f/0x30 [ 8680.053291] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 8680.056371] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8680.081241] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8680.084873] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8680.094669] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8680.099247] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8680.121814] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 8680.128165] RAX: 0000000000006586 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8680.133206] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 8680.136720] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 8680.141094] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 8680.144593] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 8680.149651] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 8680.153614] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8680.156383] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 8680.158841] Call Trace: [ 8680.159689] [ 8680.160811] ? show_regs.cold.9+0x22/0x2f [ 8680.165182] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8680.167177] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8680.168991] ? watchdog+0x40/0x40 [ 8680.170277] ? hrtimer_interrupt+0x128/0x2c0 [ 8680.172073] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8680.174108] ? apic_timer_interrupt+0xf/0x20 [ 8680.175916] [ 8680.176760] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8680.178977] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8680.181408] do_raw_spin_lock+0x177/0x190 [ 8680.183022] _raw_spin_lock+0x21/0x30 [ 8680.184051] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8680.186073] ? do_raw_spin_unlock+0x75/0x190 [ 8680.187920] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8680.189774] ? do_wait_intr+0xf0/0xf0 [ 8680.190770] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8680.192568] mdc_reint+0x2d/0x110 [mdc] [ 8680.194213] mdc_unlink+0x658/0x870 [mdc] [ 8680.195625] lmv_unlink+0x34d/0xbd0 [lmv] [ 8680.197023] ll_unlink+0x24d/0x7f0 [lustre] [ 8680.198386] vfs_unlink+0x202/0x330 [ 8680.199383] do_unlinkat+0x2fa/0x460 [ 8680.200479] __x64_sys_unlink+0x37/0x50 [ 8680.201597] do_syscall_64+0xc1/0x440 [ 8680.203104] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8680.205186] RIP: 0033:0x7f151bdee10b [ 8680.207387] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8680.217107] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8680.220358] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 8680.223287] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 8680.225822] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8680.228622] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8680.231353] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8682.980993] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6752s! [ 8683.000097] Showing busy workqueues and worker pools: [ 8683.005694] workqueue events: flags=0x0 [ 8683.010171] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8683.014323] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 8683.021490] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8683.029659] pending: kfree_rcu_monitor [ 8683.034241] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8683.043212] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 8683.050377] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8683.056769] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 8683.062959] workqueue events_long: flags=0x0 [ 8683.067345] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8683.073133] pending: lu_objects_flush [obdclass] [ 8683.075668] workqueue events_unbound: flags=0x2 [ 8683.076994] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8683.078675] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8683.092552] workqueue events_power_efficient: flags=0x80 [ 8683.096633] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8683.103332] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 8683.110115] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8683.115190] pending: neigh_periodic_work [ 8683.119084] workqueue mm_percpu_wq: flags=0x8 [ 8683.120440] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8683.128267] pending: vmstat_update, lru_add_drain_per_cpu [ 8683.131637] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8683.137339] pending: vmstat_update [ 8683.140330] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8683.143476] pending: lru_add_drain_per_cpu [ 8683.145350] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8683.150759] pending: lru_add_drain_per_cpu BAR(48) [ 8683.155621] workqueue writeback: flags=0x4a [ 8683.160168] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 8683.164112] pending: wb_workfn [ 8683.167240] workqueue xprtiod: flags=0xa [ 8683.171371] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8683.180210] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8683.194565] workqueue ipv6_addrconf: flags=0x40008 [ 8683.202736] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8683.212728] pending: addrconf_verify_work [ 8683.217836] workqueue knbd0-recv: flags=0x1a [ 8683.224600] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8683.227801] in-flight: 658:recv_work [ 8683.231374] workqueue ptlrpc_pinger: flags=0x2 [ 8683.233334] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8683.237106] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8683.239971] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6752s workers=4 idle: 32984 37 8 [ 8683.244748] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8667s workers=2 idle: 661 [ 8688.057033] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 8688.060391] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8688.080446] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8688.085281] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8688.088936] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8688.091106] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8688.098921] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8688.102415] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8688.105631] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 8688.108676] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8688.112246] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 8688.114896] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 8688.118355] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 8688.121986] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8688.124474] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 8688.127339] Call Trace: [ 8688.128314] [ 8688.129116] ? show_regs.cold.9+0x22/0x2f [ 8688.130488] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8688.132523] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8688.134215] ? watchdog+0x40/0x40 [ 8688.135526] ? hrtimer_interrupt+0x128/0x2c0 [ 8688.137101] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8688.138986] ? apic_timer_interrupt+0xf/0x20 [ 8688.141153] [ 8688.142057] ? native_safe_halt+0x2e/0x50 [ 8688.143694] kvm_wait+0xcd/0xe0 [ 8688.145514] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8688.147732] do_raw_spin_lock+0x177/0x190 [ 8688.149757] _raw_spin_lock+0x21/0x30 [ 8688.152118] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8688.154488] ? do_raw_spin_unlock+0x75/0x190 [ 8688.156326] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8688.158661] ? do_wait_intr+0xf0/0xf0 [ 8688.160680] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8688.163047] mdc_close+0x53b/0xd70 [mdc] [ 8688.164720] lmv_close+0x18d/0x490 [lmv] [ 8688.165847] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8688.168592] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8688.170561] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8688.172576] ll_md_close+0x274/0x800 [lustre] [ 8688.174262] ll_file_release+0x128/0xc50 [lustre] [ 8688.176328] __fput+0x10d/0x440 [ 8688.177503] ____fput+0x12/0x20 [ 8688.178683] task_work_run+0xc6/0x110 [ 8688.179644] exit_to_usermode_loop+0x1dd/0x1f0 [ 8688.180636] do_syscall_64+0x3d6/0x440 [ 8688.181577] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8688.182847] RIP: 0033:0x7f151bee3cd8 [ 8688.183966] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8688.189961] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8688.192918] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 8688.194956] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 8688.198919] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8688.202268] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8688.204913] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8692.056044] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 8692.070489] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8692.116415] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8692.122931] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8692.127588] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8692.141840] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8692.153852] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8692.177334] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8692.181574] RAX: 0000000000001d34 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8692.188394] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 8692.193353] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 8692.203387] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 8692.221554] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 8692.229425] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 8692.245835] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8692.258422] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8692.269293] Call Trace: [ 8692.272350] [ 8692.273943] ? show_regs.cold.9+0x22/0x2f [ 8692.276420] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8692.280542] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8692.289362] ? watchdog+0x40/0x40 [ 8692.290574] ? hrtimer_interrupt+0x128/0x2c0 [ 8692.301424] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8692.307513] ? apic_timer_interrupt+0xf/0x20 [ 8692.314899] [ 8692.317836] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8692.328461] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8692.334586] do_raw_spin_lock+0x177/0x190 [ 8692.340615] _raw_spin_lock+0x21/0x30 [ 8692.342853] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8692.348980] process_one_work+0x2c8/0x700 [ 8692.358013] worker_thread+0x296/0x6e0 [ 8692.359371] ? rescuer_thread+0x570/0x570 [ 8692.360821] kthread+0x1d1/0x200 [ 8692.367956] ? set_kthread_struct+0x70/0x70 [ 8692.373181] ret_from_fork+0x1f/0x30 [ 8700.057442] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 8700.062927] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8700.092568] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8700.104899] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8700.113275] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8700.117443] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8700.131046] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8700.135732] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 8700.141466] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 8700.147069] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 8700.151983] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 8700.159802] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8700.166280] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 8700.173382] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8700.177984] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8700.182532] Call Trace: [ 8700.184181] [ 8700.185450] ? show_regs.cold.9+0x22/0x2f [ 8700.188463] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8700.192224] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8700.193937] ? watchdog+0x40/0x40 [ 8700.195138] ? hrtimer_interrupt+0x128/0x2c0 [ 8700.199211] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8700.202377] ? apic_timer_interrupt+0xf/0x20 [ 8700.205056] [ 8700.206760] ? native_safe_halt+0x2e/0x50 [ 8700.209423] kvm_wait+0xcd/0xe0 [ 8700.211337] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8700.214052] do_raw_spin_lock+0x177/0x190 [ 8700.217924] _raw_spin_lock+0x21/0x30 [ 8700.220690] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8700.223363] obd_mod_free+0x46/0x140 [ptlrpc] [ 8700.227130] mdc_commit_open+0x91/0xb0 [mdc] [ 8700.230627] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8700.234409] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8700.237851] after_reply+0xe2c/0x1780 [ptlrpc] [ 8700.241046] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8700.246205] ? __next_timer_interrupt+0x160/0x160 [ 8700.251163] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8700.256099] ? do_wait_intr+0xf0/0xf0 [ 8700.258211] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8700.262228] kthread+0x1d1/0x200 [ 8700.264021] ? set_kthread_struct+0x70/0x70 [ 8700.266535] ret_from_fork+0x1f/0x30 [ 8708.052040] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 8708.057326] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8708.097742] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8708.112812] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8708.121156] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8708.123862] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8708.143242] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8708.147735] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8708.160063] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 8708.168820] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 8708.176563] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 8708.186546] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 8708.196633] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 8708.203172] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8708.211423] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 8708.217043] Call Trace: [ 8708.220470] [ 8708.221553] ? show_regs.cold.9+0x22/0x2f [ 8708.225695] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8708.231440] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8708.236799] ? watchdog+0x40/0x40 [ 8708.239736] ? hrtimer_interrupt+0x128/0x2c0 [ 8708.242524] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8708.245488] ? apic_timer_interrupt+0xf/0x20 [ 8708.247437] [ 8708.248539] ? native_safe_halt+0x2e/0x50 [ 8708.255497] kvm_wait+0xcd/0xe0 [ 8708.256793] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8708.261909] do_raw_spin_lock+0x177/0x190 [ 8708.265759] _raw_spin_lock+0x21/0x30 [ 8708.274177] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8708.282414] ? do_raw_spin_unlock+0x75/0x190 [ 8708.283968] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8708.291489] ? do_wait_intr+0xf0/0xf0 [ 8708.292784] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8708.298269] mdc_reint+0x2d/0x110 [mdc] [ 8708.299734] mdc_unlink+0x658/0x870 [mdc] [ 8708.306161] lmv_unlink+0x34d/0xbd0 [lmv] [ 8708.316796] ll_unlink+0x24d/0x7f0 [lustre] [ 8708.319077] vfs_unlink+0x202/0x330 [ 8708.327032] do_unlinkat+0x2fa/0x460 [ 8708.329142] __x64_sys_unlink+0x37/0x50 [ 8708.333028] do_syscall_64+0xc1/0x440 [ 8708.335152] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8708.339411] RIP: 0033:0x7f151bdee10b [ 8708.341746] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8708.362911] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8708.373602] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 8708.381390] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 8708.386721] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8708.391399] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8708.394588] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8713.700412] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6783s! [ 8713.705707] Showing busy workqueues and worker pools: [ 8713.708885] workqueue events: flags=0x0 [ 8713.717140] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8713.722400] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 8713.728464] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8713.734923] pending: kfree_rcu_monitor [ 8713.744362] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8713.756381] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 8713.765912] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8713.775430] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 8713.780497] workqueue events_long: flags=0x0 [ 8713.782757] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8713.790090] pending: lu_objects_flush [obdclass] [ 8713.793412] workqueue events_unbound: flags=0x2 [ 8713.795359] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8713.799314] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8713.808699] workqueue events_power_efficient: flags=0x80 [ 8713.811574] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8713.815516] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 8713.818731] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8713.820906] pending: neigh_periodic_work [ 8713.822633] workqueue mm_percpu_wq: flags=0x8 [ 8713.824783] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8713.828765] pending: vmstat_update, lru_add_drain_per_cpu [ 8713.831979] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8713.836929] pending: vmstat_update [ 8713.838860] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8713.842484] pending: lru_add_drain_per_cpu [ 8713.844743] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8713.848239] pending: lru_add_drain_per_cpu BAR(48) [ 8713.850781] workqueue writeback: flags=0x4a [ 8713.853227] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 8713.856712] pending: wb_workfn [ 8713.858387] workqueue xprtiod: flags=0xa [ 8713.860264] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8713.863896] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8713.868017] workqueue ipv6_addrconf: flags=0x40008 [ 8713.870475] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8713.873916] pending: addrconf_verify_work [ 8713.876128] workqueue knbd0-recv: flags=0x1a [ 8713.878179] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8713.881473] in-flight: 658:recv_work [ 8713.883726] workqueue ptlrpc_pinger: flags=0x2 [ 8713.885901] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8713.888902] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8713.891254] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6783s workers=4 idle: 32984 37 8 [ 8713.896303] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8698s workers=2 idle: 661 [ 8716.058026] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 8716.064705] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8716.100950] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8716.107180] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8716.114403] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8716.117971] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8716.137482] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8716.144533] RAX: 0000000000006b16 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8716.147154] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 8716.154462] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8716.161810] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 8716.169968] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 8716.176200] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 8716.185727] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8716.194239] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 8716.198704] Call Trace: [ 8716.199688] [ 8716.200427] ? show_regs.cold.9+0x22/0x2f [ 8716.202643] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8716.204241] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8716.206831] ? watchdog+0x40/0x40 [ 8716.208196] ? hrtimer_interrupt+0x128/0x2c0 [ 8716.210143] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8716.211710] ? apic_timer_interrupt+0xf/0x20 [ 8716.213191] [ 8716.214598] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8716.217408] do_raw_spin_lock+0x177/0x190 [ 8716.219418] _raw_spin_lock+0x21/0x30 [ 8716.221236] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8716.224512] ? do_raw_spin_unlock+0x75/0x190 [ 8716.226735] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8716.229560] ? do_wait_intr+0xf0/0xf0 [ 8716.231318] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8716.237701] mdc_close+0x53b/0xd70 [mdc] [ 8716.239658] lmv_close+0x18d/0x490 [lmv] [ 8716.241614] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8716.244351] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8716.248053] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8716.250800] ll_md_close+0x274/0x800 [lustre] [ 8716.253068] ll_file_release+0x128/0xc50 [lustre] [ 8716.255487] __fput+0x10d/0x440 [ 8716.257134] ____fput+0x12/0x20 [ 8716.258428] task_work_run+0xc6/0x110 [ 8716.260134] exit_to_usermode_loop+0x1dd/0x1f0 [ 8716.261963] do_syscall_64+0x3d6/0x440 [ 8716.271085] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8716.273488] RIP: 0033:0x7f151bee3cd8 [ 8716.276366] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8716.292321] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8716.296605] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 8716.302565] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 8716.308601] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8716.314119] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8716.323734] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8720.057066] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 8720.062889] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8720.132900] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8720.148068] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8720.155813] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8720.167057] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8720.170884] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8720.190581] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8720.198945] RAX: 0000000000006d1d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8720.208053] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 8720.221099] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 8720.234364] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 8720.244321] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 8720.251601] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 8720.259904] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8720.263064] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8720.267270] Call Trace: [ 8720.268821] [ 8720.270029] ? show_regs.cold.9+0x22/0x2f [ 8720.273016] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8720.275365] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8720.278409] ? watchdog+0x40/0x40 [ 8720.279911] ? hrtimer_interrupt+0x128/0x2c0 [ 8720.284918] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8720.288874] ? apic_timer_interrupt+0xf/0x20 [ 8720.294125] [ 8720.296061] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8720.302543] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8720.308003] do_raw_spin_lock+0x177/0x190 [ 8720.310442] _raw_spin_lock+0x21/0x30 [ 8720.312878] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8720.318891] process_one_work+0x2c8/0x700 [ 8720.320913] worker_thread+0x296/0x6e0 [ 8720.325296] ? rescuer_thread+0x570/0x570 [ 8720.331268] kthread+0x1d1/0x200 [ 8720.334686] ? set_kthread_struct+0x70/0x70 [ 8720.338637] ret_from_fork+0x1f/0x30 [ 8728.055108] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 8728.058116] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8728.090568] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8728.095567] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8728.100363] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8728.102358] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8728.110360] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8728.112248] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 8728.118549] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 8728.128677] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 8728.137485] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 8728.149457] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8728.156324] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 8728.166139] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8728.169703] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8728.172316] Call Trace: [ 8728.182264] [ 8728.183243] ? show_regs.cold.9+0x22/0x2f [ 8728.187545] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8728.194351] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8728.195912] ? watchdog+0x40/0x40 [ 8728.198876] ? hrtimer_interrupt+0x128/0x2c0 [ 8728.203717] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8728.208725] ? apic_timer_interrupt+0xf/0x20 [ 8728.216605] [ 8728.217442] ? native_safe_halt+0x2e/0x50 [ 8728.219675] kvm_wait+0xcd/0xe0 [ 8728.221306] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8728.225269] do_raw_spin_lock+0x177/0x190 [ 8728.228471] _raw_spin_lock+0x21/0x30 [ 8728.230295] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8728.233284] obd_mod_free+0x46/0x140 [ptlrpc] [ 8728.238883] mdc_commit_open+0x91/0xb0 [mdc] [ 8728.244092] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8728.248361] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8728.257398] after_reply+0xe2c/0x1780 [ptlrpc] [ 8728.265435] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8728.270734] ? __next_timer_interrupt+0x160/0x160 [ 8728.277351] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8728.281071] ? do_wait_intr+0xf0/0xf0 [ 8728.282455] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8728.290145] kthread+0x1d1/0x200 [ 8728.296026] ? set_kthread_struct+0x70/0x70 [ 8728.302130] ret_from_fork+0x1f/0x30 [ 8740.055586] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 8740.062583] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8740.124850] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8740.131584] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8740.138636] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8740.142585] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8740.153047] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8740.156795] RAX: 0000000000007f4a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8740.161063] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 8740.164601] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 8740.167792] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 8740.171342] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 8740.174658] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 8740.178191] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8740.180809] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 8740.184431] Call Trace: [ 8740.185270] [ 8740.186599] ? show_regs.cold.9+0x22/0x2f [ 8740.192549] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8740.205486] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8740.212606] ? watchdog+0x40/0x40 [ 8740.215206] ? hrtimer_interrupt+0x128/0x2c0 [ 8740.220145] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8740.221990] ? apic_timer_interrupt+0xf/0x20 [ 8740.231661] [ 8740.234231] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8740.244825] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8740.250282] do_raw_spin_lock+0x177/0x190 [ 8740.254147] _raw_spin_lock+0x21/0x30 [ 8740.264212] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8740.281098] ? do_raw_spin_unlock+0x75/0x190 [ 8740.292585] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8740.296722] ? do_wait_intr+0xf0/0xf0 [ 8740.299060] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8740.301025] mdc_reint+0x2d/0x110 [mdc] [ 8740.306274] mdc_unlink+0x658/0x870 [mdc] [ 8740.308385] lmv_unlink+0x34d/0xbd0 [lmv] [ 8740.311060] ll_unlink+0x24d/0x7f0 [lustre] [ 8740.313386] vfs_unlink+0x202/0x330 [ 8740.315165] do_unlinkat+0x2fa/0x460 [ 8740.317326] __x64_sys_unlink+0x37/0x50 [ 8740.320553] do_syscall_64+0xc1/0x440 [ 8740.322272] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8740.325273] RIP: 0033:0x7f151bdee10b [ 8740.326864] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8740.335643] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8740.339554] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 8740.347850] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 8740.355195] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8740.359612] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8740.366200] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8743.906422] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6813s! [ 8743.913995] Showing busy workqueues and worker pools: [ 8743.922699] workqueue events: flags=0x0 [ 8743.927643] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8743.932917] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 8743.939118] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8743.944890] pending: kfree_rcu_monitor [ 8743.947047] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8743.953024] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 8743.959068] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8743.963848] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 8743.975201] workqueue events_long: flags=0x0 [ 8743.982077] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8743.985065] pending: lu_objects_flush [obdclass] [ 8743.988937] workqueue events_unbound: flags=0x2 [ 8743.999632] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8744.006707] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8744.016179] workqueue events_power_efficient: flags=0x80 [ 8744.019948] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8744.024634] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 8744.033178] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8744.039491] pending: neigh_periodic_work [ 8744.042736] workqueue mm_percpu_wq: flags=0x8 [ 8744.044790] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8744.057893] pending: vmstat_update, lru_add_drain_per_cpu [ 8744.060904] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8744.064374] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 8744.066939] pending: vmstat_update [ 8744.067009] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8744.067025] pending: lru_add_drain_per_cpu [ 8744.067035] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8744.067043] pending: lru_add_drain_per_cpu BAR(48) [ 8744.067057] workqueue writeback: flags=0x4a [ 8744.067060] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 8744.067068] pending: wb_workfn [ 8744.067086] workqueue xprtiod: flags=0xa [ 8744.067088] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8744.067095] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8744.067119] workqueue ipv6_addrconf: flags=0x40008 [ 8744.067122] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8744.067129] pending: addrconf_verify_work [ 8744.067143] workqueue knbd0-recv: flags=0x1a [ 8744.067146] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8744.067152] in-flight: 658:recv_work [ 8744.067162] workqueue ptlrpc_pinger: flags=0x2 [ 8744.067165] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8744.067169] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8744.067486] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6813s workers=4 idle: 32984 37 8 [ 8744.067499] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8728s workers=2 idle: 661 [ 8744.199905] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8744.233992] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8744.238971] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8744.243057] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8744.245970] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8744.262333] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8744.264113] RAX: 0000000000003c9d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8744.271327] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 8744.274755] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8744.277869] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 8744.281069] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 8744.284175] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 8744.287390] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8744.289648] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 8744.292739] Call Trace: [ 8744.293787] [ 8744.294735] ? show_regs.cold.9+0x22/0x2f [ 8744.296519] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8744.300125] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8744.310709] ? watchdog+0x40/0x40 [ 8744.311965] ? hrtimer_interrupt+0x128/0x2c0 [ 8744.319170] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8744.322934] ? apic_timer_interrupt+0xf/0x20 [ 8744.328671] [ 8744.330211] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8744.336979] do_raw_spin_lock+0x177/0x190 [ 8744.341399] _raw_spin_lock+0x21/0x30 [ 8744.345493] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8744.349126] ? do_raw_spin_unlock+0x75/0x190 [ 8744.352637] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8744.360752] ? do_wait_intr+0xf0/0xf0 [ 8744.364783] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8744.368106] mdc_close+0x53b/0xd70 [mdc] [ 8744.371501] lmv_close+0x18d/0x490 [lmv] [ 8744.375040] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8744.379380] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8744.385462] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8744.389148] ll_md_close+0x274/0x800 [lustre] [ 8744.392712] ll_file_release+0x128/0xc50 [lustre] [ 8744.394664] __fput+0x10d/0x440 [ 8744.403036] ____fput+0x12/0x20 [ 8744.404310] task_work_run+0xc6/0x110 [ 8744.409971] exit_to_usermode_loop+0x1dd/0x1f0 [ 8744.412637] do_syscall_64+0x3d6/0x440 [ 8744.414996] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8744.420685] RIP: 0033:0x7f151bee3cd8 [ 8744.423786] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8744.441642] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8744.452866] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 8744.463350] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 8744.469187] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8744.480919] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8744.487278] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8748.056280] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 8748.068944] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8748.130553] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8748.140004] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8748.149011] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8748.160585] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8748.164828] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8748.178841] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8748.182525] RAX: 0000000000007d12 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8748.186178] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 8748.190652] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 8748.198804] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 8748.207979] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 8748.213959] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 8748.220844] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8748.224671] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8748.227869] Call Trace: [ 8748.229504] [ 8748.230549] ? show_regs.cold.9+0x22/0x2f [ 8748.232813] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8748.234989] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8748.237526] ? watchdog+0x40/0x40 [ 8748.238758] ? hrtimer_interrupt+0x128/0x2c0 [ 8748.240922] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8748.243857] ? apic_timer_interrupt+0xf/0x20 [ 8748.246631] [ 8748.247762] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8748.256868] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8748.266802] do_raw_spin_lock+0x177/0x190 [ 8748.268365] _raw_spin_lock+0x21/0x30 [ 8748.280200] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8748.287679] process_one_work+0x2c8/0x700 [ 8748.291014] worker_thread+0x296/0x6e0 [ 8748.292524] ? rescuer_thread+0x570/0x570 [ 8748.294254] kthread+0x1d1/0x200 [ 8748.295521] ? set_kthread_struct+0x70/0x70 [ 8748.297362] ret_from_fork+0x1f/0x30 [ 8756.055051] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 8756.059412] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8756.083864] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8756.089581] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8756.094254] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8756.096560] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8756.107686] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8756.111854] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 8756.116218] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 8756.120747] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 8756.124883] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 8756.129185] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8756.132870] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 8756.139907] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8756.143831] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8756.148326] Call Trace: [ 8756.154509] [ 8756.155268] ? show_regs.cold.9+0x22/0x2f [ 8756.158752] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8756.162700] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8756.165549] ? watchdog+0x40/0x40 [ 8756.167482] ? hrtimer_interrupt+0x128/0x2c0 [ 8756.169806] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8756.172656] ? apic_timer_interrupt+0xf/0x20 [ 8756.175217] [ 8756.177395] ? native_safe_halt+0x2e/0x50 [ 8756.179741] kvm_wait+0xcd/0xe0 [ 8756.181721] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8756.184495] do_raw_spin_lock+0x177/0x190 [ 8756.187375] _raw_spin_lock+0x21/0x30 [ 8756.189548] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8756.192681] obd_mod_free+0x46/0x140 [ptlrpc] [ 8756.196272] mdc_commit_open+0x91/0xb0 [mdc] [ 8756.198711] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8756.202686] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8756.207425] after_reply+0xe2c/0x1780 [ptlrpc] [ 8756.210633] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8756.214615] ? __next_timer_interrupt+0x160/0x160 [ 8756.217474] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8756.220512] ? do_wait_intr+0xf0/0xf0 [ 8756.222650] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8756.226229] kthread+0x1d1/0x200 [ 8756.231044] ? set_kthread_struct+0x70/0x70 [ 8756.235505] ret_from_fork+0x1f/0x30 [ 8772.054293] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 8772.058403] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8772.061373] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 8772.096570] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8772.096595] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8772.096599] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8772.096612] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8772.096616] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8772.096622] RAX: 0000000000005984 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8772.096625] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 8772.096627] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 8772.096629] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 8772.096632] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 8772.096636] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 8772.096639] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8772.096642] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 8772.096651] Call Trace: [ 8772.096656] [ 8772.096662] ? show_regs.cold.9+0x22/0x2f [ 8772.096672] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8772.096678] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8772.096682] ? watchdog+0x40/0x40 [ 8772.096687] ? hrtimer_interrupt+0x128/0x2c0 [ 8772.096691] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8772.096701] ? apic_timer_interrupt+0xf/0x20 [ 8772.096706] [ 8772.096708] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8772.096712] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8772.096716] do_raw_spin_lock+0x177/0x190 [ 8772.096721] _raw_spin_lock+0x21/0x30 [ 8772.096729] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8772.097004] ? do_raw_spin_unlock+0x75/0x190 [ 8772.097011] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8772.097491] ? do_wait_intr+0xf0/0xf0 [ 8772.097499] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8772.097708] mdc_reint+0x2d/0x110 [mdc] [ 8772.097748] mdc_unlink+0x658/0x870 [mdc] [ 8772.097773] lmv_unlink+0x34d/0xbd0 [lmv] [ 8772.097807] ll_unlink+0x24d/0x7f0 [lustre] [ 8772.097929] vfs_unlink+0x202/0x330 [ 8772.097937] do_unlinkat+0x2fa/0x460 [ 8772.097943] __x64_sys_unlink+0x37/0x50 [ 8772.097947] do_syscall_64+0xc1/0x440 [ 8772.097964] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8772.097972] RIP: 0033:0x7f151bdee10b [ 8772.097981] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8772.097984] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8772.097988] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 8772.097990] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 8772.097992] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8772.097994] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8772.097996] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8772.379157] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8772.415213] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8772.426934] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8772.432593] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8772.435023] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8772.448881] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8772.455286] RAX: 0000000000003d18 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8772.458819] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 8772.460424] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8772.463325] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 8772.466616] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 8772.469791] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 8772.473466] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8772.475457] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 8772.477845] Call Trace: [ 8772.479335] [ 8772.479940] ? show_regs.cold.9+0x22/0x2f [ 8772.481308] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8772.485047] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8772.487477] ? watchdog+0x40/0x40 [ 8772.490098] ? hrtimer_interrupt+0x128/0x2c0 [ 8772.492892] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8772.495055] ? apic_timer_interrupt+0xf/0x20 [ 8772.498265] [ 8772.499427] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8772.504045] do_raw_spin_lock+0x177/0x190 [ 8772.506877] _raw_spin_lock+0x21/0x30 [ 8772.511551] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8772.515012] ? do_raw_spin_unlock+0x75/0x190 [ 8772.520629] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8772.524071] ? do_wait_intr+0xf0/0xf0 [ 8772.525937] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8772.528629] mdc_close+0x53b/0xd70 [mdc] [ 8772.530255] lmv_close+0x18d/0x490 [lmv] [ 8772.532034] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8772.534663] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8772.538499] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8772.540614] ll_md_close+0x274/0x800 [lustre] [ 8772.543222] ll_file_release+0x128/0xc50 [lustre] [ 8772.548071] __fput+0x10d/0x440 [ 8772.549648] ____fput+0x12/0x20 [ 8772.553893] task_work_run+0xc6/0x110 [ 8772.555450] exit_to_usermode_loop+0x1dd/0x1f0 [ 8772.559258] do_syscall_64+0x3d6/0x440 [ 8772.561753] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8772.564397] RIP: 0033:0x7f151bee3cd8 [ 8772.567355] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8772.577482] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8772.582781] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 8772.589158] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 8772.601903] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8772.605903] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8772.611587] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8774.114204] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6843s! [ 8774.117984] Showing busy workqueues and worker pools: [ 8774.126363] workqueue events: flags=0x0 [ 8774.129916] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8774.139762] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 8774.150578] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8774.157336] pending: kfree_rcu_monitor [ 8774.160919] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8774.164304] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 8774.173838] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8774.184678] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 8774.195711] workqueue events_long: flags=0x0 [ 8774.202309] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8774.210512] pending: lu_objects_flush [obdclass] [ 8774.219768] workqueue events_unbound: flags=0x2 [ 8774.226150] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8774.237132] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8774.253844] workqueue events_power_efficient: flags=0x80 [ 8774.264283] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8774.271343] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 8774.279019] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8774.290646] pending: neigh_periodic_work [ 8774.297178] workqueue mm_percpu_wq: flags=0x8 [ 8774.303802] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8774.313424] pending: vmstat_update, lru_add_drain_per_cpu [ 8774.320916] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8774.330799] pending: vmstat_update [ 8774.336043] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8774.342904] pending: lru_add_drain_per_cpu [ 8774.346876] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8774.353892] pending: lru_add_drain_per_cpu BAR(48) [ 8774.359927] workqueue writeback: flags=0x4a [ 8774.362829] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 8774.367665] pending: wb_workfn [ 8774.370903] workqueue xprtiod: flags=0xa [ 8774.373078] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8774.377172] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8774.381911] workqueue ipv6_addrconf: flags=0x40008 [ 8774.387673] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8774.392025] pending: addrconf_verify_work [ 8774.395053] workqueue knbd0-recv: flags=0x1a [ 8774.396891] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8774.404137] in-flight: 658:recv_work [ 8774.408679] workqueue ptlrpc_pinger: flags=0x2 [ 8774.416023] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8774.418361] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8774.431641] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6843s workers=4 idle: 32984 37 8 [ 8774.437578] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8759s workers=2 idle: 661 [ 8776.056472] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 8776.070882] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8776.112265] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8776.117648] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8776.120867] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8776.124570] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8776.129565] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8776.140041] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8776.144246] RAX: 0000000000007f87 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8776.148093] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 8776.151767] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 8776.155321] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 8776.159765] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 8776.164260] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 8776.168347] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8776.170840] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8776.174073] Call Trace: [ 8776.175208] [ 8776.176099] ? show_regs.cold.9+0x22/0x2f [ 8776.177911] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8776.181171] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8776.188602] ? watchdog+0x40/0x40 [ 8776.190473] ? hrtimer_interrupt+0x128/0x2c0 [ 8776.193850] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8776.196591] ? apic_timer_interrupt+0xf/0x20 [ 8776.200901] [ 8776.202819] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8776.205224] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8776.207702] do_raw_spin_lock+0x177/0x190 [ 8776.209683] _raw_spin_lock+0x21/0x30 [ 8776.211458] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8776.213904] process_one_work+0x2c8/0x700 [ 8776.215628] worker_thread+0x296/0x6e0 [ 8776.217044] ? rescuer_thread+0x570/0x570 [ 8776.218520] kthread+0x1d1/0x200 [ 8776.219856] ? set_kthread_struct+0x70/0x70 [ 8776.221554] ret_from_fork+0x1f/0x30 [ 8784.056618] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 8784.063663] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8784.088186] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8784.093095] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8784.097137] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8784.099806] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8784.108028] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8784.113553] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 8784.120281] RDX: 0000000000003d1b RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 8784.132381] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 8784.138435] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 8784.145298] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8784.149742] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 8784.153380] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8784.155804] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8784.158995] Call Trace: [ 8784.159781] [ 8784.160722] ? show_regs.cold.9+0x22/0x2f [ 8784.162522] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8784.164114] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8784.165525] ? watchdog+0x40/0x40 [ 8784.166711] ? hrtimer_interrupt+0x128/0x2c0 [ 8784.168173] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8784.170021] ? apic_timer_interrupt+0xf/0x20 [ 8784.171117] [ 8784.171717] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8784.173443] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8784.175091] do_raw_spin_lock+0x177/0x190 [ 8784.176297] _raw_spin_lock+0x21/0x30 [ 8784.177343] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8784.179319] obd_mod_free+0x46/0x140 [ptlrpc] [ 8784.180694] mdc_commit_open+0x91/0xb0 [mdc] [ 8784.182139] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8784.184114] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8784.185808] after_reply+0xe2c/0x1780 [ptlrpc] [ 8784.187304] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8784.189383] ? __next_timer_interrupt+0x160/0x160 [ 8784.191041] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8784.192651] ? do_wait_intr+0xf0/0xf0 [ 8784.193690] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8784.195587] kthread+0x1d1/0x200 [ 8784.196539] ? set_kthread_struct+0x70/0x70 [ 8784.197961] ret_from_fork+0x1f/0x30 [ 8800.052026] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 8800.054524] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8800.057584] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 8800.087297] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8800.087313] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8800.087316] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8800.087334] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8800.087338] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8800.087344] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8800.087346] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 8800.087348] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 8800.087350] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 8800.087422] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 8800.087428] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 8800.087431] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8800.087437] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 8800.087445] Call Trace: [ 8800.087450] [ 8800.087456] ? show_regs.cold.9+0x22/0x2f [ 8800.087466] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8800.087473] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8800.087477] ? watchdog+0x40/0x40 [ 8800.087481] ? hrtimer_interrupt+0x128/0x2c0 [ 8800.087485] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8800.087493] ? apic_timer_interrupt+0xf/0x20 [ 8800.087497] [ 8800.087499] ? native_safe_halt+0x2e/0x50 [ 8800.087504] kvm_wait+0xcd/0xe0 [ 8800.087512] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8800.087519] do_raw_spin_lock+0x177/0x190 [ 8800.087524] _raw_spin_lock+0x21/0x30 [ 8800.087531] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8800.088147] ? do_raw_spin_unlock+0x75/0x190 [ 8800.088154] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8800.088478] ? do_wait_intr+0xf0/0xf0 [ 8800.088486] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8800.088976] mdc_reint+0x2d/0x110 [mdc] [ 8800.089015] mdc_unlink+0x658/0x870 [mdc] [ 8800.089043] lmv_unlink+0x34d/0xbd0 [lmv] [ 8800.089078] ll_unlink+0x24d/0x7f0 [lustre] [ 8800.089242] vfs_unlink+0x202/0x330 [ 8800.089251] do_unlinkat+0x2fa/0x460 [ 8800.089261] __x64_sys_unlink+0x37/0x50 [ 8800.089264] do_syscall_64+0xc1/0x440 [ 8800.089271] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8800.089279] RIP: 0033:0x7f151bdee10b [ 8800.089287] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8800.089291] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8800.089297] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 8800.089299] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 8800.089302] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8800.089304] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8800.089307] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8800.328159] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8800.354017] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8800.359583] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8800.362408] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8800.368131] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8800.399253] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8800.416093] RAX: 0000000000007091 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8800.421415] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 8800.427618] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8800.433569] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 8800.440361] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 8800.450560] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 8800.460521] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8800.468037] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 8800.474430] Call Trace: [ 8800.479914] [ 8800.480749] ? show_regs.cold.9+0x22/0x2f [ 8800.482285] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8800.487337] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8800.496505] ? watchdog+0x40/0x40 [ 8800.498261] ? hrtimer_interrupt+0x128/0x2c0 [ 8800.503303] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8800.505646] ? apic_timer_interrupt+0xf/0x20 [ 8800.511026] [ 8800.513376] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8800.518930] do_raw_spin_lock+0x177/0x190 [ 8800.521379] _raw_spin_lock+0x21/0x30 [ 8800.528451] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8800.530638] ? do_raw_spin_unlock+0x75/0x190 [ 8800.534102] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8800.542427] ? do_wait_intr+0xf0/0xf0 [ 8800.547734] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8800.549800] mdc_close+0x53b/0xd70 [mdc] [ 8800.554417] lmv_close+0x18d/0x490 [lmv] [ 8800.560504] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8800.563422] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8800.573803] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8800.579062] ll_md_close+0x274/0x800 [lustre] [ 8800.583312] ll_file_release+0x128/0xc50 [lustre] [ 8800.586486] __fput+0x10d/0x440 [ 8800.587717] ____fput+0x12/0x20 [ 8800.591249] task_work_run+0xc6/0x110 [ 8800.595159] exit_to_usermode_loop+0x1dd/0x1f0 [ 8800.599155] do_syscall_64+0x3d6/0x440 [ 8800.600653] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8800.604729] RIP: 0033:0x7f151bee3cd8 [ 8800.607642] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8800.624333] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8800.630112] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 8800.635515] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 8800.641144] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8800.646645] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8800.650554] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8804.056642] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8804.062752] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8804.095834] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8804.103674] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8804.109600] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8804.115269] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8804.119440] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8804.132608] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8804.137232] RAX: 0000000000002913 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8804.141311] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 8804.144734] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 8804.150903] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 8804.156756] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 8804.163600] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 8804.169540] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8804.174523] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8804.178465] Call Trace: [ 8804.180697] [ 8804.182173] ? show_regs.cold.9+0x22/0x2f [ 8804.184571] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8804.187199] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8804.190930] ? watchdog+0x40/0x40 [ 8804.192302] ? hrtimer_interrupt+0x128/0x2c0 [ 8804.194842] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8804.196834] ? apic_timer_interrupt+0xf/0x20 [ 8804.199859] [ 8804.201279] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8804.203274] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8804.205134] do_raw_spin_lock+0x177/0x190 [ 8804.206972] _raw_spin_lock+0x21/0x30 [ 8804.208484] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8804.211227] process_one_work+0x2c8/0x700 [ 8804.212950] worker_thread+0x296/0x6e0 [ 8804.214541] ? rescuer_thread+0x570/0x570 [ 8804.216204] kthread+0x1d1/0x200 [ 8804.217595] ? set_kthread_struct+0x70/0x70 [ 8804.219168] ret_from_fork+0x1f/0x30 [ 8804.832412] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6874s! [ 8804.837579] Showing busy workqueues and worker pools: [ 8804.843117] workqueue events: flags=0x0 [ 8804.844501] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8804.848504] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 8804.855814] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8804.860622] pending: kfree_rcu_monitor [ 8804.865575] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8804.872630] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 8804.888031] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8804.895598] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 8804.899266] workqueue events_long: flags=0x0 [ 8804.908733] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8804.911188] pending: lu_objects_flush [obdclass] [ 8804.922929] workqueue events_unbound: flags=0x2 [ 8804.924465] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8804.943840] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8804.955821] workqueue events_power_efficient: flags=0x80 [ 8804.961923] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8804.965329] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 8804.970054] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8804.975542] pending: neigh_periodic_work [ 8804.980107] workqueue mm_percpu_wq: flags=0x8 [ 8804.985487] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8804.990126] pending: vmstat_update, lru_add_drain_per_cpu [ 8804.995500] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8805.000055] pending: vmstat_update [ 8805.004265] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8805.007267] pending: lru_add_drain_per_cpu [ 8805.015038] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8805.024481] pending: lru_add_drain_per_cpu BAR(48) [ 8805.029455] workqueue writeback: flags=0x4a [ 8805.032741] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 8805.037127] pending: wb_workfn [ 8805.042369] workqueue xprtiod: flags=0xa [ 8805.047182] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8805.054392] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8805.065386] workqueue ipv6_addrconf: flags=0x40008 [ 8805.066743] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8805.068493] pending: addrconf_verify_work [ 8805.072386] workqueue knbd0-recv: flags=0x1a [ 8805.078027] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8805.080448] in-flight: 658:recv_work [ 8805.086756] workqueue ptlrpc_pinger: flags=0x2 [ 8805.088602] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8805.098320] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8805.100608] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6874s workers=4 idle: 32984 37 8 [ 8805.105312] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8789s workers=2 idle: 661 [ 8812.059610] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 8812.067132] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8812.108365] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8812.122931] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8812.127963] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8812.132162] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8812.151239] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8812.156902] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 8812.164330] RDX: 00000000000015bd RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 8812.173989] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 8812.179687] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 8812.191062] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8812.194494] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 8812.202705] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8812.211879] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8812.214332] Call Trace: [ 8812.219825] [ 8812.222215] ? show_regs.cold.9+0x22/0x2f [ 8812.226878] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8812.232757] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8812.234663] ? watchdog+0x40/0x40 [ 8812.238626] ? hrtimer_interrupt+0x128/0x2c0 [ 8812.242420] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8812.248062] ? apic_timer_interrupt+0xf/0x20 [ 8812.251206] [ 8812.252237] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8812.258281] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8812.261483] do_raw_spin_lock+0x177/0x190 [ 8812.265089] _raw_spin_lock+0x21/0x30 [ 8812.270472] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8812.281269] obd_mod_free+0x46/0x140 [ptlrpc] [ 8812.285555] mdc_commit_open+0x91/0xb0 [mdc] [ 8812.287608] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8812.293448] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8812.299473] after_reply+0xe2c/0x1780 [ptlrpc] [ 8812.302208] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8812.309455] ? __next_timer_interrupt+0x160/0x160 [ 8812.313843] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8812.317710] ? do_wait_intr+0xf0/0xf0 [ 8812.319910] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8812.323193] kthread+0x1d1/0x200 [ 8812.325286] ? set_kthread_struct+0x70/0x70 [ 8812.327711] ret_from_fork+0x1f/0x30 [ 8824.792144] rcu: INFO: rcu_sched self-detected stall on CPU [ 8824.796130] rcu: 1-...!: (3919400 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 8824.801844] (t=6900145 jiffies g=249657 q=4353) [ 8824.806151] rcu: rcu_sched kthread starved for 6895097 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 8824.817376] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8824.824969] rcu: RCU grace-period kthread stack dump: [ 8824.828460] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8824.836242] Call Trace: [ 8824.839245] __schedule+0x351/0xcb0 [ 8824.840525] schedule+0xc0/0x180 [ 8824.841910] schedule_timeout+0xb4/0x190 [ 8824.843534] ? do_raw_spin_unlock+0x75/0x190 [ 8824.845175] ? __next_timer_interrupt+0x160/0x160 [ 8824.847543] ? prepare_to_swait_event+0xf3/0x140 [ 8824.849068] rcu_gp_kthread+0x8dd/0x15b0 [ 8824.850553] ? force_qs_rnp+0x2c0/0x2c0 [ 8824.852213] kthread+0x1d1/0x200 [ 8824.853594] ? set_kthread_struct+0x70/0x70 [ 8824.855171] ret_from_fork+0x1f/0x30 [ 8824.857207] rcu: Stack dump where RCU GP kthread last ran: [ 8824.864966] Sending NMI from CPU 1 to CPUs 0: [ 8824.868372] NMI backtrace for cpu 0 [ 8824.868378] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8824.868380] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8824.868382] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8824.868384] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8824.868387] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 [ 8824.868389] RAX: 000000000000649e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8824.868391] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 8824.868392] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 8824.868393] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 8824.868394] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 8824.868396] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 8824.868397] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8824.868398] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 8824.868399] Call Trace: [ 8824.868400] [ 8824.868401] ? show_regs.cold.9+0x22/0x2f [ 8824.868402] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 8824.868403] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 8824.868404] ? nmi_handle+0xbd/0x210 [ 8824.868405] ? default_do_nmi+0x4b/0x1b0 [ 8824.868406] ? do_nmi+0x280/0x380 [ 8824.868407] ? end_repeat_nmi+0x16/0x1f [ 8824.868408] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8824.868410] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8824.868411] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8824.868412] [ 8824.868413] do_raw_spin_lock+0x177/0x190 [ 8824.868414] _raw_spin_lock+0x21/0x30 [ 8824.868415] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8824.868416] ? do_raw_spin_unlock+0x75/0x190 [ 8824.868417] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8824.868418] ? do_wait_intr+0xf0/0xf0 [ 8824.868420] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8824.868421] mdc_reint+0x2d/0x110 [mdc] [ 8824.868422] mdc_unlink+0x658/0x870 [mdc] [ 8824.868423] lmv_unlink+0x34d/0xbd0 [lmv] [ 8824.868424] ll_unlink+0x24d/0x7f0 [lustre] [ 8824.868426] vfs_unlink+0x202/0x330 [ 8824.868427] do_unlinkat+0x2fa/0x460 [ 8824.868428] __x64_sys_unlink+0x37/0x50 [ 8824.868429] do_syscall_64+0xc1/0x440 [ 8824.868430] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8824.868431] RIP: 0033:0x7f151bdee10b [ 8824.868433] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8824.868435] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8824.868438] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 8824.868440] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 8824.868441] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8824.868443] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8824.868444] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8824.868830] NMI backtrace for cpu 1 [ 8825.213658] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8825.230960] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8825.239283] Call Trace: [ 8825.242628] [ 8825.244395] ? dump_stack+0xbb/0x10e [ 8825.247760] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8825.254200] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8825.258758] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8825.262468] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8825.266376] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8825.272658] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 8825.275318] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 8825.284376] ? update_process_times+0x6d/0xc0 [ 8825.287101] ? tick_sched_handle.isra.18+0x27/0x90 [ 8825.290317] ? tick_sched_timer+0x53/0xb0 [ 8825.294375] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8825.298513] ? tick_sched_handle.isra.18+0x90/0x90 [ 8825.304632] ? hrtimer_interrupt+0x128/0x2c0 [ 8825.307594] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8825.311514] ? apic_timer_interrupt+0xf/0x20 [ 8825.314256] [ 8825.316367] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8825.321188] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8825.326502] ? do_raw_spin_lock+0x177/0x190 [ 8825.329054] ? _raw_spin_lock+0x21/0x30 [ 8825.331640] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8825.335235] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 8825.338587] ? mdc_commit_open+0x91/0xb0 [mdc] [ 8825.341248] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8825.347107] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8825.351618] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 8825.353979] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8825.364294] ? __next_timer_interrupt+0x160/0x160 [ 8825.372302] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8825.377406] ? do_wait_intr+0xf0/0xf0 [ 8825.381814] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8825.383925] ? kthread+0x1d1/0x200 [ 8825.387084] ? set_kthread_struct+0x70/0x70 [ 8825.389628] ? ret_from_fork+0x1f/0x30 [ 8828.057643] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 8828.078566] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8828.123868] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8828.133945] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8828.157440] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8828.162879] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8828.182509] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8828.189705] RAX: 0000000000007d0e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8828.194812] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 8828.203942] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8828.210939] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 8828.219991] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 8828.224499] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 8828.231365] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8828.236352] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 8828.239030] Call Trace: [ 8828.240908] [ 8828.248402] ? show_regs.cold.9+0x22/0x2f [ 8828.251319] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8828.257739] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8828.260184] ? watchdog+0x40/0x40 [ 8828.262800] ? hrtimer_interrupt+0x128/0x2c0 [ 8828.266136] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8828.268866] ? apic_timer_interrupt+0xf/0x20 [ 8828.271380] [ 8828.273940] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8828.276664] do_raw_spin_lock+0x177/0x190 [ 8828.282339] _raw_spin_lock+0x21/0x30 [ 8828.287693] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8828.292312] ? do_raw_spin_unlock+0x75/0x190 [ 8828.294632] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8828.297478] ? do_wait_intr+0xf0/0xf0 [ 8828.299025] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8828.301125] mdc_close+0x53b/0xd70 [mdc] [ 8828.305798] lmv_close+0x18d/0x490 [lmv] [ 8828.310821] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8828.317776] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8828.322201] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8828.326387] ll_md_close+0x274/0x800 [lustre] [ 8828.336828] ll_file_release+0x128/0xc50 [lustre] [ 8828.339155] __fput+0x10d/0x440 [ 8828.341816] ____fput+0x12/0x20 [ 8828.343508] task_work_run+0xc6/0x110 [ 8828.347877] exit_to_usermode_loop+0x1dd/0x1f0 [ 8828.352643] do_syscall_64+0x3d6/0x440 [ 8828.356207] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8828.361421] RIP: 0033:0x7f151bee3cd8 [ 8828.363616] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8828.372217] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8828.375383] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 8828.379630] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 8828.381491] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8828.386007] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8828.390056] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8832.056822] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8832.067599] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8832.093450] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8832.099237] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8832.105053] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8832.108442] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8832.111693] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8832.121698] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8832.126235] RAX: 0000000000005b83 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8832.130712] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 8832.134610] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 8832.139115] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 8832.142554] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 8832.146005] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 8832.150964] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8832.154591] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8832.158322] Call Trace: [ 8832.159828] [ 8832.161186] ? show_regs.cold.9+0x22/0x2f [ 8832.164044] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8832.167684] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8832.170766] ? watchdog+0x40/0x40 [ 8832.173148] ? hrtimer_interrupt+0x128/0x2c0 [ 8832.176013] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8832.180239] ? apic_timer_interrupt+0xf/0x20 [ 8832.183159] [ 8832.185066] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8832.188224] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8832.190769] do_raw_spin_lock+0x177/0x190 [ 8832.192736] _raw_spin_lock+0x21/0x30 [ 8832.195113] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8832.198885] process_one_work+0x2c8/0x700 [ 8832.200624] worker_thread+0x296/0x6e0 [ 8832.202000] ? rescuer_thread+0x570/0x570 [ 8832.203876] kthread+0x1d1/0x200 [ 8832.205850] ? set_kthread_struct+0x70/0x70 [ 8832.208249] ret_from_fork+0x1f/0x30 [ 8835.554108] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6904s! [ 8835.557783] Showing busy workqueues and worker pools: [ 8835.563324] workqueue events: flags=0x0 [ 8835.564187] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8835.565537] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 8835.570758] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8835.573454] pending: kfree_rcu_monitor [ 8835.575422] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8835.577913] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 8835.580970] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8835.582997] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 8835.585505] workqueue events_long: flags=0x0 [ 8835.587057] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8835.590436] pending: lu_objects_flush [obdclass] [ 8835.593860] workqueue events_unbound: flags=0x2 [ 8835.597029] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8835.600758] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8835.604679] workqueue events_power_efficient: flags=0x80 [ 8835.607608] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8835.610054] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 8835.612611] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8835.614428] pending: neigh_periodic_work [ 8835.615644] workqueue mm_percpu_wq: flags=0x8 [ 8835.616821] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8835.618295] pending: vmstat_update, lru_add_drain_per_cpu [ 8835.620570] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8835.622640] pending: vmstat_update [ 8835.624826] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8835.628008] pending: lru_add_drain_per_cpu [ 8835.630307] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8835.633136] pending: lru_add_drain_per_cpu BAR(48) [ 8835.635872] workqueue writeback: flags=0x4a [ 8835.638405] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 8835.641050] pending: wb_workfn [ 8835.642329] workqueue xprtiod: flags=0xa [ 8835.643620] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8835.645377] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8835.647111] workqueue ipv6_addrconf: flags=0x40008 [ 8835.648204] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8835.650928] pending: addrconf_verify_work [ 8835.651994] workqueue knbd0-recv: flags=0x1a [ 8835.654296] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8835.656594] in-flight: 658:recv_work [ 8835.657787] workqueue ptlrpc_pinger: flags=0x2 [ 8835.661756] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8835.667017] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8835.675195] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6905s workers=4 idle: 32984 37 8 [ 8835.680521] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8820s workers=2 idle: 661 [ 8852.055974] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 8852.061682] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8852.115003] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8852.129270] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8852.141333] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8852.156291] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8852.182630] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8852.189191] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 8852.198644] RDX: 00000000000043ab RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 8852.213475] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 8852.222941] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 8852.234335] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8852.250814] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 8852.267040] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8852.275075] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8852.282571] Call Trace: [ 8852.285486] [ 8852.291775] ? show_regs.cold.9+0x22/0x2f [ 8852.296725] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8852.298204] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8852.305126] ? watchdog+0x40/0x40 [ 8852.316365] ? hrtimer_interrupt+0x128/0x2c0 [ 8852.319785] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8852.327903] ? apic_timer_interrupt+0xf/0x20 [ 8852.341902] [ 8852.343716] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8852.358623] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8852.362676] do_raw_spin_lock+0x177/0x190 [ 8852.372724] _raw_spin_lock+0x21/0x30 [ 8852.377617] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8852.381113] obd_mod_free+0x46/0x140 [ptlrpc] [ 8852.389531] mdc_commit_open+0x91/0xb0 [mdc] [ 8852.396140] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8852.411954] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8852.417622] after_reply+0xe2c/0x1780 [ptlrpc] [ 8852.421224] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8852.425937] ? __next_timer_interrupt+0x160/0x160 [ 8852.431549] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8852.435533] ? do_wait_intr+0xf0/0xf0 [ 8852.444945] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8852.449983] kthread+0x1d1/0x200 [ 8852.452041] ? set_kthread_struct+0x70/0x70 [ 8852.455739] ret_from_fork+0x1f/0x30 [ 8856.057115] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 8856.085839] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8856.188110] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8856.204987] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8856.221940] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8856.228837] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8856.278996] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8856.287008] RAX: 000000000000057f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8856.291978] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 8856.302723] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8856.306456] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 8856.309830] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 8856.318053] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 8856.322850] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8856.329968] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 8856.338164] Call Trace: [ 8856.339620] [ 8856.340608] ? show_regs.cold.9+0x22/0x2f [ 8856.345191] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8856.348032] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8856.350198] ? watchdog+0x40/0x40 [ 8856.351832] ? hrtimer_interrupt+0x128/0x2c0 [ 8856.354034] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8856.356748] ? apic_timer_interrupt+0xf/0x20 [ 8856.358389] [ 8856.359243] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8856.361607] do_raw_spin_lock+0x177/0x190 [ 8856.363619] _raw_spin_lock+0x21/0x30 [ 8856.365210] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8856.367590] ? do_raw_spin_unlock+0x75/0x190 [ 8856.369280] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8856.371845] ? do_wait_intr+0xf0/0xf0 [ 8856.373478] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8856.376043] mdc_close+0x53b/0xd70 [mdc] [ 8856.377557] lmv_close+0x18d/0x490 [lmv] [ 8856.378548] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8856.380101] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8856.382112] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8856.383872] ll_md_close+0x274/0x800 [lustre] [ 8856.385775] ll_file_release+0x128/0xc50 [lustre] [ 8856.387017] __fput+0x10d/0x440 [ 8856.388398] ____fput+0x12/0x20 [ 8856.389116] task_work_run+0xc6/0x110 [ 8856.392610] exit_to_usermode_loop+0x1dd/0x1f0 [ 8856.394545] do_syscall_64+0x3d6/0x440 [ 8856.402461] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8856.410732] RIP: 0033:0x7f151bee3cd8 [ 8856.418203] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8856.438703] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8856.442777] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 8856.445981] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 8856.449999] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8856.452465] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8856.455754] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8860.055847] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 8860.057631] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8860.089254] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8860.089465] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8860.089472] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8860.089475] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8860.089487] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8860.089492] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8860.089496] RAX: 0000000000002c83 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8860.089498] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 8860.089500] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 8860.089502] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 8860.089504] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 8860.089507] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 8860.089510] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8860.089512] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 8860.089520] Call Trace: [ 8860.089794] [ 8860.089802] ? show_regs.cold.9+0x22/0x2f [ 8860.089811] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8860.089818] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8860.089823] ? watchdog+0x40/0x40 [ 8860.089827] ? hrtimer_interrupt+0x128/0x2c0 [ 8860.089831] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8860.089840] ? apic_timer_interrupt+0xf/0x20 [ 8860.089844] [ 8860.089847] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8860.089852] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8860.089856] do_raw_spin_lock+0x177/0x190 [ 8860.089861] _raw_spin_lock+0x21/0x30 [ 8860.089867] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8860.098591] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8860.098675] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8860.098683] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8860.098690] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8860.098959] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8860.098971] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8860.098975] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8860.098980] RAX: 0000000000004d0f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8860.098983] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 8860.098985] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 8860.098987] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 8860.098990] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 8860.098993] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 8860.098996] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8860.098998] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8860.099007] Call Trace: [ 8860.099012] [ 8860.099017] ? show_regs.cold.9+0x22/0x2f [ 8860.099027] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8860.099034] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8860.099038] ? watchdog+0x40/0x40 [ 8860.099042] ? hrtimer_interrupt+0x128/0x2c0 [ 8860.099046] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8860.099055] ? apic_timer_interrupt+0xf/0x20 [ 8860.099060] [ 8860.099063] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8860.099068] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8860.099075] do_raw_spin_lock+0x177/0x190 [ 8860.099080] _raw_spin_lock+0x21/0x30 [ 8860.099086] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8860.177049] ? do_raw_spin_unlock+0x75/0x190 [ 8860.177071] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8860.182075] ? do_wait_intr+0xf0/0xf0 [ 8860.182089] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8860.182457] mdc_reint+0x2d/0x110 [mdc] [ 8860.182493] mdc_unlink+0x658/0x870 [mdc] [ 8860.182520] lmv_unlink+0x34d/0xbd0 [lmv] [ 8860.182552] ll_unlink+0x24d/0x7f0 [lustre] [ 8860.182678] vfs_unlink+0x202/0x330 [ 8860.182687] do_unlinkat+0x2fa/0x460 [ 8860.182692] __x64_sys_unlink+0x37/0x50 [ 8860.182696] do_syscall_64+0xc1/0x440 [ 8860.182703] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8860.182712] RIP: 0033:0x7f151bdee10b [ 8860.182719] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8860.182724] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8860.182728] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 8860.182730] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 8860.182733] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8860.182744] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8860.182747] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8860.962074] process_one_work+0x2c8/0x700 [ 8860.965318] worker_thread+0x296/0x6e0 [ 8860.969729] ? rescuer_thread+0x570/0x570 [ 8860.977950] kthread+0x1d1/0x200 [ 8860.979011] ? set_kthread_struct+0x70/0x70 [ 8860.985931] ret_from_fork+0x1f/0x30 [ 8865.771984] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6935s! [ 8865.782863] Showing busy workqueues and worker pools: [ 8865.787431] workqueue events: flags=0x0 [ 8865.788920] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8865.795490] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 8865.804951] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8865.811685] pending: kfree_rcu_monitor [ 8865.817795] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8865.822026] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 8865.835978] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8865.849405] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 8865.865493] workqueue events_long: flags=0x0 [ 8865.874874] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8865.883562] pending: lu_objects_flush [obdclass] [ 8865.885585] workqueue events_unbound: flags=0x2 [ 8865.897883] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8865.912422] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8865.934426] workqueue events_power_efficient: flags=0x80 [ 8865.940517] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8865.947911] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 8865.957627] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8865.970892] pending: neigh_periodic_work [ 8865.975985] workqueue mm_percpu_wq: flags=0x8 [ 8865.979942] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8865.987993] pending: vmstat_update, lru_add_drain_per_cpu [ 8865.995818] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8866.004474] pending: vmstat_update [ 8866.009247] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8866.016623] pending: lru_add_drain_per_cpu [ 8866.021635] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8866.029649] pending: lru_add_drain_per_cpu BAR(48) [ 8866.032440] workqueue writeback: flags=0x4a [ 8866.039733] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 8866.045822] pending: wb_workfn [ 8866.048403] workqueue xprtiod: flags=0xa [ 8866.060870] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8866.076977] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8866.088776] workqueue ipv6_addrconf: flags=0x40008 [ 8866.094510] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8866.101025] pending: addrconf_verify_work [ 8866.104986] workqueue knbd0-recv: flags=0x1a [ 8866.108952] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8866.116908] in-flight: 658:recv_work [ 8866.120455] workqueue ptlrpc_pinger: flags=0x2 [ 8866.122128] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8866.131509] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8866.141021] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6935s workers=4 idle: 32984 37 8 [ 8866.153369] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8850s workers=2 idle: 661 [ 8880.057734] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 8880.079303] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8880.130447] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8880.147258] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8880.152812] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8880.157023] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8880.187715] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8880.199121] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 8880.208424] RDX: 0000000000002b34 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 8880.223491] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 8880.228580] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 8880.233687] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8880.238194] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 8880.243239] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8880.245242] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8880.256055] Call Trace: [ 8880.258195] [ 8880.260036] ? show_regs.cold.9+0x22/0x2f [ 8880.266087] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8880.277985] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8880.281483] ? watchdog+0x40/0x40 [ 8880.282678] ? hrtimer_interrupt+0x128/0x2c0 [ 8880.298370] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8880.310174] ? apic_timer_interrupt+0xf/0x20 [ 8880.320323] [ 8880.327153] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8880.330630] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8880.335109] do_raw_spin_lock+0x177/0x190 [ 8880.350001] _raw_spin_lock+0x21/0x30 [ 8880.362747] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8880.372237] obd_mod_free+0x46/0x140 [ptlrpc] [ 8880.377739] mdc_commit_open+0x91/0xb0 [mdc] [ 8880.381022] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8880.386182] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8880.392876] after_reply+0xe2c/0x1780 [ptlrpc] [ 8880.403276] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8880.415225] ? __next_timer_interrupt+0x160/0x160 [ 8880.424280] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8880.428647] ? do_wait_intr+0xf0/0xf0 [ 8880.432126] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8880.436339] kthread+0x1d1/0x200 [ 8880.439981] ? set_kthread_struct+0x70/0x70 [ 8880.445284] ret_from_fork+0x1f/0x30 [ 8884.057032] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 8884.068651] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8884.127979] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8884.144128] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8884.155354] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8884.158492] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8884.178657] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8884.185700] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8884.193528] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 8884.199949] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8884.212067] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 8884.226937] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 8884.236152] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 8884.238593] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8884.246626] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 8884.250044] Call Trace: [ 8884.251157] [ 8884.252011] ? show_regs.cold.9+0x22/0x2f [ 8884.254937] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8884.258635] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8884.262924] ? watchdog+0x40/0x40 [ 8884.265405] ? hrtimer_interrupt+0x128/0x2c0 [ 8884.267512] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8884.270384] ? apic_timer_interrupt+0xf/0x20 [ 8884.273705] [ 8884.275347] ? native_safe_halt+0x2e/0x50 [ 8884.278163] kvm_wait+0xcd/0xe0 [ 8884.279841] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8884.282649] do_raw_spin_lock+0x177/0x190 [ 8884.285030] _raw_spin_lock+0x21/0x30 [ 8884.286794] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8884.289786] ? do_raw_spin_unlock+0x75/0x190 [ 8884.291653] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8884.294268] ? do_wait_intr+0xf0/0xf0 [ 8884.296305] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8884.298550] mdc_close+0x53b/0xd70 [mdc] [ 8884.300131] lmv_close+0x18d/0x490 [lmv] [ 8884.301758] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8884.304387] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8884.306826] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8884.309419] ll_md_close+0x274/0x800 [lustre] [ 8884.311562] ll_file_release+0x128/0xc50 [lustre] [ 8884.314048] __fput+0x10d/0x440 [ 8884.315005] ____fput+0x12/0x20 [ 8884.316298] task_work_run+0xc6/0x110 [ 8884.317410] exit_to_usermode_loop+0x1dd/0x1f0 [ 8884.319143] do_syscall_64+0x3d6/0x440 [ 8884.320459] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8884.322170] RIP: 0033:0x7f151bee3cd8 [ 8884.323399] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8884.331156] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8884.334365] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 8884.337758] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 8884.341595] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8884.344614] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8884.348171] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8888.056043] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8888.058900] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8888.111985] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8888.132545] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8888.146248] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8888.150471] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8888.158162] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8888.195753] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8888.201290] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8888.208295] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 8888.216414] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 8888.221724] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 8888.228742] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 8888.234009] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 8888.245052] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8888.250806] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8888.258556] Call Trace: [ 8888.260495] [ 8888.261508] ? show_regs.cold.9+0x22/0x2f [ 8888.266562] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8888.270405] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8888.274009] ? watchdog+0x40/0x40 [ 8888.278899] ? hrtimer_interrupt+0x128/0x2c0 [ 8888.281517] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8888.285566] ? apic_timer_interrupt+0xf/0x20 [ 8888.292448] [ 8888.294201] ? native_safe_halt+0x2e/0x50 [ 8888.300466] kvm_wait+0xcd/0xe0 [ 8888.302587] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8888.305933] do_raw_spin_lock+0x177/0x190 [ 8888.310222] _raw_spin_lock+0x21/0x30 [ 8888.314296] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8888.318615] process_one_work+0x2c8/0x700 [ 8888.324280] worker_thread+0x296/0x6e0 [ 8888.330992] ? rescuer_thread+0x570/0x570 [ 8888.336290] kthread+0x1d1/0x200 [ 8888.339365] ? set_kthread_struct+0x70/0x70 [ 8888.345438] ret_from_fork+0x1f/0x30 [ 8892.053417] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 8892.062085] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8892.111679] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8892.125512] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8892.132729] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8892.139124] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8892.172850] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8892.188111] RAX: 0000000000003983 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8892.196136] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 8892.205929] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 8892.217210] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 8892.221711] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 8892.230189] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 8892.237368] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8892.241256] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 8892.249213] Call Trace: [ 8892.251249] [ 8892.252411] ? show_regs.cold.9+0x22/0x2f [ 8892.256141] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8892.260354] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8892.263416] ? watchdog+0x40/0x40 [ 8892.266037] ? hrtimer_interrupt+0x128/0x2c0 [ 8892.270903] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8892.278466] ? apic_timer_interrupt+0xf/0x20 [ 8892.285494] [ 8892.287371] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8892.294210] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8892.304186] do_raw_spin_lock+0x177/0x190 [ 8892.313024] _raw_spin_lock+0x21/0x30 [ 8892.315768] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8892.322176] ? do_raw_spin_unlock+0x75/0x190 [ 8892.326083] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8892.335253] ? do_wait_intr+0xf0/0xf0 [ 8892.336634] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8892.341028] mdc_reint+0x2d/0x110 [mdc] [ 8892.343371] mdc_unlink+0x658/0x870 [mdc] [ 8892.345294] lmv_unlink+0x34d/0xbd0 [lmv] [ 8892.347562] ll_unlink+0x24d/0x7f0 [lustre] [ 8892.349308] vfs_unlink+0x202/0x330 [ 8892.354544] do_unlinkat+0x2fa/0x460 [ 8892.361966] __x64_sys_unlink+0x37/0x50 [ 8892.364456] do_syscall_64+0xc1/0x440 [ 8892.365788] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8892.371263] RIP: 0033:0x7f151bdee10b [ 8892.381619] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8892.406642] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8892.414215] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 8892.426683] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 8892.433053] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8892.440144] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8892.445386] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8896.482649] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6965s! [ 8896.494355] Showing busy workqueues and worker pools: [ 8896.496402] workqueue events: flags=0x0 [ 8896.498607] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8896.503372] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 8896.508607] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8896.525800] pending: kfree_rcu_monitor [ 8896.529162] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8896.535086] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 8896.539844] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8896.542978] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 8896.546688] workqueue events_long: flags=0x0 [ 8896.548246] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8896.551412] pending: lu_objects_flush [obdclass] [ 8896.554450] workqueue events_unbound: flags=0x2 [ 8896.556887] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8896.559962] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8896.565254] workqueue events_power_efficient: flags=0x80 [ 8896.567615] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8896.572488] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 8896.576572] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8896.579849] pending: neigh_periodic_work [ 8896.582037] workqueue mm_percpu_wq: flags=0x8 [ 8896.584098] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8896.587018] pending: vmstat_update, lru_add_drain_per_cpu [ 8896.590117] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8896.593282] pending: vmstat_update [ 8896.595282] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8896.597952] pending: lru_add_drain_per_cpu [ 8896.599504] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8896.602674] pending: lru_add_drain_per_cpu BAR(48) [ 8896.606534] workqueue writeback: flags=0x4a [ 8896.609960] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 8896.617654] pending: wb_workfn [ 8896.619484] workqueue xprtiod: flags=0xa [ 8896.625531] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8896.631620] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8896.635827] workqueue ipv6_addrconf: flags=0x40008 [ 8896.638478] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8896.641963] pending: addrconf_verify_work [ 8896.644382] workqueue knbd0-recv: flags=0x1a [ 8896.646675] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8896.649656] in-flight: 658:recv_work [ 8896.655749] workqueue ptlrpc_pinger: flags=0x2 [ 8896.659730] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8896.665090] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8896.669629] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6966s workers=4 idle: 32984 37 8 [ 8896.673406] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8881s workers=2 idle: 661 [ 8908.055377] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 8908.058928] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8908.082911] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8908.088829] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8908.093374] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8908.096454] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8908.108406] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8908.112440] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 8908.117786] RDX: 0000000000007793 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 8908.121080] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 8908.124857] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 8908.129080] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8908.132851] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 8908.136786] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8908.139966] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8908.142425] Call Trace: [ 8908.143578] [ 8908.144740] ? show_regs.cold.9+0x22/0x2f [ 8908.147194] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8908.148971] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8908.150668] ? watchdog+0x40/0x40 [ 8908.152067] ? hrtimer_interrupt+0x128/0x2c0 [ 8908.154025] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8908.156402] ? apic_timer_interrupt+0xf/0x20 [ 8908.158292] [ 8908.159253] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8908.161964] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8908.164162] do_raw_spin_lock+0x177/0x190 [ 8908.166022] _raw_spin_lock+0x21/0x30 [ 8908.167719] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8908.170343] obd_mod_free+0x46/0x140 [ptlrpc] [ 8908.172391] mdc_commit_open+0x91/0xb0 [mdc] [ 8908.174312] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8908.176517] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8908.179000] after_reply+0xe2c/0x1780 [ptlrpc] [ 8908.180847] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8908.182874] ? __next_timer_interrupt+0x160/0x160 [ 8908.184855] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8908.186267] ? do_wait_intr+0xf0/0xf0 [ 8908.187358] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8908.189492] kthread+0x1d1/0x200 [ 8908.190410] ? set_kthread_struct+0x70/0x70 [ 8908.191576] ret_from_fork+0x1f/0x30 [ 8912.062758] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 8912.068053] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8912.118110] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8912.130090] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8912.139690] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8912.148928] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8912.169924] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8912.177739] RAX: 00000000000062d0 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8912.187804] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 8912.193252] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8912.206174] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 8912.214698] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 8912.220186] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 8912.227280] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8912.231442] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 8912.238389] Call Trace: [ 8912.248946] [ 8912.249708] ? show_regs.cold.9+0x22/0x2f [ 8912.251155] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8912.256788] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8912.258338] ? watchdog+0x40/0x40 [ 8912.273630] ? hrtimer_interrupt+0x128/0x2c0 [ 8912.275219] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8912.290054] ? apic_timer_interrupt+0xf/0x20 [ 8912.291594] [ 8912.304646] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8912.306586] do_raw_spin_lock+0x177/0x190 [ 8912.319366] _raw_spin_lock+0x21/0x30 [ 8912.320619] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8912.331415] ? do_raw_spin_unlock+0x75/0x190 [ 8912.334339] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8912.342115] ? do_wait_intr+0xf0/0xf0 [ 8912.346570] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8912.348569] mdc_close+0x53b/0xd70 [mdc] [ 8912.356571] lmv_close+0x18d/0x490 [lmv] [ 8912.359385] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8912.367735] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8912.379389] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8912.385566] ll_md_close+0x274/0x800 [lustre] [ 8912.396773] ll_file_release+0x128/0xc50 [lustre] [ 8912.399300] __fput+0x10d/0x440 [ 8912.404878] ____fput+0x12/0x20 [ 8912.406977] task_work_run+0xc6/0x110 [ 8912.408492] exit_to_usermode_loop+0x1dd/0x1f0 [ 8912.413523] do_syscall_64+0x3d6/0x440 [ 8912.416095] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8912.421287] RIP: 0033:0x7f151bee3cd8 [ 8912.425460] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8912.440844] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8912.449538] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 8912.464667] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 8912.472927] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8912.483765] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8912.486721] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8916.056577] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8916.059510] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8916.078636] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8916.082676] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8916.085532] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8916.088405] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8916.090904] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8916.098530] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8916.101451] RAX: 0000000000007f88 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8916.104142] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 8916.106921] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 8916.110034] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 8916.113013] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 8916.116036] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 8916.118672] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8916.121241] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8916.124324] Call Trace: [ 8916.125541] [ 8916.126500] ? show_regs.cold.9+0x22/0x2f [ 8916.128325] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8916.130316] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8916.132285] ? watchdog+0x40/0x40 [ 8916.133701] ? hrtimer_interrupt+0x128/0x2c0 [ 8916.135159] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8916.137111] ? apic_timer_interrupt+0xf/0x20 [ 8916.138751] [ 8916.139520] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8916.142257] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8916.144371] do_raw_spin_lock+0x177/0x190 [ 8916.146236] _raw_spin_lock+0x21/0x30 [ 8916.148993] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8916.152566] process_one_work+0x2c8/0x700 [ 8916.154302] worker_thread+0x296/0x6e0 [ 8916.156041] ? rescuer_thread+0x570/0x570 [ 8916.157584] kthread+0x1d1/0x200 [ 8916.159084] ? set_kthread_struct+0x70/0x70 [ 8916.161034] ret_from_fork+0x1f/0x30 [ 8924.052111] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 8924.057091] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8924.080900] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8924.086747] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8924.092168] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8924.094277] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8924.104420] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8924.107842] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8924.110884] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 8924.114336] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 8924.118789] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 8924.124347] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 8924.129808] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 8924.134127] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8924.137021] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 8924.139769] Call Trace: [ 8924.141255] [ 8924.142151] ? show_regs.cold.9+0x22/0x2f [ 8924.143588] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8924.146016] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8924.148105] ? watchdog+0x40/0x40 [ 8924.149708] ? hrtimer_interrupt+0x128/0x2c0 [ 8924.151751] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8924.154060] ? apic_timer_interrupt+0xf/0x20 [ 8924.156323] [ 8924.157087] ? native_safe_halt+0x2e/0x50 [ 8924.159253] kvm_wait+0xcd/0xe0 [ 8924.160751] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8924.163465] do_raw_spin_lock+0x177/0x190 [ 8924.165418] _raw_spin_lock+0x21/0x30 [ 8924.167452] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8924.170253] ? do_raw_spin_unlock+0x75/0x190 [ 8924.172344] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8924.174776] ? do_wait_intr+0xf0/0xf0 [ 8924.176096] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8924.178739] mdc_reint+0x2d/0x110 [mdc] [ 8924.180672] mdc_unlink+0x658/0x870 [mdc] [ 8924.182592] lmv_unlink+0x34d/0xbd0 [lmv] [ 8924.184466] ll_unlink+0x24d/0x7f0 [lustre] [ 8924.186859] vfs_unlink+0x202/0x330 [ 8924.188773] do_unlinkat+0x2fa/0x460 [ 8924.190885] __x64_sys_unlink+0x37/0x50 [ 8924.193102] do_syscall_64+0xc1/0x440 [ 8924.194914] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8924.198799] RIP: 0033:0x7f151bdee10b [ 8924.201094] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8924.217529] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8924.220476] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 8924.227368] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 8924.231022] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8924.234629] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8924.241318] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8926.690193] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6996s! [ 8926.693390] Showing busy workqueues and worker pools: [ 8926.695919] workqueue events: flags=0x0 [ 8926.697681] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8926.701166] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 8926.705408] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8926.708414] pending: kfree_rcu_monitor [ 8926.710264] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8926.713844] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 8926.719606] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8926.723170] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 8926.727962] workqueue events_long: flags=0x0 [ 8926.730235] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8926.733407] pending: lu_objects_flush [obdclass] [ 8926.736847] workqueue events_unbound: flags=0x2 [ 8926.739123] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8926.742709] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8926.749782] workqueue events_power_efficient: flags=0x80 [ 8926.752462] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8926.755947] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 8926.759697] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8926.763799] pending: neigh_periodic_work [ 8926.766233] workqueue mm_percpu_wq: flags=0x8 [ 8926.767879] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8926.770400] pending: vmstat_update, lru_add_drain_per_cpu [ 8926.771794] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8926.773841] pending: vmstat_update [ 8926.775375] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8926.778412] pending: lru_add_drain_per_cpu [ 8926.782596] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8926.784284] pending: lru_add_drain_per_cpu BAR(48) [ 8926.785656] workqueue writeback: flags=0x4a [ 8926.786648] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 8926.789007] pending: wb_workfn [ 8926.790404] workqueue xprtiod: flags=0xa [ 8926.791545] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8926.793796] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8926.796320] workqueue ipv6_addrconf: flags=0x40008 [ 8926.798348] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8926.801651] pending: addrconf_verify_work [ 8926.803681] workqueue knbd0-recv: flags=0x1a [ 8926.805419] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8926.808668] in-flight: 658:recv_work [ 8926.810526] workqueue ptlrpc_pinger: flags=0x2 [ 8926.812442] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8926.815726] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8926.818546] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6996s workers=4 idle: 32984 37 8 [ 8926.822759] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8911s workers=2 idle: 661 [ 8936.055190] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 8936.060944] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8936.088152] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8936.093542] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8936.098127] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8936.100783] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8936.107819] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8936.111840] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 8936.115582] RDX: 0000000000002a11 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 8936.118197] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 8936.124476] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 8936.128644] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8936.133130] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 8936.139231] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8936.141632] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8936.145335] Call Trace: [ 8936.147071] [ 8936.148881] ? show_regs.cold.9+0x22/0x2f [ 8936.151904] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8936.154812] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8936.156728] ? watchdog+0x40/0x40 [ 8936.158916] ? hrtimer_interrupt+0x128/0x2c0 [ 8936.160738] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8936.163248] ? apic_timer_interrupt+0xf/0x20 [ 8936.166103] [ 8936.167918] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8936.172605] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8936.174786] do_raw_spin_lock+0x177/0x190 [ 8936.176464] _raw_spin_lock+0x21/0x30 [ 8936.178343] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8936.180453] obd_mod_free+0x46/0x140 [ptlrpc] [ 8936.182221] mdc_commit_open+0x91/0xb0 [mdc] [ 8936.183792] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8936.186080] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8936.188309] after_reply+0xe2c/0x1780 [ptlrpc] [ 8936.190214] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8936.192513] ? __next_timer_interrupt+0x160/0x160 [ 8936.194308] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8936.196424] ? do_wait_intr+0xf0/0xf0 [ 8936.197822] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8936.199605] kthread+0x1d1/0x200 [ 8936.200476] ? set_kthread_struct+0x70/0x70 [ 8936.201536] ret_from_fork+0x1f/0x30 [ 8940.061629] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 8940.069281] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8940.114886] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8940.132820] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8940.148256] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8940.157743] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8940.199146] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8940.211765] RAX: 0000000000007f42 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8940.217802] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 8940.237198] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8940.248021] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 8940.267166] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 8940.282903] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 8940.301199] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8940.305586] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 8940.308034] Call Trace: [ 8940.321890] [ 8940.322965] ? show_regs.cold.9+0x22/0x2f [ 8940.328006] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8940.335802] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8940.345027] ? watchdog+0x40/0x40 [ 8940.352285] ? hrtimer_interrupt+0x128/0x2c0 [ 8940.366887] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8940.375930] ? apic_timer_interrupt+0xf/0x20 [ 8940.386237] [ 8940.396348] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8940.407248] do_raw_spin_lock+0x177/0x190 [ 8940.411130] _raw_spin_lock+0x21/0x30 [ 8940.420179] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8940.422322] ? do_raw_spin_unlock+0x75/0x190 [ 8940.440010] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8940.454025] ? do_wait_intr+0xf0/0xf0 [ 8940.455084] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8940.467563] mdc_close+0x53b/0xd70 [mdc] [ 8940.469429] lmv_close+0x18d/0x490 [lmv] [ 8940.472629] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8940.480398] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8940.484553] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8940.487293] ll_md_close+0x274/0x800 [lustre] [ 8940.491861] ll_file_release+0x128/0xc50 [lustre] [ 8940.496537] __fput+0x10d/0x440 [ 8940.497876] ____fput+0x12/0x20 [ 8940.499875] task_work_run+0xc6/0x110 [ 8940.501940] exit_to_usermode_loop+0x1dd/0x1f0 [ 8940.507281] do_syscall_64+0x3d6/0x440 [ 8940.509826] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8940.512897] RIP: 0033:0x7f151bee3cd8 [ 8940.515355] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8940.525669] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8940.531001] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 8940.536533] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 8940.541487] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8940.546433] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8940.552365] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8944.057541] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8944.070626] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8944.123558] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8944.134626] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8944.144140] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8944.149772] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8944.155937] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8944.164875] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8944.170042] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8944.173573] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 8944.177516] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 8944.191555] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 8944.196570] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 8944.201617] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 8944.208139] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8944.212336] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8944.217372] Call Trace: [ 8944.218918] [ 8944.220386] ? show_regs.cold.9+0x22/0x2f [ 8944.223207] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8944.226453] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8944.229434] ? watchdog+0x40/0x40 [ 8944.231701] ? hrtimer_interrupt+0x128/0x2c0 [ 8944.234907] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8944.237539] ? apic_timer_interrupt+0xf/0x20 [ 8944.240303] [ 8944.241903] ? native_safe_halt+0x2e/0x50 [ 8944.243337] kvm_wait+0xcd/0xe0 [ 8944.244395] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8944.247996] do_raw_spin_lock+0x177/0x190 [ 8944.249951] _raw_spin_lock+0x21/0x30 [ 8944.251261] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8944.253637] process_one_work+0x2c8/0x700 [ 8944.254943] worker_thread+0x296/0x6e0 [ 8944.256255] ? rescuer_thread+0x570/0x570 [ 8944.257906] kthread+0x1d1/0x200 [ 8944.259537] ? set_kthread_struct+0x70/0x70 [ 8944.261458] ret_from_fork+0x1f/0x30 [ 8952.055405] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 8952.058115] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8952.109740] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8952.117105] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8952.123830] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8952.130699] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8952.147641] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8952.154832] RAX: 000000000000381c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8952.160623] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 8952.165872] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 8952.170848] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 8952.190929] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 8952.195781] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 8952.203991] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8952.210466] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 8952.217424] Call Trace: [ 8952.219054] [ 8952.220118] ? show_regs.cold.9+0x22/0x2f [ 8952.233658] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8952.237527] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8952.243687] ? watchdog+0x40/0x40 [ 8952.246128] ? hrtimer_interrupt+0x128/0x2c0 [ 8952.248793] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8952.251010] ? apic_timer_interrupt+0xf/0x20 [ 8952.252626] [ 8952.256721] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8952.266984] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8952.269748] do_raw_spin_lock+0x177/0x190 [ 8952.271234] _raw_spin_lock+0x21/0x30 [ 8952.277751] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8952.291786] ? do_raw_spin_unlock+0x75/0x190 [ 8952.296163] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8952.304962] ? do_wait_intr+0xf0/0xf0 [ 8952.306297] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8952.309707] mdc_reint+0x2d/0x110 [mdc] [ 8952.312862] mdc_unlink+0x658/0x870 [mdc] [ 8952.316107] lmv_unlink+0x34d/0xbd0 [lmv] [ 8952.318890] ll_unlink+0x24d/0x7f0 [lustre] [ 8952.325108] vfs_unlink+0x202/0x330 [ 8952.327845] do_unlinkat+0x2fa/0x460 [ 8952.329766] __x64_sys_unlink+0x37/0x50 [ 8952.334372] do_syscall_64+0xc1/0x440 [ 8952.336088] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8952.343827] RIP: 0033:0x7f151bdee10b [ 8952.349859] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8952.372109] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8952.380772] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 8952.389574] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 8952.396136] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8952.403089] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8952.411506] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8956.901159] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7026s! [ 8956.916786] Showing busy workqueues and worker pools: [ 8956.923466] workqueue events: flags=0x0 [ 8956.925886] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8956.930375] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 8956.936658] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8956.942836] pending: kfree_rcu_monitor [ 8956.948469] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8956.955501] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 8956.971212] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8956.982829] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 8956.999437] workqueue events_long: flags=0x0 [ 8957.006430] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8957.018412] pending: lu_objects_flush [obdclass] [ 8957.032708] workqueue events_unbound: flags=0x2 [ 8957.041611] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8957.053931] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8957.064813] workqueue events_power_efficient: flags=0x80 [ 8957.071720] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8957.077534] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 8957.088603] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8957.103784] pending: neigh_periodic_work [ 8957.110104] workqueue mm_percpu_wq: flags=0x8 [ 8957.112477] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8957.119951] pending: vmstat_update, lru_add_drain_per_cpu [ 8957.129392] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8957.142730] pending: vmstat_update [ 8957.147917] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8957.155761] pending: lru_add_drain_per_cpu [ 8957.161751] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8957.165896] pending: lru_add_drain_per_cpu BAR(48) [ 8957.170429] workqueue writeback: flags=0x4a [ 8957.181132] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 8957.190865] pending: wb_workfn [ 8957.200661] workqueue xprtiod: flags=0xa [ 8957.202191] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8957.224274] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8957.241416] workqueue ipv6_addrconf: flags=0x40008 [ 8957.258022] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8957.262457] pending: addrconf_verify_work [ 8957.270604] workqueue knbd0-recv: flags=0x1a [ 8957.274591] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8957.288520] in-flight: 658:recv_work [ 8957.296760] workqueue ptlrpc_pinger: flags=0x2 [ 8957.300418] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8957.308481] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8957.318673] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7026s workers=4 idle: 32984 37 8 [ 8957.339820] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8942s workers=2 idle: 661 [ 8964.057867] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 8964.065843] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8964.104415] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8964.115819] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8964.120146] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8964.127517] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8964.135757] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8964.141148] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 8964.145549] RDX: 00000000000053e2 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 8964.147913] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 8964.156371] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 8964.159063] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8964.165360] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 8964.169267] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8964.173811] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8964.177290] Call Trace: [ 8964.178513] [ 8964.179998] ? show_regs.cold.9+0x22/0x2f [ 8964.183463] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8964.185259] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8964.186719] ? watchdog+0x40/0x40 [ 8964.188281] ? hrtimer_interrupt+0x128/0x2c0 [ 8964.189777] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8964.196599] ? apic_timer_interrupt+0xf/0x20 [ 8964.198600] [ 8964.199660] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8964.201908] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8964.206838] do_raw_spin_lock+0x177/0x190 [ 8964.208249] _raw_spin_lock+0x21/0x30 [ 8964.223762] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8964.225709] obd_mod_free+0x46/0x140 [ptlrpc] [ 8964.236870] mdc_commit_open+0x91/0xb0 [mdc] [ 8964.242579] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8964.247652] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8964.252423] after_reply+0xe2c/0x1780 [ptlrpc] [ 8964.255907] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8964.261566] ? __next_timer_interrupt+0x160/0x160 [ 8964.263118] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8964.269525] ? do_wait_intr+0xf0/0xf0 [ 8964.272352] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8964.279036] kthread+0x1d1/0x200 [ 8964.282465] ? set_kthread_struct+0x70/0x70 [ 8964.284643] ret_from_fork+0x1f/0x30 [ 8968.060633] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 8968.068804] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8968.125009] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8968.151844] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8968.168470] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8968.180932] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8968.225485] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8968.236398] RAX: 0000000000006116 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8968.249455] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 8968.252112] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8968.274314] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 8968.279229] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 8968.286592] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 8968.303440] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8968.314720] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 8968.325323] Call Trace: [ 8968.327303] [ 8968.329213] ? show_regs.cold.9+0x22/0x2f [ 8968.331650] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8968.335380] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8968.342875] ? watchdog+0x40/0x40 [ 8968.346687] ? hrtimer_interrupt+0x128/0x2c0 [ 8968.348666] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8968.352879] ? apic_timer_interrupt+0xf/0x20 [ 8968.356702] [ 8968.364486] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8968.375715] do_raw_spin_lock+0x177/0x190 [ 8968.380723] _raw_spin_lock+0x21/0x30 [ 8968.386449] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8968.395252] ? do_raw_spin_unlock+0x75/0x190 [ 8968.399667] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8968.411842] ? do_wait_intr+0xf0/0xf0 [ 8968.416812] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8968.420292] mdc_close+0x53b/0xd70 [mdc] [ 8968.424754] lmv_close+0x18d/0x490 [lmv] [ 8968.426621] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8968.439785] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8968.441938] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8968.447972] ll_md_close+0x274/0x800 [lustre] [ 8968.449693] ll_file_release+0x128/0xc50 [lustre] [ 8968.453889] __fput+0x10d/0x440 [ 8968.457879] ____fput+0x12/0x20 [ 8968.462393] task_work_run+0xc6/0x110 [ 8968.466702] exit_to_usermode_loop+0x1dd/0x1f0 [ 8968.470745] do_syscall_64+0x3d6/0x440 [ 8968.472154] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8968.479041] RIP: 0033:0x7f151bee3cd8 [ 8968.482459] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8968.503959] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8968.510673] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 8968.524367] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 8968.533308] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8968.540749] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8968.559339] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8972.056065] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 8972.059716] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8972.105360] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8972.123341] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8972.132001] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8972.142620] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 8972.150042] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 8972.168213] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8972.175386] RAX: 0000000000002a78 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8972.178817] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 8972.183813] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 8972.196405] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 8972.208450] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 8972.214133] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 8972.226169] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8972.239985] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8972.252811] Call Trace: [ 8972.254323] [ 8972.259417] ? show_regs.cold.9+0x22/0x2f [ 8972.264376] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8972.268546] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8972.277569] ? watchdog+0x40/0x40 [ 8972.278849] ? hrtimer_interrupt+0x128/0x2c0 [ 8972.288900] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8972.290677] ? apic_timer_interrupt+0xf/0x20 [ 8972.304241] [ 8972.307698] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 8972.317292] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8972.328754] do_raw_spin_lock+0x177/0x190 [ 8972.336324] _raw_spin_lock+0x21/0x30 [ 8972.343620] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8972.350183] process_one_work+0x2c8/0x700 [ 8972.355760] worker_thread+0x296/0x6e0 [ 8972.357790] ? rescuer_thread+0x570/0x570 [ 8972.364384] kthread+0x1d1/0x200 [ 8972.366356] ? set_kthread_struct+0x70/0x70 [ 8972.369628] ret_from_fork+0x1f/0x30 [ 8984.055089] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 8984.063485] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8984.125641] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8984.141665] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8984.152369] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8984.160626] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8984.179166] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8984.188183] RAX: 0000000000005b71 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8984.194689] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 8984.203020] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 8984.209977] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 8984.218192] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 8984.221770] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 8984.237014] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8984.249798] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 8984.252492] Call Trace: [ 8984.261528] [ 8984.264800] ? show_regs.cold.9+0x22/0x2f [ 8984.270701] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8984.278224] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8984.286468] ? watchdog+0x40/0x40 [ 8984.295470] ? hrtimer_interrupt+0x128/0x2c0 [ 8984.296878] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8984.308419] ? apic_timer_interrupt+0xf/0x20 [ 8984.320142] [ 8984.320914] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8984.322888] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8984.334617] do_raw_spin_lock+0x177/0x190 [ 8984.336305] _raw_spin_lock+0x21/0x30 [ 8984.337884] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8984.342399] ? do_raw_spin_unlock+0x75/0x190 [ 8984.344831] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8984.349840] ? do_wait_intr+0xf0/0xf0 [ 8984.358786] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8984.369161] mdc_reint+0x2d/0x110 [mdc] [ 8984.376453] mdc_unlink+0x658/0x870 [mdc] [ 8984.378000] lmv_unlink+0x34d/0xbd0 [lmv] [ 8984.388420] ll_unlink+0x24d/0x7f0 [lustre] [ 8984.389626] vfs_unlink+0x202/0x330 [ 8984.390799] do_unlinkat+0x2fa/0x460 [ 8984.400444] __x64_sys_unlink+0x37/0x50 [ 8984.403090] do_syscall_64+0xc1/0x440 [ 8984.408688] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8984.410556] RIP: 0033:0x7f151bdee10b [ 8984.423990] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8984.441967] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8984.445277] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 8984.453257] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 8984.457774] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8984.469861] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8984.476492] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8987.625105] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7057s! [ 8987.632720] Showing busy workqueues and worker pools: [ 8987.640774] workqueue events: flags=0x0 [ 8987.651551] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8987.655712] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 8987.660377] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8987.669752] pending: kfree_rcu_monitor [ 8987.675736] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8987.682757] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 8987.686942] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8987.691417] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 8987.704169] workqueue events_long: flags=0x0 [ 8987.712343] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8987.720057] pending: lu_objects_flush [obdclass] [ 8987.723836] workqueue events_unbound: flags=0x2 [ 8987.727873] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 8987.736195] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 8987.753615] workqueue events_power_efficient: flags=0x80 [ 8987.758761] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8987.771766] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 8987.775456] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8987.781724] pending: neigh_periodic_work [ 8987.787600] workqueue mm_percpu_wq: flags=0x8 [ 8987.791520] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8987.800276] pending: vmstat_update, lru_add_drain_per_cpu [ 8987.805579] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8987.810721] pending: vmstat_update [ 8987.812088] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8987.822772] pending: lru_add_drain_per_cpu [ 8987.834537] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8987.836953] pending: lru_add_drain_per_cpu BAR(48) [ 8987.839371] workqueue writeback: flags=0x4a [ 8987.840798] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 8987.843474] pending: wb_workfn [ 8987.844723] workqueue xprtiod: flags=0xa [ 8987.846462] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 8987.849316] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 8987.852390] workqueue ipv6_addrconf: flags=0x40008 [ 8987.857326] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8987.862514] pending: addrconf_verify_work [ 8987.864458] workqueue knbd0-recv: flags=0x1a [ 8987.866481] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8987.871549] in-flight: 658:recv_work [ 8987.875367] workqueue ptlrpc_pinger: flags=0x2 [ 8987.876894] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8987.880693] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 8987.884484] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7057s workers=4 idle: 32984 37 8 [ 8987.888306] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8972s workers=2 idle: 661 [ 8992.069827] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 8992.078177] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8992.118809] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8992.133515] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8992.147366] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8992.153162] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8992.171799] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8992.180972] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 8992.187844] RDX: 00000000000076ac RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 8992.196749] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 8992.200426] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 8992.205781] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8992.213456] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 8992.215414] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8992.222737] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 8992.229403] Call Trace: [ 8992.230571] [ 8992.231615] ? show_regs.cold.9+0x22/0x2f [ 8992.234481] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8992.238496] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8992.242266] ? watchdog+0x40/0x40 [ 8992.243463] ? hrtimer_interrupt+0x128/0x2c0 [ 8992.248191] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8992.254654] ? apic_timer_interrupt+0xf/0x20 [ 8992.257350] [ 8992.257974] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8992.266399] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8992.272703] do_raw_spin_lock+0x177/0x190 [ 8992.276195] _raw_spin_lock+0x21/0x30 [ 8992.280300] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8992.285837] obd_mod_free+0x46/0x140 [ptlrpc] [ 8992.289257] mdc_commit_open+0x91/0xb0 [mdc] [ 8992.291865] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8992.294213] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8992.296614] after_reply+0xe2c/0x1780 [ptlrpc] [ 8992.304809] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8992.315281] ? __next_timer_interrupt+0x160/0x160 [ 8992.323551] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8992.334435] ? do_wait_intr+0xf0/0xf0 [ 8992.336438] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8992.342415] kthread+0x1d1/0x200 [ 8992.344404] ? set_kthread_struct+0x70/0x70 [ 8992.349896] ret_from_fork+0x1f/0x30 [ 8996.057098] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 8996.066242] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8996.116294] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8996.131847] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8996.143241] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8996.149775] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8996.163720] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8996.169040] RAX: 000000000000790b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 8996.175622] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 8996.181242] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8996.187848] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 8996.193028] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 8996.198281] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 8996.203930] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8996.208050] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 8996.213746] Call Trace: [ 8996.215769] [ 8996.217499] ? show_regs.cold.9+0x22/0x2f [ 8996.220188] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8996.223232] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8996.227264] ? watchdog+0x40/0x40 [ 8996.230265] ? hrtimer_interrupt+0x128/0x2c0 [ 8996.233753] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8996.237857] ? apic_timer_interrupt+0xf/0x20 [ 8996.241268] [ 8996.242489] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8996.246590] do_raw_spin_lock+0x177/0x190 [ 8996.249247] _raw_spin_lock+0x21/0x30 [ 8996.251567] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8996.255238] ? do_raw_spin_unlock+0x75/0x190 [ 8996.258407] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8996.262266] ? do_wait_intr+0xf0/0xf0 [ 8996.265206] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8996.269534] mdc_close+0x53b/0xd70 [mdc] [ 8996.272528] lmv_close+0x18d/0x490 [lmv] [ 8996.277070] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8996.282650] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 8996.288094] ll_md_real_close+0xa5/0x2a0 [lustre] [ 8996.292157] ll_md_close+0x274/0x800 [lustre] [ 8996.295221] ll_file_release+0x128/0xc50 [lustre] [ 8996.298338] __fput+0x10d/0x440 [ 8996.301237] ____fput+0x12/0x20 [ 8996.303579] task_work_run+0xc6/0x110 [ 8996.306579] exit_to_usermode_loop+0x1dd/0x1f0 [ 8996.309569] do_syscall_64+0x3d6/0x440 [ 8996.312301] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8996.320240] RIP: 0033:0x7f151bee3cd8 [ 8996.324944] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 8996.355227] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 8996.360156] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 8996.367152] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 8996.371312] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 8996.375984] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 8996.380871] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9000.056032] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9000.060607] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9000.085503] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9000.092778] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9000.103447] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9000.109942] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9000.114354] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9000.128852] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9000.134691] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9000.139648] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 9000.143426] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 9000.148960] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 9000.157198] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 9000.160229] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 9000.165547] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9000.173194] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9000.183426] Call Trace: [ 9000.185995] [ 9000.186962] ? show_regs.cold.9+0x22/0x2f [ 9000.188841] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9000.191999] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9000.194928] ? watchdog+0x40/0x40 [ 9000.198258] ? hrtimer_interrupt+0x128/0x2c0 [ 9000.202071] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9000.203820] ? apic_timer_interrupt+0xf/0x20 [ 9000.207293] [ 9000.208222] ? native_safe_halt+0x2e/0x50 [ 9000.210400] kvm_wait+0xcd/0xe0 [ 9000.212175] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9000.216946] do_raw_spin_lock+0x177/0x190 [ 9000.221855] _raw_spin_lock+0x21/0x30 [ 9000.225530] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9000.228732] process_one_work+0x2c8/0x700 [ 9000.231448] worker_thread+0x296/0x6e0 [ 9000.234975] ? rescuer_thread+0x570/0x570 [ 9000.236695] kthread+0x1d1/0x200 [ 9000.237773] ? set_kthread_struct+0x70/0x70 [ 9000.241233] ret_from_fork+0x1f/0x30 [ 9004.795132] rcu: INFO: rcu_sched self-detected stall on CPU [ 9004.798363] rcu: 1-...!: (4016156 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 9004.807258] (t=7080148 jiffies g=249657 q=4404) [ 9004.812393] rcu: rcu_sched kthread starved for 7075100 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 9004.815401] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 9004.837313] rcu: RCU grace-period kthread stack dump: [ 9004.845488] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 9004.857976] Call Trace: [ 9004.858783] __schedule+0x351/0xcb0 [ 9004.863605] schedule+0xc0/0x180 [ 9004.870145] schedule_timeout+0xb4/0x190 [ 9004.872996] ? do_raw_spin_unlock+0x75/0x190 [ 9004.877489] ? __next_timer_interrupt+0x160/0x160 [ 9004.880846] ? prepare_to_swait_event+0xf3/0x140 [ 9004.884197] rcu_gp_kthread+0x8dd/0x15b0 [ 9004.887341] ? force_qs_rnp+0x2c0/0x2c0 [ 9004.888976] kthread+0x1d1/0x200 [ 9004.890568] ? set_kthread_struct+0x70/0x70 [ 9004.893443] ret_from_fork+0x1f/0x30 [ 9004.895403] rcu: Stack dump where RCU GP kthread last ran: [ 9004.897627] Sending NMI from CPU 1 to CPUs 2: [ 9004.903876] NMI backtrace for cpu 2 [ 9004.903886] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9004.903889] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9004.903890] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9004.903893] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9004.903895] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9004.903898] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 [ 9004.903899] RAX: 0000000000007d95 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9004.903901] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 9004.903902] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 9004.903905] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 9004.903906] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 9004.903907] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 9004.903909] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9004.903911] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9004.903912] Call Trace: [ 9004.903913] [ 9004.903913] ? show_regs.cold.9+0x22/0x2f [ 9004.903915] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 9004.903916] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 9004.903917] ? nmi_handle+0xbd/0x210 [ 9004.903918] ? default_do_nmi+0x4b/0x1b0 [ 9004.903919] ? do_nmi+0x280/0x380 [ 9004.903920] ? end_repeat_nmi+0x16/0x1f [ 9004.903921] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9004.903923] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9004.903924] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9004.903925] [ 9004.903926] do_raw_spin_lock+0x177/0x190 [ 9004.903927] _raw_spin_lock+0x21/0x30 [ 9004.903928] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9004.903929] process_one_work+0x2c8/0x700 [ 9004.903930] worker_thread+0x296/0x6e0 [ 9004.903931] ? rescuer_thread+0x570/0x570 [ 9004.903932] kthread+0x1d1/0x200 [ 9004.903933] ? set_kthread_struct+0x70/0x70 [ 9004.903934] ret_from_fork+0x1f/0x30 [ 9004.904303] NMI backtrace for cpu 1 [ 9005.116192] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9005.120813] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9005.128451] Call Trace: [ 9005.131025] [ 9005.132063] ? dump_stack+0xbb/0x10e [ 9005.133774] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9005.140391] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9005.144557] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9005.148763] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9005.152817] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9005.155628] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 9005.159286] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 9005.163500] ? update_process_times+0x6d/0xc0 [ 9005.166618] ? tick_sched_handle.isra.18+0x27/0x90 [ 9005.168362] ? tick_sched_timer+0x53/0xb0 [ 9005.173645] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9005.175288] ? tick_sched_handle.isra.18+0x90/0x90 [ 9005.182780] ? hrtimer_interrupt+0x128/0x2c0 [ 9005.186192] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9005.189461] ? apic_timer_interrupt+0xf/0x20 [ 9005.190944] [ 9005.191745] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9005.194506] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9005.197314] ? do_raw_spin_lock+0x177/0x190 [ 9005.199139] ? _raw_spin_lock+0x21/0x30 [ 9005.200804] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9005.208353] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9005.211200] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9005.213918] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9005.216461] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9005.223939] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9005.226551] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9005.233342] ? __next_timer_interrupt+0x160/0x160 [ 9005.236998] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9005.240352] ? do_wait_intr+0xf0/0xf0 [ 9005.245552] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9005.249453] ? kthread+0x1d1/0x200 [ 9005.252041] ? set_kthread_struct+0x70/0x70 [ 9005.255387] ? ret_from_fork+0x1f/0x30 [ 9012.053241] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 9012.063130] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9012.090872] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9012.096253] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9012.100050] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9012.102878] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9012.113996] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9012.118070] RAX: 000000000000377a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9012.121286] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 9012.124187] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 9012.127243] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 9012.130358] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 9012.133546] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 9012.137334] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9012.140153] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 9012.142869] Call Trace: [ 9012.143563] [ 9012.144817] ? show_regs.cold.9+0x22/0x2f [ 9012.147032] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9012.149113] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9012.151199] ? watchdog+0x40/0x40 [ 9012.152935] ? hrtimer_interrupt+0x128/0x2c0 [ 9012.154812] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9012.156904] ? apic_timer_interrupt+0xf/0x20 [ 9012.159085] [ 9012.160294] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9012.163019] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9012.165513] do_raw_spin_lock+0x177/0x190 [ 9012.167526] _raw_spin_lock+0x21/0x30 [ 9012.169400] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9012.172163] ? do_raw_spin_unlock+0x75/0x190 [ 9012.175077] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9012.179124] ? do_wait_intr+0xf0/0xf0 [ 9012.183199] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9012.191352] mdc_reint+0x2d/0x110 [mdc] [ 9012.194604] mdc_unlink+0x658/0x870 [mdc] [ 9012.203507] lmv_unlink+0x34d/0xbd0 [lmv] [ 9012.208206] ll_unlink+0x24d/0x7f0 [lustre] [ 9012.212241] vfs_unlink+0x202/0x330 [ 9012.215028] do_unlinkat+0x2fa/0x460 [ 9012.218140] __x64_sys_unlink+0x37/0x50 [ 9012.220863] do_syscall_64+0xc1/0x440 [ 9012.226442] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9012.231428] RIP: 0033:0x7f151bdee10b [ 9012.233478] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9012.257293] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9012.261344] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 9012.265863] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 9012.277167] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9012.284894] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9012.290452] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9018.337707] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7087s! [ 9018.345025] Showing busy workqueues and worker pools: [ 9018.348438] workqueue events: flags=0x0 [ 9018.352745] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9018.358626] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 9018.364934] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9018.367553] pending: kfree_rcu_monitor [ 9018.370123] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9018.374097] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 9018.381154] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9018.386047] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 9018.390814] workqueue events_long: flags=0x0 [ 9018.395085] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9018.403691] pending: lu_objects_flush [obdclass] [ 9018.411894] workqueue events_unbound: flags=0x2 [ 9018.416074] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9018.435967] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9018.467983] workqueue events_power_efficient: flags=0x80 [ 9018.472661] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9018.480182] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 9018.492578] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9018.500573] pending: neigh_periodic_work [ 9018.506360] workqueue mm_percpu_wq: flags=0x8 [ 9018.509447] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9018.523160] pending: vmstat_update, lru_add_drain_per_cpu [ 9018.532543] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9018.546459] pending: vmstat_update [ 9018.556192] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9018.566524] pending: lru_add_drain_per_cpu [ 9018.569524] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9018.580558] pending: lru_add_drain_per_cpu BAR(48) [ 9018.590622] workqueue writeback: flags=0x4a [ 9018.598021] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 9018.611208] pending: wb_workfn [ 9018.617072] workqueue xprtiod: flags=0xa [ 9018.622533] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9018.628556] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9018.638994] workqueue ipv6_addrconf: flags=0x40008 [ 9018.642725] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9018.651002] pending: addrconf_verify_work [ 9018.655057] workqueue knbd0-recv: flags=0x1a [ 9018.665486] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9018.672767] in-flight: 658:recv_work [ 9018.681407] workqueue ptlrpc_pinger: flags=0x2 [ 9018.687404] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9018.694227] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9018.702737] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7088s workers=4 idle: 32984 37 8 [ 9018.709901] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9003s workers=2 idle: 661 [ 9024.057973] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 9024.065417] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9024.113041] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9024.120093] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9024.126032] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9024.130184] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9024.140987] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9024.146358] RAX: 0000000000007c25 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9024.154360] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 9024.162430] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9024.166132] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 9024.169391] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 9024.173245] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 9024.179501] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9024.191147] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 9024.196462] Call Trace: [ 9024.205060] [ 9024.205855] ? show_regs.cold.9+0x22/0x2f [ 9024.207235] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9024.220477] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9024.232623] ? watchdog+0x40/0x40 [ 9024.243321] ? hrtimer_interrupt+0x128/0x2c0 [ 9024.259066] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9024.260982] ? apic_timer_interrupt+0xf/0x20 [ 9024.276086] [ 9024.278559] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9024.289110] do_raw_spin_lock+0x177/0x190 [ 9024.291348] _raw_spin_lock+0x21/0x30 [ 9024.293305] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9024.296352] ? do_raw_spin_unlock+0x75/0x190 [ 9024.298479] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9024.304638] ? do_wait_intr+0xf0/0xf0 [ 9024.308696] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9024.320127] mdc_close+0x53b/0xd70 [mdc] [ 9024.324384] lmv_close+0x18d/0x490 [lmv] [ 9024.330045] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9024.336720] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9024.341093] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9024.343518] ll_md_close+0x274/0x800 [lustre] [ 9024.348753] ll_file_release+0x128/0xc50 [lustre] [ 9024.354680] __fput+0x10d/0x440 [ 9024.356228] ____fput+0x12/0x20 [ 9024.357671] task_work_run+0xc6/0x110 [ 9024.359282] exit_to_usermode_loop+0x1dd/0x1f0 [ 9024.363122] do_syscall_64+0x3d6/0x440 [ 9024.364765] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9024.367130] RIP: 0033:0x7f151bee3cd8 [ 9024.368952] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9024.378186] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9024.382473] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 9024.387152] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 9024.391214] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9024.394743] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9024.398074] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9032.055328] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 9032.060833] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9032.067082] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 [ 9032.078696] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9032.078789] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9032.078801] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9032.078807] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9032.079188] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9032.079205] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9032.079209] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9032.079214] RAX: 0000000000007e91 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9032.079216] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 9032.079219] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 9032.079221] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 9032.079224] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 9032.079227] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 9032.079229] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9032.079231] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9032.079241] Call Trace: [ 9032.079246] [ 9032.079251] ? show_regs.cold.9+0x22/0x2f [ 9032.079267] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9032.079273] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9032.079278] ? watchdog+0x40/0x40 [ 9032.079282] ? hrtimer_interrupt+0x128/0x2c0 [ 9032.079286] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9032.079299] ? apic_timer_interrupt+0xf/0x20 [ 9032.079304] [ 9032.079307] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9032.079312] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9032.079406] do_raw_spin_lock+0x177/0x190 [ 9032.079411] _raw_spin_lock+0x21/0x30 [ 9032.079416] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9032.079837] process_one_work+0x2c8/0x700 [ 9032.079845] worker_thread+0x296/0x6e0 [ 9032.079933] ? rescuer_thread+0x570/0x570 [ 9032.079938] kthread+0x1d1/0x200 [ 9032.079942] ? set_kthread_struct+0x70/0x70 [ 9032.079946] ret_from_fork+0x1f/0x30 [ 9032.301627] auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9032.322768] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9032.328663] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9032.332665] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9032.335827] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9032.349097] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9032.352747] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 9032.355752] RDX: 000000000000365f RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 9032.359066] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 9032.362547] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 9032.366338] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9032.369849] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 9032.373361] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9032.376420] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9032.380724] Call Trace: [ 9032.382064] [ 9032.383141] ? show_regs.cold.9+0x22/0x2f [ 9032.385330] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9032.387072] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9032.388973] ? watchdog+0x40/0x40 [ 9032.390029] ? hrtimer_interrupt+0x128/0x2c0 [ 9032.391581] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9032.394115] ? apic_timer_interrupt+0xf/0x20 [ 9032.396462] [ 9032.397369] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9032.400713] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9032.403507] do_raw_spin_lock+0x177/0x190 [ 9032.405490] _raw_spin_lock+0x21/0x30 [ 9032.407546] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9032.410310] obd_mod_free+0x46/0x140 [ptlrpc] [ 9032.413014] mdc_commit_open+0x91/0xb0 [mdc] [ 9032.415288] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9032.418303] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9032.421493] after_reply+0xe2c/0x1780 [ptlrpc] [ 9032.424115] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9032.427715] ? __next_timer_interrupt+0x160/0x160 [ 9032.430325] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9032.432759] ? do_wait_intr+0xf0/0xf0 [ 9032.434597] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9032.437335] kthread+0x1d1/0x200 [ 9032.438895] ? set_kthread_struct+0x70/0x70 [ 9032.440901] ret_from_fork+0x1f/0x30 [ 9044.052062] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 9044.055917] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9044.096177] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9044.100885] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9044.107050] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9044.110643] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9044.122706] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9044.126796] RAX: 0000000000002788 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9044.130832] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 9044.135687] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 9044.139773] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 9044.149192] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 9044.155103] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 9044.159026] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9044.162052] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 9044.166694] Call Trace: [ 9044.167966] [ 9044.169218] ? show_regs.cold.9+0x22/0x2f [ 9044.171571] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9044.174587] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9044.180064] ? watchdog+0x40/0x40 [ 9044.189631] ? hrtimer_interrupt+0x128/0x2c0 [ 9044.192916] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9044.199710] ? apic_timer_interrupt+0xf/0x20 [ 9044.201922] [ 9044.205776] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9044.211493] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9044.215715] do_raw_spin_lock+0x177/0x190 [ 9044.221404] _raw_spin_lock+0x21/0x30 [ 9044.223502] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9044.230525] ? do_raw_spin_unlock+0x75/0x190 [ 9044.235574] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9044.254626] ? do_wait_intr+0xf0/0xf0 [ 9044.256519] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9044.263035] mdc_reint+0x2d/0x110 [mdc] [ 9044.271674] mdc_unlink+0x658/0x870 [mdc] [ 9044.276194] lmv_unlink+0x34d/0xbd0 [lmv] [ 9044.286806] ll_unlink+0x24d/0x7f0 [lustre] [ 9044.294928] vfs_unlink+0x202/0x330 [ 9044.304828] do_unlinkat+0x2fa/0x460 [ 9044.307981] __x64_sys_unlink+0x37/0x50 [ 9044.320622] do_syscall_64+0xc1/0x440 [ 9044.328635] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9044.337151] RIP: 0033:0x7f151bdee10b [ 9044.342704] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9044.373874] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9044.389966] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 9044.402950] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 9044.415930] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9044.428780] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9044.446347] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9049.061389] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7118s! [ 9049.070934] Showing busy workqueues and worker pools: [ 9049.076023] workqueue events: flags=0x0 [ 9049.079490] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9049.091356] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 9049.100526] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9049.107576] pending: kfree_rcu_monitor [ 9049.110125] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9049.114844] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 9049.120508] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9049.124244] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 9049.127842] workqueue events_long: flags=0x0 [ 9049.130119] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9049.133919] pending: lu_objects_flush [obdclass] [ 9049.140473] workqueue events_unbound: flags=0x2 [ 9049.146980] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9049.155439] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9049.170192] workqueue events_power_efficient: flags=0x80 [ 9049.179443] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9049.193575] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 9049.201925] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9049.208969] pending: neigh_periodic_work [ 9049.215033] workqueue mm_percpu_wq: flags=0x8 [ 9049.219829] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9049.225978] pending: vmstat_update, lru_add_drain_per_cpu [ 9049.229900] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9049.241590] pending: vmstat_update [ 9049.250048] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9049.263628] pending: lru_add_drain_per_cpu [ 9049.271477] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9049.278062] pending: lru_add_drain_per_cpu BAR(48) [ 9049.281507] workqueue writeback: flags=0x4a [ 9049.282969] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 9049.300609] pending: wb_workfn [ 9049.305604] workqueue xprtiod: flags=0xa [ 9049.311675] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9049.318754] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9049.328976] workqueue ipv6_addrconf: flags=0x40008 [ 9049.338686] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9049.346019] pending: addrconf_verify_work [ 9049.351456] workqueue knbd0-recv: flags=0x1a [ 9049.357552] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9049.360897] in-flight: 658:recv_work [ 9049.362995] workqueue ptlrpc_pinger: flags=0x2 [ 9049.365465] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9049.368713] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9049.372543] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7118s workers=4 idle: 32984 37 8 [ 9049.380223] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9034s workers=2 idle: 661 [ 9052.057544] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 9052.066671] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9052.120137] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9052.139180] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9052.154003] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9052.158051] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9052.184555] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9052.196479] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9052.212963] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 9052.237708] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9052.250983] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 9052.269662] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 9052.287276] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 9052.299618] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9052.308967] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 9052.319978] Call Trace: [ 9052.320992] [ 9052.336403] ? show_regs.cold.9+0x22/0x2f [ 9052.339591] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9052.351409] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9052.356914] ? watchdog+0x40/0x40 [ 9052.358192] ? hrtimer_interrupt+0x128/0x2c0 [ 9052.369726] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9052.378405] ? apic_timer_interrupt+0xf/0x20 [ 9052.394682] [ 9052.399517] ? native_safe_halt+0x2e/0x50 [ 9052.406535] kvm_wait+0xcd/0xe0 [ 9052.417625] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9052.425081] do_raw_spin_lock+0x177/0x190 [ 9052.426602] _raw_spin_lock+0x21/0x30 [ 9052.438127] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9052.456034] ? do_raw_spin_unlock+0x75/0x190 [ 9052.465751] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9052.479528] ? do_wait_intr+0xf0/0xf0 [ 9052.484646] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9052.490446] mdc_close+0x53b/0xd70 [mdc] [ 9052.493774] lmv_close+0x18d/0x490 [lmv] [ 9052.496918] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9052.499920] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9052.504066] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9052.509162] ll_md_close+0x274/0x800 [lustre] [ 9052.512519] ll_file_release+0x128/0xc50 [lustre] [ 9052.517747] __fput+0x10d/0x440 [ 9052.519082] ____fput+0x12/0x20 [ 9052.526702] task_work_run+0xc6/0x110 [ 9052.531640] exit_to_usermode_loop+0x1dd/0x1f0 [ 9052.539997] do_syscall_64+0x3d6/0x440 [ 9052.545778] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9052.557647] RIP: 0033:0x7f151bee3cd8 [ 9052.560499] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9052.582729] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9052.593753] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 9052.601505] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 9052.605766] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9052.612567] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9052.620095] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9060.055011] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 9060.056141] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9060.061044] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul [ 9060.064861] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9060.064940] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9060.064947] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9060.064953] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9060.065213] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9060.065222] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9060.065226] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9060.065230] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9060.065233] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 9060.065235] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 9060.065237] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 9060.065240] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 9060.065243] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 9060.065245] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9060.065248] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9060.066874] Call Trace: [ 9060.066884] [ 9060.066890] ? show_regs.cold.9+0x22/0x2f [ 9060.066903] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9060.066910] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9060.066913] ? watchdog+0x40/0x40 [ 9060.066919] ? hrtimer_interrupt+0x128/0x2c0 [ 9060.066923] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9060.066930] ? apic_timer_interrupt+0xf/0x20 [ 9060.066935] [ 9060.066939] ? native_safe_halt+0x2e/0x50 [ 9060.067414] kvm_wait+0xcd/0xe0 [ 9060.137608] crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9060.137639] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9060.137648] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9060.137651] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9060.137666] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9060.137670] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9060.137675] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 9060.137677] RDX: 0000000000000228 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 9060.137680] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 9060.137682] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 9060.137684] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9060.137687] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 9060.137690] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9060.137693] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9060.137701] Call Trace: [ 9060.137708] [ 9060.137715] ? show_regs.cold.9+0x22/0x2f [ 9060.137725] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9060.137731] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9060.137736] ? watchdog+0x40/0x40 [ 9060.137740] ? hrtimer_interrupt+0x128/0x2c0 [ 9060.137744] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9060.137752] ? apic_timer_interrupt+0xf/0x20 [ 9060.137757] [ 9060.137761] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9060.137766] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9060.137771] do_raw_spin_lock+0x177/0x190 [ 9060.137775] _raw_spin_lock+0x21/0x30 [ 9060.137783] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9060.138075] obd_mod_free+0x46/0x140 [ptlrpc] [ 9060.209085] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9060.209117] do_raw_spin_lock+0x177/0x190 [ 9060.209122] _raw_spin_lock+0x21/0x30 [ 9060.209133] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9060.212453] process_one_work+0x2c8/0x700 [ 9060.212465] worker_thread+0x296/0x6e0 [ 9060.212477] ? rescuer_thread+0x570/0x570 [ 9060.212482] kthread+0x1d1/0x200 [ 9060.212486] ? set_kthread_struct+0x70/0x70 [ 9060.212490] ret_from_fork+0x1f/0x30 [ 9060.879842] mdc_commit_open+0x91/0xb0 [mdc] [ 9060.891251] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9060.895496] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9060.905710] after_reply+0xe2c/0x1780 [ptlrpc] [ 9060.917203] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9060.924036] ? __next_timer_interrupt+0x160/0x160 [ 9060.933815] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9060.945703] ? do_wait_intr+0xf0/0xf0 [ 9060.953114] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9060.961749] kthread+0x1d1/0x200 [ 9060.968449] ? set_kthread_struct+0x70/0x70 [ 9060.976114] ret_from_fork+0x1f/0x30 [ 9076.054571] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 9076.070427] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9076.142851] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9076.161908] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9076.179193] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9076.183346] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9076.203047] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9076.216419] RAX: 000000000000247b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9076.225911] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 9076.234179] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 9076.242435] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 9076.249660] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 9076.257169] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 9076.264605] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9076.274531] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 9076.280798] Call Trace: [ 9076.282220] [ 9076.283481] ? show_regs.cold.9+0x22/0x2f [ 9076.285772] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9076.287610] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9076.289267] ? watchdog+0x40/0x40 [ 9076.291222] ? hrtimer_interrupt+0x128/0x2c0 [ 9076.293694] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9076.297516] ? apic_timer_interrupt+0xf/0x20 [ 9076.299738] [ 9076.300688] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9076.306369] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9076.310878] do_raw_spin_lock+0x177/0x190 [ 9076.315145] _raw_spin_lock+0x21/0x30 [ 9076.323552] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9076.330194] ? do_raw_spin_unlock+0x75/0x190 [ 9076.339369] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9076.343521] ? do_wait_intr+0xf0/0xf0 [ 9076.350238] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9076.360067] mdc_reint+0x2d/0x110 [mdc] [ 9076.362692] mdc_unlink+0x658/0x870 [mdc] [ 9076.367605] lmv_unlink+0x34d/0xbd0 [lmv] [ 9076.370976] ll_unlink+0x24d/0x7f0 [lustre] [ 9076.380654] vfs_unlink+0x202/0x330 [ 9076.384856] do_unlinkat+0x2fa/0x460 [ 9076.385898] __x64_sys_unlink+0x37/0x50 [ 9076.390226] do_syscall_64+0xc1/0x440 [ 9076.393603] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9076.399913] RIP: 0033:0x7f151bdee10b [ 9076.404499] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9076.429455] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9076.440306] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 9076.446455] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 9076.457381] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9076.464777] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9076.469511] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9079.778430] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7149s! [ 9079.783930] Showing busy workqueues and worker pools: [ 9079.786299] workqueue events: flags=0x0 [ 9079.788420] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9079.801596] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 9079.814705] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9079.819836] pending: kfree_rcu_monitor [ 9079.824791] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9079.836484] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 9079.851567] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9079.861302] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 9079.870088] workqueue events_long: flags=0x0 [ 9079.872199] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9079.876480] pending: lu_objects_flush [obdclass] [ 9079.879452] workqueue events_unbound: flags=0x2 [ 9079.883569] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9079.887706] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9079.896476] workqueue events_power_efficient: flags=0x80 [ 9079.899746] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9079.904372] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 9079.906693] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9079.914085] pending: neigh_periodic_work [ 9079.918552] workqueue mm_percpu_wq: flags=0x8 [ 9079.922571] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9079.932573] pending: vmstat_update, lru_add_drain_per_cpu [ 9079.936387] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9079.947519] pending: vmstat_update [ 9079.951536] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9079.958437] pending: lru_add_drain_per_cpu [ 9079.967464] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9079.982641] pending: lru_add_drain_per_cpu BAR(48) [ 9079.985632] workqueue writeback: flags=0x4a [ 9079.992679] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 9080.002294] pending: wb_workfn [ 9080.004426] workqueue xprtiod: flags=0xa [ 9080.006431] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9080.010277] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9080.014602] workqueue ipv6_addrconf: flags=0x40008 [ 9080.021414] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9080.023867] pending: addrconf_verify_work [ 9080.032798] workqueue knbd0-recv: flags=0x1a [ 9080.038044] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9080.047476] in-flight: 658:recv_work [ 9080.050120] workqueue ptlrpc_pinger: flags=0x2 [ 9080.053825] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9080.057061] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 9080.060164] in-flight: [ 9080.064642] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9080.064787] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9080.064795] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9080.064798] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9080.064812] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9080.064816] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9080.064821] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9080.064823] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 9080.064825] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9080.064827] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 9080.064830] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 9080.064833] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 9080.064836] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9080.064838] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 9080.064847] Call Trace: [ 9080.064853] [ 9080.064858] ? show_regs.cold.9+0x22/0x2f [ 9080.064868] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9080.064874] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9080.064879] ? watchdog+0x40/0x40 [ 9080.064884] ? hrtimer_interrupt+0x128/0x2c0 [ 9080.064888] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9080.064895] ? apic_timer_interrupt+0xf/0x20 [ 9080.064899] [ 9080.064902] ? native_safe_halt+0x2e/0x50 [ 9080.064906] kvm_wait+0xcd/0xe0 [ 9080.064915] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9080.064924] do_raw_spin_lock+0x177/0x190 [ 9080.070657] 38:ptlrpc_pinger_main [ptlrpc] [ 9080.071113] pool 8: [ 9080.096405] _raw_spin_lock+0x21/0x30 [ 9080.096430] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9080.096842] ? do_raw_spin_unlock+0x75/0x190 [ 9080.096851] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9080.097393] ? do_wait_intr+0xf0/0xf0 [ 9080.097406] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9080.097827] mdc_close+0x53b/0xd70 [mdc] [ 9080.097883] lmv_close+0x18d/0x490 [lmv] [ 9080.097913] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9080.111405] cpus=0-3 flags=0x4 nice=0 hung=7149s workers=4 idle: 32984 37 8 [ 9080.111457] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9064s workers=2 idle: 661 [ 9080.215774] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9080.218660] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9080.220454] ll_md_close+0x274/0x800 [lustre] [ 9080.221697] ll_file_release+0x128/0xc50 [lustre] [ 9080.223366] __fput+0x10d/0x440 [ 9080.224524] ____fput+0x12/0x20 [ 9080.225694] task_work_run+0xc6/0x110 [ 9080.227326] exit_to_usermode_loop+0x1dd/0x1f0 [ 9080.229351] do_syscall_64+0x3d6/0x440 [ 9080.231327] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9080.233578] RIP: 0033:0x7f151bee3cd8 [ 9080.235329] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9080.244510] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9080.248479] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 9080.252294] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 9080.255680] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9080.259403] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9080.262562] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9088.055067] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 9088.057491] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9088.059518] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9088.076604] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9088.076619] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9088.076622] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9088.076637] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9088.076642] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9088.076648] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 9088.076651] RDX: 000000000000375e RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 9088.076654] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 9088.076656] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 9088.076659] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9088.076662] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 9088.076665] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9088.076667] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9088.076677] Call Trace: [ 9088.076683] [ 9088.076690] ? show_regs.cold.9+0x22/0x2f [ 9088.076700] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9088.076708] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9088.076713] ? watchdog+0x40/0x40 [ 9088.076718] ? hrtimer_interrupt+0x128/0x2c0 [ 9088.076723] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9088.076733] ? apic_timer_interrupt+0xf/0x20 [ 9088.076739] [ 9088.076742] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9088.076748] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9088.076754] do_raw_spin_lock+0x177/0x190 [ 9088.076759] _raw_spin_lock+0x21/0x30 [ 9088.076767] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9088.077178] obd_mod_free+0x46/0x140 [ptlrpc] [ 9088.077445] mdc_commit_open+0x91/0xb0 [mdc] [ 9088.077493] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9088.077753] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9088.078017] after_reply+0xe2c/0x1780 [ptlrpc] [ 9088.078292] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9088.078566] ? __next_timer_interrupt+0x160/0x160 [ 9088.078578] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9088.078775] ? do_wait_intr+0xf0/0xf0 [ 9088.078782] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9088.079100] kthread+0x1d1/0x200 [ 9088.079107] ? set_kthread_struct+0x70/0x70 [ 9088.079112] ret_from_fork+0x1f/0x30 [ 9088.207185] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9088.229368] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9088.233449] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9088.237071] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9088.240052] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9088.242176] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9088.249288] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9088.252009] RAX: 0000000000007e88 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9088.255111] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 9088.258218] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 9088.261527] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 9088.264232] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 9088.267609] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 9088.270858] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9088.273853] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9088.280307] Call Trace: [ 9088.281457] [ 9088.283588] ? show_regs.cold.9+0x22/0x2f [ 9088.285126] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9088.288652] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9088.290336] ? watchdog+0x40/0x40 [ 9088.293294] ? hrtimer_interrupt+0x128/0x2c0 [ 9088.296010] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9088.299126] ? apic_timer_interrupt+0xf/0x20 [ 9088.300765] [ 9088.302668] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9088.308006] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9088.311332] do_raw_spin_lock+0x177/0x190 [ 9088.313311] _raw_spin_lock+0x21/0x30 [ 9088.315215] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9088.317705] process_one_work+0x2c8/0x700 [ 9088.319642] worker_thread+0x296/0x6e0 [ 9088.321504] ? rescuer_thread+0x570/0x570 [ 9088.323390] kthread+0x1d1/0x200 [ 9088.325494] ? set_kthread_struct+0x70/0x70 [ 9088.328188] ret_from_fork+0x1f/0x30 [ 9108.052046] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 9108.055166] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9108.060352] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 9108.073817] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9108.073859] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9108.073862] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9108.073916] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9108.073920] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9108.073925] RAX: 0000000000007e91 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9108.073928] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 9108.073930] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 9108.073932] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 9108.073934] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 9108.073938] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 9108.073941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9108.073943] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 9108.073952] Call Trace: [ 9108.073973] [ 9108.073979] ? show_regs.cold.9+0x22/0x2f [ 9108.073989] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9108.073995] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9108.073999] ? watchdog+0x40/0x40 [ 9108.074003] ? hrtimer_interrupt+0x128/0x2c0 [ 9108.074007] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9108.074014] ? apic_timer_interrupt+0xf/0x20 [ 9108.074019] [ 9108.074021] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9108.074025] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9108.074030] do_raw_spin_lock+0x177/0x190 [ 9108.074035] _raw_spin_lock+0x21/0x30 [ 9108.074042] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9108.074327] ? do_raw_spin_unlock+0x75/0x190 [ 9108.074334] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9108.074528] ? do_wait_intr+0xf0/0xf0 [ 9108.074535] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9108.074725] mdc_reint+0x2d/0x110 [mdc] [ 9108.074766] mdc_unlink+0x658/0x870 [mdc] [ 9108.074790] lmv_unlink+0x34d/0xbd0 [lmv] [ 9108.074821] ll_unlink+0x24d/0x7f0 [lustre] [ 9108.079284] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9108.079359] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9108.079366] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9108.079369] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9108.079382] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9108.079386] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9108.079390] RAX: 000000000000434f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9108.079392] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 9108.079394] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9108.079397] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 9108.079400] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 9108.079403] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 9108.079405] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9108.079408] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 9108.079417] Call Trace: [ 9108.079422] [ 9108.079428] ? show_regs.cold.9+0x22/0x2f [ 9108.079437] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9108.079443] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9108.079447] ? watchdog+0x40/0x40 [ 9108.079452] ? hrtimer_interrupt+0x128/0x2c0 [ 9108.079456] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9108.079465] ? apic_timer_interrupt+0xf/0x20 [ 9108.079469] [ 9108.079472] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9108.079480] do_raw_spin_lock+0x177/0x190 [ 9108.079484] _raw_spin_lock+0x21/0x30 [ 9108.079489] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9108.079909] ? do_raw_spin_unlock+0x75/0x190 [ 9108.079916] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9108.080123] ? do_wait_intr+0xf0/0xf0 [ 9108.080131] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9108.080324] mdc_close+0x53b/0xd70 [mdc] [ 9108.080364] lmv_close+0x18d/0x490 [lmv] [ 9108.080390] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9108.080573] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9108.080691] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9108.081199] ll_md_close+0x274/0x800 [lustre] [ 9108.081290] ll_file_release+0x128/0xc50 [lustre] [ 9108.081381] __fput+0x10d/0x440 [ 9108.081390] ____fput+0x12/0x20 [ 9108.081394] task_work_run+0xc6/0x110 [ 9108.081402] exit_to_usermode_loop+0x1dd/0x1f0 [ 9108.081409] do_syscall_64+0x3d6/0x440 [ 9108.081413] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9108.081422] RIP: 0033:0x7f151bee3cd8 [ 9108.081479] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9108.081485] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9108.081491] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 9108.081493] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 9108.081495] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9108.081497] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9108.081499] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9108.411413] vfs_unlink+0x202/0x330 [ 9108.414353] do_unlinkat+0x2fa/0x460 [ 9108.415682] __x64_sys_unlink+0x37/0x50 [ 9108.417366] do_syscall_64+0xc1/0x440 [ 9108.422267] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9108.426962] RIP: 0033:0x7f151bdee10b [ 9108.428643] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9108.441435] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9108.446191] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 9108.449427] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 9108.452551] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9108.459675] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9108.462579] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9110.497056] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7179s! [ 9110.500865] Showing busy workqueues and worker pools: [ 9110.503042] workqueue events: flags=0x0 [ 9110.506375] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9110.509432] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 9110.514606] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9110.518020] pending: kfree_rcu_monitor [ 9110.519987] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9110.522824] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 9110.528024] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9110.532088] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 9110.537607] workqueue events_long: flags=0x0 [ 9110.540406] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9110.544349] pending: lu_objects_flush [obdclass] [ 9110.547197] workqueue events_unbound: flags=0x2 [ 9110.549515] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9110.552894] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9110.562751] workqueue events_power_efficient: flags=0x80 [ 9110.567932] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9110.576193] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 9110.587575] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9110.596024] pending: neigh_periodic_work [ 9110.599846] workqueue mm_percpu_wq: flags=0x8 [ 9110.601793] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9110.608364] pending: vmstat_update, lru_add_drain_per_cpu [ 9110.620789] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9110.626071] pending: vmstat_update [ 9110.630622] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9110.635061] pending: lru_add_drain_per_cpu [ 9110.642678] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9110.664603] pending: lru_add_drain_per_cpu BAR(48) [ 9110.669495] workqueue writeback: flags=0x4a [ 9110.671398] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 9110.677615] pending: wb_workfn [ 9110.681659] workqueue xprtiod: flags=0xa [ 9110.686111] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9110.691987] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9110.694668] workqueue ipv6_addrconf: flags=0x40008 [ 9110.697379] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9110.700637] pending: addrconf_verify_work [ 9110.703875] workqueue knbd0-recv: flags=0x1a [ 9110.706467] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9110.713142] in-flight: 658:recv_work [ 9110.715120] workqueue ptlrpc_pinger: flags=0x2 [ 9110.716929] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9110.719620] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9110.723012] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7180s workers=4 idle: 32984 37 8 [ 9110.726349] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9095s workers=2 idle: 661 [ 9116.056014] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9116.056803] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 9116.059254] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9116.059332] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9116.059340] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9116.059347] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9116.059647] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9116.059658] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9116.059661] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9116.059665] RAX: 0000000000002d8a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9116.059668] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 9116.059670] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 9116.059673] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 9116.059675] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 9116.059679] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 9116.059681] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9116.059684] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9116.059694] Call Trace: [ 9116.059699] [ 9116.059705] ? show_regs.cold.9+0x22/0x2f [ 9116.059714] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9116.059721] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9116.059726] ? watchdog+0x40/0x40 [ 9116.059730] ? hrtimer_interrupt+0x128/0x2c0 [ 9116.059734] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9116.069611] Modules linked in: [ 9116.121853] ? apic_timer_interrupt+0xf/0x20 [ 9116.121878] [ 9116.121883] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9116.121891] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9116.121900] do_raw_spin_lock+0x177/0x190 [ 9116.121907] _raw_spin_lock+0x21/0x30 [ 9116.121912] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9116.123053] process_one_work+0x2c8/0x700 [ 9116.123062] worker_thread+0x296/0x6e0 [ 9116.123067] ? rescuer_thread+0x570/0x570 [ 9116.123071] kthread+0x1d1/0x200 [ 9116.123075] ? set_kthread_struct+0x70/0x70 [ 9116.123078] ret_from_fork+0x1f/0x30 [ 9116.371020] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9116.403854] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9116.408934] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9116.419513] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9116.421493] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9116.435852] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9116.444284] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 9116.452038] RDX: 000000000000061d RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 9116.456109] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 9116.462931] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 9116.469881] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9116.479093] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 9116.483075] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9116.494287] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9116.496774] Call Trace: [ 9116.497710] [ 9116.499678] ? show_regs.cold.9+0x22/0x2f [ 9116.501572] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9116.506415] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9116.508366] ? watchdog+0x40/0x40 [ 9116.513973] ? hrtimer_interrupt+0x128/0x2c0 [ 9116.519596] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9116.527052] ? apic_timer_interrupt+0xf/0x20 [ 9116.532018] [ 9116.534035] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9116.541551] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9116.543566] do_raw_spin_lock+0x177/0x190 [ 9116.550864] _raw_spin_lock+0x21/0x30 [ 9116.552737] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9116.555192] obd_mod_free+0x46/0x140 [ptlrpc] [ 9116.564319] mdc_commit_open+0x91/0xb0 [mdc] [ 9116.568195] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9116.576195] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9116.583525] after_reply+0xe2c/0x1780 [ptlrpc] [ 9116.586887] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9116.599102] ? __next_timer_interrupt+0x160/0x160 [ 9116.605994] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9116.610057] ? do_wait_intr+0xf0/0xf0 [ 9116.614254] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9116.627737] kthread+0x1d1/0x200 [ 9116.632622] ? set_kthread_struct+0x70/0x70 [ 9116.639377] ret_from_fork+0x1f/0x30 [ 9136.056190] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 9136.057395] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 9136.061284] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) [ 9136.067319] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9136.067412] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9136.067420] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9136.067424] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9136.067452] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9136.067456] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9136.067461] RAX: 0000000000007d8e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9136.067464] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 9136.067467] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9136.067469] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 9136.067472] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 9136.067480] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 9136.067483] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9136.067492] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 9136.067500] Call Trace: [ 9136.067505] [ 9136.067515] ? show_regs.cold.9+0x22/0x2f [ 9136.067527] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9136.067533] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9136.067537] ? watchdog+0x40/0x40 [ 9136.067541] ? hrtimer_interrupt+0x128/0x2c0 [ 9136.067550] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9136.067568] ? apic_timer_interrupt+0xf/0x20 [ 9136.067576] [ 9136.067579] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9136.067591] do_raw_spin_lock+0x177/0x190 [ 9136.067599] _raw_spin_lock+0x21/0x30 [ 9136.067604] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9136.067942] ? do_raw_spin_unlock+0x75/0x190 [ 9136.067949] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9136.068205] ? do_wait_intr+0xf0/0xf0 [ 9136.068220] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9136.068432] mdc_close+0x53b/0xd70 [mdc] [ 9136.068471] lmv_close+0x18d/0x490 [lmv] [ 9136.068502] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9136.068710] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9136.068935] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9136.069037] ll_md_close+0x274/0x800 [lustre] [ 9136.069144] ll_file_release+0x128/0xc50 [lustre] [ 9136.069250] __fput+0x10d/0x440 [ 9136.069260] ____fput+0x12/0x20 [ 9136.069264] task_work_run+0xc6/0x110 [ 9136.069277] exit_to_usermode_loop+0x1dd/0x1f0 [ 9136.069284] do_syscall_64+0x3d6/0x440 [ 9136.069289] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9136.069296] RIP: 0033:0x7f151bee3cd8 [ 9136.069304] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9136.069308] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9136.069313] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 9136.069315] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 9136.069317] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9136.069319] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9136.069322] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9136.491505] ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9136.515312] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9136.521684] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9136.527910] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9136.531516] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9136.545617] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9136.549896] RAX: 0000000000007a30 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9136.552948] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 9136.558610] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 9136.572994] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 9136.582566] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 9136.594059] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 9136.607391] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9136.612389] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 9136.622764] Call Trace: [ 9136.625355] [ 9136.626232] ? show_regs.cold.9+0x22/0x2f [ 9136.631746] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9136.633020] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9136.634175] ? watchdog+0x40/0x40 [ 9136.638790] ? hrtimer_interrupt+0x128/0x2c0 [ 9136.640403] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9136.644802] ? apic_timer_interrupt+0xf/0x20 [ 9136.650357] [ 9136.653414] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9136.659900] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9136.662120] do_raw_spin_lock+0x177/0x190 [ 9136.663540] _raw_spin_lock+0x21/0x30 [ 9136.666720] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9136.670100] ? do_raw_spin_unlock+0x75/0x190 [ 9136.672750] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9136.677055] ? do_wait_intr+0xf0/0xf0 [ 9136.678451] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9136.683054] mdc_reint+0x2d/0x110 [mdc] [ 9136.687653] mdc_unlink+0x658/0x870 [mdc] [ 9136.690559] lmv_unlink+0x34d/0xbd0 [lmv] [ 9136.692534] ll_unlink+0x24d/0x7f0 [lustre] [ 9136.695769] vfs_unlink+0x202/0x330 [ 9136.697139] do_unlinkat+0x2fa/0x460 [ 9136.699157] __x64_sys_unlink+0x37/0x50 [ 9136.703619] do_syscall_64+0xc1/0x440 [ 9136.705778] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9136.707858] RIP: 0033:0x7f151bdee10b [ 9136.709097] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9136.730466] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9136.743565] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 9136.756309] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 9136.766435] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9136.770743] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9136.774998] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9141.217071] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7210s! [ 9141.220548] Showing busy workqueues and worker pools: [ 9141.222865] workqueue events: flags=0x0 [ 9141.224824] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9141.227867] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 9141.232168] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9141.235611] pending: kfree_rcu_monitor [ 9141.237670] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9141.240872] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 9141.245230] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9141.248529] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 9141.251910] workqueue events_long: flags=0x0 [ 9141.253901] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9141.257269] pending: lu_objects_flush [obdclass] [ 9141.259924] workqueue events_unbound: flags=0x2 [ 9141.262237] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9141.265296] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9141.270075] workqueue events_power_efficient: flags=0x80 [ 9141.272719] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9141.275727] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 9141.278985] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9141.282114] pending: neigh_periodic_work [ 9141.284118] workqueue mm_percpu_wq: flags=0x8 [ 9141.286217] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9141.289478] pending: vmstat_update, lru_add_drain_per_cpu [ 9141.292099] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9141.295207] pending: vmstat_update [ 9141.297094] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9141.300258] pending: lru_add_drain_per_cpu [ 9141.302437] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9141.305667] pending: lru_add_drain_per_cpu BAR(48) [ 9141.307896] workqueue writeback: flags=0x4a [ 9141.309807] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 9141.312742] pending: wb_workfn [ 9141.314308] workqueue xprtiod: flags=0xa [ 9141.316223] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9141.319187] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9141.323214] workqueue ipv6_addrconf: flags=0x40008 [ 9141.325806] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9141.328768] pending: addrconf_verify_work [ 9141.330940] workqueue knbd0-recv: flags=0x1a [ 9141.333074] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9141.336272] in-flight: 658:recv_work [ 9141.337991] workqueue ptlrpc_pinger: flags=0x2 [ 9141.340115] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9141.344016] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9141.347106] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7210s workers=4 idle: 32984 37 8 [ 9141.350908] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9126s workers=2 idle: 661 [ 9144.055019] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 9144.057353] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl [ 9144.057400] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 9144.057405] i2c_piix4 pcspkr squashfs [ 9144.095616] Modules linked in: [ 9144.097998] ata_generic [ 9144.100489] lustre(O) [ 9144.101772] crct10dif_pclmul [ 9144.102688] osp(O) [ 9144.103421] crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9144.103443] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9144.103447] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9144.103449] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9144.103462] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9144.103465] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9144.103468] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 9144.103470] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 9144.103471] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 9144.103473] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 9144.103475] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9144.103488] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 9144.103491] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9144.103494] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9144.103501] Call Trace: [ 9144.103507] [ 9144.103512] ? show_regs.cold.9+0x22/0x2f [ 9144.103523] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9144.103529] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9144.103533] ? watchdog+0x40/0x40 [ 9144.103538] ? hrtimer_interrupt+0x128/0x2c0 [ 9144.103542] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9144.103548] ? apic_timer_interrupt+0xf/0x20 [ 9144.103552] [ 9144.103555] ? native_safe_halt+0x2e/0x50 [ 9144.103559] kvm_wait+0xcd/0xe0 [ 9144.103565] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9144.103573] do_raw_spin_lock+0x177/0x190 [ 9144.103577] _raw_spin_lock+0x21/0x30 [ 9144.103583] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9144.103861] obd_mod_free+0x46/0x140 [ptlrpc] [ 9144.104212] mdc_commit_open+0x91/0xb0 [mdc] [ 9144.104251] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9144.104443] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9144.104647] after_reply+0xe2c/0x1780 [ptlrpc] [ 9144.104844] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9144.104989] ? __next_timer_interrupt+0x160/0x160 [ 9144.104996] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9144.105211] ? do_wait_intr+0xf0/0xf0 [ 9144.105218] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9144.105349] kthread+0x1d1/0x200 [ 9144.105356] ? set_kthread_struct+0x70/0x70 [ 9144.105359] ret_from_fork+0x1f/0x30 [ 9144.297749] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9144.329953] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9144.340546] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9144.349859] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9144.355946] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9144.358790] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9144.366941] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9144.369827] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9144.372874] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 9144.376667] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 9144.379950] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 9144.383046] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 9144.385916] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 9144.389069] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9144.391201] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9144.400760] Call Trace: [ 9144.402449] [ 9144.404320] ? show_regs.cold.9+0x22/0x2f [ 9144.407717] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9144.414835] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9144.418822] ? watchdog+0x40/0x40 [ 9144.423527] ? hrtimer_interrupt+0x128/0x2c0 [ 9144.425587] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9144.426968] ? apic_timer_interrupt+0xf/0x20 [ 9144.428764] [ 9144.429407] ? native_safe_halt+0x2e/0x50 [ 9144.430694] kvm_wait+0xcd/0xe0 [ 9144.436042] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9144.438448] do_raw_spin_lock+0x177/0x190 [ 9144.443580] _raw_spin_lock+0x21/0x30 [ 9144.450498] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9144.457633] process_one_work+0x2c8/0x700 [ 9144.462847] worker_thread+0x296/0x6e0 [ 9144.466672] ? rescuer_thread+0x570/0x570 [ 9144.468856] kthread+0x1d1/0x200 [ 9144.473889] ? set_kthread_struct+0x70/0x70 [ 9144.477736] ret_from_fork+0x1f/0x30 [ 9164.058588] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 9164.065892] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9164.092971] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9164.100374] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9164.103848] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9164.106051] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9164.125809] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9164.129850] RAX: 0000000000004b35 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9164.132863] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 9164.139655] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9164.146108] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 9164.149942] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 9164.156593] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 9164.160993] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9164.163444] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 9164.171229] Call Trace: [ 9164.171924] [ 9164.173769] ? show_regs.cold.9+0x22/0x2f [ 9164.180011] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9164.181510] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9164.182767] ? watchdog+0x40/0x40 [ 9164.185921] ? hrtimer_interrupt+0x128/0x2c0 [ 9164.189129] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9164.194621] ? apic_timer_interrupt+0xf/0x20 [ 9164.197031] [ 9164.198637] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9164.201341] do_raw_spin_lock+0x177/0x190 [ 9164.202819] _raw_spin_lock+0x21/0x30 [ 9164.206620] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9164.209450] ? do_raw_spin_unlock+0x75/0x190 [ 9164.215563] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9164.224104] ? do_wait_intr+0xf0/0xf0 [ 9164.228897] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9164.235570] mdc_close+0x53b/0xd70 [mdc] [ 9164.240925] lmv_close+0x18d/0x490 [lmv] [ 9164.242275] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9164.244084] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9164.255837] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9164.259466] ll_md_close+0x274/0x800 [lustre] [ 9164.267636] ll_file_release+0x128/0xc50 [lustre] [ 9164.272092] __fput+0x10d/0x440 [ 9164.274974] ____fput+0x12/0x20 [ 9164.276296] task_work_run+0xc6/0x110 [ 9164.277815] exit_to_usermode_loop+0x1dd/0x1f0 [ 9164.280636] do_syscall_64+0x3d6/0x440 [ 9164.281993] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9164.284527] RIP: 0033:0x7f151bee3cd8 [ 9164.286037] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9164.293552] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9164.296353] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 9164.299044] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 9164.301379] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9164.304025] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9164.307063] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9168.053950] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 9168.059943] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9168.104361] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9168.122971] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9168.138368] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9168.144268] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9168.165416] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9168.172822] RAX: 0000000000007f8c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9168.176446] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 9168.180050] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 9168.183702] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 9168.187235] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 9168.192899] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 9168.197988] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9168.204369] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 9168.213547] Call Trace: [ 9168.215929] [ 9168.218012] ? show_regs.cold.9+0x22/0x2f [ 9168.222960] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9168.229405] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9168.233742] ? watchdog+0x40/0x40 [ 9168.240302] ? hrtimer_interrupt+0x128/0x2c0 [ 9168.244222] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9168.249818] ? apic_timer_interrupt+0xf/0x20 [ 9168.252853] [ 9168.256048] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9168.260019] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9168.267855] do_raw_spin_lock+0x177/0x190 [ 9168.271465] _raw_spin_lock+0x21/0x30 [ 9168.274543] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9168.280007] ? do_raw_spin_unlock+0x75/0x190 [ 9168.287757] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9168.290044] ? do_wait_intr+0xf0/0xf0 [ 9168.291681] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9168.294065] mdc_reint+0x2d/0x110 [mdc] [ 9168.295777] mdc_unlink+0x658/0x870 [mdc] [ 9168.296896] lmv_unlink+0x34d/0xbd0 [lmv] [ 9168.298202] ll_unlink+0x24d/0x7f0 [lustre] [ 9168.300359] vfs_unlink+0x202/0x330 [ 9168.302032] do_unlinkat+0x2fa/0x460 [ 9168.303874] __x64_sys_unlink+0x37/0x50 [ 9168.305823] do_syscall_64+0xc1/0x440 [ 9168.307216] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9168.309951] RIP: 0033:0x7f151bdee10b [ 9168.311730] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9168.320841] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9168.325043] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 9168.328581] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 9168.331996] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9168.336188] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9168.340202] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9171.425209] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7240s! [ 9171.431173] Showing busy workqueues and worker pools: [ 9171.436807] workqueue events: flags=0x0 [ 9171.439389] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9171.451385] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 9171.460757] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9171.464894] pending: kfree_rcu_monitor [ 9171.467327] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9171.470606] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 9171.475631] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9171.479805] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 9171.485309] workqueue events_long: flags=0x0 [ 9171.488344] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9171.493615] pending: lu_objects_flush [obdclass] [ 9171.497480] workqueue events_unbound: flags=0x2 [ 9171.505788] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9171.513383] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9171.523070] workqueue events_power_efficient: flags=0x80 [ 9171.529451] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9171.536754] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 9171.539426] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9171.554904] pending: neigh_periodic_work [ 9171.556441] workqueue mm_percpu_wq: flags=0x8 [ 9171.557975] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9171.567015] pending: vmstat_update, lru_add_drain_per_cpu [ 9171.571057] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9171.575460] pending: vmstat_update [ 9171.579019] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9171.585315] pending: lru_add_drain_per_cpu [ 9171.589013] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9171.593478] pending: lru_add_drain_per_cpu BAR(48) [ 9171.598379] workqueue writeback: flags=0x4a [ 9171.603433] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 9171.613022] pending: wb_workfn [ 9171.616258] workqueue xprtiod: flags=0xa [ 9171.618946] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9171.622432] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9171.630990] workqueue ipv6_addrconf: flags=0x40008 [ 9171.633052] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9171.636660] pending: addrconf_verify_work [ 9171.639466] workqueue knbd0-recv: flags=0x1a [ 9171.641886] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9171.649313] in-flight: 658:recv_work [ 9171.653040] workqueue ptlrpc_pinger: flags=0x2 [ 9171.655441] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9171.658508] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9171.666327] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7241s workers=4 idle: 32984 37 8 [ 9171.673604] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9156s workers=2 idle: 661 [ 9172.055040] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 9172.057584] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 9172.058584] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9172.058969] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9172.058977] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9172.058980] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9172.058994] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9172.058998] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9172.059003] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 9172.059006] RDX: 0000000000001bc3 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 9172.059008] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 9172.059011] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 9172.059013] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9172.059017] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 9172.059020] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9172.059022] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9172.059030] Call Trace: [ 9172.059035] [ 9172.059041] ? show_regs.cold.9+0x22/0x2f [ 9172.059051] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9172.059057] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9172.059061] ? watchdog+0x40/0x40 [ 9172.059064] ? hrtimer_interrupt+0x128/0x2c0 [ 9172.059067] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9172.059074] ? apic_timer_interrupt+0xf/0x20 [ 9172.059077] [ 9172.059079] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9172.059082] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9172.059085] do_raw_spin_lock+0x177/0x190 [ 9172.059089] _raw_spin_lock+0x21/0x30 [ 9172.059093] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9172.059414] obd_mod_free+0x46/0x140 [ptlrpc] [ 9172.059594] mdc_commit_open+0x91/0xb0 [mdc] [ 9172.059628] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9172.059945] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9172.060287] after_reply+0xe2c/0x1780 [ptlrpc] [ 9172.060559] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9172.060828] ? __next_timer_interrupt+0x160/0x160 [ 9172.060838] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9172.061112] ? do_wait_intr+0xf0/0xf0 [ 9172.061120] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9172.062444] kthread+0x1d1/0x200 [ 9172.062451] ? set_kthread_struct+0x70/0x70 [ 9172.062456] ret_from_fork+0x1f/0x30 [ 9172.217451] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9172.240836] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9172.246409] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9172.250662] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9172.255170] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9172.258104] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9172.267929] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9172.272276] RAX: 000000000000144c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9172.276354] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 9172.280398] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 9172.284446] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 9172.288579] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 9172.294465] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 9172.299877] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9172.303759] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9172.308957] Call Trace: [ 9172.310339] [ 9172.311753] ? show_regs.cold.9+0x22/0x2f [ 9172.314193] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9172.316891] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9172.319518] ? watchdog+0x40/0x40 [ 9172.322074] ? hrtimer_interrupt+0x128/0x2c0 [ 9172.324143] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9172.326851] ? apic_timer_interrupt+0xf/0x20 [ 9172.329427] [ 9172.330895] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9172.333879] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9172.336785] do_raw_spin_lock+0x177/0x190 [ 9172.339819] _raw_spin_lock+0x21/0x30 [ 9172.341855] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9172.345916] process_one_work+0x2c8/0x700 [ 9172.348913] worker_thread+0x296/0x6e0 [ 9172.351322] ? rescuer_thread+0x570/0x570 [ 9172.353551] kthread+0x1d1/0x200 [ 9172.356556] ? set_kthread_struct+0x70/0x70 [ 9172.359005] ret_from_fork+0x1f/0x30 [ 9184.798031] rcu: INFO: rcu_sched self-detected stall on CPU [ 9184.800210] rcu: 1-...!: (4122118 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 9184.803792] (t=7260151 jiffies g=249657 q=4476) [ 9184.805695] rcu: rcu_sched kthread starved for 7255103 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 9184.809763] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 9184.813713] rcu: RCU grace-period kthread stack dump: [ 9184.815842] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 9184.819518] Call Trace: [ 9184.820562] __schedule+0x351/0xcb0 [ 9184.822069] schedule+0xc0/0x180 [ 9184.823424] schedule_timeout+0xb4/0x190 [ 9184.825103] ? do_raw_spin_unlock+0x75/0x190 [ 9184.826842] ? __next_timer_interrupt+0x160/0x160 [ 9184.828621] ? prepare_to_swait_event+0xf3/0x140 [ 9184.830384] rcu_gp_kthread+0x8dd/0x15b0 [ 9184.831821] ? force_qs_rnp+0x2c0/0x2c0 [ 9184.833405] kthread+0x1d1/0x200 [ 9184.834623] ? set_kthread_struct+0x70/0x70 [ 9184.836216] ret_from_fork+0x1f/0x30 [ 9184.837558] rcu: Stack dump where RCU GP kthread last ran: [ 9184.839853] Sending NMI from CPU 1 to CPUs 3: [ 9184.841579] NMI backtrace for cpu 3 skipped: idling at native_safe_halt+0x2e/0x50 [ 9184.844447] NMI backtrace for cpu 1 [ 9184.846067] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9184.850693] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9184.854153] Call Trace: [ 9184.855140] [ 9184.855957] ? dump_stack+0xbb/0x10e [ 9184.857678] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9184.859775] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9184.861930] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9184.864535] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9184.866939] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9184.869193] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 9184.871546] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 9184.873760] ? update_process_times+0x6d/0xc0 [ 9184.875578] ? tick_sched_handle.isra.18+0x27/0x90 [ 9184.877681] ? tick_sched_timer+0x53/0xb0 [ 9184.879413] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9184.881278] ? tick_sched_handle.isra.18+0x90/0x90 [ 9184.883214] ? hrtimer_interrupt+0x128/0x2c0 [ 9184.884879] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9184.887179] ? apic_timer_interrupt+0xf/0x20 [ 9184.888835] [ 9184.889732] ? native_safe_halt+0x2e/0x50 [ 9184.891469] ? kvm_wait+0xcd/0xe0 [ 9184.892765] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9184.894937] ? do_raw_spin_lock+0x177/0x190 [ 9184.896718] ? _raw_spin_lock+0x21/0x30 [ 9184.898272] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9184.900406] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9184.902876] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9184.904822] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9184.907278] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9184.910375] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9184.913232] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9184.915543] ? __next_timer_interrupt+0x160/0x160 [ 9184.917911] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9184.919857] ? do_wait_intr+0xf0/0xf0 [ 9184.921511] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9184.924297] ? kthread+0x1d1/0x200 [ 9184.925698] ? set_kthread_struct+0x70/0x70 [ 9184.927492] ? ret_from_fork+0x1f/0x30 [ 9196.053710] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 9196.066627] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9196.111388] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9196.118234] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9196.124594] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9196.127346] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9196.137293] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9196.140363] RAX: 0000000000007b12 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9196.147246] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 9196.155830] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 9196.162147] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 9196.169173] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 9196.175381] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 9196.182913] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9196.186311] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 9196.201146] Call Trace: [ 9196.205523] [ 9196.209361] ? show_regs.cold.9+0x22/0x2f [ 9196.217119] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9196.224407] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9196.233722] ? watchdog+0x40/0x40 [ 9196.235200] ? hrtimer_interrupt+0x128/0x2c0 [ 9196.238204] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9196.245997] ? apic_timer_interrupt+0xf/0x20 [ 9196.248409] [ 9196.249953] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9196.252969] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9196.256316] do_raw_spin_lock+0x177/0x190 [ 9196.258552] _raw_spin_lock+0x21/0x30 [ 9196.262330] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9196.265646] ? do_raw_spin_unlock+0x75/0x190 [ 9196.267242] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9196.274220] ? do_wait_intr+0xf0/0xf0 [ 9196.277644] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9196.280886] mdc_reint+0x2d/0x110 [mdc] [ 9196.285448] mdc_unlink+0x658/0x870 [mdc] [ 9196.287376] lmv_unlink+0x34d/0xbd0 [lmv] [ 9196.289820] ll_unlink+0x24d/0x7f0 [lustre] [ 9196.292875] vfs_unlink+0x202/0x330 [ 9196.298849] do_unlinkat+0x2fa/0x460 [ 9196.300273] __x64_sys_unlink+0x37/0x50 [ 9196.304989] do_syscall_64+0xc1/0x440 [ 9196.307337] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9196.312016] RIP: 0033:0x7f151bdee10b [ 9196.313710] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9196.326291] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9196.332980] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 9196.341295] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 9196.344619] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9196.347338] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9196.350233] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9200.056541] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 9200.063692] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9200.102745] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9200.113781] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9200.119687] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9200.126061] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9200.128274] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9200.135111] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 9200.137573] RAX: 0000000000004114 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9200.139809] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 9200.142428] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 9200.144717] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 9200.146957] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 9200.149360] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 9200.151991] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9200.153813] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9200.156068] Call Trace: [ 9200.156678] [ 9200.157654] ? show_regs.cold.9+0x22/0x2f [ 9200.159039] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9200.160687] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9200.162369] ? watchdog+0x40/0x40 [ 9200.163434] ? hrtimer_interrupt+0x128/0x2c0 [ 9200.165212] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9200.167019] ? apic_timer_interrupt+0xf/0x20 [ 9200.168431] [ 9200.169334] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9200.170946] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9200.172733] do_raw_spin_lock+0x177/0x190 [ 9200.174417] _raw_spin_lock+0x21/0x30 [ 9200.175659] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9200.177540] process_one_work+0x2c8/0x700 [ 9200.178911] worker_thread+0x296/0x6e0 [ 9200.184595] ? rescuer_thread+0x570/0x570 [ 9200.186284] kthread+0x1d1/0x200 [ 9200.187712] ? set_kthread_struct+0x70/0x70 [ 9200.192127] ret_from_fork+0x1f/0x30 [ 9202.146529] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7271s! [ 9202.160972] Showing busy workqueues and worker pools: [ 9202.185767] workqueue events: flags=0x0 [ 9202.206207] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9202.222651] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 9202.241866] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9202.254043] pending: kfree_rcu_monitor [ 9202.267993] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9202.283702] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 9202.308110] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9202.314360] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 9202.325408] workqueue events_long: flags=0x0 [ 9202.331093] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9202.335864] pending: lu_objects_flush [obdclass] [ 9202.338706] workqueue events_unbound: flags=0x2 [ 9202.342090] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9202.348265] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9202.364167] workqueue events_power_efficient: flags=0x80 [ 9202.371471] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9202.381475] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 9202.391981] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9202.398219] pending: neigh_periodic_work [ 9202.406790] workqueue mm_percpu_wq: flags=0x8 [ 9202.411752] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9202.417669] pending: vmstat_update, lru_add_drain_per_cpu [ 9202.422584] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9202.427782] pending: vmstat_update [ 9202.429962] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9202.434521] pending: lru_add_drain_per_cpu [ 9202.443207] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9202.460878] pending: lru_add_drain_per_cpu BAR(48) [ 9202.470573] workqueue writeback: flags=0x4a [ 9202.476153] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 9202.485176] pending: wb_workfn [ 9202.489687] workqueue xprtiod: flags=0xa [ 9202.493166] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9202.498877] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9202.506267] workqueue ipv6_addrconf: flags=0x40008 [ 9202.516609] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9202.525715] pending: addrconf_verify_work [ 9202.529410] workqueue knbd0-recv: flags=0x1a [ 9202.538095] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9202.548957] in-flight: 658:recv_work [ 9202.555230] workqueue ptlrpc_pinger: flags=0x2 [ 9202.561615] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9202.572728] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9202.577816] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7271s workers=4 idle: 32984 37 8 [ 9202.590172] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9187s workers=2 idle: 661 [ 9212.056813] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 9212.062412] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 9212.065729] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9212.065813] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9212.065831] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9212.065834] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9212.065848] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9212.066080] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9212.066086] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 9212.066088] RDX: 000000000000318b RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 9212.066091] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 9212.066093] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 9212.066095] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9212.066098] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 9212.066101] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9212.066104] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9212.066111] Call Trace: [ 9212.066116] [ 9212.066122] ? show_regs.cold.9+0x22/0x2f [ 9212.066132] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9212.066139] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9212.066143] ? watchdog+0x40/0x40 [ 9212.066147] ? hrtimer_interrupt+0x128/0x2c0 [ 9212.066151] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9212.066159] ? apic_timer_interrupt+0xf/0x20 [ 9212.066164] [ 9212.066167] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9212.066172] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9212.066177] do_raw_spin_lock+0x177/0x190 [ 9212.066182] _raw_spin_lock+0x21/0x30 [ 9212.066188] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9212.066455] obd_mod_free+0x46/0x140 [ptlrpc] [ 9212.066929] mdc_commit_open+0x91/0xb0 [mdc] [ 9212.066967] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9212.067171] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9212.067374] after_reply+0xe2c/0x1780 [ptlrpc] [ 9212.067576] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9212.068207] ? __next_timer_interrupt+0x160/0x160 [ 9212.068217] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9212.068477] ? do_wait_intr+0xf0/0xf0 [ 9212.068485] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9212.068768] kthread+0x1d1/0x200 [ 9212.068773] ? set_kthread_struct+0x70/0x70 [ 9212.068777] ret_from_fork+0x1f/0x30 [ 9212.264680] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9212.300533] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9212.306791] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9212.310660] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9212.314581] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9212.335845] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9212.347743] RAX: 0000000000006a8b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9212.355280] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 9212.362580] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9212.371492] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 9212.378829] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 9212.384612] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 9212.393453] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9212.398940] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 9212.405723] Call Trace: [ 9212.406906] [ 9212.410023] ? show_regs.cold.9+0x22/0x2f [ 9212.412068] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9212.413992] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9212.415956] ? watchdog+0x40/0x40 [ 9212.418298] ? hrtimer_interrupt+0x128/0x2c0 [ 9212.421073] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9212.423768] ? apic_timer_interrupt+0xf/0x20 [ 9212.425843] [ 9212.426713] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9212.428821] do_raw_spin_lock+0x177/0x190 [ 9212.430706] _raw_spin_lock+0x21/0x30 [ 9212.432201] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9212.435240] ? do_raw_spin_unlock+0x75/0x190 [ 9212.436919] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9212.439117] ? do_wait_intr+0xf0/0xf0 [ 9212.440623] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9212.443001] mdc_close+0x53b/0xd70 [mdc] [ 9212.444967] lmv_close+0x18d/0x490 [lmv] [ 9212.446121] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9212.448231] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9212.450892] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9212.453189] ll_md_close+0x274/0x800 [lustre] [ 9212.455193] ll_file_release+0x128/0xc50 [lustre] [ 9212.457281] __fput+0x10d/0x440 [ 9212.459295] ____fput+0x12/0x20 [ 9212.460614] task_work_run+0xc6/0x110 [ 9212.462296] exit_to_usermode_loop+0x1dd/0x1f0 [ 9212.464004] do_syscall_64+0x3d6/0x440 [ 9212.465903] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9212.467618] RIP: 0033:0x7f151bee3cd8 [ 9212.469315] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9212.478052] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9212.482264] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 9212.486856] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 9212.491176] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9212.494424] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9212.497981] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9228.054687] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 9228.064062] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 9228.068195] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9228.068950] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9228.068959] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9228.068962] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9228.068978] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9228.068985] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9228.068989] RAX: 000000000000699c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9228.068991] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 9228.068992] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 9228.068994] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 9228.068996] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 9228.069000] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 9228.069002] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9228.069004] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 9228.069014] Call Trace: [ 9228.069020] [ 9228.069025] ? show_regs.cold.9+0x22/0x2f [ 9228.069035] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9228.069043] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9228.069046] ? watchdog+0x40/0x40 [ 9228.069052] ? hrtimer_interrupt+0x128/0x2c0 [ 9228.069056] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9228.069063] ? apic_timer_interrupt+0xf/0x20 [ 9228.069068] [ 9228.069071] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9228.069075] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9228.069079] do_raw_spin_lock+0x177/0x190 [ 9228.069083] _raw_spin_lock+0x21/0x30 [ 9228.069089] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9228.071746] ? do_raw_spin_unlock+0x75/0x190 [ 9228.071756] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9228.082279] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9228.082360] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9228.082369] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9228.082516] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9228.082917] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9228.082928] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9228.082933] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9228.082937] RAX: 0000000000007d11 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9228.082939] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 9228.082941] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 9228.082943] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 9228.082945] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 9228.082948] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 9228.082951] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9228.082953] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9228.082962] Call Trace: [ 9228.082967] [ 9228.082973] ? show_regs.cold.9+0x22/0x2f [ 9228.082984] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9228.082991] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9228.082995] ? watchdog+0x40/0x40 [ 9228.082999] ? hrtimer_interrupt+0x128/0x2c0 [ 9228.083003] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9228.083012] ? apic_timer_interrupt+0xf/0x20 [ 9228.083016] [ 9228.083019] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9228.083023] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9228.083028] do_raw_spin_lock+0x177/0x190 [ 9228.083033] _raw_spin_lock+0x21/0x30 [ 9228.083039] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9228.085793] process_one_work+0x2c8/0x700 [ 9228.085806] worker_thread+0x296/0x6e0 [ 9228.085810] ? rescuer_thread+0x570/0x570 [ 9228.085814] kthread+0x1d1/0x200 [ 9228.085818] ? set_kthread_struct+0x70/0x70 [ 9228.085822] ret_from_fork+0x1f/0x30 [ 9228.883650] ? do_wait_intr+0xf0/0xf0 [ 9228.890175] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9228.900891] mdc_reint+0x2d/0x110 [mdc] [ 9228.909871] mdc_unlink+0x658/0x870 [mdc] [ 9228.911440] lmv_unlink+0x34d/0xbd0 [lmv] [ 9228.925793] ll_unlink+0x24d/0x7f0 [lustre] [ 9228.930000] vfs_unlink+0x202/0x330 [ 9228.936047] do_unlinkat+0x2fa/0x460 [ 9228.937300] __x64_sys_unlink+0x37/0x50 [ 9228.953429] do_syscall_64+0xc1/0x440 [ 9228.954817] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9228.956625] RIP: 0033:0x7f151bdee10b [ 9228.961962] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9228.984322] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9228.994966] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 9229.010884] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 9229.020790] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9229.032693] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9229.045919] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9232.869724] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7302s! [ 9232.876456] Showing busy workqueues and worker pools: [ 9232.881510] workqueue events: flags=0x0 [ 9232.885190] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9232.893295] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 9232.900976] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9232.910160] pending: kfree_rcu_monitor [ 9232.912178] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9232.923503] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 9232.932512] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9232.937499] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 9232.941637] workqueue events_long: flags=0x0 [ 9232.943985] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9232.948189] pending: lu_objects_flush [obdclass] [ 9232.952238] workqueue events_unbound: flags=0x2 [ 9232.954704] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9232.957109] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9232.963809] workqueue events_power_efficient: flags=0x80 [ 9232.967054] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9232.970652] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 9232.975782] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9232.987390] pending: neigh_periodic_work [ 9232.992644] workqueue mm_percpu_wq: flags=0x8 [ 9233.001169] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9233.008157] pending: vmstat_update, lru_add_drain_per_cpu [ 9233.012094] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9233.017479] pending: vmstat_update [ 9233.026626] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9233.042129] pending: lru_add_drain_per_cpu [ 9233.049745] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9233.063161] pending: lru_add_drain_per_cpu BAR(48) [ 9233.069045] workqueue writeback: flags=0x4a [ 9233.073507] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 9233.078110] pending: wb_workfn [ 9233.088191] workqueue xprtiod: flags=0xa [ 9233.095073] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9233.102037] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9233.115630] workqueue ipv6_addrconf: flags=0x40008 [ 9233.119761] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9233.129223] pending: addrconf_verify_work [ 9233.138291] workqueue knbd0-recv: flags=0x1a [ 9233.143385] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9233.151794] in-flight: 658:recv_work [ 9233.154547] workqueue ptlrpc_pinger: flags=0x2 [ 9233.160901] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9233.169016] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9233.179673] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7302s workers=4 idle: 32984 37 8 [ 9233.193111] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9217s workers=2 idle: 661 [ 9240.057391] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 9240.062081] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9240.064969] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 9240.114220] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9240.114234] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9240.114237] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9240.114251] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9240.114255] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9240.114261] RAX: 0000000000007e1b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9240.114264] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 9240.114266] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9240.114268] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 9240.114270] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 9240.114274] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 9240.114277] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9240.114280] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 9240.114288] Call Trace: [ 9240.114294] [ 9240.114299] ? show_regs.cold.9+0x22/0x2f [ 9240.114308] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9240.114316] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9240.114320] ? watchdog+0x40/0x40 [ 9240.114325] ? hrtimer_interrupt+0x128/0x2c0 [ 9240.114330] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9240.114337] ? apic_timer_interrupt+0xf/0x20 [ 9240.114342] [ 9240.114344] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9240.114353] do_raw_spin_lock+0x177/0x190 [ 9240.114358] _raw_spin_lock+0x21/0x30 [ 9240.114363] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9240.115243] ? do_raw_spin_unlock+0x75/0x190 [ 9240.115249] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9240.118260] ? do_wait_intr+0xf0/0xf0 [ 9240.118273] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9240.138372] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9240.138463] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9240.138471] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9240.138474] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9240.138487] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9240.138492] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9240.138496] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 9240.138499] RDX: 000000000000144b RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 9240.138501] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 9240.138518] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 9240.138520] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9240.138524] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 9240.138527] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9240.138530] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9240.138539] Call Trace: [ 9240.138552] [ 9240.138558] ? show_regs.cold.9+0x22/0x2f [ 9240.138567] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9240.138573] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9240.138577] ? watchdog+0x40/0x40 [ 9240.138581] ? hrtimer_interrupt+0x128/0x2c0 [ 9240.138586] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9240.138595] ? apic_timer_interrupt+0xf/0x20 [ 9240.138600] [ 9240.138603] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9240.138608] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9240.138611] do_raw_spin_lock+0x177/0x190 [ 9240.138616] _raw_spin_lock+0x21/0x30 [ 9240.138622] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9240.138886] obd_mod_free+0x46/0x140 [ptlrpc] [ 9240.139483] mdc_commit_open+0x91/0xb0 [mdc] [ 9240.139665] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9240.139883] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9240.140751] after_reply+0xe2c/0x1780 [ptlrpc] [ 9240.140955] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9240.158512] mdc_close+0x53b/0xd70 [mdc] [ 9240.158578] lmv_close+0x18d/0x490 [lmv] [ 9240.158609] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9240.158766] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9240.158870] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9240.158952] ll_md_close+0x274/0x800 [lustre] [ 9240.159034] ll_file_release+0x128/0xc50 [lustre] [ 9240.159115] __fput+0x10d/0x440 [ 9240.159127] ____fput+0x12/0x20 [ 9240.159131] task_work_run+0xc6/0x110 [ 9240.159140] exit_to_usermode_loop+0x1dd/0x1f0 [ 9240.159147] do_syscall_64+0x3d6/0x440 [ 9240.159151] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9240.159159] RIP: 0033:0x7f151bee3cd8 [ 9240.159166] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9240.159171] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9240.159177] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 9240.159179] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 9240.159182] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9240.159184] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9240.159186] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9240.952797] ? __next_timer_interrupt+0x160/0x160 [ 9240.959454] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9240.963555] ? do_wait_intr+0xf0/0xf0 [ 9240.971880] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9240.977182] kthread+0x1d1/0x200 [ 9240.979694] ? set_kthread_struct+0x70/0x70 [ 9240.993495] ret_from_fork+0x1f/0x30 [ 9256.056058] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9256.059488] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9256.079948] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9256.084906] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9256.089342] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9256.092048] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9256.094837] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9256.104852] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9256.107971] RAX: 0000000000002583 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9256.111845] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 9256.118938] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 9256.123133] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 9256.126016] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 9256.128651] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 9256.132093] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9256.145459] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9256.149397] Call Trace: [ 9256.150490] [ 9256.152115] ? show_regs.cold.9+0x22/0x2f [ 9256.155037] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9256.158227] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9256.161797] ? watchdog+0x40/0x40 [ 9256.163994] ? hrtimer_interrupt+0x128/0x2c0 [ 9256.168162] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9256.175127] ? apic_timer_interrupt+0xf/0x20 [ 9256.182272] [ 9256.184976] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9256.193309] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9256.195859] do_raw_spin_lock+0x177/0x190 [ 9256.201906] _raw_spin_lock+0x21/0x30 [ 9256.204768] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9256.212589] process_one_work+0x2c8/0x700 [ 9256.217244] worker_thread+0x296/0x6e0 [ 9256.221589] ? rescuer_thread+0x570/0x570 [ 9256.224075] kthread+0x1d1/0x200 [ 9256.225312] ? set_kthread_struct+0x70/0x70 [ 9256.232216] ret_from_fork+0x1f/0x30 [ 9260.052067] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 9260.055557] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9260.077879] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9260.083463] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9260.087819] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9260.090081] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9260.098938] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9260.102676] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9260.105945] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 9260.109471] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 9260.112364] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 9260.115887] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 9260.119227] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 9260.122935] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9260.125713] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 9260.128933] Call Trace: [ 9260.130188] [ 9260.131205] ? show_regs.cold.9+0x22/0x2f [ 9260.133147] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9260.135795] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9260.137859] ? watchdog+0x40/0x40 [ 9260.139207] ? hrtimer_interrupt+0x128/0x2c0 [ 9260.141286] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9260.143411] ? apic_timer_interrupt+0xf/0x20 [ 9260.145114] [ 9260.145946] ? native_safe_halt+0x2e/0x50 [ 9260.147964] kvm_wait+0xcd/0xe0 [ 9260.149442] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9260.151822] do_raw_spin_lock+0x177/0x190 [ 9260.153558] _raw_spin_lock+0x21/0x30 [ 9260.155282] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9260.157921] ? do_raw_spin_unlock+0x75/0x190 [ 9260.159779] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9260.162345] ? do_wait_intr+0xf0/0xf0 [ 9260.163943] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9260.166534] mdc_reint+0x2d/0x110 [mdc] [ 9260.168055] mdc_unlink+0x658/0x870 [mdc] [ 9260.170034] lmv_unlink+0x34d/0xbd0 [lmv] [ 9260.171574] ll_unlink+0x24d/0x7f0 [lustre] [ 9260.173784] vfs_unlink+0x202/0x330 [ 9260.175304] do_unlinkat+0x2fa/0x460 [ 9260.176990] __x64_sys_unlink+0x37/0x50 [ 9260.178561] do_syscall_64+0xc1/0x440 [ 9260.180239] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9260.182536] RIP: 0033:0x7f151bdee10b [ 9260.184653] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9260.193199] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9260.197034] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 9260.200063] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 9260.202876] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9260.205925] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9260.209796] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9263.586043] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7332s! [ 9263.589496] Showing busy workqueues and worker pools: [ 9263.593254] workqueue events: flags=0x0 [ 9263.595682] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9263.599435] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 9263.604489] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9263.608326] pending: kfree_rcu_monitor [ 9263.610946] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9263.614639] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 9263.620292] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9263.625545] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 9263.630083] workqueue events_long: flags=0x0 [ 9263.633436] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9263.637494] pending: lu_objects_flush [obdclass] [ 9263.640330] workqueue events_unbound: flags=0x2 [ 9263.643455] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9263.646996] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9263.653284] workqueue events_power_efficient: flags=0x80 [ 9263.655971] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9263.659693] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 9263.663735] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9263.667612] pending: neigh_periodic_work [ 9263.670026] workqueue mm_percpu_wq: flags=0x8 [ 9263.673308] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9263.677531] pending: vmstat_update, lru_add_drain_per_cpu [ 9263.680498] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9263.685096] pending: vmstat_update [ 9263.687629] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9263.693030] pending: lru_add_drain_per_cpu [ 9263.696149] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9263.700165] pending: lru_add_drain_per_cpu BAR(48) [ 9263.703408] workqueue writeback: flags=0x4a [ 9263.705618] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 9263.708753] pending: wb_workfn [ 9263.710687] workqueue xprtiod: flags=0xa [ 9263.712620] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9263.716024] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9263.720201] workqueue ipv6_addrconf: flags=0x40008 [ 9263.725291] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9263.730599] pending: addrconf_verify_work [ 9263.733987] workqueue knbd0-recv: flags=0x1a [ 9263.736439] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9263.739600] in-flight: 658:recv_work [ 9263.741812] workqueue ptlrpc_pinger: flags=0x2 [ 9263.744653] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9263.759615] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9263.763608] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7333s workers=4 idle: 32984 37 8 [ 9263.776687] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9248s workers=2 idle: 661 [ 9268.056401] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 9268.057725] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 9268.066030] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9268.066091] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9268.066096] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9268.066098] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9268.066110] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9268.066113] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9268.066116] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 9268.066118] RDX: 0000000000007bfa RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 9268.066119] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 9268.066121] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 9268.066123] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9268.066125] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 9268.066127] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9268.066129] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9268.066135] Call Trace: [ 9268.066139] [ 9268.066144] ? show_regs.cold.9+0x22/0x2f [ 9268.066151] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9268.066155] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9268.066158] ? watchdog+0x40/0x40 [ 9268.066161] ? hrtimer_interrupt+0x128/0x2c0 [ 9268.066164] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9268.066170] ? apic_timer_interrupt+0xf/0x20 [ 9268.066174] [ 9268.066175] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9268.066178] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9268.066181] do_raw_spin_lock+0x177/0x190 [ 9268.066184] _raw_spin_lock+0x21/0x30 [ 9268.066188] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9268.070651] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9268.070726] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9268.070733] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9268.070737] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9268.070748] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9268.070753] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9268.070758] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9268.070760] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 9268.070763] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9268.070765] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 9268.070767] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 9268.070771] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 9268.070774] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9268.070777] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 9268.070785] Call Trace: [ 9268.070791] [ 9268.070797] ? show_regs.cold.9+0x22/0x2f [ 9268.070808] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9268.070815] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9268.070819] ? watchdog+0x40/0x40 [ 9268.070824] ? hrtimer_interrupt+0x128/0x2c0 [ 9268.070827] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9268.070834] ? apic_timer_interrupt+0xf/0x20 [ 9268.070839] [ 9268.070842] ? native_safe_halt+0x2e/0x50 [ 9268.070846] kvm_wait+0xcd/0xe0 [ 9268.070853] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9268.070862] do_raw_spin_lock+0x177/0x190 [ 9268.070867] _raw_spin_lock+0x21/0x30 [ 9268.070871] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9268.071163] ? do_raw_spin_unlock+0x75/0x190 [ 9268.071169] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9268.071373] ? do_wait_intr+0xf0/0xf0 [ 9268.071381] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9268.071577] mdc_close+0x53b/0xd70 [mdc] [ 9268.071620] lmv_close+0x18d/0x490 [lmv] [ 9268.071652] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9268.071816] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9268.071928] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9268.072010] ll_md_close+0x274/0x800 [lustre] [ 9268.072094] ll_file_release+0x128/0xc50 [lustre] [ 9268.072180] __fput+0x10d/0x440 [ 9268.072191] ____fput+0x12/0x20 [ 9268.072195] task_work_run+0xc6/0x110 [ 9268.072203] exit_to_usermode_loop+0x1dd/0x1f0 [ 9268.072210] do_syscall_64+0x3d6/0x440 [ 9268.072215] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9268.072222] RIP: 0033:0x7f151bee3cd8 [ 9268.072230] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9268.072234] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9268.072240] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 9268.072242] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 9268.072244] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9268.072246] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9268.072248] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9268.525507] obd_mod_free+0x46/0x140 [ptlrpc] [ 9268.528868] mdc_commit_open+0x91/0xb0 [mdc] [ 9268.532419] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9268.538430] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9268.543342] after_reply+0xe2c/0x1780 [ptlrpc] [ 9268.548213] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9268.555045] ? __next_timer_interrupt+0x160/0x160 [ 9268.560546] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9268.565314] ? do_wait_intr+0xf0/0xf0 [ 9268.567458] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9268.574374] kthread+0x1d1/0x200 [ 9268.576433] ? set_kthread_struct+0x70/0x70 [ 9268.577918] ret_from_fork+0x1f/0x30 [ 9284.056202] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9284.060449] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9284.091384] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9284.099558] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9284.106634] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9284.122240] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9284.124896] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9284.139626] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9284.142696] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9284.146509] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 9284.153077] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 9284.161750] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 9284.166852] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 9284.172547] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 9284.176460] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9284.179010] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9284.182412] Call Trace: [ 9284.183833] [ 9284.184921] ? show_regs.cold.9+0x22/0x2f [ 9284.186868] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9284.189656] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9284.193522] ? watchdog+0x40/0x40 [ 9284.195574] ? hrtimer_interrupt+0x128/0x2c0 [ 9284.198337] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9284.203705] ? apic_timer_interrupt+0xf/0x20 [ 9284.205612] [ 9284.207506] ? native_safe_halt+0x2e/0x50 [ 9284.209624] kvm_wait+0xcd/0xe0 [ 9284.210702] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9284.212351] do_raw_spin_lock+0x177/0x190 [ 9284.213469] _raw_spin_lock+0x21/0x30 [ 9284.214421] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9284.216450] process_one_work+0x2c8/0x700 [ 9284.219634] worker_thread+0x296/0x6e0 [ 9284.222337] ? rescuer_thread+0x570/0x570 [ 9284.227753] kthread+0x1d1/0x200 [ 9284.235071] ? set_kthread_struct+0x70/0x70 [ 9284.239280] ret_from_fork+0x1f/0x30 [ 9288.055422] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 9288.066203] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9288.114447] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9288.129053] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9288.139397] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9288.142998] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9288.163369] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9288.172941] RAX: 0000000000004119 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9288.180418] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 9288.188037] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 9288.195947] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 9288.203325] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 9288.211228] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 9288.219389] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9288.227934] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 9288.236808] Call Trace: [ 9288.239444] [ 9288.241883] ? show_regs.cold.9+0x22/0x2f [ 9288.248919] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9288.250728] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9288.253224] ? watchdog+0x40/0x40 [ 9288.256856] ? hrtimer_interrupt+0x128/0x2c0 [ 9288.260277] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9288.268187] ? apic_timer_interrupt+0xf/0x20 [ 9288.273003] [ 9288.273871] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9288.282867] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9288.291405] do_raw_spin_lock+0x177/0x190 [ 9288.299458] _raw_spin_lock+0x21/0x30 [ 9288.302643] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9288.310887] ? do_raw_spin_unlock+0x75/0x190 [ 9288.319628] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9288.325272] ? do_wait_intr+0xf0/0xf0 [ 9288.328523] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9288.335028] mdc_reint+0x2d/0x110 [mdc] [ 9288.338465] mdc_unlink+0x658/0x870 [mdc] [ 9288.342363] lmv_unlink+0x34d/0xbd0 [lmv] [ 9288.344758] ll_unlink+0x24d/0x7f0 [lustre] [ 9288.346404] vfs_unlink+0x202/0x330 [ 9288.350874] do_unlinkat+0x2fa/0x460 [ 9288.353823] __x64_sys_unlink+0x37/0x50 [ 9288.359186] do_syscall_64+0xc1/0x440 [ 9288.363870] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9288.367910] RIP: 0033:0x7f151bdee10b [ 9288.370254] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9288.394680] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9288.402777] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 9288.411257] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 9288.418739] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9288.422006] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9288.425679] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9293.796713] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7363s! [ 9293.809978] Showing busy workqueues and worker pools: [ 9293.814720] workqueue events: flags=0x0 [ 9293.817877] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9293.823677] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 9293.828910] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9293.834892] pending: kfree_rcu_monitor [ 9293.838126] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9293.846325] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 9293.858682] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9293.867470] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 9293.871706] workqueue events_long: flags=0x0 [ 9293.873035] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9293.878372] pending: lu_objects_flush [obdclass] [ 9293.884186] workqueue events_unbound: flags=0x2 [ 9293.887279] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9293.894604] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9293.904568] workqueue events_power_efficient: flags=0x80 [ 9293.911646] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9293.922574] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 9293.929424] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9293.935310] pending: neigh_periodic_work [ 9293.939571] workqueue mm_percpu_wq: flags=0x8 [ 9293.943569] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9293.949651] pending: vmstat_update, lru_add_drain_per_cpu [ 9293.951988] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9293.957818] pending: vmstat_update [ 9293.964299] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9293.976193] pending: lru_add_drain_per_cpu [ 9293.979769] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9293.986710] pending: lru_add_drain_per_cpu BAR(48) [ 9293.989601] workqueue writeback: flags=0x4a [ 9293.992751] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 9293.997504] pending: wb_workfn [ 9293.998910] workqueue xprtiod: flags=0xa [ 9294.004321] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9294.013122] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9294.016810] workqueue ipv6_addrconf: flags=0x40008 [ 9294.021270] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9294.026692] pending: addrconf_verify_work [ 9294.030464] workqueue knbd0-recv: flags=0x1a [ 9294.032505] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9294.036854] in-flight: 658:recv_work [ 9294.041097] workqueue ptlrpc_pinger: flags=0x2 [ 9294.049638] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9294.051849] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9294.066557] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7363s workers=4 idle: 32984 37 8 [ 9294.075093] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9278s workers=2 idle: 661 [ 9296.056683] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 9296.062851] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9296.067140] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 9296.109672] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9296.109692] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9296.109695] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9296.109711] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9296.113305] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9296.113387] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9296.113395] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9296.113397] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9296.113411] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9296.113416] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9296.113421] RAX: 0000000000004f85 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9296.113423] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 9296.113426] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9296.113428] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 9296.113430] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 9296.113434] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 9296.113438] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9296.113440] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 9296.113448] Call Trace: [ 9296.113454] [ 9296.113459] ? show_regs.cold.9+0x22/0x2f [ 9296.113470] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9296.113476] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9296.113480] ? watchdog+0x40/0x40 [ 9296.113485] ? hrtimer_interrupt+0x128/0x2c0 [ 9296.113489] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9296.113497] ? apic_timer_interrupt+0xf/0x20 [ 9296.113502] [ 9296.113505] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9296.113511] do_raw_spin_lock+0x177/0x190 [ 9296.113516] _raw_spin_lock+0x21/0x30 [ 9296.113520] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9296.119415] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9296.119432] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 9296.119435] RDX: 0000000000003fd4 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 9296.119437] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 9296.119439] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 9296.119442] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9296.119445] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 9296.119448] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9296.119451] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9296.119461] Call Trace: [ 9296.119470] [ 9296.119476] ? show_regs.cold.9+0x22/0x2f [ 9296.119490] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9296.119497] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9296.119501] ? watchdog+0x40/0x40 [ 9296.119506] ? hrtimer_interrupt+0x128/0x2c0 [ 9296.119510] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9296.119519] ? apic_timer_interrupt+0xf/0x20 [ 9296.119523] [ 9296.119526] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9296.119532] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9296.119537] do_raw_spin_lock+0x177/0x190 [ 9296.119542] _raw_spin_lock+0x21/0x30 [ 9296.119549] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9296.125744] ? do_raw_spin_unlock+0x75/0x190 [ 9296.125782] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9296.126540] ? do_wait_intr+0xf0/0xf0 [ 9296.126550] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9296.127066] mdc_close+0x53b/0xd70 [mdc] [ 9296.127229] lmv_close+0x18d/0x490 [lmv] [ 9296.127263] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9296.127445] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9296.127556] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9296.127639] ll_md_close+0x274/0x800 [lustre] [ 9296.127723] ll_file_release+0x128/0xc50 [lustre] [ 9296.128012] __fput+0x10d/0x440 [ 9296.128025] ____fput+0x12/0x20 [ 9296.128029] task_work_run+0xc6/0x110 [ 9296.128037] exit_to_usermode_loop+0x1dd/0x1f0 [ 9296.128045] do_syscall_64+0x3d6/0x440 [ 9296.128049] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9296.128057] RIP: 0033:0x7f151bee3cd8 [ 9296.128065] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9296.128069] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9296.128074] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 9296.128077] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 9296.128079] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9296.128081] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9296.128083] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9296.399604] obd_mod_free+0x46/0x140 [ptlrpc] [ 9296.402682] mdc_commit_open+0x91/0xb0 [mdc] [ 9296.404799] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9296.408265] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9296.411680] after_reply+0xe2c/0x1780 [ptlrpc] [ 9296.414186] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9296.417710] ? __next_timer_interrupt+0x160/0x160 [ 9296.420565] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9296.423815] ? do_wait_intr+0xf0/0xf0 [ 9296.425650] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9296.428239] kthread+0x1d1/0x200 [ 9296.429540] ? set_kthread_struct+0x70/0x70 [ 9296.436339] ret_from_fork+0x1f/0x30 [ 9312.066075] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9312.073468] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9312.131190] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9312.146594] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9312.167275] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9312.177682] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9312.179868] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9312.188146] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9312.196498] RAX: 0000000000007c0d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9312.203108] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 9312.209321] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 9312.213569] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 9312.218106] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 9312.228356] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 9312.233327] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9312.241228] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9312.249259] Call Trace: [ 9312.251288] [ 9312.251792] ? show_regs.cold.9+0x22/0x2f [ 9312.260051] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9312.267555] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9312.272539] ? watchdog+0x40/0x40 [ 9312.277095] ? hrtimer_interrupt+0x128/0x2c0 [ 9312.280476] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9312.285423] ? apic_timer_interrupt+0xf/0x20 [ 9312.289311] [ 9312.290395] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9312.299358] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9312.309702] do_raw_spin_lock+0x177/0x190 [ 9312.313326] _raw_spin_lock+0x21/0x30 [ 9312.316985] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9312.324420] process_one_work+0x2c8/0x700 [ 9312.331594] worker_thread+0x296/0x6e0 [ 9312.333571] ? rescuer_thread+0x570/0x570 [ 9312.337809] kthread+0x1d1/0x200 [ 9312.339415] ? set_kthread_struct+0x70/0x70 [ 9312.342964] ret_from_fork+0x1f/0x30 [ 9320.052232] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 9320.055172] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9320.111643] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9320.129216] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9320.147921] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9320.151956] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9320.178735] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9320.190483] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9320.195593] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 9320.205440] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 9320.217127] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 9320.230492] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 9320.244688] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 9320.254173] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9320.257588] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 9320.265755] Call Trace: [ 9320.267558] [ 9320.268740] ? show_regs.cold.9+0x22/0x2f [ 9320.270962] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9320.273361] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9320.275981] ? watchdog+0x40/0x40 [ 9320.277722] ? hrtimer_interrupt+0x128/0x2c0 [ 9320.280375] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9320.282903] ? apic_timer_interrupt+0xf/0x20 [ 9320.284967] [ 9320.285740] ? native_safe_halt+0x2e/0x50 [ 9320.291046] kvm_wait+0xcd/0xe0 [ 9320.292758] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9320.298893] do_raw_spin_lock+0x177/0x190 [ 9320.303268] _raw_spin_lock+0x21/0x30 [ 9320.307134] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9320.317016] ? do_raw_spin_unlock+0x75/0x190 [ 9320.318617] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9320.325323] ? do_wait_intr+0xf0/0xf0 [ 9320.327488] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9320.330551] mdc_reint+0x2d/0x110 [mdc] [ 9320.332714] mdc_unlink+0x658/0x870 [mdc] [ 9320.334755] lmv_unlink+0x34d/0xbd0 [lmv] [ 9320.336516] ll_unlink+0x24d/0x7f0 [lustre] [ 9320.338900] vfs_unlink+0x202/0x330 [ 9320.341517] do_unlinkat+0x2fa/0x460 [ 9320.342737] __x64_sys_unlink+0x37/0x50 [ 9320.345350] do_syscall_64+0xc1/0x440 [ 9320.346577] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9320.353643] RIP: 0033:0x7f151bdee10b [ 9320.354972] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9320.366530] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9320.377151] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 9320.385131] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 9320.391434] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9320.398594] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9320.410701] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9324.055078] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 9324.058264] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9324.061116] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 9324.102068] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9324.102083] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9324.102087] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9324.102102] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9324.102106] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9324.102112] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 9324.102115] RDX: 0000000000005c76 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 9324.102118] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 9324.102121] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 9324.102123] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9324.102127] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 9324.102130] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9324.102145] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9324.102154] Call Trace: [ 9324.102159] [ 9324.102165] ? show_regs.cold.9+0x22/0x2f [ 9324.105656] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9324.105753] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9324.105761] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9324.105764] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9324.105781] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9324.105785] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9324.105790] RAX: 0000000000002113 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9324.105792] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 9324.105800] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9324.105803] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 9324.105809] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 9324.105813] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 9324.105815] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9324.105818] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 9324.105827] Call Trace: [ 9324.105833] [ 9324.105839] ? show_regs.cold.9+0x22/0x2f [ 9324.105848] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9324.105862] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9324.105866] ? watchdog+0x40/0x40 [ 9324.105874] ? hrtimer_interrupt+0x128/0x2c0 [ 9324.105879] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9324.105887] ? apic_timer_interrupt+0xf/0x20 [ 9324.105893] [ 9324.105897] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9324.105904] do_raw_spin_lock+0x177/0x190 [ 9324.105914] _raw_spin_lock+0x21/0x30 [ 9324.105920] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9324.106253] ? do_raw_spin_unlock+0x75/0x190 [ 9324.106259] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9324.106893] ? do_wait_intr+0xf0/0xf0 [ 9324.106901] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9324.107312] mdc_close+0x53b/0xd70 [mdc] [ 9324.107358] lmv_close+0x18d/0x490 [lmv] [ 9324.118463] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9324.118483] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9324.118719] ? watchdog+0x40/0x40 [ 9324.124678] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9324.128445] ? hrtimer_interrupt+0x128/0x2c0 [ 9324.128463] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9324.128473] ? apic_timer_interrupt+0xf/0x20 [ 9324.128477] [ 9324.128482] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9324.128739] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9324.128745] do_raw_spin_lock+0x177/0x190 [ 9324.128751] _raw_spin_lock+0x21/0x30 [ 9324.128759] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9324.136847] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9324.136988] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9324.137079] ll_md_close+0x274/0x800 [lustre] [ 9324.139682] obd_mod_free+0x46/0x140 [ptlrpc] [ 9324.139942] mdc_commit_open+0x91/0xb0 [mdc] [ 9324.139971] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9324.140093] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9324.140217] after_reply+0xe2c/0x1780 [ptlrpc] [ 9324.140341] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9324.141074] ? __next_timer_interrupt+0x160/0x160 [ 9324.141082] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9324.141213] ? do_wait_intr+0xf0/0xf0 [ 9324.141220] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9324.141349] kthread+0x1d1/0x200 [ 9324.141353] ? set_kthread_struct+0x70/0x70 [ 9324.141356] ret_from_fork+0x1f/0x30 [ 9324.148026] ll_file_release+0x128/0xc50 [lustre] [ 9324.148164] __fput+0x10d/0x440 [ 9324.148173] ____fput+0x12/0x20 [ 9324.148178] task_work_run+0xc6/0x110 [ 9324.148186] exit_to_usermode_loop+0x1dd/0x1f0 [ 9324.148192] do_syscall_64+0x3d6/0x440 [ 9324.148197] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9324.148205] RIP: 0033:0x7f151bee3cd8 [ 9324.148212] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9324.148216] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9324.148222] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 9324.148224] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 9324.148226] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9324.148229] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9324.148231] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9324.516676] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7393s! [ 9324.522884] Showing busy workqueues and worker pools: [ 9324.525095] workqueue events: flags=0x0 [ 9324.531073] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9324.535281] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 9324.541770] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9324.545088] pending: kfree_rcu_monitor [ 9324.547387] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9324.551348] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 9324.555844] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9324.559322] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 9324.565425] workqueue events_long: flags=0x0 [ 9324.567954] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9324.572574] pending: lu_objects_flush [obdclass] [ 9324.576897] workqueue events_unbound: flags=0x2 [ 9324.582440] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9324.589118] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9324.594658] workqueue events_power_efficient: flags=0x80 [ 9324.597485] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9324.600861] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 9324.604991] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9324.608090] pending: neigh_periodic_work [ 9324.610944] workqueue mm_percpu_wq: flags=0x8 [ 9324.612949] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9324.615886] pending: vmstat_update, lru_add_drain_per_cpu [ 9324.619265] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9324.622929] pending: vmstat_update [ 9324.624358] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9324.627964] pending: lru_add_drain_per_cpu [ 9324.632986] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9324.639093] pending: lru_add_drain_per_cpu BAR(48) [ 9324.643415] workqueue writeback: flags=0x4a [ 9324.645390] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 9324.647526] pending: wb_workfn [ 9324.653467] workqueue xprtiod: flags=0xa [ 9324.658295] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9324.663595] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9324.669299] workqueue ipv6_addrconf: flags=0x40008 [ 9324.677144] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9324.683973] pending: addrconf_verify_work [ 9324.689848] workqueue knbd0-recv: flags=0x1a [ 9324.701022] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9324.704870] in-flight: 658:recv_work [ 9324.706512] workqueue ptlrpc_pinger: flags=0x2 [ 9324.710897] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9324.716450] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9324.719407] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7394s workers=4 idle: 32984 37 8 [ 9324.726356] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9309s workers=2 idle: 661 [ 9340.061653] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9340.069044] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9340.119108] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9340.129026] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9340.138011] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9340.143171] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9340.151029] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9340.167902] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9340.172058] RAX: 0000000000007886 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9340.176009] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 9340.180120] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 9340.183589] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 9340.187283] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 9340.190677] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 9340.194942] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9340.198034] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9340.202170] Call Trace: [ 9340.209679] [ 9340.213704] ? show_regs.cold.9+0x22/0x2f [ 9340.216767] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9340.224354] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9340.228097] ? watchdog+0x40/0x40 [ 9340.232837] ? hrtimer_interrupt+0x128/0x2c0 [ 9340.236097] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9340.241497] ? apic_timer_interrupt+0xf/0x20 [ 9340.249494] [ 9340.253595] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9340.264763] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9340.277401] do_raw_spin_lock+0x177/0x190 [ 9340.282741] _raw_spin_lock+0x21/0x30 [ 9340.287393] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9340.295035] process_one_work+0x2c8/0x700 [ 9340.298597] worker_thread+0x296/0x6e0 [ 9340.302627] ? rescuer_thread+0x570/0x570 [ 9340.304795] kthread+0x1d1/0x200 [ 9340.311267] ? set_kthread_struct+0x70/0x70 [ 9340.318257] ret_from_fork+0x1f/0x30 [ 9352.052087] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 9352.056002] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 9352.057150] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 9352.057161] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9352.057232] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9352.057238] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9352.057241] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9352.057256] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9352.057260] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9352.057265] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9352.057267] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 9352.057270] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9352.057272] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 9352.057274] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 9352.057277] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 9352.057280] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9352.057283] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 9352.057292] Call Trace: [ 9352.057299] [ 9352.057307] ? show_regs.cold.9+0x22/0x2f [ 9352.057319] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9352.057338] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9352.057344] ? watchdog+0x40/0x40 [ 9352.057349] ? hrtimer_interrupt+0x128/0x2c0 [ 9352.057353] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9352.057361] ? apic_timer_interrupt+0xf/0x20 [ 9352.057365] [ 9352.057369] ? native_safe_halt+0x2e/0x50 [ 9352.057373] kvm_wait+0xcd/0xe0 [ 9352.057381] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9352.057391] do_raw_spin_lock+0x177/0x190 [ 9352.057395] _raw_spin_lock+0x21/0x30 [ 9352.057400] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9352.057828] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 [ 9352.057921] ? do_raw_spin_unlock+0x75/0x190 [ 9352.057928] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9352.058123] ? do_wait_intr+0xf0/0xf0 [ 9352.058131] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9352.058327] mdc_close+0x53b/0xd70 [mdc] [ 9352.058368] lmv_close+0x18d/0x490 [lmv] [ 9352.058394] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9352.058930] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9352.059052] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9352.059132] ll_md_close+0x274/0x800 [lustre] [ 9352.059220] dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9352.059213] ll_file_release+0x128/0xc50 [lustre] [ 9352.059255] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9352.059262] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9352.059265] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9352.059276] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9352.059280] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9352.059284] RAX: 0000000000006bc8 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9352.059286] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 9352.059288] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 9352.059290] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 9352.059294] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 9352.059299] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 9352.059303] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9352.059299] __fput+0x10d/0x440 [ 9352.059305] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 9352.059314] ____fput+0x12/0x20 [ 9352.059318] Call Trace: [ 9352.059325] [ 9352.059328] ? show_regs.cold.9+0x22/0x2f [ 9352.059336] task_work_run+0xc6/0x110 [ 9352.059339] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9352.059350] exit_to_usermode_loop+0x1dd/0x1f0 [ 9352.059352] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9352.059362] ? watchdog+0x40/0x40 [ 9352.059366] do_syscall_64+0x3d6/0x440 [ 9352.059371] ? hrtimer_interrupt+0x128/0x2c0 [ 9352.059375] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9352.059381] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9352.059388] RIP: 0033:0x7f151bee3cd8 [ 9352.059393] ? apic_timer_interrupt+0xf/0x20 [ 9352.059399] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9352.059403] [ 9352.059407] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9352.059411] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9352.059416] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 9352.059421] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9352.059423] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 9352.059429] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9352.059432] do_raw_spin_lock+0x177/0x190 [ 9352.059434] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9352.059439] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9352.059442] _raw_spin_lock+0x21/0x30 [ 9352.059451] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9352.059742] ? do_raw_spin_unlock+0x75/0x190 [ 9352.059748] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9352.060303] ? do_wait_intr+0xf0/0xf0 [ 9352.060313] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9352.060537] mdc_reint+0x2d/0x110 [mdc] [ 9352.060580] mdc_unlink+0x658/0x870 [mdc] [ 9352.060611] lmv_unlink+0x34d/0xbd0 [lmv] [ 9352.060643] ll_unlink+0x24d/0x7f0 [lustre] [ 9352.060754] vfs_unlink+0x202/0x330 [ 9352.060764] do_unlinkat+0x2fa/0x460 [ 9352.060770] __x64_sys_unlink+0x37/0x50 [ 9352.071099] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9352.071184] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9352.071193] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9352.071196] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9352.071212] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9352.071217] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9352.071222] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 9352.071225] RDX: 0000000000001e06 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 9352.071227] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 9352.071230] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 9352.071233] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9352.071236] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 9352.071239] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9352.071241] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9352.071252] Call Trace: [ 9352.071259] [ 9352.071265] ? show_regs.cold.9+0x22/0x2f [ 9352.071276] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9352.071295] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9352.071300] ? watchdog+0x40/0x40 [ 9352.071304] ? hrtimer_interrupt+0x128/0x2c0 [ 9352.071309] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9352.071317] ? apic_timer_interrupt+0xf/0x20 [ 9352.071492] [ 9352.071497] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9352.071501] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9352.071506] do_raw_spin_lock+0x177/0x190 [ 9352.071511] _raw_spin_lock+0x21/0x30 [ 9352.071517] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9352.071979] obd_mod_free+0x46/0x140 [ptlrpc] [ 9352.077367] do_syscall_64+0xc1/0x440 [ 9352.077394] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9352.077404] RIP: 0033:0x7f151bdee10b [ 9352.077413] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9352.077418] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9352.077424] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 9352.077427] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 9352.077429] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9352.077431] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9352.077434] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9353.085115] mdc_commit_open+0x91/0xb0 [mdc] [ 9353.087946] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9353.091679] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9353.094957] after_reply+0xe2c/0x1780 [ptlrpc] [ 9353.098807] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9353.105834] ? __next_timer_interrupt+0x160/0x160 [ 9353.109049] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9353.112688] ? do_wait_intr+0xf0/0xf0 [ 9353.114495] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9353.117972] kthread+0x1d1/0x200 [ 9353.119776] ? set_kthread_struct+0x70/0x70 [ 9353.120985] ret_from_fork+0x1f/0x30 [ 9355.232617] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7424s! [ 9355.241169] Showing busy workqueues and worker pools: [ 9355.243481] workqueue events: flags=0x0 [ 9355.246182] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9355.250041] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 9355.260780] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9355.269214] pending: kfree_rcu_monitor [ 9355.270597] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9355.275237] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 9355.281771] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9355.295275] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 9355.305042] workqueue events_long: flags=0x0 [ 9355.309542] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9355.325004] pending: lu_objects_flush [obdclass] [ 9355.342092] workqueue events_unbound: flags=0x2 [ 9355.344648] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9355.351514] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9355.357010] workqueue events_power_efficient: flags=0x80 [ 9355.359174] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9355.368190] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 9355.375880] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9355.379594] pending: neigh_periodic_work [ 9355.381743] workqueue mm_percpu_wq: flags=0x8 [ 9355.383579] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9355.395601] pending: vmstat_update, lru_add_drain_per_cpu [ 9355.401534] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9355.407424] pending: vmstat_update [ 9355.408584] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9355.411733] pending: lru_add_drain_per_cpu [ 9355.413565] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9355.417188] pending: lru_add_drain_per_cpu BAR(48) [ 9355.426633] workqueue writeback: flags=0x4a [ 9355.434202] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 9355.437008] pending: wb_workfn [ 9355.438033] workqueue xprtiod: flags=0xa [ 9355.439909] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9355.444854] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9355.453613] workqueue ipv6_addrconf: flags=0x40008 [ 9355.458591] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9355.465367] pending: addrconf_verify_work [ 9355.469868] workqueue knbd0-recv: flags=0x1a [ 9355.473855] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9355.477792] in-flight: 658:recv_work [ 9355.478797] workqueue ptlrpc_pinger: flags=0x2 [ 9355.480264] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9355.483541] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9355.486455] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7424s workers=4 idle: 32984 37 8 [ 9355.489052] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9340s workers=2 idle: 661 [ 9364.803210] rcu: INFO: rcu_sched self-detected stall on CPU [ 9364.811394] rcu: 1-...!: (4219994 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 9364.820411] (t=7440156 jiffies g=249657 q=4538) [ 9364.823720] rcu: rcu_sched kthread starved for 7435108 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 9364.830685] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 9364.842318] rcu: RCU grace-period kthread stack dump: [ 9364.848125] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 9364.855658] Call Trace: [ 9364.858260] __schedule+0x351/0xcb0 [ 9364.862047] schedule+0xc0/0x180 [ 9364.863867] schedule_timeout+0xb4/0x190 [ 9364.868021] ? do_raw_spin_unlock+0x75/0x190 [ 9364.875584] ? __next_timer_interrupt+0x160/0x160 [ 9364.881657] ? prepare_to_swait_event+0xf3/0x140 [ 9364.884472] rcu_gp_kthread+0x8dd/0x15b0 [ 9364.889991] ? force_qs_rnp+0x2c0/0x2c0 [ 9364.891281] kthread+0x1d1/0x200 [ 9364.899705] ? set_kthread_struct+0x70/0x70 [ 9364.902513] ret_from_fork+0x1f/0x30 [ 9364.914301] rcu: Stack dump where RCU GP kthread last ran: [ 9364.925884] NMI backtrace for cpu 1 [ 9364.927345] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9364.934357] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9364.948151] Call Trace: [ 9364.952734] [ 9364.955698] ? dump_stack+0xbb/0x10e [ 9364.957749] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9364.966720] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9364.969208] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9364.974220] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9364.979363] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9364.983778] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 9364.990105] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 9364.994819] ? update_process_times+0x6d/0xc0 [ 9364.997352] ? tick_sched_handle.isra.18+0x27/0x90 [ 9365.003709] ? tick_sched_timer+0x53/0xb0 [ 9365.008330] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9365.012262] ? tick_sched_handle.isra.18+0x90/0x90 [ 9365.016948] ? hrtimer_interrupt+0x128/0x2c0 [ 9365.018909] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9365.022462] ? apic_timer_interrupt+0xf/0x20 [ 9365.026744] [ 9365.027826] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9365.031404] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9365.034278] ? do_raw_spin_lock+0x177/0x190 [ 9365.036410] ? _raw_spin_lock+0x21/0x30 [ 9365.037521] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9365.040167] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9365.042015] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9365.043583] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9365.051833] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9365.058659] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9365.060477] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9365.070159] ? __next_timer_interrupt+0x160/0x160 [ 9365.076298] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9365.082007] ? do_wait_intr+0xf0/0xf0 [ 9365.083288] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9365.088161] ? kthread+0x1d1/0x200 [ 9365.092285] ? set_kthread_struct+0x70/0x70 [ 9365.099979] ? ret_from_fork+0x1f/0x30 [ 9365.104066] NMI backtrace for cpu 1 [ 9365.109500] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9365.117241] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9365.129476] Call Trace: [ 9365.132534] [ 9365.135099] ? dump_stack+0xbb/0x10e [ 9365.141507] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9365.150397] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9365.159727] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9365.169661] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9365.178811] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9365.184477] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 9365.190961] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 9365.197531] ? update_process_times+0x6d/0xc0 [ 9365.199141] ? tick_sched_handle.isra.18+0x27/0x90 [ 9365.205174] ? tick_sched_timer+0x53/0xb0 [ 9365.210388] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9365.216742] ? tick_sched_handle.isra.18+0x90/0x90 [ 9365.222444] ? hrtimer_interrupt+0x128/0x2c0 [ 9365.227389] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9365.231238] ? apic_timer_interrupt+0xf/0x20 [ 9365.235411] [ 9365.241953] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9365.251422] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9365.259408] ? do_raw_spin_lock+0x177/0x190 [ 9365.266228] ? _raw_spin_lock+0x21/0x30 [ 9365.268605] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9365.278876] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9365.284542] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9365.298235] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9365.311091] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9365.315173] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9365.327749] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9365.347825] ? __next_timer_interrupt+0x160/0x160 [ 9365.355799] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9365.361344] ? do_wait_intr+0xf0/0xf0 [ 9365.363462] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9365.366213] ? kthread+0x1d1/0x200 [ 9365.367451] ? set_kthread_struct+0x70/0x70 [ 9365.368997] ? ret_from_fork+0x1f/0x30 [ 9368.056012] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9368.062786] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9368.104863] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9368.112801] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9368.119540] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9368.127768] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9368.132830] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9368.147835] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9368.151792] RAX: 00000000000040f8 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9368.155513] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 9368.158073] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 9368.160126] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 9368.162293] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 9368.164508] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 9368.166901] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9368.168387] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9368.170101] Call Trace: [ 9368.170774] [ 9368.171314] ? show_regs.cold.9+0x22/0x2f [ 9368.172275] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9368.173556] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9368.174821] ? watchdog+0x40/0x40 [ 9368.175841] ? hrtimer_interrupt+0x128/0x2c0 [ 9368.177042] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9368.178439] ? apic_timer_interrupt+0xf/0x20 [ 9368.179863] [ 9368.180479] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9368.182156] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9368.183660] do_raw_spin_lock+0x177/0x190 [ 9368.184674] _raw_spin_lock+0x21/0x30 [ 9368.185633] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9368.187114] process_one_work+0x2c8/0x700 [ 9368.188119] worker_thread+0x296/0x6e0 [ 9368.189315] ? rescuer_thread+0x570/0x570 [ 9368.190219] kthread+0x1d1/0x200 [ 9368.191115] ? set_kthread_struct+0x70/0x70 [ 9368.194992] ret_from_fork+0x1f/0x30 [ 9380.056643] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 9380.057022] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 9380.068435] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) [ 9380.075938] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9380.076029] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9380.076038] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9380.076041] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9380.076055] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9380.076060] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9380.076064] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9380.076067] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 9380.076069] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9380.076071] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 9380.076074] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 9380.076077] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 9380.076079] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9380.076082] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 9380.076090] Call Trace: [ 9380.076096] [ 9380.076102] ? show_regs.cold.9+0x22/0x2f [ 9380.076112] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9380.076241] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9380.076246] ? watchdog+0x40/0x40 [ 9380.076251] ? hrtimer_interrupt+0x128/0x2c0 [ 9380.076255] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9380.076261] ? apic_timer_interrupt+0xf/0x20 [ 9380.076265] [ 9380.076268] ? native_safe_halt+0x2e/0x50 [ 9380.076272] kvm_wait+0xcd/0xe0 [ 9380.076281] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9380.076290] do_raw_spin_lock+0x177/0x190 [ 9380.076294] _raw_spin_lock+0x21/0x30 [ 9380.076298] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9380.076876] ? do_raw_spin_unlock+0x75/0x190 [ 9380.076884] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9380.077909] ? do_wait_intr+0xf0/0xf0 [ 9380.077918] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9380.105335] lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9380.105460] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9380.105468] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9380.105471] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9380.105485] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9380.137927] mdc_close+0x53b/0xd70 [mdc] [ 9380.138000] lmv_close+0x18d/0x490 [lmv] [ 9380.138031] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9380.138306] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9380.146659] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9380.158854] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9380.164283] RAX: 0000000000004083 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9380.164294] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 9380.164297] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 9380.164299] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 9380.164302] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 9380.164306] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 9380.164309] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9380.164311] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 9380.164319] Call Trace: [ 9380.164326] [ 9380.164331] ? show_regs.cold.9+0x22/0x2f [ 9380.164345] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9380.164350] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9380.164354] ? watchdog+0x40/0x40 [ 9380.164358] ? hrtimer_interrupt+0x128/0x2c0 [ 9380.164362] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9380.164371] ? apic_timer_interrupt+0xf/0x20 [ 9380.164377] [ 9380.164380] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9380.164388] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9380.164393] do_raw_spin_lock+0x177/0x190 [ 9380.164398] _raw_spin_lock+0x21/0x30 [ 9380.164406] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9380.164693] ? do_raw_spin_unlock+0x75/0x190 [ 9380.164700] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9380.164914] ? do_wait_intr+0xf0/0xf0 [ 9380.164924] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9380.165127] mdc_reint+0x2d/0x110 [mdc] [ 9380.165165] mdc_unlink+0x658/0x870 [mdc] [ 9380.165206] lmv_unlink+0x34d/0xbd0 [lmv] [ 9380.165239] ll_unlink+0x24d/0x7f0 [lustre] [ 9380.165367] vfs_unlink+0x202/0x330 [ 9380.165379] do_unlinkat+0x2fa/0x460 [ 9380.165384] __x64_sys_unlink+0x37/0x50 [ 9380.165388] do_syscall_64+0xc1/0x440 [ 9380.165394] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9380.165401] RIP: 0033:0x7f151bdee10b [ 9380.165408] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9380.165412] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9380.165417] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 9380.165420] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 9380.165422] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9380.165424] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9380.165427] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9380.631391] ll_md_close+0x274/0x800 [lustre] [ 9380.634564] ll_file_release+0x128/0xc50 [lustre] [ 9380.638420] __fput+0x10d/0x440 [ 9380.644454] ____fput+0x12/0x20 [ 9380.648426] task_work_run+0xc6/0x110 [ 9380.654125] exit_to_usermode_loop+0x1dd/0x1f0 [ 9380.656804] do_syscall_64+0x3d6/0x440 [ 9380.659130] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9380.661919] RIP: 0033:0x7f151bee3cd8 [ 9380.663973] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9380.676409] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9380.680431] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 9380.684413] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 9380.689172] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9380.694195] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9380.697855] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9385.959654] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7455s! [ 9385.973981] Showing busy workqueues and worker pools: [ 9385.979379] workqueue events: flags=0x0 [ 9385.985912] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9385.997380] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 9386.005090] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9386.017429] pending: kfree_rcu_monitor [ 9386.024791] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9386.035510] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 9386.055950] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9386.059932] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 9386.068976] workqueue events_long: flags=0x0 [ 9386.074613] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9386.078836] pending: lu_objects_flush [obdclass] [ 9386.084459] workqueue events_unbound: flags=0x2 [ 9386.086901] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9386.095907] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9386.103936] workqueue events_power_efficient: flags=0x80 [ 9386.108628] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9386.113464] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 9386.119063] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9386.122614] pending: neigh_periodic_work [ 9386.126835] workqueue mm_percpu_wq: flags=0x8 [ 9386.131931] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9386.135759] pending: vmstat_update, lru_add_drain_per_cpu [ 9386.146216] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9386.148643] pending: vmstat_update [ 9386.152480] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9386.159986] pending: lru_add_drain_per_cpu [ 9386.167053] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9386.177983] pending: lru_add_drain_per_cpu BAR(48) [ 9386.182684] workqueue writeback: flags=0x4a [ 9386.190911] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 9386.193113] pending: wb_workfn [ 9386.196805] workqueue xprtiod: flags=0xa [ 9386.197830] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9386.206910] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9386.218589] workqueue ipv6_addrconf: flags=0x40008 [ 9386.225603] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9386.235022] pending: addrconf_verify_work [ 9386.237595] workqueue knbd0-recv: flags=0x1a [ 9386.245180] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9386.255350] in-flight: 658:recv_work [ 9386.261468] workqueue ptlrpc_pinger: flags=0x2 [ 9386.268673] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9386.270898] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9386.282448] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7455s workers=4 idle: 32984 37 8 [ 9386.291430] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9371s workers=2 idle: 661 [ 9392.064619] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 9392.073475] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9392.121065] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9392.136382] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9392.144856] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9392.148891] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9392.176222] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9392.189212] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 9392.198823] RDX: 00000000000063bd RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 9392.203878] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 9392.207106] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 9392.214127] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9392.219809] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 9392.227305] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9392.229965] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9392.232738] Call Trace: [ 9392.236234] [ 9392.237233] ? show_regs.cold.9+0x22/0x2f [ 9392.239168] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9392.243130] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9392.245986] ? watchdog+0x40/0x40 [ 9392.247851] ? hrtimer_interrupt+0x128/0x2c0 [ 9392.249458] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9392.260341] ? apic_timer_interrupt+0xf/0x20 [ 9392.266954] [ 9392.272282] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9392.280774] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9392.288750] do_raw_spin_lock+0x177/0x190 [ 9392.292192] _raw_spin_lock+0x21/0x30 [ 9392.298112] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9392.300226] obd_mod_free+0x46/0x140 [ptlrpc] [ 9392.305832] mdc_commit_open+0x91/0xb0 [mdc] [ 9392.310891] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9392.317191] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9392.324198] after_reply+0xe2c/0x1780 [ptlrpc] [ 9392.329508] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9392.331904] ? __next_timer_interrupt+0x160/0x160 [ 9392.337027] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9392.339717] ? do_wait_intr+0xf0/0xf0 [ 9392.345836] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9392.353799] kthread+0x1d1/0x200 [ 9392.357220] ? set_kthread_struct+0x70/0x70 [ 9392.359822] ret_from_fork+0x1f/0x30 [ 9396.063594] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9396.070351] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9396.101827] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9396.107748] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9396.112768] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9396.116238] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9396.119892] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9396.142587] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9396.148166] RAX: 0000000000007d45 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9396.155249] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 9396.161238] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 9396.167395] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 9396.177427] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 9396.185621] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 9396.193009] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9396.196527] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9396.201519] Call Trace: [ 9396.206278] [ 9396.207513] ? show_regs.cold.9+0x22/0x2f [ 9396.209199] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9396.214187] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9396.217207] ? watchdog+0x40/0x40 [ 9396.219371] ? hrtimer_interrupt+0x128/0x2c0 [ 9396.222964] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9396.229371] ? apic_timer_interrupt+0xf/0x20 [ 9396.232607] [ 9396.234178] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9396.239161] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9396.242147] do_raw_spin_lock+0x177/0x190 [ 9396.244230] _raw_spin_lock+0x21/0x30 [ 9396.246195] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9396.249323] process_one_work+0x2c8/0x700 [ 9396.251353] worker_thread+0x296/0x6e0 [ 9396.253111] ? rescuer_thread+0x570/0x570 [ 9396.255084] kthread+0x1d1/0x200 [ 9396.256814] ? set_kthread_struct+0x70/0x70 [ 9396.258421] ret_from_fork+0x1f/0x30 [ 9408.057398] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 9408.068596] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9408.095384] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9408.102147] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9408.107359] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9408.110233] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9408.118849] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9408.122074] RAX: 000000000000670a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9408.125323] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 9408.128522] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9408.131747] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 9408.135587] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 9408.139714] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 9408.144724] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9408.147634] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 9408.151501] Call Trace: [ 9408.152750] [ 9408.154130] ? show_regs.cold.9+0x22/0x2f [ 9408.155693] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9408.158120] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9408.160158] ? watchdog+0x40/0x40 [ 9408.161949] ? hrtimer_interrupt+0x128/0x2c0 [ 9408.163909] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9408.165468] ? apic_timer_interrupt+0xf/0x20 [ 9408.166840] [ 9408.167762] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9408.170620] do_raw_spin_lock+0x177/0x190 [ 9408.172260] _raw_spin_lock+0x21/0x30 [ 9408.178687] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9408.184185] ? do_raw_spin_unlock+0x75/0x190 [ 9408.185840] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9408.189724] ? do_wait_intr+0xf0/0xf0 [ 9408.192340] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9408.195157] mdc_close+0x53b/0xd70 [mdc] [ 9408.197470] lmv_close+0x18d/0x490 [lmv] [ 9408.199481] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9408.202397] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9408.205079] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9408.208232] ll_md_close+0x274/0x800 [lustre] [ 9408.210503] ll_file_release+0x128/0xc50 [lustre] [ 9408.213151] __fput+0x10d/0x440 [ 9408.214666] ____fput+0x12/0x20 [ 9408.216411] task_work_run+0xc6/0x110 [ 9408.218078] exit_to_usermode_loop+0x1dd/0x1f0 [ 9408.220397] do_syscall_64+0x3d6/0x440 [ 9408.222223] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9408.224623] RIP: 0033:0x7f151bee3cd8 [ 9408.226289] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9408.235299] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9408.238938] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 9408.242313] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 9408.246373] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9408.249272] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9408.252609] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9412.052381] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 9412.057498] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9412.095404] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9412.108201] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9412.115934] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9412.121064] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9412.136730] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9412.143562] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9412.153595] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 9412.162060] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 9412.170972] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 9412.177922] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 9412.189553] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 9412.211035] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9412.222040] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 9412.231296] Call Trace: [ 9412.235303] [ 9412.238436] ? show_regs.cold.9+0x22/0x2f [ 9412.243472] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9412.246233] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9412.252485] ? watchdog+0x40/0x40 [ 9412.257401] ? hrtimer_interrupt+0x128/0x2c0 [ 9412.258919] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9412.265602] ? apic_timer_interrupt+0xf/0x20 [ 9412.269816] [ 9412.271179] ? native_safe_halt+0x2e/0x50 [ 9412.274105] kvm_wait+0xcd/0xe0 [ 9412.278831] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9412.282805] do_raw_spin_lock+0x177/0x190 [ 9412.285017] _raw_spin_lock+0x21/0x30 [ 9412.288861] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9412.292730] ? do_raw_spin_unlock+0x75/0x190 [ 9412.295525] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9412.303509] ? do_wait_intr+0xf0/0xf0 [ 9412.309109] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9412.315162] mdc_reint+0x2d/0x110 [mdc] [ 9412.319822] mdc_unlink+0x658/0x870 [mdc] [ 9412.324509] lmv_unlink+0x34d/0xbd0 [lmv] [ 9412.328905] ll_unlink+0x24d/0x7f0 [lustre] [ 9412.332593] vfs_unlink+0x202/0x330 [ 9412.339198] do_unlinkat+0x2fa/0x460 [ 9412.340462] __x64_sys_unlink+0x37/0x50 [ 9412.343233] do_syscall_64+0xc1/0x440 [ 9412.348969] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9412.353168] RIP: 0033:0x7f151bdee10b [ 9412.356585] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9412.370234] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9412.375315] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 9412.377729] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 9412.383486] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9412.386832] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9412.393642] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9416.684023] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7486s! [ 9416.692379] Showing busy workqueues and worker pools: [ 9416.697552] workqueue events: flags=0x0 [ 9416.700143] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9416.708244] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 9416.722453] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9416.725542] pending: kfree_rcu_monitor [ 9416.730453] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9416.732487] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 9416.742563] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9416.751589] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 9416.756143] workqueue events_long: flags=0x0 [ 9416.758472] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9416.762636] pending: lu_objects_flush [obdclass] [ 9416.768011] workqueue events_unbound: flags=0x2 [ 9416.776524] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9416.785471] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9416.802025] workqueue events_power_efficient: flags=0x80 [ 9416.818033] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9416.821361] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 9416.824568] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9416.829116] pending: neigh_periodic_work [ 9416.833282] workqueue mm_percpu_wq: flags=0x8 [ 9416.835175] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9416.843984] pending: vmstat_update, lru_add_drain_per_cpu [ 9416.849915] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9416.861696] pending: vmstat_update [ 9416.865543] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9416.870231] pending: lru_add_drain_per_cpu [ 9416.876372] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9416.889233] pending: lru_add_drain_per_cpu BAR(48) [ 9416.892958] workqueue writeback: flags=0x4a [ 9416.901063] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 9416.904961] pending: wb_workfn [ 9416.907180] workqueue xprtiod: flags=0xa [ 9416.911156] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9416.918193] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9416.927265] workqueue ipv6_addrconf: flags=0x40008 [ 9416.934669] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9416.945514] pending: addrconf_verify_work [ 9416.948484] workqueue knbd0-recv: flags=0x1a [ 9416.953726] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9416.962120] in-flight: 658:recv_work [ 9416.966168] workqueue ptlrpc_pinger: flags=0x2 [ 9416.970985] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9416.978406] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9416.987117] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7486s workers=4 idle: 32984 37 8 [ 9417.000376] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9401s workers=2 idle: 661 [ 9420.058322] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 9420.061901] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9420.106490] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9420.126764] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9420.140901] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9420.146444] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9420.164414] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9420.178618] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 9420.183672] RDX: 0000000000004705 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 9420.188020] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 9420.190271] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 9420.194089] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9420.197373] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 9420.204557] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9420.209354] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9420.212492] Call Trace: [ 9420.214159] [ 9420.215060] ? show_regs.cold.9+0x22/0x2f [ 9420.217424] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9420.219497] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9420.221580] ? watchdog+0x40/0x40 [ 9420.223125] ? hrtimer_interrupt+0x128/0x2c0 [ 9420.225111] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9420.228548] ? apic_timer_interrupt+0xf/0x20 [ 9420.233632] [ 9420.234403] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9420.239392] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9420.243845] do_raw_spin_lock+0x177/0x190 [ 9420.245347] _raw_spin_lock+0x21/0x30 [ 9420.246428] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9420.248954] obd_mod_free+0x46/0x140 [ptlrpc] [ 9420.251144] mdc_commit_open+0x91/0xb0 [mdc] [ 9420.252983] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9420.255177] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9420.257567] after_reply+0xe2c/0x1780 [ptlrpc] [ 9420.259883] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9420.262114] ? __next_timer_interrupt+0x160/0x160 [ 9420.263972] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9420.266173] ? do_wait_intr+0xf0/0xf0 [ 9420.267756] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9420.270207] kthread+0x1d1/0x200 [ 9420.271377] ? set_kthread_struct+0x70/0x70 [ 9420.273254] ret_from_fork+0x1f/0x30 [ 9424.058033] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9424.062967] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9424.113100] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9424.123642] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9424.135396] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9424.146397] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9424.155385] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9424.179154] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9424.185378] RAX: 0000000000004287 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9424.189400] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 9424.193842] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 9424.198452] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 9424.203060] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 9424.205424] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 9424.208279] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9424.210859] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9424.213904] Call Trace: [ 9424.214852] [ 9424.215355] ? show_regs.cold.9+0x22/0x2f [ 9424.216746] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9424.218523] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9424.222679] ? watchdog+0x40/0x40 [ 9424.223713] ? hrtimer_interrupt+0x128/0x2c0 [ 9424.229492] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9424.231303] ? apic_timer_interrupt+0xf/0x20 [ 9424.235740] [ 9424.236387] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9424.239195] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9424.245221] do_raw_spin_lock+0x177/0x190 [ 9424.246257] _raw_spin_lock+0x21/0x30 [ 9424.247346] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9424.254328] process_one_work+0x2c8/0x700 [ 9424.256578] worker_thread+0x296/0x6e0 [ 9424.258193] ? rescuer_thread+0x570/0x570 [ 9424.259924] kthread+0x1d1/0x200 [ 9424.261367] ? set_kthread_struct+0x70/0x70 [ 9424.264662] ret_from_fork+0x1f/0x30 [ 9436.059195] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 9436.063115] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9436.116625] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9436.125708] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9436.133531] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9436.137571] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9436.156147] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 9436.164206] RAX: 0000000000007e0c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9436.173286] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 9436.180263] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9436.184265] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 9436.187872] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 9436.191857] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 9436.195862] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9436.198559] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 9436.201778] Call Trace: [ 9436.202413] [ 9436.203314] ? show_regs.cold.9+0x22/0x2f [ 9436.205348] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9436.207988] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9436.210389] ? watchdog+0x40/0x40 [ 9436.212080] ? hrtimer_interrupt+0x128/0x2c0 [ 9436.214038] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9436.216288] ? apic_timer_interrupt+0xf/0x20 [ 9436.218366] [ 9436.219549] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9436.222157] do_raw_spin_lock+0x177/0x190 [ 9436.223984] _raw_spin_lock+0x21/0x30 [ 9436.225797] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9436.228534] ? do_raw_spin_unlock+0x75/0x190 [ 9436.230216] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9436.232781] ? do_wait_intr+0xf0/0xf0 [ 9436.234796] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9436.237803] mdc_close+0x53b/0xd70 [mdc] [ 9436.240176] lmv_close+0x18d/0x490 [lmv] [ 9436.243180] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9436.246092] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9436.249723] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9436.252322] ll_md_close+0x274/0x800 [lustre] [ 9436.260716] ll_file_release+0x128/0xc50 [lustre] [ 9436.267649] __fput+0x10d/0x440 [ 9436.269349] ____fput+0x12/0x20 [ 9436.275199] task_work_run+0xc6/0x110 [ 9436.277131] exit_to_usermode_loop+0x1dd/0x1f0 [ 9436.281516] do_syscall_64+0x3d6/0x440 [ 9436.282603] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9436.284969] RIP: 0033:0x7f151bee3cd8 [ 9436.287099] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9436.296228] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9436.300977] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 9436.307039] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 9436.311187] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9436.314958] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9436.317775] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9444.052738] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 9444.057062] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9444.084106] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9444.092740] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9444.098639] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9444.102073] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9444.110971] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9444.114885] RAX: 0000000000004306 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9444.118728] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 9444.121992] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 9444.125548] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 9444.129066] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 9444.132864] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 9444.138161] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9444.141552] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 9444.146682] Call Trace: [ 9444.147900] [ 9444.149274] ? show_regs.cold.9+0x22/0x2f [ 9444.151087] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9444.152848] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9444.154658] ? watchdog+0x40/0x40 [ 9444.155657] ? hrtimer_interrupt+0x128/0x2c0 [ 9444.157659] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9444.159224] ? apic_timer_interrupt+0xf/0x20 [ 9444.162044] [ 9444.162945] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9444.164959] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9444.167494] do_raw_spin_lock+0x177/0x190 [ 9444.169740] _raw_spin_lock+0x21/0x30 [ 9444.171187] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9444.173192] ? do_raw_spin_unlock+0x75/0x190 [ 9444.180068] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9444.182625] ? do_wait_intr+0xf0/0xf0 [ 9444.183714] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9444.185701] mdc_reint+0x2d/0x110 [mdc] [ 9444.187894] mdc_unlink+0x658/0x870 [mdc] [ 9444.189691] lmv_unlink+0x34d/0xbd0 [lmv] [ 9444.191511] ll_unlink+0x24d/0x7f0 [lustre] [ 9444.193550] vfs_unlink+0x202/0x330 [ 9444.194954] do_unlinkat+0x2fa/0x460 [ 9444.196650] __x64_sys_unlink+0x37/0x50 [ 9444.198234] do_syscall_64+0xc1/0x440 [ 9444.199775] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9444.201778] RIP: 0033:0x7f151bdee10b [ 9444.202962] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9444.210901] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9444.219047] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 9444.229826] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 9444.231996] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9444.241595] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9444.250172] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9447.392649] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7516s! [ 9447.400754] Showing busy workqueues and worker pools: [ 9447.406694] workqueue events: flags=0x0 [ 9447.408816] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9447.414232] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 9447.421800] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9447.425473] pending: kfree_rcu_monitor [ 9447.427020] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9447.430259] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 9447.435185] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9447.438385] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 9447.442022] workqueue events_long: flags=0x0 [ 9447.444379] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9447.447502] pending: lu_objects_flush [obdclass] [ 9447.449906] workqueue events_unbound: flags=0x2 [ 9447.451927] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9447.455085] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9447.460137] workqueue events_power_efficient: flags=0x80 [ 9447.462302] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9447.464828] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 9447.467388] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9447.469363] pending: neigh_periodic_work [ 9447.470387] workqueue mm_percpu_wq: flags=0x8 [ 9447.471754] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9447.473471] pending: vmstat_update, lru_add_drain_per_cpu [ 9447.475606] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9447.476939] pending: vmstat_update [ 9447.478120] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9447.481083] pending: lru_add_drain_per_cpu [ 9447.483287] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9447.486343] pending: lru_add_drain_per_cpu BAR(48) [ 9447.488766] workqueue writeback: flags=0x4a [ 9447.490255] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 9447.491934] pending: wb_workfn [ 9447.492940] workqueue xprtiod: flags=0xa [ 9447.494578] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9447.497290] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9447.501127] workqueue ipv6_addrconf: flags=0x40008 [ 9447.503234] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9447.506841] pending: addrconf_verify_work [ 9447.509422] workqueue knbd0-recv: flags=0x1a [ 9447.511750] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9447.516873] in-flight: 658:recv_work [ 9447.521941] workqueue ptlrpc_pinger: flags=0x2 [ 9447.524363] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9447.530795] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9447.537420] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7516s workers=4 idle: 32984 37 8 [ 9447.545041] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9432s workers=2 idle: 661 [ 9448.063324] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 9448.070368] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9448.107913] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9448.118839] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9448.129143] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9448.131383] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9448.149285] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9448.152445] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 9448.156069] RDX: 000000000000484f RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 9448.161069] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 9448.170249] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 9448.175045] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9448.180594] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 9448.186255] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9448.188276] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9448.198895] Call Trace: [ 9448.200937] [ 9448.202726] ? show_regs.cold.9+0x22/0x2f [ 9448.204134] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9448.208299] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9448.219259] ? watchdog+0x40/0x40 [ 9448.220518] ? hrtimer_interrupt+0x128/0x2c0 [ 9448.221998] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9448.231615] ? apic_timer_interrupt+0xf/0x20 [ 9448.233078] [ 9448.236590] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9448.240932] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9448.253008] do_raw_spin_lock+0x177/0x190 [ 9448.257943] _raw_spin_lock+0x21/0x30 [ 9448.259298] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9448.265853] obd_mod_free+0x46/0x140 [ptlrpc] [ 9448.269453] mdc_commit_open+0x91/0xb0 [mdc] [ 9448.274092] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9448.279879] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9448.288659] after_reply+0xe2c/0x1780 [ptlrpc] [ 9448.292165] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9448.296422] ? __next_timer_interrupt+0x160/0x160 [ 9448.301196] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9448.309310] ? do_wait_intr+0xf0/0xf0 [ 9448.314373] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9448.321703] kthread+0x1d1/0x200 [ 9448.323937] ? set_kthread_struct+0x70/0x70 [ 9448.327450] ret_from_fork+0x1f/0x30 [ 9452.056063] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9452.061682] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9452.135204] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9452.148487] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9452.154116] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9452.158967] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9452.161587] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9452.181482] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9452.189345] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9452.201856] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 9452.214525] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 9452.228978] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 9452.238822] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 9452.248582] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 9452.261938] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9452.265147] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9452.268431] Call Trace: [ 9452.269552] [ 9452.270321] ? show_regs.cold.9+0x22/0x2f [ 9452.272125] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9452.275803] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9452.280172] ? watchdog+0x40/0x40 [ 9452.283293] ? hrtimer_interrupt+0x128/0x2c0 [ 9452.289866] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9452.296621] ? apic_timer_interrupt+0xf/0x20 [ 9452.301220] [ 9452.302139] ? native_safe_halt+0x2e/0x50 [ 9452.304941] kvm_wait+0xcd/0xe0 [ 9452.311099] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9452.321212] do_raw_spin_lock+0x177/0x190 [ 9452.328348] _raw_spin_lock+0x21/0x30 [ 9452.336183] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9452.344638] process_one_work+0x2c8/0x700 [ 9452.348806] worker_thread+0x296/0x6e0 [ 9452.354047] ? rescuer_thread+0x570/0x570 [ 9452.356891] kthread+0x1d1/0x200 [ 9452.364675] ? set_kthread_struct+0x70/0x70 [ 9452.368222] ret_from_fork+0x1f/0x30 [ 9464.057683] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 9464.064801] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9464.126865] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9464.138121] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9464.155836] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9464.165618] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9464.185090] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9464.191954] RAX: 0000000000007abe RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9464.203774] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 9464.206878] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9464.223485] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 9464.234658] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 9464.244995] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 9464.250636] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9464.261044] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 9464.270706] Call Trace: [ 9464.273662] [ 9464.274612] ? show_regs.cold.9+0x22/0x2f [ 9464.276453] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9464.281413] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9464.286053] ? watchdog+0x40/0x40 [ 9464.289939] ? hrtimer_interrupt+0x128/0x2c0 [ 9464.292730] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9464.297713] ? apic_timer_interrupt+0xf/0x20 [ 9464.299760] [ 9464.302559] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9464.306996] do_raw_spin_lock+0x177/0x190 [ 9464.311157] _raw_spin_lock+0x21/0x30 [ 9464.312996] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9464.318504] ? do_raw_spin_unlock+0x75/0x190 [ 9464.330064] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9464.337428] ? do_wait_intr+0xf0/0xf0 [ 9464.343058] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9464.350671] mdc_close+0x53b/0xd70 [mdc] [ 9464.352863] lmv_close+0x18d/0x490 [lmv] [ 9464.356728] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9464.360019] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9464.367324] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9464.378098] ll_md_close+0x274/0x800 [lustre] [ 9464.381868] ll_file_release+0x128/0xc50 [lustre] [ 9464.387706] __fput+0x10d/0x440 [ 9464.389355] ____fput+0x12/0x20 [ 9464.390835] task_work_run+0xc6/0x110 [ 9464.392736] exit_to_usermode_loop+0x1dd/0x1f0 [ 9464.394963] do_syscall_64+0x3d6/0x440 [ 9464.396805] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9464.402766] RIP: 0033:0x7f151bee3cd8 [ 9464.405687] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9464.431986] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9464.440085] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 9464.450683] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 9464.461655] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9464.471357] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9464.480407] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9472.052039] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 9472.057042] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9472.082120] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9472.087994] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9472.093528] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9472.098417] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9472.109496] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9472.119380] RAX: 0000000000006513 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9472.122786] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 9472.128794] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 9472.132306] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 9472.136577] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 9472.142821] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 9472.149037] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9472.152384] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 9472.158076] Call Trace: [ 9472.159432] [ 9472.160671] ? show_regs.cold.9+0x22/0x2f [ 9472.164982] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9472.170014] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9472.174653] ? watchdog+0x40/0x40 [ 9472.176552] ? hrtimer_interrupt+0x128/0x2c0 [ 9472.179017] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9472.182487] ? apic_timer_interrupt+0xf/0x20 [ 9472.187880] [ 9472.190102] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9472.192942] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9472.195272] do_raw_spin_lock+0x177/0x190 [ 9472.197237] _raw_spin_lock+0x21/0x30 [ 9472.200714] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9472.206391] ? do_raw_spin_unlock+0x75/0x190 [ 9472.209872] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9472.215431] ? do_wait_intr+0xf0/0xf0 [ 9472.218812] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9472.223274] mdc_reint+0x2d/0x110 [mdc] [ 9472.225570] mdc_unlink+0x658/0x870 [mdc] [ 9472.226978] lmv_unlink+0x34d/0xbd0 [lmv] [ 9472.228876] ll_unlink+0x24d/0x7f0 [lustre] [ 9472.232466] vfs_unlink+0x202/0x330 [ 9472.236492] do_unlinkat+0x2fa/0x460 [ 9472.238719] __x64_sys_unlink+0x37/0x50 [ 9472.241228] do_syscall_64+0xc1/0x440 [ 9472.242613] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9472.245284] RIP: 0033:0x7f151bdee10b [ 9472.248026] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9472.260502] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9472.265235] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 9472.272472] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 9472.275727] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9472.282454] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9472.288902] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9476.055175] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 9476.058316] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9476.101737] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9476.112035] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9476.118953] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9476.121688] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9476.140753] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9476.148543] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 9476.166865] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 9476.179803] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 9476.192250] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 9476.210503] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9476.213111] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 9476.237534] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9476.255227] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9476.263603] Call Trace: [ 9476.264552] [ 9476.265246] ? show_regs.cold.9+0x22/0x2f [ 9476.272067] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9476.283312] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9476.291453] ? watchdog+0x40/0x40 [ 9476.296446] ? hrtimer_interrupt+0x128/0x2c0 [ 9476.301535] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9476.305987] ? apic_timer_interrupt+0xf/0x20 [ 9476.310973] [ 9476.311762] ? native_safe_halt+0x2e/0x50 [ 9476.313334] kvm_wait+0xcd/0xe0 [ 9476.319417] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9476.327477] do_raw_spin_lock+0x177/0x190 [ 9476.330318] _raw_spin_lock+0x21/0x30 [ 9476.333226] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9476.337293] obd_mod_free+0x46/0x140 [ptlrpc] [ 9476.341322] mdc_commit_open+0x91/0xb0 [mdc] [ 9476.344953] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9476.347040] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9476.357324] after_reply+0xe2c/0x1780 [ptlrpc] [ 9476.360509] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9476.367973] ? __next_timer_interrupt+0x160/0x160 [ 9476.371488] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9476.374532] ? do_wait_intr+0xf0/0xf0 [ 9476.375866] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9476.379988] kthread+0x1d1/0x200 [ 9476.383969] ? set_kthread_struct+0x70/0x70 [ 9476.386885] ret_from_fork+0x1f/0x30 [ 9477.607664] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7547s! [ 9477.622761] Showing busy workqueues and worker pools: [ 9477.632159] workqueue events: flags=0x0 [ 9477.635859] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9477.650331] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 9477.659435] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9477.668049] pending: kfree_rcu_monitor [ 9477.676452] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9477.689214] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 9477.700197] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9477.710869] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 9477.717199] workqueue events_long: flags=0x0 [ 9477.720422] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9477.732976] pending: lu_objects_flush [obdclass] [ 9477.743102] workqueue events_unbound: flags=0x2 [ 9477.757708] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9477.775041] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9477.783650] workqueue events_power_efficient: flags=0x80 [ 9477.787455] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9477.790576] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 9477.799679] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9477.802569] pending: neigh_periodic_work [ 9477.805901] workqueue mm_percpu_wq: flags=0x8 [ 9477.807900] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9477.811673] pending: vmstat_update, lru_add_drain_per_cpu [ 9477.814211] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9477.819065] pending: vmstat_update [ 9477.820995] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9477.826233] pending: lru_add_drain_per_cpu [ 9477.827544] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9477.832537] pending: lru_add_drain_per_cpu BAR(48) [ 9477.836473] workqueue writeback: flags=0x4a [ 9477.838671] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 9477.842090] pending: wb_workfn [ 9477.843579] workqueue xprtiod: flags=0xa [ 9477.845326] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9477.853974] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9477.862084] workqueue ipv6_addrconf: flags=0x40008 [ 9477.863748] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9477.866433] pending: addrconf_verify_work [ 9477.868184] workqueue knbd0-recv: flags=0x1a [ 9477.872862] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9477.880848] in-flight: 658:recv_work [ 9477.885322] workqueue ptlrpc_pinger: flags=0x2 [ 9477.889356] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9477.900752] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9477.903735] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7547s workers=4 idle: 32984 37 8 [ 9477.908096] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9462s workers=2 idle: 661 [ 9480.056028] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9480.058967] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9480.102059] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9480.115396] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9480.122668] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9480.126362] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9480.128767] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9480.140031] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9480.146298] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9480.149675] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 9480.154247] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 9480.158442] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 9480.165637] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 9480.171309] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 9480.177078] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9480.179115] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9480.187010] Call Trace: [ 9480.193125] [ 9480.194582] ? show_regs.cold.9+0x22/0x2f [ 9480.198032] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9480.204773] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9480.211307] ? watchdog+0x40/0x40 [ 9480.213632] ? hrtimer_interrupt+0x128/0x2c0 [ 9480.216941] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9480.218747] ? apic_timer_interrupt+0xf/0x20 [ 9480.220769] [ 9480.221420] ? native_safe_halt+0x2e/0x50 [ 9480.222839] kvm_wait+0xcd/0xe0 [ 9480.224462] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9480.226213] do_raw_spin_lock+0x177/0x190 [ 9480.227355] _raw_spin_lock+0x21/0x30 [ 9480.228318] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9480.230844] process_one_work+0x2c8/0x700 [ 9480.232597] worker_thread+0x296/0x6e0 [ 9480.233930] ? rescuer_thread+0x570/0x570 [ 9480.241317] kthread+0x1d1/0x200 [ 9480.242738] ? set_kthread_struct+0x70/0x70 [ 9480.244638] ret_from_fork+0x1f/0x30 [ 9492.060137] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 9492.079792] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9492.151522] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9492.169201] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9492.178854] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9492.180823] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9492.203445] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9492.207274] RAX: 0000000000005e4e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9492.214817] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 9492.222538] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9492.232175] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 9492.241456] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 9492.249403] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 9492.259467] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9492.266084] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 9492.273696] Call Trace: [ 9492.276993] [ 9492.277711] ? show_regs.cold.9+0x22/0x2f [ 9492.284237] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9492.286010] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9492.292632] ? watchdog+0x40/0x40 [ 9492.293721] ? hrtimer_interrupt+0x128/0x2c0 [ 9492.295351] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9492.296838] ? apic_timer_interrupt+0xf/0x20 [ 9492.298323] [ 9492.299928] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9492.302579] do_raw_spin_lock+0x177/0x190 [ 9492.304703] _raw_spin_lock+0x21/0x30 [ 9492.306850] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9492.309848] ? do_raw_spin_unlock+0x75/0x190 [ 9492.311745] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9492.313918] ? do_wait_intr+0xf0/0xf0 [ 9492.315717] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9492.318313] mdc_close+0x53b/0xd70 [mdc] [ 9492.321020] lmv_close+0x18d/0x490 [lmv] [ 9492.323284] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9492.326145] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9492.328615] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9492.330598] ll_md_close+0x274/0x800 [lustre] [ 9492.332770] ll_file_release+0x128/0xc50 [lustre] [ 9492.335131] __fput+0x10d/0x440 [ 9492.338968] ____fput+0x12/0x20 [ 9492.340467] task_work_run+0xc6/0x110 [ 9492.342530] exit_to_usermode_loop+0x1dd/0x1f0 [ 9492.344391] do_syscall_64+0x3d6/0x440 [ 9492.346330] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9492.348335] RIP: 0033:0x7f151bee3cd8 [ 9492.349777] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9492.357508] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9492.360697] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 9492.363489] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 9492.366177] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9492.369539] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9492.372510] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9504.055347] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 9504.057377] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9504.059252] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 9504.116160] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9504.116174] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9504.116177] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9504.116200] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9504.116205] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9504.116211] RAX: 0000000000007b89 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9504.116214] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 9504.116216] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 9504.116218] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 9504.116221] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 9504.116225] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 9504.116228] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9504.116342] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 9504.116351] Call Trace: [ 9504.116356] [ 9504.116362] ? show_regs.cold.9+0x22/0x2f [ 9504.116373] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9504.116380] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9504.116384] ? watchdog+0x40/0x40 [ 9504.116389] ? hrtimer_interrupt+0x128/0x2c0 [ 9504.116393] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9504.116401] ? apic_timer_interrupt+0xf/0x20 [ 9504.116406] [ 9504.116410] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9504.116414] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9504.116419] do_raw_spin_lock+0x177/0x190 [ 9504.116463] _raw_spin_lock+0x21/0x30 [ 9504.116472] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9504.116791] ? do_raw_spin_unlock+0x75/0x190 [ 9504.116798] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9504.130536] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9504.130618] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9504.130627] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9504.130630] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9504.130657] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9504.130662] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9504.130666] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 9504.130668] RDX: 0000000000007840 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 9504.130671] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 9504.130673] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 9504.130675] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9504.130679] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 9504.130682] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9504.130684] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9504.130692] Call Trace: [ 9504.130698] [ 9504.130704] ? show_regs.cold.9+0x22/0x2f [ 9504.130715] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9504.130722] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9504.130726] ? watchdog+0x40/0x40 [ 9504.130731] ? hrtimer_interrupt+0x128/0x2c0 [ 9504.130735] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9504.130745] ? apic_timer_interrupt+0xf/0x20 [ 9504.130750] [ 9504.130753] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9504.130759] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9504.130764] do_raw_spin_lock+0x177/0x190 [ 9504.130768] _raw_spin_lock+0x21/0x30 [ 9504.130774] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9504.131413] obd_mod_free+0x46/0x140 [ptlrpc] [ 9504.131611] mdc_commit_open+0x91/0xb0 [mdc] [ 9504.131658] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9504.132237] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9504.147509] ? do_wait_intr+0xf0/0xf0 [ 9504.147531] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9504.147918] mdc_reint+0x2d/0x110 [mdc] [ 9504.147961] mdc_unlink+0x658/0x870 [mdc] [ 9504.147999] lmv_unlink+0x34d/0xbd0 [lmv] [ 9504.148030] ll_unlink+0x24d/0x7f0 [lustre] [ 9504.148155] vfs_unlink+0x202/0x330 [ 9504.148164] do_unlinkat+0x2fa/0x460 [ 9504.148168] __x64_sys_unlink+0x37/0x50 [ 9504.148172] do_syscall_64+0xc1/0x440 [ 9504.148179] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9504.148189] RIP: 0033:0x7f151bdee10b [ 9504.148196] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9504.148200] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9504.148205] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 9504.148207] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 9504.148209] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9504.148211] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9504.148214] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9504.674218] after_reply+0xe2c/0x1780 [ptlrpc] [ 9504.682801] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9504.688809] ? __next_timer_interrupt+0x160/0x160 [ 9504.697166] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9504.701233] ? do_wait_intr+0xf0/0xf0 [ 9504.704347] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9504.712592] kthread+0x1d1/0x200 [ 9504.714792] ? set_kthread_struct+0x70/0x70 [ 9504.722606] ret_from_fork+0x1f/0x30 [ 9508.056085] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 9508.059110] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9508.084461] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9508.093573] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9508.103831] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9508.112451] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9508.120627] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9508.146909] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9508.161082] RAX: 0000000000006c94 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9508.170260] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 9508.184254] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 9508.195424] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 9508.203626] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 9508.212134] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 9508.222996] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9508.237374] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9508.242985] Call Trace: [ 9508.247356] [ 9508.249825] ? show_regs.cold.9+0x22/0x2f [ 9508.253091] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9508.259268] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9508.261710] ? watchdog+0x40/0x40 [ 9508.268034] ? hrtimer_interrupt+0x128/0x2c0 [ 9508.272496] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9508.281337] ? apic_timer_interrupt+0xf/0x20 [ 9508.286937] [ 9508.295099] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9508.301092] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9508.308911] do_raw_spin_lock+0x177/0x190 [ 9508.315834] _raw_spin_lock+0x21/0x30 [ 9508.318404] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9508.320672] BUG: workqueue lockup - pool [ 9508.325462] process_one_work+0x2c8/0x700 [ 9508.325492] worker_thread+0x296/0x6e0 [ 9508.325497] ? rescuer_thread+0x570/0x570 [ 9508.325501] kthread+0x1d1/0x200 [ 9508.325506] ? set_kthread_struct+0x70/0x70 [ 9508.325509] ret_from_fork+0x1f/0x30 [ 9508.357086] cpus=0-3 flags=0x4 nice=0 stuck for 7577s! [ 9508.363853] Showing busy workqueues and worker pools: [ 9508.368648] workqueue events: flags=0x0 [ 9508.372547] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9508.379580] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 9508.389470] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9508.398779] pending: kfree_rcu_monitor [ 9508.404738] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9508.415209] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 9508.426264] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9508.433742] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 9508.438318] workqueue events_long: flags=0x0 [ 9508.439639] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9508.443567] pending: lu_objects_flush [obdclass] [ 9508.451536] workqueue events_unbound: flags=0x2 [ 9508.454622] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9508.459109] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9508.463579] workqueue events_power_efficient: flags=0x80 [ 9508.467544] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9508.470701] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 9508.473678] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9508.476936] pending: neigh_periodic_work [ 9508.478860] workqueue mm_percpu_wq: flags=0x8 [ 9508.480702] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9508.487621] pending: vmstat_update, lru_add_drain_per_cpu [ 9508.491010] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9508.494892] pending: vmstat_update [ 9508.501009] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9508.504958] pending: lru_add_drain_per_cpu [ 9508.507380] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9508.511042] pending: lru_add_drain_per_cpu BAR(48) [ 9508.513501] workqueue writeback: flags=0x4a [ 9508.515465] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 9508.518755] pending: wb_workfn [ 9508.520628] workqueue xprtiod: flags=0xa [ 9508.522842] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9508.526066] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9508.529651] workqueue ipv6_addrconf: flags=0x40008 [ 9508.533541] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9508.540111] pending: addrconf_verify_work [ 9508.544348] workqueue knbd0-recv: flags=0x1a [ 9508.549059] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9508.559122] in-flight: 658:recv_work [ 9508.565448] workqueue ptlrpc_pinger: flags=0x2 [ 9508.570357] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9508.575050] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9508.578062] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7577s workers=4 idle: 32984 37 8 [ 9508.590628] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9493s workers=2 idle: 661 [ 9520.057815] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 9520.061280] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9520.103642] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9520.115200] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9520.123207] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9520.133321] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9520.148852] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9520.155509] RAX: 0000000000007c99 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9520.159870] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 9520.166398] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9520.174542] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 9520.182617] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 9520.186824] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 9520.192097] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9520.195604] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 9520.198759] Call Trace: [ 9520.199622] [ 9520.200529] ? show_regs.cold.9+0x22/0x2f [ 9520.203241] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9520.209747] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9520.211453] ? watchdog+0x40/0x40 [ 9520.212961] ? hrtimer_interrupt+0x128/0x2c0 [ 9520.214509] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9520.218107] ? apic_timer_interrupt+0xf/0x20 [ 9520.219601] [ 9520.221685] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9520.228908] do_raw_spin_lock+0x177/0x190 [ 9520.231219] _raw_spin_lock+0x21/0x30 [ 9520.233423] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9520.237346] ? do_raw_spin_unlock+0x75/0x190 [ 9520.238728] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9520.241065] ? do_wait_intr+0xf0/0xf0 [ 9520.247502] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9520.253063] mdc_close+0x53b/0xd70 [mdc] [ 9520.254523] lmv_close+0x18d/0x490 [lmv] [ 9520.256081] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9520.261770] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9520.263746] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9520.265552] ll_md_close+0x274/0x800 [lustre] [ 9520.267342] ll_file_release+0x128/0xc50 [lustre] [ 9520.269307] __fput+0x10d/0x440 [ 9520.270453] ____fput+0x12/0x20 [ 9520.271522] task_work_run+0xc6/0x110 [ 9520.274489] exit_to_usermode_loop+0x1dd/0x1f0 [ 9520.276107] do_syscall_64+0x3d6/0x440 [ 9520.278223] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9520.282566] RIP: 0033:0x7f151bee3cd8 [ 9520.283964] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9520.293982] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9520.299673] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 9520.303253] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 9520.309554] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9520.313209] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9520.315413] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9532.056760] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 9532.065101] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9532.113895] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9532.127491] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9532.134808] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9532.141939] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9532.165941] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9532.174912] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 9532.182797] RDX: 0000000000004aaa RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 9532.187413] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 9532.196258] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 9532.201483] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9532.207976] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 9532.215193] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9532.217801] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9532.221906] Call Trace: [ 9532.223062] [ 9532.224779] ? show_regs.cold.9+0x22/0x2f [ 9532.228113] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9532.231570] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9532.235945] ? watchdog+0x40/0x40 [ 9532.238136] ? hrtimer_interrupt+0x128/0x2c0 [ 9532.240828] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9532.244790] ? apic_timer_interrupt+0xf/0x20 [ 9532.248411] [ 9532.250854] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9532.257027] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9532.263190] do_raw_spin_lock+0x177/0x190 [ 9532.264567] _raw_spin_lock+0x21/0x30 [ 9532.274989] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9532.285895] obd_mod_free+0x46/0x140 [ptlrpc] [ 9532.294456] mdc_commit_open+0x91/0xb0 [mdc] [ 9532.301601] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9532.303569] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9532.322046] after_reply+0xe2c/0x1780 [ptlrpc] [ 9532.330533] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9532.341960] ? __next_timer_interrupt+0x160/0x160 [ 9532.343991] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9532.349931] ? do_wait_intr+0xf0/0xf0 [ 9532.353494] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9532.360719] kthread+0x1d1/0x200 [ 9532.364928] ? set_kthread_struct+0x70/0x70 [ 9532.370569] ret_from_fork+0x1f/0x30 [ 9536.059062] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 9536.060027] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 9536.082980] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9536.083065] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9536.083071] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9536.083074] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9536.083087] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9536.083091] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9536.083095] RAX: 0000000000003e09 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9536.083098] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 9536.083100] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 9536.083102] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 9536.083104] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 9536.083107] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 9536.083110] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9536.083113] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 9536.083121] Call Trace: [ 9536.083126] [ 9536.083132] ? show_regs.cold.9+0x22/0x2f [ 9536.083141] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9536.083146] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9536.083151] ? watchdog+0x40/0x40 [ 9536.083155] ? hrtimer_interrupt+0x128/0x2c0 [ 9536.083159] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9536.083167] ? apic_timer_interrupt+0xf/0x20 [ 9536.083172] [ 9536.083175] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9536.083180] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9536.083184] do_raw_spin_lock+0x177/0x190 [ 9536.083189] _raw_spin_lock+0x21/0x30 [ 9536.083195] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9536.097412] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9536.098225] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9536.098232] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9536.098239] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9536.098720] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9536.098731] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9536.098736] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9536.098740] RAX: 0000000000006d8f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9536.098742] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 9536.099082] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 9536.099085] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 9536.099087] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 9536.099094] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 9536.099097] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9536.099099] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9536.099106] Call Trace: [ 9536.099111] [ 9536.099116] ? show_regs.cold.9+0x22/0x2f [ 9536.099125] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9536.099132] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9536.099137] ? watchdog+0x40/0x40 [ 9536.099141] ? hrtimer_interrupt+0x128/0x2c0 [ 9536.099145] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9536.099154] ? apic_timer_interrupt+0xf/0x20 [ 9536.099158] [ 9536.099162] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9536.099167] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9536.099172] do_raw_spin_lock+0x177/0x190 [ 9536.099176] _raw_spin_lock+0x21/0x30 [ 9536.099181] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9536.099728] process_one_work+0x2c8/0x700 [ 9536.099735] worker_thread+0x296/0x6e0 [ 9536.099740] ? rescuer_thread+0x570/0x570 [ 9536.099744] kthread+0x1d1/0x200 [ 9536.099748] ? set_kthread_struct+0x70/0x70 [ 9536.099757] ret_from_fork+0x1f/0x30 [ 9536.780152] ? do_raw_spin_unlock+0x75/0x190 [ 9536.794087] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9536.800357] ? do_wait_intr+0xf0/0xf0 [ 9536.801607] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9536.825163] mdc_reint+0x2d/0x110 [mdc] [ 9536.831119] mdc_unlink+0x658/0x870 [mdc] [ 9536.838013] lmv_unlink+0x34d/0xbd0 [lmv] [ 9536.848270] ll_unlink+0x24d/0x7f0 [lustre] [ 9536.849846] vfs_unlink+0x202/0x330 [ 9536.875247] do_unlinkat+0x2fa/0x460 [ 9536.876507] __x64_sys_unlink+0x37/0x50 [ 9536.879357] do_syscall_64+0xc1/0x440 [ 9536.885763] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9536.895572] RIP: 0033:0x7f151bdee10b [ 9536.896826] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9536.922120] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9536.948381] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 9536.956847] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 9536.970834] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9536.977700] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9536.985341] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9539.051621] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7608s! [ 9539.064640] Showing busy workqueues and worker pools: [ 9539.066537] workqueue events: flags=0x0 [ 9539.072507] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9539.079035] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 9539.084213] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9539.088028] pending: kfree_rcu_monitor [ 9539.092256] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9539.096733] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 9539.105408] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9539.111918] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 9539.125178] workqueue events_long: flags=0x0 [ 9539.132106] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9539.143264] pending: lu_objects_flush [obdclass] [ 9539.150122] workqueue events_unbound: flags=0x2 [ 9539.157051] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9539.160909] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9539.166474] workqueue events_power_efficient: flags=0x80 [ 9539.173073] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9539.181885] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 9539.188872] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9539.194671] pending: neigh_periodic_work [ 9539.197026] workqueue mm_percpu_wq: flags=0x8 [ 9539.199209] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9539.207643] pending: vmstat_update, lru_add_drain_per_cpu [ 9539.215192] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9539.223270] pending: vmstat_update [ 9539.228872] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9539.233239] pending: lru_add_drain_per_cpu [ 9539.236109] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9539.239955] pending: lru_add_drain_per_cpu BAR(48) [ 9539.245267] workqueue writeback: flags=0x4a [ 9539.249972] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 9539.253107] pending: wb_workfn [ 9539.256200] workqueue xprtiod: flags=0xa [ 9539.259883] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9539.271808] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9539.281824] workqueue ipv6_addrconf: flags=0x40008 [ 9539.283361] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9539.291415] pending: addrconf_verify_work [ 9539.305190] workqueue knbd0-recv: flags=0x1a [ 9539.312125] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9539.322196] in-flight: 658:recv_work [ 9539.329108] workqueue ptlrpc_pinger: flags=0x2 [ 9539.335677] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9539.342784] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9539.347968] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7608s workers=4 idle: 32984 37 8 [ 9539.359024] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9524s workers=2 idle: 661 [ 9544.806117] rcu: INFO: rcu_sched self-detected stall on CPU [ 9544.808885] rcu: 1-...!: (4318696 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 9544.814413] (t=7620159 jiffies g=249657 q=4590) [ 9544.817831] rcu: rcu_sched kthread starved for 7615111 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 9544.824812] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 9544.833244] rcu: RCU grace-period kthread stack dump: [ 9544.839569] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 9544.850037] Call Trace: [ 9544.853062] __schedule+0x351/0xcb0 [ 9544.858556] schedule+0xc0/0x180 [ 9544.860612] schedule_timeout+0xb4/0x190 [ 9544.863797] ? do_raw_spin_unlock+0x75/0x190 [ 9544.868985] ? __next_timer_interrupt+0x160/0x160 [ 9544.874030] ? prepare_to_swait_event+0xf3/0x140 [ 9544.879605] rcu_gp_kthread+0x8dd/0x15b0 [ 9544.884125] ? force_qs_rnp+0x2c0/0x2c0 [ 9544.888111] kthread+0x1d1/0x200 [ 9544.892121] ? set_kthread_struct+0x70/0x70 [ 9544.897087] ret_from_fork+0x1f/0x30 [ 9544.900794] rcu: Stack dump where RCU GP kthread last ran: [ 9544.906093] Sending NMI from CPU 1 to CPUs 2: [ 9544.910291] NMI backtrace for cpu 2 skipped: idling at native_safe_halt+0x2e/0x50 [ 9544.913771] NMI backtrace for cpu 1 [ 9544.918091] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9544.931616] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9544.939082] Call Trace: [ 9544.942481] [ 9544.943363] ? dump_stack+0xbb/0x10e [ 9544.949683] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9544.956146] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9544.959250] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9544.966261] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9544.972601] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9544.978340] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 9544.983938] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 9544.988607] ? update_process_times+0x6d/0xc0 [ 9544.992337] ? tick_sched_handle.isra.18+0x27/0x90 [ 9544.997315] ? tick_sched_timer+0x53/0xb0 [ 9545.002566] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9545.005331] ? tick_sched_handle.isra.18+0x90/0x90 [ 9545.010126] ? hrtimer_interrupt+0x128/0x2c0 [ 9545.012244] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9545.017166] ? apic_timer_interrupt+0xf/0x20 [ 9545.020245] [ 9545.021340] ? native_safe_halt+0x2e/0x50 [ 9545.026573] ? kvm_wait+0xcd/0xe0 [ 9545.029784] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9545.034250] ? do_raw_spin_lock+0x177/0x190 [ 9545.037526] ? _raw_spin_lock+0x21/0x30 [ 9545.041133] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9545.045812] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9545.049967] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9545.053672] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9545.058146] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9545.062675] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9545.066848] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9545.071943] ? __next_timer_interrupt+0x160/0x160 [ 9545.076237] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9545.080302] ? do_wait_intr+0xf0/0xf0 [ 9545.083181] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9545.087810] ? kthread+0x1d1/0x200 [ 9545.090601] ? set_kthread_struct+0x70/0x70 [ 9545.096673] ? ret_from_fork+0x1f/0x30 [ 9548.057028] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 9548.061595] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9548.103105] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9548.117731] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9548.123128] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9548.126720] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9548.139167] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9548.148137] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9548.153088] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 9548.157265] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9548.160988] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 9548.165046] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 9548.168274] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 9548.172622] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9548.181766] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 9548.192974] Call Trace: [ 9548.195868] [ 9548.198772] ? show_regs.cold.9+0x22/0x2f [ 9548.203681] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9548.206844] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9548.210959] ? watchdog+0x40/0x40 [ 9548.213269] ? hrtimer_interrupt+0x128/0x2c0 [ 9548.224127] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9548.231189] ? apic_timer_interrupt+0xf/0x20 [ 9548.237651] [ 9548.239202] ? native_safe_halt+0x2e/0x50 [ 9548.242924] kvm_wait+0xcd/0xe0 [ 9548.244746] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9548.248949] do_raw_spin_lock+0x177/0x190 [ 9548.252717] _raw_spin_lock+0x21/0x30 [ 9548.257199] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9548.263236] ? do_raw_spin_unlock+0x75/0x190 [ 9548.265969] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9548.269373] ? do_wait_intr+0xf0/0xf0 [ 9548.277224] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9548.284412] mdc_close+0x53b/0xd70 [mdc] [ 9548.288646] lmv_close+0x18d/0x490 [lmv] [ 9548.291544] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9548.299720] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9548.305666] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9548.310619] ll_md_close+0x274/0x800 [lustre] [ 9548.316232] ll_file_release+0x128/0xc50 [lustre] [ 9548.322606] __fput+0x10d/0x440 [ 9548.325235] ____fput+0x12/0x20 [ 9548.330326] task_work_run+0xc6/0x110 [ 9548.333949] exit_to_usermode_loop+0x1dd/0x1f0 [ 9548.336956] do_syscall_64+0x3d6/0x440 [ 9548.339083] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9548.342292] RIP: 0033:0x7f151bee3cd8 [ 9548.344721] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9548.368435] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9548.385170] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 9548.395226] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 9548.399655] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9548.403905] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9548.409485] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9564.053401] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 9564.057935] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9564.094103] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9564.103312] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9564.107802] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9564.112399] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9564.125799] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9564.130431] RAX: 0000000000004784 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9564.135147] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 9564.138952] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 9564.149566] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 9564.154933] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 9564.157247] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 9564.169006] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9564.174649] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 9564.182070] Call Trace: [ 9564.183235] [ 9564.185847] ? show_regs.cold.9+0x22/0x2f [ 9564.190514] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9564.194942] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9564.200086] ? watchdog+0x40/0x40 [ 9564.204304] ? hrtimer_interrupt+0x128/0x2c0 [ 9564.209196] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9564.214942] ? apic_timer_interrupt+0xf/0x20 [ 9564.219143] [ 9564.220974] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9564.226556] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9564.229204] do_raw_spin_lock+0x177/0x190 [ 9564.231123] _raw_spin_lock+0x21/0x30 [ 9564.234002] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9564.237686] ? do_raw_spin_unlock+0x75/0x190 [ 9564.240863] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9564.244916] ? do_wait_intr+0xf0/0xf0 [ 9564.248060] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9564.252696] mdc_reint+0x2d/0x110 [mdc] [ 9564.256325] mdc_unlink+0x658/0x870 [mdc] [ 9564.259641] lmv_unlink+0x34d/0xbd0 [lmv] [ 9564.263111] ll_unlink+0x24d/0x7f0 [lustre] [ 9564.266999] vfs_unlink+0x202/0x330 [ 9564.268170] do_unlinkat+0x2fa/0x460 [ 9564.271957] __x64_sys_unlink+0x37/0x50 [ 9564.277024] do_syscall_64+0xc1/0x440 [ 9564.278717] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9564.281195] RIP: 0033:0x7f151bdee10b [ 9564.285055] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9564.302015] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9564.307222] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 9564.314240] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 9564.321461] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9564.323715] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9564.332135] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9569.763039] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7639s! [ 9569.767424] Showing busy workqueues and worker pools: [ 9569.769892] workqueue events: flags=0x0 [ 9569.771927] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9569.775066] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 9569.782263] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9569.789308] pending: kfree_rcu_monitor [ 9569.790807] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9569.796099] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 9569.802101] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9569.811593] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 9569.818373] workqueue events_long: flags=0x0 [ 9569.822909] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9569.829148] pending: lu_objects_flush [obdclass] [ 9569.832411] workqueue events_unbound: flags=0x2 [ 9569.834803] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9569.838424] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9569.845200] workqueue events_power_efficient: flags=0x80 [ 9569.849914] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9569.852985] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 9569.857684] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9569.863512] pending: neigh_periodic_work [ 9569.865979] workqueue mm_percpu_wq: flags=0x8 [ 9569.868138] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9569.871925] pending: vmstat_update, lru_add_drain_per_cpu [ 9569.874735] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9569.879858] pending: vmstat_update [ 9569.881724] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9569.884924] pending: lru_add_drain_per_cpu [ 9569.887056] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9569.890298] pending: lru_add_drain_per_cpu BAR(48) [ 9569.892675] workqueue writeback: flags=0x4a [ 9569.895320] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 9569.898095] pending: wb_workfn [ 9569.899613] workqueue xprtiod: flags=0xa [ 9569.901501] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9569.905919] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9569.909841] workqueue ipv6_addrconf: flags=0x40008 [ 9569.913864] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9569.919505] pending: addrconf_verify_work [ 9569.922609] workqueue knbd0-recv: flags=0x1a [ 9569.924294] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9569.927105] in-flight: 658:recv_work [ 9569.929723] workqueue ptlrpc_pinger: flags=0x2 [ 9569.931790] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9569.934735] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9569.937238] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7639s workers=4 idle: 32984 37 8 [ 9569.940399] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9554s workers=2 idle: 661 [ 9572.058847] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9572.058944] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 9572.065238] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9572.065315] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9572.065321] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9572.065338] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9572.065609] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9572.065619] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9572.065623] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9572.065627] RAX: 0000000000007c24 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9572.065629] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 9572.065631] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 9572.065633] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 9572.065636] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 9572.065639] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 9572.065642] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9572.065644] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9572.065652] Call Trace: [ 9572.065657] [ 9572.065662] ? show_regs.cold.9+0x22/0x2f [ 9572.065678] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9572.065684] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9572.065689] ? watchdog+0x40/0x40 [ 9572.065693] ? hrtimer_interrupt+0x128/0x2c0 [ 9572.065698] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9572.065706] ? apic_timer_interrupt+0xf/0x20 [ 9572.065711] [ 9572.065717] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9572.065722] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9572.065727] do_raw_spin_lock+0x177/0x190 [ 9572.065732] _raw_spin_lock+0x21/0x30 [ 9572.065737] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9572.065950] process_one_work+0x2c8/0x700 [ 9572.065957] worker_thread+0x296/0x6e0 [ 9572.065961] ? rescuer_thread+0x570/0x570 [ 9572.065965] kthread+0x1d1/0x200 [ 9572.065970] ? set_kthread_struct+0x70/0x70 [ 9572.065973] ret_from_fork+0x1f/0x30 [ 9572.165981] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9572.182948] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9572.186247] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9572.189434] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9572.192042] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9572.199017] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9572.202223] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 9572.204795] RDX: 0000000000007a03 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 9572.207580] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 9572.210415] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 9572.213110] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9572.215576] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 9572.218610] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9572.220513] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9572.223044] Call Trace: [ 9572.223807] [ 9572.224703] ? show_regs.cold.9+0x22/0x2f [ 9572.226098] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9572.227749] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9572.229359] ? watchdog+0x40/0x40 [ 9572.230787] ? hrtimer_interrupt+0x128/0x2c0 [ 9572.232094] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9572.234044] ? apic_timer_interrupt+0xf/0x20 [ 9572.235525] [ 9572.236191] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9572.237953] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9572.240232] do_raw_spin_lock+0x177/0x190 [ 9572.241849] _raw_spin_lock+0x21/0x30 [ 9572.243442] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9572.245343] obd_mod_free+0x46/0x140 [ptlrpc] [ 9572.247336] mdc_commit_open+0x91/0xb0 [mdc] [ 9572.249044] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9572.250854] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9572.253114] after_reply+0xe2c/0x1780 [ptlrpc] [ 9572.254789] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9572.256862] ? __next_timer_interrupt+0x160/0x160 [ 9572.258747] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9572.260852] ? do_wait_intr+0xf0/0xf0 [ 9572.262197] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9572.264209] kthread+0x1d1/0x200 [ 9572.265521] ? set_kthread_struct+0x70/0x70 [ 9572.267001] ret_from_fork+0x1f/0x30 [ 9576.057716] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 9576.060192] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9576.087814] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9576.091803] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9576.095387] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9576.097463] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9576.105375] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9576.108301] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9576.110580] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 9576.113594] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9576.116581] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 9576.125394] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 9576.135179] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 9576.143653] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9576.149555] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 9576.154397] Call Trace: [ 9576.155402] [ 9576.156337] ? show_regs.cold.9+0x22/0x2f [ 9576.159878] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9576.163063] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9576.165085] ? watchdog+0x40/0x40 [ 9576.166449] ? hrtimer_interrupt+0x128/0x2c0 [ 9576.168605] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9576.174346] ? apic_timer_interrupt+0xf/0x20 [ 9576.176295] [ 9576.177042] ? native_safe_halt+0x2e/0x50 [ 9576.179114] kvm_wait+0xcd/0xe0 [ 9576.180129] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9576.181414] do_raw_spin_lock+0x177/0x190 [ 9576.182803] _raw_spin_lock+0x21/0x30 [ 9576.184060] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9576.190503] ? do_raw_spin_unlock+0x75/0x190 [ 9576.193022] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9576.203131] ? do_wait_intr+0xf0/0xf0 [ 9576.205377] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9576.214938] mdc_close+0x53b/0xd70 [mdc] [ 9576.219811] lmv_close+0x18d/0x490 [lmv] [ 9576.221669] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9576.228804] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9576.231902] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9576.233471] ll_md_close+0x274/0x800 [lustre] [ 9576.235605] ll_file_release+0x128/0xc50 [lustre] [ 9576.237452] __fput+0x10d/0x440 [ 9576.238459] ____fput+0x12/0x20 [ 9576.244437] task_work_run+0xc6/0x110 [ 9576.245977] exit_to_usermode_loop+0x1dd/0x1f0 [ 9576.247677] do_syscall_64+0x3d6/0x440 [ 9576.249599] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9576.251743] RIP: 0033:0x7f151bee3cd8 [ 9576.253423] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9576.261440] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9576.264710] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 9576.269111] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 9576.272541] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9576.274961] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9576.277642] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9596.052244] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 9596.057341] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9596.096408] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9596.104845] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9596.109768] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9596.118471] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9596.146644] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9596.155127] RAX: 000000000000600b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9596.161239] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 9596.166801] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 9596.171200] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 9596.176983] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 9596.185325] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 9596.192189] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9596.195223] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 9596.198703] Call Trace: [ 9596.199689] [ 9596.201111] ? show_regs.cold.9+0x22/0x2f [ 9596.202852] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9596.205716] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9596.208550] ? watchdog+0x40/0x40 [ 9596.210449] ? hrtimer_interrupt+0x128/0x2c0 [ 9596.212656] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9596.215504] ? apic_timer_interrupt+0xf/0x20 [ 9596.217915] [ 9596.218897] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9596.221723] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9596.224646] do_raw_spin_lock+0x177/0x190 [ 9596.226895] _raw_spin_lock+0x21/0x30 [ 9596.229308] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9596.232389] ? do_raw_spin_unlock+0x75/0x190 [ 9596.234024] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9596.237354] ? do_wait_intr+0xf0/0xf0 [ 9596.238838] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9596.242545] mdc_reint+0x2d/0x110 [mdc] [ 9596.244533] mdc_unlink+0x658/0x870 [mdc] [ 9596.248290] lmv_unlink+0x34d/0xbd0 [lmv] [ 9596.251905] ll_unlink+0x24d/0x7f0 [lustre] [ 9596.257834] vfs_unlink+0x202/0x330 [ 9596.259654] do_unlinkat+0x2fa/0x460 [ 9596.262499] __x64_sys_unlink+0x37/0x50 [ 9596.264692] do_syscall_64+0xc1/0x440 [ 9596.267113] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9596.268985] RIP: 0033:0x7f151bdee10b [ 9596.271142] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9596.279589] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9596.283371] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 9596.286679] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 9596.290300] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9596.294205] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9596.297432] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9599.975043] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7669s! [ 9599.977554] Showing busy workqueues and worker pools: [ 9599.979507] workqueue events: flags=0x0 [ 9599.981087] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9599.983982] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 9599.987811] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9599.990595] pending: kfree_rcu_monitor [ 9599.992248] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9599.994344] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 9599.997531] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9600.000032] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 9600.003205] workqueue events_long: flags=0x0 [ 9600.004798] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9600.007648] pending: lu_objects_flush [obdclass] [ 9600.010013] workqueue events_unbound: flags=0x2 [ 9600.011762] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9600.013909] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9600.018155] workqueue events_power_efficient: flags=0x80 [ 9600.019835] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9600.022531] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 9600.025571] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9600.028309] pending: neigh_periodic_work [ 9600.030230] workqueue mm_percpu_wq: flags=0x8 [ 9600.032035] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9600.034870] pending: vmstat_update, lru_add_drain_per_cpu [ 9600.037397] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9600.039893] pending: vmstat_update [ 9600.041021] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9600.043772] pending: lru_add_drain_per_cpu [ 9600.045747] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9600.048483] pending: lru_add_drain_per_cpu BAR(48) [ 9600.050730] workqueue writeback: flags=0x4a [ 9600.052491] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 9600.055007] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 9600.055229] pending: [ 9600.056028] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9600.056035] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9600.056119] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9600.056126] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9600.056133] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9600.056404] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9600.056414] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9600.056418] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9600.056422] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9600.056424] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 9600.056426] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 9600.056428] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 9600.056431] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 9600.056434] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 9600.056437] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9600.056439] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9600.056448] Call Trace: [ 9600.056453] [ 9600.056458] ? show_regs.cold.9+0x22/0x2f [ 9600.056469] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9600.056476] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9600.056481] ? watchdog+0x40/0x40 [ 9600.056485] ? hrtimer_interrupt+0x128/0x2c0 [ 9600.056489] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9600.056495] ? apic_timer_interrupt+0xf/0x20 [ 9600.056500] [ 9600.056504] ? native_safe_halt+0x2e/0x50 [ 9600.056508] kvm_wait+0xcd/0xe0 [ 9600.056514] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9600.056522] do_raw_spin_lock+0x177/0x190 [ 9600.056527] _raw_spin_lock+0x21/0x30 [ 9600.056531] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9600.056744] process_one_work+0x2c8/0x700 [ 9600.056751] worker_thread+0x296/0x6e0 [ 9600.056756] ? rescuer_thread+0x570/0x570 [ 9600.056760] kthread+0x1d1/0x200 [ 9600.056776] ? set_kthread_struct+0x70/0x70 [ 9600.056780] ret_from_fork+0x1f/0x30 [ 9600.065457] Modules linked in: [ 9600.066528] wb_workfn [ 9600.066574] workqueue xprtiod: flags=0xa [ 9600.066578] pwq 8: [ 9600.075357] lustre(O) [ 9600.094234] cpus=0-3 [ 9600.101992] osp(O) [ 9600.105565] flags=0x4 nice=0 [ 9600.109209] ofd(O) [ 9600.111243] active=3/256 refcnt=5 [ 9600.131999] lod(O) [ 9600.134454] pending: [ 9600.139811] mdt(O) [ 9600.142164] xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9600.142219] workqueue ipv6_addrconf: flags=0x40008 [ 9600.142226] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9600.142236] pending: addrconf_verify_work [ 9600.142249] workqueue knbd0-recv: flags=0x1a [ 9600.147466] mdd(O) mgs(O) [ 9600.150510] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9600.150529] in-flight: 658:recv_work [ 9600.150545] workqueue ptlrpc_pinger: flags=0x2 [ 9600.150548] pwq 8: [ 9600.156188] lquota(O) [ 9600.159507] cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9600.159525] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9600.163904] lfsck(O) [ 9600.166858] [ 9600.166898] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7669s workers=4 [ 9600.168636] obdecho(O) [ 9600.169649] idle: 32984 37 8 [ 9600.169681] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9584s workers=2 idle: 661 [ 9600.285718] mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9600.319299] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9600.327463] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9600.332955] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9600.340587] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9600.362825] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9600.369205] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 9600.375614] RDX: 0000000000003340 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 9600.380711] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 9600.385133] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 9600.390662] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9600.394809] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 9600.398028] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9600.400793] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9600.404631] Call Trace: [ 9600.407237] [ 9600.409714] ? show_regs.cold.9+0x22/0x2f [ 9600.412614] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9600.417609] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9600.424644] ? watchdog+0x40/0x40 [ 9600.427689] ? hrtimer_interrupt+0x128/0x2c0 [ 9600.430411] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9600.435749] ? apic_timer_interrupt+0xf/0x20 [ 9600.437278] [ 9600.438024] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9600.445201] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9600.448006] do_raw_spin_lock+0x177/0x190 [ 9600.450067] _raw_spin_lock+0x21/0x30 [ 9600.452797] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9600.458828] obd_mod_free+0x46/0x140 [ptlrpc] [ 9600.461464] mdc_commit_open+0x91/0xb0 [mdc] [ 9600.464728] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9600.469499] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9600.471665] after_reply+0xe2c/0x1780 [ptlrpc] [ 9600.484027] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9600.489062] ? __next_timer_interrupt+0x160/0x160 [ 9600.505140] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9600.513168] ? do_wait_intr+0xf0/0xf0 [ 9600.514872] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9600.521318] kthread+0x1d1/0x200 [ 9600.522455] ? set_kthread_struct+0x70/0x70 [ 9600.523853] ret_from_fork+0x1f/0x30 [ 9604.057030] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 9604.059768] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9604.083982] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9604.089044] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9604.093182] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9604.094799] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9604.103947] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9604.106733] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9604.108644] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 9604.111993] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9604.115111] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 9604.118755] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 9604.121815] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 9604.125312] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9604.128035] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 9604.130395] Call Trace: [ 9604.131190] [ 9604.132000] ? show_regs.cold.9+0x22/0x2f [ 9604.133653] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9604.136028] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9604.137877] ? watchdog+0x40/0x40 [ 9604.139467] ? hrtimer_interrupt+0x128/0x2c0 [ 9604.141314] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9604.143803] ? apic_timer_interrupt+0xf/0x20 [ 9604.145324] [ 9604.146302] ? native_safe_halt+0x2e/0x50 [ 9604.147532] kvm_wait+0xcd/0xe0 [ 9604.148632] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9604.151188] do_raw_spin_lock+0x177/0x190 [ 9604.153228] _raw_spin_lock+0x21/0x30 [ 9604.154631] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9604.157026] ? do_raw_spin_unlock+0x75/0x190 [ 9604.158783] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9604.161196] ? do_wait_intr+0xf0/0xf0 [ 9604.162812] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9604.164779] mdc_close+0x53b/0xd70 [mdc] [ 9604.165832] lmv_close+0x18d/0x490 [lmv] [ 9604.166974] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9604.169305] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9604.171591] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9604.174272] ll_md_close+0x274/0x800 [lustre] [ 9604.176338] ll_file_release+0x128/0xc50 [lustre] [ 9604.178388] __fput+0x10d/0x440 [ 9604.180012] ____fput+0x12/0x20 [ 9604.181387] task_work_run+0xc6/0x110 [ 9604.183166] exit_to_usermode_loop+0x1dd/0x1f0 [ 9604.184971] do_syscall_64+0x3d6/0x440 [ 9604.186373] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9604.189284] RIP: 0033:0x7f151bee3cd8 [ 9604.191136] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9604.199962] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9604.204304] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 9604.207608] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 9604.210795] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9604.214056] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9604.217481] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9628.052021] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 9628.054587] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9628.060228] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9628.061605] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 9628.061616] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9628.061685] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9628.061690] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9628.061693] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9628.061707] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9628.061711] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9628.061716] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 9628.061719] RDX: 000000000000576c RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 9628.061721] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 9628.061724] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 9628.061726] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9628.061729] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 9628.061732] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9628.061735] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9628.061744] Call Trace: [ 9628.061749] [ 9628.061755] ? show_regs.cold.9+0x22/0x2f [ 9628.061764] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9628.061771] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9628.061775] ? watchdog+0x40/0x40 [ 9628.061779] ? hrtimer_interrupt+0x128/0x2c0 [ 9628.061791] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9628.061800] ? apic_timer_interrupt+0xf/0x20 [ 9628.061805] [ 9628.061808] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9628.061813] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9628.061817] do_raw_spin_lock+0x177/0x190 [ 9628.061822] _raw_spin_lock+0x21/0x30 [ 9628.061827] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9628.062132] obd_mod_free+0x46/0x140 [ptlrpc] [ 9628.062343] mdc_commit_open+0x91/0xb0 [mdc] [ 9628.062385] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9628.062778] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9628.062987] after_reply+0xe2c/0x1780 [ptlrpc] [ 9628.063300] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9628.063513] ? __next_timer_interrupt+0x160/0x160 [ 9628.063673] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9628.063895] ? do_wait_intr+0xf0/0xf0 [ 9628.063905] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9628.064601] kthread+0x1d1/0x200 [ 9628.064607] ? set_kthread_struct+0x70/0x70 [ 9628.064612] ret_from_fork+0x1f/0x30 [ 9628.075371] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9628.075387] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9628.075390] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9628.075404] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9628.075408] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9628.075414] RAX: 0000000000002f06 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9628.075416] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 9628.075418] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 9628.075420] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 9628.075423] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 9628.075426] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 9628.075430] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9628.075432] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 9628.075441] Call Trace: [ 9628.075446] [ 9628.075452] ? show_regs.cold.9+0x22/0x2f [ 9628.075461] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9628.075468] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9628.075473] ? watchdog+0x40/0x40 [ 9628.075477] ? hrtimer_interrupt+0x128/0x2c0 [ 9628.075480] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9628.075488] ? apic_timer_interrupt+0xf/0x20 [ 9628.075493] [ 9628.075496] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9628.075500] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9628.075505] do_raw_spin_lock+0x177/0x190 [ 9628.075509] _raw_spin_lock+0x21/0x30 [ 9628.075517] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9628.075933] ? do_raw_spin_unlock+0x75/0x190 [ 9628.075938] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9628.076199] ? do_wait_intr+0xf0/0xf0 [ 9628.076207] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9628.076400] mdc_reint+0x2d/0x110 [mdc] [ 9628.076438] mdc_unlink+0x658/0x870 [mdc] [ 9628.076465] lmv_unlink+0x34d/0xbd0 [lmv] [ 9628.076495] ll_unlink+0x24d/0x7f0 [lustre] [ 9628.076608] vfs_unlink+0x202/0x330 [ 9628.076746] do_unlinkat+0x2fa/0x460 [ 9628.076754] __x64_sys_unlink+0x37/0x50 [ 9628.076758] do_syscall_64+0xc1/0x440 [ 9628.076764] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9628.076830] RIP: 0033:0x7f151bdee10b [ 9628.076838] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9628.076842] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9628.076847] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 9628.076849] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 9628.076851] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9628.076854] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9628.076856] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9628.359130] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9628.382438] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9628.389321] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9628.393513] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9628.400151] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9628.403158] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9628.411385] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9628.415189] RAX: 000000000000440d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9628.418593] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 9628.422008] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 9628.425487] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 9628.430993] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 9628.434156] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 9628.438280] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9628.441124] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9628.444720] Call Trace: [ 9628.446405] [ 9628.447336] ? show_regs.cold.9+0x22/0x2f [ 9628.449761] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9628.452001] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9628.454153] ? watchdog+0x40/0x40 [ 9628.455895] ? hrtimer_interrupt+0x128/0x2c0 [ 9628.457814] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9628.460775] ? apic_timer_interrupt+0xf/0x20 [ 9628.462629] [ 9628.463581] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9628.466293] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9628.468531] do_raw_spin_lock+0x177/0x190 [ 9628.470230] _raw_spin_lock+0x21/0x30 [ 9628.471758] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9628.474193] process_one_work+0x2c8/0x700 [ 9628.476154] worker_thread+0x296/0x6e0 [ 9628.477514] ? rescuer_thread+0x570/0x570 [ 9628.479201] kthread+0x1d1/0x200 [ 9628.480337] ? set_kthread_struct+0x70/0x70 [ 9628.481836] ret_from_fork+0x1f/0x30 [ 9630.184315] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7699s! [ 9630.195557] Showing busy workqueues and worker pools: [ 9630.198508] workqueue events: flags=0x0 [ 9630.201522] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9630.204330] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 9630.207903] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9630.210201] pending: kfree_rcu_monitor [ 9630.211891] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9630.214021] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 9630.219270] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9630.223412] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 9630.227575] workqueue events_long: flags=0x0 [ 9630.229594] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9630.232074] pending: lu_objects_flush [obdclass] [ 9630.234549] workqueue events_unbound: flags=0x2 [ 9630.236293] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9630.247755] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9630.258229] workqueue events_power_efficient: flags=0x80 [ 9630.260899] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9630.265250] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 9630.267860] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9630.270788] pending: neigh_periodic_work [ 9630.274508] workqueue mm_percpu_wq: flags=0x8 [ 9630.280067] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9630.283142] pending: vmstat_update, lru_add_drain_per_cpu [ 9630.284323] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9630.285697] pending: vmstat_update [ 9630.298873] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9630.301144] pending: lru_add_drain_per_cpu [ 9630.304072] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9630.308167] pending: lru_add_drain_per_cpu BAR(48) [ 9630.313156] workqueue writeback: flags=0x4a [ 9630.315831] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 9630.327582] pending: wb_workfn [ 9630.329137] workqueue xprtiod: flags=0xa [ 9630.330537] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9630.333320] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9630.337942] workqueue ipv6_addrconf: flags=0x40008 [ 9630.340774] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9630.347577] pending: addrconf_verify_work [ 9630.357256] workqueue knbd0-recv: flags=0x1a [ 9630.358626] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9630.365875] in-flight: 658:recv_work [ 9630.368229] workqueue ptlrpc_pinger: flags=0x2 [ 9630.369650] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9630.382715] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9630.386412] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7699s workers=4 idle: 32984 37 8 [ 9630.390307] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9615s workers=2 idle: 661 [ 9632.057866] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 9632.065680] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9632.154208] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9632.170453] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9632.178806] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9632.184160] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9632.200445] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9632.207775] RAX: 0000000000007317 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9632.209506] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 9632.215507] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9632.222934] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 9632.225294] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 9632.228193] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 9632.236524] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9632.239679] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 9632.242946] Call Trace: [ 9632.244021] [ 9632.244905] ? show_regs.cold.9+0x22/0x2f [ 9632.249187] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9632.251570] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9632.253708] ? watchdog+0x40/0x40 [ 9632.255799] ? hrtimer_interrupt+0x128/0x2c0 [ 9632.260882] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9632.264997] ? apic_timer_interrupt+0xf/0x20 [ 9632.266969] [ 9632.268260] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9632.270494] do_raw_spin_lock+0x177/0x190 [ 9632.278329] _raw_spin_lock+0x21/0x30 [ 9632.280193] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9632.283211] ? do_raw_spin_unlock+0x75/0x190 [ 9632.284832] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9632.288863] ? do_wait_intr+0xf0/0xf0 [ 9632.290700] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9632.292912] mdc_close+0x53b/0xd70 [mdc] [ 9632.296675] lmv_close+0x18d/0x490 [lmv] [ 9632.299027] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9632.303877] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9632.308446] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9632.312440] ll_md_close+0x274/0x800 [lustre] [ 9632.314860] ll_file_release+0x128/0xc50 [lustre] [ 9632.321421] __fput+0x10d/0x440 [ 9632.322981] ____fput+0x12/0x20 [ 9632.324055] task_work_run+0xc6/0x110 [ 9632.325309] exit_to_usermode_loop+0x1dd/0x1f0 [ 9632.329583] do_syscall_64+0x3d6/0x440 [ 9632.332508] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9632.337423] RIP: 0033:0x7f151bee3cd8 [ 9632.343475] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9632.372754] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9632.375498] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 9632.393544] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 9632.400501] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9632.411243] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9632.422565] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9656.052266] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 9656.057110] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 9656.057178] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9656.057186] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9656.057599] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9656.057606] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9656.057623] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9656.058127] Modules linked in: [ 9656.058158] [ 9656.058162] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9656.058195] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9656.058200] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9656.058205] RAX: 0000000000007c86 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9656.058208] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 9656.058210] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 9656.058212] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 9656.058215] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 9656.058219] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 9656.058222] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9656.058237] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9656.058246] Call Trace: [ 9656.058262] [ 9656.058267] ? show_regs.cold.9+0x22/0x2f [ 9656.058793] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9656.058800] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9656.059182] ? watchdog+0x40/0x40 [ 9656.059187] ? hrtimer_interrupt+0x128/0x2c0 [ 9656.059191] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9656.059200] ? apic_timer_interrupt+0xf/0x20 [ 9656.059205] [ 9656.059209] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9656.059215] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9656.059336] do_raw_spin_lock+0x177/0x190 [ 9656.059407] _raw_spin_lock+0x21/0x30 [ 9656.059413] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9656.059692] process_one_work+0x2c8/0x700 [ 9656.059701] worker_thread+0x296/0x6e0 [ 9656.059705] ? rescuer_thread+0x570/0x570 [ 9656.059708] kthread+0x1d1/0x200 [ 9656.059713] ? set_kthread_struct+0x70/0x70 [ 9656.059716] ret_from_fork+0x1f/0x30 [ 9656.061704] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9656.061791] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9656.061799] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9656.061802] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9656.061817] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9656.061821] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9656.061825] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 9656.061828] RDX: 0000000000007bf5 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 9656.061831] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 9656.061833] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 9656.061836] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9656.061839] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 9656.061842] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9656.061845] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9656.061852] Call Trace: [ 9656.061856] [ 9656.061861] ? show_regs.cold.9+0x22/0x2f [ 9656.061879] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9656.061885] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9656.061888] ? watchdog+0x40/0x40 [ 9656.061893] ? hrtimer_interrupt+0x128/0x2c0 [ 9656.061897] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9656.061903] ? apic_timer_interrupt+0xf/0x20 [ 9656.061907] [ 9656.061910] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9656.061915] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9656.061919] do_raw_spin_lock+0x177/0x190 [ 9656.061928] _raw_spin_lock+0x21/0x30 [ 9656.061934] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9656.062255] obd_mod_free+0x46/0x140 [ptlrpc] [ 9656.062478] mdc_commit_open+0x91/0xb0 [mdc] [ 9656.062527] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9656.062813] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9656.063079] after_reply+0xe2c/0x1780 [ptlrpc] [ 9656.063285] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9656.063488] ? __next_timer_interrupt+0x160/0x160 [ 9656.063497] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9656.063726] ? do_wait_intr+0xf0/0xf0 [ 9656.063736] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9656.064026] kthread+0x1d1/0x200 [ 9656.064033] ? set_kthread_struct+0x70/0x70 [ 9656.064037] ret_from_fork+0x1f/0x30 [ 9656.646342] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9656.695479] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9656.706752] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9656.723267] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9656.731938] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9656.750661] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9656.755541] RAX: 0000000000007507 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9656.761019] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 9656.765689] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 9656.768184] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 9656.777359] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 9656.788595] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 9656.798760] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9656.802721] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 9656.807629] Call Trace: [ 9656.808615] [ 9656.810243] ? show_regs.cold.9+0x22/0x2f [ 9656.816251] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9656.826068] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9656.831052] ? watchdog+0x40/0x40 [ 9656.836684] ? hrtimer_interrupt+0x128/0x2c0 [ 9656.840188] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9656.851549] ? apic_timer_interrupt+0xf/0x20 [ 9656.859738] [ 9656.861260] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9656.870444] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9656.872425] do_raw_spin_lock+0x177/0x190 [ 9656.886179] _raw_spin_lock+0x21/0x30 [ 9656.891357] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9656.901963] ? do_raw_spin_unlock+0x75/0x190 [ 9656.903497] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9656.912402] ? do_wait_intr+0xf0/0xf0 [ 9656.914562] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9656.924328] mdc_reint+0x2d/0x110 [mdc] [ 9656.932900] mdc_unlink+0x658/0x870 [mdc] [ 9656.941298] lmv_unlink+0x34d/0xbd0 [lmv] [ 9656.950627] ll_unlink+0x24d/0x7f0 [lustre] [ 9656.952189] vfs_unlink+0x202/0x330 [ 9656.960100] do_unlinkat+0x2fa/0x460 [ 9656.967568] __x64_sys_unlink+0x37/0x50 [ 9656.977024] do_syscall_64+0xc1/0x440 [ 9656.978369] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9656.987184] RIP: 0033:0x7f151bdee10b [ 9656.995335] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9657.014348] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9657.017063] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 9657.026283] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 9657.031491] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9657.036657] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9657.044701] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9660.057245] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 9660.069247] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9660.114820] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9660.123401] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9660.135251] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9660.141078] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9660.159002] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9660.165202] RAX: 000000000000534c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9660.172948] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 9660.187891] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9660.198958] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 9660.212022] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 9660.219938] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 9660.224481] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9660.232202] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 9660.237677] Call Trace: [ 9660.239116] [ 9660.241462] ? show_regs.cold.9+0x22/0x2f [ 9660.242859] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9660.247893] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9660.252622] ? watchdog+0x40/0x40 [ 9660.253701] ? hrtimer_interrupt+0x128/0x2c0 [ 9660.255098] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9660.261720] ? apic_timer_interrupt+0xf/0x20 [ 9660.267257] [ 9660.268205] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9660.271837] do_raw_spin_lock+0x177/0x190 [ 9660.273186] _raw_spin_lock+0x21/0x30 [ 9660.275200] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9660.284953] ? do_raw_spin_unlock+0x75/0x190 [ 9660.293110] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9660.299794] ? do_wait_intr+0xf0/0xf0 [ 9660.301124] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9660.304990] mdc_close+0x53b/0xd70 [mdc] [ 9660.309514] lmv_close+0x18d/0x490 [lmv] [ 9660.316791] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9660.327868] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9660.330046] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9660.351105] ll_md_close+0x274/0x800 [lustre] [ 9660.353641] ll_file_release+0x128/0xc50 [lustre] [ 9660.357884] __fput+0x10d/0x440 [ 9660.363813] ____fput+0x12/0x20 [ 9660.366946] task_work_run+0xc6/0x110 [ 9660.370074] exit_to_usermode_loop+0x1dd/0x1f0 [ 9660.375936] do_syscall_64+0x3d6/0x440 [ 9660.379729] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9660.387581] RIP: 0033:0x7f151bee3cd8 [ 9660.390022] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9660.416196] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9660.426547] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 9660.443535] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 9660.453119] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9660.460537] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9660.468187] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9660.898762] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7730s! [ 9660.909115] Showing busy workqueues and worker pools: [ 9660.917681] workqueue events: flags=0x0 [ 9660.921982] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9660.930869] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 9660.938937] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9660.945283] pending: kfree_rcu_monitor [ 9660.948981] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9660.955482] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 9660.965957] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9660.973850] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 9660.978816] workqueue events_long: flags=0x0 [ 9660.983604] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9660.986896] pending: lu_objects_flush [obdclass] [ 9660.990856] workqueue events_unbound: flags=0x2 [ 9660.995781] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9661.001751] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9661.010303] workqueue events_power_efficient: flags=0x80 [ 9661.019061] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9661.021422] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 9661.031757] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9661.039305] pending: neigh_periodic_work [ 9661.044304] workqueue mm_percpu_wq: flags=0x8 [ 9661.047893] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9661.053842] pending: vmstat_update, lru_add_drain_per_cpu [ 9661.057147] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9661.063784] pending: vmstat_update [ 9661.066092] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9661.074935] pending: lru_add_drain_per_cpu [ 9661.079303] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9661.084797] pending: lru_add_drain_per_cpu BAR(48) [ 9661.088755] workqueue writeback: flags=0x4a [ 9661.090175] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 9661.095723] pending: wb_workfn [ 9661.098993] workqueue xprtiod: flags=0xa [ 9661.103284] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9661.109592] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9661.118566] workqueue ipv6_addrconf: flags=0x40008 [ 9661.121757] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9661.125670] pending: addrconf_verify_work [ 9661.134372] workqueue knbd0-recv: flags=0x1a [ 9661.137987] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9661.146064] in-flight: 658:recv_work [ 9661.151293] workqueue ptlrpc_pinger: flags=0x2 [ 9661.155846] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9661.163253] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9661.168544] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7730s workers=4 idle: 32984 37 8 [ 9661.182178] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9645s workers=2 idle: 661 [ 9684.055149] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 9684.062268] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9684.062802] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9684.084203] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9684.084217] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9684.084221] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9684.084235] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9684.084250] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9684.084256] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 9684.084258] RDX: 0000000000000e4f RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 9684.084260] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 9684.084262] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 9684.084264] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9684.084267] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 9684.084270] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9684.084272] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9684.084278] Call Trace: [ 9684.084284] [ 9684.084289] ? show_regs.cold.9+0x22/0x2f [ 9684.084298] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9684.084304] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9684.084309] ? watchdog+0x40/0x40 [ 9684.084313] ? hrtimer_interrupt+0x128/0x2c0 [ 9684.084316] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9684.084323] ? apic_timer_interrupt+0xf/0x20 [ 9684.084328] [ 9684.084331] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9684.084336] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9684.084341] do_raw_spin_lock+0x177/0x190 [ 9684.084345] _raw_spin_lock+0x21/0x30 [ 9684.084353] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9684.084768] obd_mod_free+0x46/0x140 [ptlrpc] [ 9684.084978] mdc_commit_open+0x91/0xb0 [mdc] [ 9684.085020] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9684.085378] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9684.085667] after_reply+0xe2c/0x1780 [ptlrpc] [ 9684.085882] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9684.086346] ? __next_timer_interrupt+0x160/0x160 [ 9684.086357] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9684.086572] ? do_wait_intr+0xf0/0xf0 [ 9684.086583] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9684.086782] kthread+0x1d1/0x200 [ 9684.086788] ? set_kthread_struct+0x70/0x70 [ 9684.086792] ret_from_fork+0x1f/0x30 [ 9684.360761] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9684.409070] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9684.415364] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9684.419524] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9684.423412] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9684.427185] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9684.447299] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9684.450659] RAX: 000000000000480e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9684.455478] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 9684.459558] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 9684.463635] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 9684.469170] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 9684.478391] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 9684.490332] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9684.497819] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9684.516404] Call Trace: [ 9684.520521] [ 9684.526550] ? show_regs.cold.9+0x22/0x2f [ 9684.532431] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9684.539586] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9684.543051] ? watchdog+0x40/0x40 [ 9684.551788] ? hrtimer_interrupt+0x128/0x2c0 [ 9684.556259] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9684.564953] ? apic_timer_interrupt+0xf/0x20 [ 9684.566512] [ 9684.572462] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9684.594230] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9684.596422] do_raw_spin_lock+0x177/0x190 [ 9684.599067] _raw_spin_lock+0x21/0x30 [ 9684.602010] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9684.604945] process_one_work+0x2c8/0x700 [ 9684.606787] worker_thread+0x296/0x6e0 [ 9684.609257] ? rescuer_thread+0x570/0x570 [ 9684.610630] kthread+0x1d1/0x200 [ 9684.611651] ? set_kthread_struct+0x70/0x70 [ 9684.613666] ret_from_fork+0x1f/0x30 [ 9688.052176] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 9688.057009] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 9688.061287] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9688.061370] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9688.061379] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9688.061382] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9688.061397] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9688.061401] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9688.061405] RAX: 000000000000431a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9688.061408] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 9688.061410] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 9688.061412] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 9688.061415] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 9688.061418] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 9688.061422] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9688.061424] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 9688.061434] Call Trace: [ 9688.061440] [ 9688.061445] ? show_regs.cold.9+0x22/0x2f [ 9688.061456] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9688.061462] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9688.061467] ? watchdog+0x40/0x40 [ 9688.061472] ? hrtimer_interrupt+0x128/0x2c0 [ 9688.061477] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9688.061485] ? apic_timer_interrupt+0xf/0x20 [ 9688.061491] [ 9688.061494] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9688.061500] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9688.061505] do_raw_spin_lock+0x177/0x190 [ 9688.061510] _raw_spin_lock+0x21/0x30 [ 9688.061516] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9688.072428] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9688.072518] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9688.072526] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9688.072529] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9688.072544] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9688.072548] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9688.072553] RAX: 0000000000007775 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9688.072555] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 9688.072557] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9688.072560] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 9688.072562] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 9688.072566] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 9688.072570] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9688.072572] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 9688.072581] Call Trace: [ 9688.072587] [ 9688.072593] ? show_regs.cold.9+0x22/0x2f [ 9688.072604] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9688.072611] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9688.072616] ? watchdog+0x40/0x40 [ 9688.072621] ? hrtimer_interrupt+0x128/0x2c0 [ 9688.072625] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9688.072634] ? apic_timer_interrupt+0xf/0x20 [ 9688.072639] [ 9688.072642] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9688.072649] do_raw_spin_lock+0x177/0x190 [ 9688.072654] _raw_spin_lock+0x21/0x30 [ 9688.072658] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9688.124455] ? do_raw_spin_unlock+0x75/0x190 [ 9688.124477] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9688.126788] ? do_wait_intr+0xf0/0xf0 [ 9688.126800] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9688.127444] mdc_reint+0x2d/0x110 [mdc] [ 9688.127481] mdc_unlink+0x658/0x870 [mdc] [ 9688.127510] lmv_unlink+0x34d/0xbd0 [lmv] [ 9688.127542] ll_unlink+0x24d/0x7f0 [lustre] [ 9688.127675] vfs_unlink+0x202/0x330 [ 9688.127684] do_unlinkat+0x2fa/0x460 [ 9688.127689] __x64_sys_unlink+0x37/0x50 [ 9688.127693] do_syscall_64+0xc1/0x440 [ 9688.127699] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9688.127708] RIP: 0033:0x7f151bdee10b [ 9688.127715] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9688.127719] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9688.127725] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 9688.127728] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 9688.127731] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9688.127733] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9688.127735] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9688.745414] ? do_raw_spin_unlock+0x75/0x190 [ 9688.751848] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9688.755353] ? do_wait_intr+0xf0/0xf0 [ 9688.757704] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9688.761473] mdc_close+0x53b/0xd70 [mdc] [ 9688.763526] lmv_close+0x18d/0x490 [lmv] [ 9688.764922] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9688.768310] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9688.774456] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9688.779464] ll_md_close+0x274/0x800 [lustre] [ 9688.783809] ll_file_release+0x128/0xc50 [lustre] [ 9688.787384] __fput+0x10d/0x440 [ 9688.790657] ____fput+0x12/0x20 [ 9688.792413] task_work_run+0xc6/0x110 [ 9688.794918] exit_to_usermode_loop+0x1dd/0x1f0 [ 9688.796947] do_syscall_64+0x3d6/0x440 [ 9688.798631] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9688.801221] RIP: 0033:0x7f151bee3cd8 [ 9688.802839] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9688.809809] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9688.813170] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 9688.816007] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 9688.822475] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9688.826689] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9688.828910] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9691.618273] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7761s! [ 9691.624765] Showing busy workqueues and worker pools: [ 9691.627803] workqueue events: flags=0x0 [ 9691.632938] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9691.636958] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 9691.643553] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9691.663746] pending: kfree_rcu_monitor [ 9691.669941] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9691.678256] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 9691.691157] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9691.699854] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 9691.704075] workqueue events_long: flags=0x0 [ 9691.706913] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9691.710833] pending: lu_objects_flush [obdclass] [ 9691.712876] workqueue events_unbound: flags=0x2 [ 9691.715313] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9691.718257] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9691.729979] workqueue events_power_efficient: flags=0x80 [ 9691.733500] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9691.743008] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 9691.747684] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9691.752457] pending: neigh_periodic_work [ 9691.754687] workqueue mm_percpu_wq: flags=0x8 [ 9691.758945] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9691.765339] pending: vmstat_update, lru_add_drain_per_cpu [ 9691.768061] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9691.774007] pending: vmstat_update [ 9691.777907] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9691.783018] pending: lru_add_drain_per_cpu [ 9691.787515] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9691.791013] pending: lru_add_drain_per_cpu BAR(48) [ 9691.795460] workqueue writeback: flags=0x4a [ 9691.797414] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 9691.803741] pending: wb_workfn [ 9691.806179] workqueue xprtiod: flags=0xa [ 9691.807923] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9691.810840] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9691.814209] workqueue ipv6_addrconf: flags=0x40008 [ 9691.819859] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9691.822238] pending: addrconf_verify_work [ 9691.825306] workqueue knbd0-recv: flags=0x1a [ 9691.828610] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9691.835283] in-flight: 658:recv_work [ 9691.836930] workqueue ptlrpc_pinger: flags=0x2 [ 9691.839883] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9691.842823] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9691.845425] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7761s workers=4 idle: 32984 37 8 [ 9691.849431] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9676s workers=2 idle: 661 [ 9712.055011] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 9712.056026] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 9712.059334] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9712.059393] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9712.059399] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9712.059401] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9712.059413] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9712.059416] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9712.059419] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 9712.059421] RDX: 00000000000030bb RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 9712.059422] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 9712.059424] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 9712.059426] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9712.059428] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 9712.059430] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9712.059432] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9712.059439] Call Trace: [ 9712.059444] [ 9712.059449] ? show_regs.cold.9+0x22/0x2f [ 9712.059457] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9712.059462] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9712.059465] ? watchdog+0x40/0x40 [ 9712.059468] ? hrtimer_interrupt+0x128/0x2c0 [ 9712.059471] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9712.059486] ? apic_timer_interrupt+0xf/0x20 [ 9712.059491] [ 9712.066810] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9712.066904] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9712.066912] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9712.066918] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9712.067212] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9712.067221] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9712.067225] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9712.067229] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9712.067231] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 9712.067233] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 9712.067236] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 9712.067238] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 9712.067242] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 9712.067245] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9712.067247] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9712.069775] Call Trace: [ 9712.069788] [ 9712.069793] ? show_regs.cold.9+0x22/0x2f [ 9712.069807] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9712.069813] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9712.069816] ? watchdog+0x40/0x40 [ 9712.069821] ? hrtimer_interrupt+0x128/0x2c0 [ 9712.069825] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9712.069833] ? apic_timer_interrupt+0xf/0x20 [ 9712.069838] [ 9712.069841] ? native_safe_halt+0x2e/0x50 [ 9712.069845] kvm_wait+0xcd/0xe0 [ 9712.069853] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9712.069863] do_raw_spin_lock+0x177/0x190 [ 9712.069875] _raw_spin_lock+0x21/0x30 [ 9712.069880] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9712.070222] process_one_work+0x2c8/0x700 [ 9712.070230] worker_thread+0x296/0x6e0 [ 9712.070234] ? rescuer_thread+0x570/0x570 [ 9712.070238] kthread+0x1d1/0x200 [ 9712.070243] ? set_kthread_struct+0x70/0x70 [ 9712.070247] ret_from_fork+0x1f/0x30 [ 9712.287896] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9712.290506] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9712.293189] do_raw_spin_lock+0x177/0x190 [ 9712.295152] _raw_spin_lock+0x21/0x30 [ 9712.296730] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9712.299852] obd_mod_free+0x46/0x140 [ptlrpc] [ 9712.302579] mdc_commit_open+0x91/0xb0 [mdc] [ 9712.305382] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9712.308676] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9712.311822] after_reply+0xe2c/0x1780 [ptlrpc] [ 9712.314203] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9712.315695] ? __next_timer_interrupt+0x160/0x160 [ 9712.318326] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9712.320485] ? do_wait_intr+0xf0/0xf0 [ 9712.321906] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9712.324729] kthread+0x1d1/0x200 [ 9712.326886] ? set_kthread_struct+0x70/0x70 [ 9712.329361] ret_from_fork+0x1f/0x30 [ 9716.056079] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 9716.058587] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 9716.063384] Modules linked in: [ 9716.068261] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) [ 9716.069641] lustre(O) [ 9716.069644] ksocklnd(O) lnet(O) [ 9716.082905] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9716.083217] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9716.083226] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9716.083229] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9716.083244] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9716.083249] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9716.083254] RAX: 0000000000003f84 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9716.083256] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 9716.083259] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 9716.083262] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 9716.083265] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 9716.083269] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 9716.083273] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9716.083275] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 9716.083285] Call Trace: [ 9716.083291] [ 9716.083296] ? show_regs.cold.9+0x22/0x2f [ 9716.083306] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9716.083313] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9716.083318] ? watchdog+0x40/0x40 [ 9716.083322] ? hrtimer_interrupt+0x128/0x2c0 [ 9716.083325] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9716.083335] ? apic_timer_interrupt+0xf/0x20 [ 9716.083340] [ 9716.083343] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9716.083348] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9716.083353] do_raw_spin_lock+0x177/0x190 [ 9716.083358] _raw_spin_lock+0x21/0x30 [ 9716.083364] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9716.083701] ? do_raw_spin_unlock+0x75/0x190 [ 9716.083707] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9716.084014] ? do_wait_intr+0xf0/0xf0 [ 9716.084021] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9716.084206] mdc_reint+0x2d/0x110 [mdc] [ 9716.084241] mdc_unlink+0x658/0x870 [mdc] [ 9716.084268] lmv_unlink+0x34d/0xbd0 [lmv] [ 9716.084299] ll_unlink+0x24d/0x7f0 [lustre] [ 9716.084390] vfs_unlink+0x202/0x330 [ 9716.084396] do_unlinkat+0x2fa/0x460 [ 9716.084399] __x64_sys_unlink+0x37/0x50 [ 9716.084402] do_syscall_64+0xc1/0x440 [ 9716.084407] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9716.084411] RIP: 0033:0x7f151bdee10b [ 9716.084416] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9716.084419] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9716.084422] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 9716.084424] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 9716.084425] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9716.084427] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9716.084428] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9716.086666] libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9716.086706] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9716.086712] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9716.086715] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9716.086727] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9716.086730] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9716.086734] RAX: 0000000000007f86 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9716.086736] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 9716.086738] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9716.086740] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 9716.086743] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 9716.086746] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 9716.315761] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9716.318614] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 9716.321120] Call Trace: [ 9716.322111] [ 9716.322584] ? show_regs.cold.9+0x22/0x2f [ 9716.323850] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9716.326206] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9716.328201] ? watchdog+0x40/0x40 [ 9716.329519] ? hrtimer_interrupt+0x128/0x2c0 [ 9716.332718] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9716.335256] ? apic_timer_interrupt+0xf/0x20 [ 9716.336968] [ 9716.339583] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9716.341942] do_raw_spin_lock+0x177/0x190 [ 9716.344040] _raw_spin_lock+0x21/0x30 [ 9716.346457] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9716.350055] ? do_raw_spin_unlock+0x75/0x190 [ 9716.352205] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9716.358634] ? do_wait_intr+0xf0/0xf0 [ 9716.360625] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9716.364433] mdc_close+0x53b/0xd70 [mdc] [ 9716.366508] lmv_close+0x18d/0x490 [lmv] [ 9716.368222] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9716.372667] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9716.377153] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9716.380597] ll_md_close+0x274/0x800 [lustre] [ 9716.383587] ll_file_release+0x128/0xc50 [lustre] [ 9716.387578] __fput+0x10d/0x440 [ 9716.392207] ____fput+0x12/0x20 [ 9716.395049] task_work_run+0xc6/0x110 [ 9716.396558] exit_to_usermode_loop+0x1dd/0x1f0 [ 9716.398333] do_syscall_64+0x3d6/0x440 [ 9716.402567] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9716.404997] RIP: 0033:0x7f151bee3cd8 [ 9716.406002] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9716.415634] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9716.419656] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 9716.426320] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 9716.430799] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9716.434387] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9716.437473] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9722.336408] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7791s! [ 9722.339830] Showing busy workqueues and worker pools: [ 9722.341974] workqueue events: flags=0x0 [ 9722.343509] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9722.346182] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 9722.350694] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9722.354059] pending: kfree_rcu_monitor [ 9722.355715] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9722.358719] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 9722.363180] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9722.366195] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 9722.369448] workqueue events_long: flags=0x0 [ 9722.372048] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9722.375610] pending: lu_objects_flush [obdclass] [ 9722.378353] workqueue events_unbound: flags=0x2 [ 9722.380167] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9722.383275] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9722.389543] workqueue events_power_efficient: flags=0x80 [ 9722.392537] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9722.396412] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 9722.400580] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9722.404474] pending: neigh_periodic_work [ 9722.407128] workqueue mm_percpu_wq: flags=0x8 [ 9722.409063] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9722.411616] pending: vmstat_update, lru_add_drain_per_cpu [ 9722.414116] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9722.416746] pending: vmstat_update [ 9722.418622] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9722.421309] pending: lru_add_drain_per_cpu [ 9722.423100] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9722.427457] pending: lru_add_drain_per_cpu BAR(48) [ 9722.429691] workqueue writeback: flags=0x4a [ 9722.431381] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 9722.433313] pending: wb_workfn [ 9722.434864] workqueue xprtiod: flags=0xa [ 9722.437252] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9722.439502] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9722.442924] workqueue ipv6_addrconf: flags=0x40008 [ 9722.444632] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9722.447474] pending: addrconf_verify_work [ 9722.449276] workqueue knbd0-recv: flags=0x1a [ 9722.451908] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9722.454888] in-flight: 658:recv_work [ 9722.459726] workqueue ptlrpc_pinger: flags=0x2 [ 9722.462097] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9722.465228] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9722.468530] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7791s workers=4 idle: 32984 37 8 [ 9722.472525] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9707s workers=2 idle: 661 [ 9724.809063] rcu: INFO: rcu_sched self-detected stall on CPU [ 9724.814760] rcu: 1-...!: (4430216 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 9724.826450] (t=7800162 jiffies g=249657 q=4631) [ 9724.831027] rcu: rcu_sched kthread starved for 7795114 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 9724.841342] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 9724.850367] rcu: RCU grace-period kthread stack dump: [ 9724.855930] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 9724.861869] Call Trace: [ 9724.862828] __schedule+0x351/0xcb0 [ 9724.864767] schedule+0xc0/0x180 [ 9724.870644] schedule_timeout+0xb4/0x190 [ 9724.876417] ? do_raw_spin_unlock+0x75/0x190 [ 9724.878905] ? __next_timer_interrupt+0x160/0x160 [ 9724.881933] ? prepare_to_swait_event+0xf3/0x140 [ 9724.885265] rcu_gp_kthread+0x8dd/0x15b0 [ 9724.888940] ? force_qs_rnp+0x2c0/0x2c0 [ 9724.893930] kthread+0x1d1/0x200 [ 9724.896482] ? set_kthread_struct+0x70/0x70 [ 9724.901673] ret_from_fork+0x1f/0x30 [ 9724.904305] rcu: Stack dump where RCU GP kthread last ran: [ 9724.911430] Sending NMI from CPU 1 to CPUs 2: [ 9724.914161] NMI backtrace for cpu 2 skipped: idling at native_safe_halt+0x2e/0x50 [ 9724.915902] NMI backtrace for cpu 1 [ 9724.924098] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9724.930666] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9724.937668] Call Trace: [ 9724.939446] [ 9724.940600] ? dump_stack+0xbb/0x10e [ 9724.943492] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9724.947934] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9724.951290] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9724.954950] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9724.958529] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9724.961369] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 9724.964298] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 9724.967414] ? update_process_times+0x6d/0xc0 [ 9724.969446] ? tick_sched_handle.isra.18+0x27/0x90 [ 9724.975384] ? tick_sched_timer+0x53/0xb0 [ 9724.977211] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9724.980559] ? tick_sched_handle.isra.18+0x90/0x90 [ 9724.983523] ? hrtimer_interrupt+0x128/0x2c0 [ 9724.988442] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9724.997924] ? apic_timer_interrupt+0xf/0x20 [ 9725.001087] [ 9725.001969] ? native_safe_halt+0x2e/0x50 [ 9725.003611] ? kvm_wait+0xcd/0xe0 [ 9725.005914] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9725.009106] ? do_raw_spin_lock+0x177/0x190 [ 9725.015963] ? _raw_spin_lock+0x21/0x30 [ 9725.019884] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9725.023090] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9725.028359] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9725.032672] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9725.035905] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9725.040373] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9725.044286] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9725.049266] ? __next_timer_interrupt+0x160/0x160 [ 9725.052083] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9725.057038] ? do_wait_intr+0xf0/0xf0 [ 9725.061357] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9725.065054] ? kthread+0x1d1/0x200 [ 9725.071095] ? set_kthread_struct+0x70/0x70 [ 9725.075114] ? ret_from_fork+0x1f/0x30 [ 9744.057073] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 9744.067090] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9744.104203] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9744.111746] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9744.117076] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9744.121064] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9744.135313] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9744.139486] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9744.143956] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 9744.149193] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9744.153276] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 9744.158133] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 9744.162289] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 9744.167262] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9744.177966] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 9744.180148] Call Trace: [ 9744.183115] [ 9744.184489] ? show_regs.cold.9+0x22/0x2f [ 9744.187079] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9744.190109] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9744.192896] ? watchdog+0x40/0x40 [ 9744.194338] ? hrtimer_interrupt+0x128/0x2c0 [ 9744.198132] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9744.201976] ? apic_timer_interrupt+0xf/0x20 [ 9744.205128] [ 9744.207411] ? native_safe_halt+0x2e/0x50 [ 9744.210228] kvm_wait+0xcd/0xe0 [ 9744.212309] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9744.216416] do_raw_spin_lock+0x177/0x190 [ 9744.219089] _raw_spin_lock+0x21/0x30 [ 9744.221320] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9744.227387] ? do_raw_spin_unlock+0x75/0x190 [ 9744.228707] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9744.233234] ? do_wait_intr+0xf0/0xf0 [ 9744.240098] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9744.243319] mdc_close+0x53b/0xd70 [mdc] [ 9744.246023] lmv_close+0x18d/0x490 [lmv] [ 9744.248236] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9744.251344] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9744.255170] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9744.259072] ll_md_close+0x274/0x800 [lustre] [ 9744.261598] ll_file_release+0x128/0xc50 [lustre] [ 9744.265225] __fput+0x10d/0x440 [ 9744.267117] ____fput+0x12/0x20 [ 9744.269016] task_work_run+0xc6/0x110 [ 9744.271970] exit_to_usermode_loop+0x1dd/0x1f0 [ 9744.274908] do_syscall_64+0x3d6/0x440 [ 9744.277018] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9744.279083] RIP: 0033:0x7f151bee3cd8 [ 9744.281041] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9744.288235] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9744.291237] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 9744.294146] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 9744.297060] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9744.301034] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9744.305954] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9748.053903] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 9748.057502] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9748.083298] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9748.091960] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9748.096771] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9748.101085] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9748.110874] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9748.118958] RAX: 0000000000005c85 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9748.122460] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 9748.125816] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 9748.129957] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 9748.134201] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 9748.146068] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 9748.154828] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9748.158895] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 9748.164809] Call Trace: [ 9748.166212] [ 9748.170181] ? show_regs.cold.9+0x22/0x2f [ 9748.172854] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9748.174412] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9748.182058] ? watchdog+0x40/0x40 [ 9748.183592] ? hrtimer_interrupt+0x128/0x2c0 [ 9748.186187] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9748.188869] ? apic_timer_interrupt+0xf/0x20 [ 9748.190309] [ 9748.191035] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9748.194868] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9748.196218] do_raw_spin_lock+0x177/0x190 [ 9748.199994] _raw_spin_lock+0x21/0x30 [ 9748.202300] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9748.204933] ? do_raw_spin_unlock+0x75/0x190 [ 9748.207108] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9748.209554] ? do_wait_intr+0xf0/0xf0 [ 9748.211039] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9748.213254] mdc_reint+0x2d/0x110 [mdc] [ 9748.214873] mdc_unlink+0x658/0x870 [mdc] [ 9748.216602] lmv_unlink+0x34d/0xbd0 [lmv] [ 9748.218204] ll_unlink+0x24d/0x7f0 [lustre] [ 9748.220024] vfs_unlink+0x202/0x330 [ 9748.221581] do_unlinkat+0x2fa/0x460 [ 9748.223038] __x64_sys_unlink+0x37/0x50 [ 9748.224648] do_syscall_64+0xc1/0x440 [ 9748.226248] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9748.228303] RIP: 0033:0x7f151bdee10b [ 9748.229780] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9748.236829] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9748.239771] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 9748.242451] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 9748.245275] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9748.248268] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9748.251129] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9752.055689] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 9752.056014] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [ 9752.060074] Modules linked in: [ 9752.063164] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9752.063237] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9752.063244] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9752.063251] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9752.063768] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9752.063781] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9752.063785] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9752.063795] RAX: 000000000000362d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9752.063798] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 9752.063801] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 9752.063804] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 9752.063807] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 9752.063810] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 9752.063813] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9752.063815] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9752.063822] Call Trace: [ 9752.063828] [ 9752.063833] ? show_regs.cold.9+0x22/0x2f [ 9752.063843] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9752.063849] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9752.063853] ? watchdog+0x40/0x40 [ 9752.063858] ? hrtimer_interrupt+0x128/0x2c0 [ 9752.063862] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9752.063871] ? apic_timer_interrupt+0xf/0x20 [ 9752.063876] [ 9752.063879] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9752.063884] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9752.063890] do_raw_spin_lock+0x177/0x190 [ 9752.063895] _raw_spin_lock+0x21/0x30 [ 9752.063900] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9752.064126] process_one_work+0x2c8/0x700 [ 9752.064133] worker_thread+0x296/0x6e0 [ 9752.064138] ? rescuer_thread+0x570/0x570 [ 9752.064142] kthread+0x1d1/0x200 [ 9752.064147] ? set_kthread_struct+0x70/0x70 [ 9752.064150] ret_from_fork+0x1f/0x30 [ 9752.201432] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9752.223746] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9752.228425] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9752.232897] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9752.235265] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9752.244632] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9752.247993] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 9752.249815] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 9752.252869] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 9752.255980] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 9752.259808] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9752.263260] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 9752.267100] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9752.269391] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9752.272972] Call Trace: [ 9752.274123] [ 9752.275241] ? show_regs.cold.9+0x22/0x2f [ 9752.277117] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9752.279728] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9752.281880] ? watchdog+0x40/0x40 [ 9752.282943] ? hrtimer_interrupt+0x128/0x2c0 [ 9752.284614] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9752.286292] ? apic_timer_interrupt+0xf/0x20 [ 9752.289961] [ 9752.291111] ? native_safe_halt+0x2e/0x50 [ 9752.293957] kvm_wait+0xcd/0xe0 [ 9752.296858] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9752.299677] do_raw_spin_lock+0x177/0x190 [ 9752.301707] _raw_spin_lock+0x21/0x30 [ 9752.303794] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9752.306382] obd_mod_free+0x46/0x140 [ptlrpc] [ 9752.308116] mdc_commit_open+0x91/0xb0 [mdc] [ 9752.310721] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9752.313936] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9752.317266] after_reply+0xe2c/0x1780 [ptlrpc] [ 9752.320377] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9752.322991] ? __next_timer_interrupt+0x160/0x160 [ 9752.325361] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9752.328677] ? do_wait_intr+0xf0/0xf0 [ 9752.330746] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9752.333799] kthread+0x1d1/0x200 [ 9752.335298] ? set_kthread_struct+0x70/0x70 [ 9752.337739] ret_from_fork+0x1f/0x30 [ 9752.545025] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7821s! [ 9752.552879] Showing busy workqueues and worker pools: [ 9752.555828] workqueue events: flags=0x0 [ 9752.558819] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9752.561473] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 9752.564338] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9752.566921] pending: kfree_rcu_monitor [ 9752.568572] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9752.570813] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 9752.573932] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9752.576335] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 9752.578886] workqueue events_long: flags=0x0 [ 9752.580350] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9752.583038] pending: lu_objects_flush [obdclass] [ 9752.584825] workqueue events_unbound: flags=0x2 [ 9752.586550] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9752.588584] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9752.592169] workqueue events_power_efficient: flags=0x80 [ 9752.594206] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9752.596897] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 9752.599009] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9752.600995] pending: neigh_periodic_work [ 9752.602399] workqueue mm_percpu_wq: flags=0x8 [ 9752.603655] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9752.605689] pending: vmstat_update, lru_add_drain_per_cpu [ 9752.607293] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9752.609360] pending: vmstat_update [ 9752.610670] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9752.612510] pending: lru_add_drain_per_cpu [ 9752.614006] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9752.616210] pending: lru_add_drain_per_cpu BAR(48) [ 9752.618312] workqueue writeback: flags=0x4a [ 9752.620040] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 9752.622337] pending: wb_workfn [ 9752.623683] workqueue xprtiod: flags=0xa [ 9752.625183] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9752.627981] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9752.631198] workqueue ipv6_addrconf: flags=0x40008 [ 9752.632975] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9752.635176] pending: addrconf_verify_work [ 9752.636686] workqueue knbd0-recv: flags=0x1a [ 9752.638098] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9752.640186] in-flight: 658:recv_work [ 9752.641669] workqueue ptlrpc_pinger: flags=0x2 [ 9752.643102] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9752.645338] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9752.647415] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7822s workers=4 idle: 32984 37 8 [ 9752.650018] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9737s workers=2 idle: 661 [ 9772.057289] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 9772.060549] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9772.091824] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9772.101393] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9772.117556] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9772.124514] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9772.158880] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9772.169324] RAX: 0000000000007718 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9772.178983] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 9772.186700] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9772.193220] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 9772.203820] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 9772.207927] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 9772.211930] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9772.216906] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 9772.221076] Call Trace: [ 9772.224047] [ 9772.225503] ? show_regs.cold.9+0x22/0x2f [ 9772.229445] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9772.233365] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9772.234916] ? watchdog+0x40/0x40 [ 9772.235701] ? hrtimer_interrupt+0x128/0x2c0 [ 9772.240469] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9772.243125] ? apic_timer_interrupt+0xf/0x20 [ 9772.245172] [ 9772.247458] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9772.252555] do_raw_spin_lock+0x177/0x190 [ 9772.261947] _raw_spin_lock+0x21/0x30 [ 9772.265974] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9772.269172] ? do_raw_spin_unlock+0x75/0x190 [ 9772.272594] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9772.277211] ? do_wait_intr+0xf0/0xf0 [ 9772.282932] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9772.285512] mdc_close+0x53b/0xd70 [mdc] [ 9772.290641] lmv_close+0x18d/0x490 [lmv] [ 9772.292475] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9772.301643] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9772.306602] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9772.315035] ll_md_close+0x274/0x800 [lustre] [ 9772.319169] ll_file_release+0x128/0xc50 [lustre] [ 9772.325595] __fput+0x10d/0x440 [ 9772.329597] ____fput+0x12/0x20 [ 9772.332205] task_work_run+0xc6/0x110 [ 9772.334475] exit_to_usermode_loop+0x1dd/0x1f0 [ 9772.337834] do_syscall_64+0x3d6/0x440 [ 9772.342044] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9772.346484] RIP: 0033:0x7f151bee3cd8 [ 9772.350776] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9772.364591] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9772.371019] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 9772.376743] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 9772.383116] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9772.390110] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9772.396036] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9780.053620] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 9780.056015] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9780.058394] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 9780.058408] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9780.059674] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9780.059681] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9780.059683] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9780.059696] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9780.059700] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9780.059704] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 9780.059707] RDX: 0000000000003bea RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 9780.059709] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 9780.059711] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 9780.059714] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9780.059717] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 9780.059721] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9780.059724] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9780.059733] Call Trace: [ 9780.059738] [ 9780.059744] ? show_regs.cold.9+0x22/0x2f [ 9780.059754] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9780.059760] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9780.059764] ? watchdog+0x40/0x40 [ 9780.059768] ? hrtimer_interrupt+0x128/0x2c0 [ 9780.059772] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9780.059781] ? apic_timer_interrupt+0xf/0x20 [ 9780.059786] [ 9780.059789] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9780.059793] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9780.059798] do_raw_spin_lock+0x177/0x190 [ 9780.059803] _raw_spin_lock+0x21/0x30 [ 9780.059808] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9780.063972] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9780.064044] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9780.064051] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9780.064054] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9780.064067] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9780.064071] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9780.064075] RAX: 0000000000007a09 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9780.064077] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 9780.064079] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 9780.064081] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 9780.064084] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 9780.064087] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 9780.064090] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9780.064092] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 9780.064101] Call Trace: [ 9780.064105] [ 9780.064108] ? show_regs.cold.9+0x22/0x2f [ 9780.064117] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9780.064124] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9780.064128] ? watchdog+0x40/0x40 [ 9780.064132] ? hrtimer_interrupt+0x128/0x2c0 [ 9780.064136] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9780.064144] ? apic_timer_interrupt+0xf/0x20 [ 9780.064149] [ 9780.064152] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9780.064156] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9780.064161] do_raw_spin_lock+0x177/0x190 [ 9780.064166] _raw_spin_lock+0x21/0x30 [ 9780.064171] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9780.064435] ? do_raw_spin_unlock+0x75/0x190 [ 9780.064441] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9780.064637] ? do_wait_intr+0xf0/0xf0 [ 9780.064645] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9780.065820] mdc_reint+0x2d/0x110 [mdc] [ 9780.065859] mdc_unlink+0x658/0x870 [mdc] [ 9780.065887] lmv_unlink+0x34d/0xbd0 [lmv] [ 9780.065917] ll_unlink+0x24d/0x7f0 [lustre] [ 9780.066030] vfs_unlink+0x202/0x330 [ 9780.066038] do_unlinkat+0x2fa/0x460 [ 9780.066042] __x64_sys_unlink+0x37/0x50 [ 9780.066046] do_syscall_64+0xc1/0x440 [ 9780.066052] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9780.066059] RIP: 0033:0x7f151bdee10b [ 9780.066066] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9780.066069] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9780.066073] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 9780.066076] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 9780.066078] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9780.066080] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9780.066082] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9780.063507] obd_mod_free+0x46/0x140 [ptlrpc] [ 9780.066437] mdc_commit_open+0x91/0xb0 [mdc] [ 9780.066469] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9780.066920] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9780.067159] after_reply+0xe2c/0x1780 [ptlrpc] [ 9780.073404] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9780.073489] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9780.073497] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9780.073505] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9780.075214] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9780.075228] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9780.075232] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 9780.075248] RAX: 00000000000025e5 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9780.086630] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9780.086895] ? __next_timer_interrupt+0x160/0x160 [ 9780.086905] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9780.087112] ? do_wait_intr+0xf0/0xf0 [ 9780.087122] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9780.087338] kthread+0x1d1/0x200 [ 9780.087346] ? set_kthread_struct+0x70/0x70 [ 9780.087350] ret_from_fork+0x1f/0x30 [ 9780.775418] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 9780.789756] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 9780.792285] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 9780.799080] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 9780.810102] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 9780.819211] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9780.821485] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9780.827423] Call Trace: [ 9780.828317] [ 9780.829035] ? show_regs.cold.9+0x22/0x2f [ 9780.830835] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9780.835848] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9780.837675] ? watchdog+0x40/0x40 [ 9780.842372] ? hrtimer_interrupt+0x128/0x2c0 [ 9780.851725] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9780.856985] ? apic_timer_interrupt+0xf/0x20 [ 9780.859801] [ 9780.866760] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9780.871054] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9780.880430] do_raw_spin_lock+0x177/0x190 [ 9780.888453] _raw_spin_lock+0x21/0x30 [ 9780.889879] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9780.899785] process_one_work+0x2c8/0x700 [ 9780.901519] worker_thread+0x296/0x6e0 [ 9780.907060] ? rescuer_thread+0x570/0x570 [ 9780.909737] kthread+0x1d1/0x200 [ 9780.917700] ? set_kthread_struct+0x70/0x70 [ 9780.922930] ret_from_fork+0x1f/0x30 [ 9782.753047] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7852s! [ 9782.756108] Showing busy workqueues and worker pools: [ 9782.758593] workqueue events: flags=0x0 [ 9782.761009] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9782.763874] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 9782.767621] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9782.769677] pending: kfree_rcu_monitor [ 9782.771541] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9782.773439] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 9782.777953] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9782.779707] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 9782.781836] workqueue events_long: flags=0x0 [ 9782.782904] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9782.784922] pending: lu_objects_flush [obdclass] [ 9782.786622] workqueue events_unbound: flags=0x2 [ 9782.787848] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9782.789619] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9782.792870] workqueue events_power_efficient: flags=0x80 [ 9782.794266] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9782.796000] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 9782.798746] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9782.800530] pending: neigh_periodic_work [ 9782.801670] workqueue mm_percpu_wq: flags=0x8 [ 9782.802831] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9782.804701] pending: vmstat_update, lru_add_drain_per_cpu [ 9782.806055] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9782.808060] pending: vmstat_update [ 9782.809026] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9782.810668] pending: lru_add_drain_per_cpu [ 9782.811772] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9782.814206] pending: lru_add_drain_per_cpu BAR(48) [ 9782.815812] workqueue writeback: flags=0x4a [ 9782.816839] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 9782.818306] pending: wb_workfn [ 9782.819206] workqueue xprtiod: flags=0xa [ 9782.820144] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9782.821582] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9782.823491] workqueue ipv6_addrconf: flags=0x40008 [ 9782.824675] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9782.826133] pending: addrconf_verify_work [ 9782.827261] workqueue knbd0-recv: flags=0x1a [ 9782.828720] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9782.831111] in-flight: 658:recv_work [ 9782.832597] workqueue ptlrpc_pinger: flags=0x2 [ 9782.834366] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9782.836788] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9782.839245] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7852s workers=4 idle: 32984 37 8 [ 9782.842787] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9767s workers=2 idle: 661 [ 9800.057035] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 9800.060331] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9800.077317] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9800.081647] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9800.085208] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9800.086785] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9800.100733] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9800.106276] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9800.109256] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 9800.112316] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9800.114920] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 9800.121472] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 9800.125804] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 9800.129975] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9800.136167] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 9800.144715] Call Trace: [ 9800.146748] [ 9800.147750] ? show_regs.cold.9+0x22/0x2f [ 9800.152817] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9800.155262] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9800.160820] ? watchdog+0x40/0x40 [ 9800.163536] ? hrtimer_interrupt+0x128/0x2c0 [ 9800.168563] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9800.172488] ? apic_timer_interrupt+0xf/0x20 [ 9800.176490] [ 9800.177408] ? native_safe_halt+0x2e/0x50 [ 9800.181394] kvm_wait+0xcd/0xe0 [ 9800.184549] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9800.189844] do_raw_spin_lock+0x177/0x190 [ 9800.191373] _raw_spin_lock+0x21/0x30 [ 9800.192916] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9800.200610] ? do_raw_spin_unlock+0x75/0x190 [ 9800.206688] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9800.208516] ? do_wait_intr+0xf0/0xf0 [ 9800.209689] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9800.221669] mdc_close+0x53b/0xd70 [mdc] [ 9800.225612] lmv_close+0x18d/0x490 [lmv] [ 9800.228864] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9800.234489] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9800.238502] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9800.244923] ll_md_close+0x274/0x800 [lustre] [ 9800.248023] ll_file_release+0x128/0xc50 [lustre] [ 9800.252345] __fput+0x10d/0x440 [ 9800.255342] ____fput+0x12/0x20 [ 9800.256431] task_work_run+0xc6/0x110 [ 9800.259312] exit_to_usermode_loop+0x1dd/0x1f0 [ 9800.262687] do_syscall_64+0x3d6/0x440 [ 9800.264873] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9800.268986] RIP: 0033:0x7f151bee3cd8 [ 9800.270192] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9800.287970] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9800.291255] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 9800.297505] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 9800.302624] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9800.305607] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9800.308390] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9808.053624] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 9808.056096] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9808.056150] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 9808.056157] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9808.056229] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9808.056235] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9808.056238] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9808.056252] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9808.056256] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9808.056261] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 9808.056264] RDX: 0000000000005c86 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 9808.056266] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 9808.056268] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 9808.056270] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9808.056273] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 9808.056276] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9808.056278] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9808.056286] Call Trace: [ 9808.056291] [ 9808.056296] ? show_regs.cold.9+0x22/0x2f [ 9808.056306] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9808.056312] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9808.056316] ? watchdog+0x40/0x40 [ 9808.056320] ? hrtimer_interrupt+0x128/0x2c0 [ 9808.056323] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9808.056331] ? apic_timer_interrupt+0xf/0x20 [ 9808.056336] [ 9808.056339] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9808.056343] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9808.056347] do_raw_spin_lock+0x177/0x190 [ 9808.056351] _raw_spin_lock+0x21/0x30 [ 9808.056357] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9808.056613] obd_mod_free+0x46/0x140 [ptlrpc] [ 9808.056838] mdc_commit_open+0x91/0xb0 [mdc] [ 9808.058433] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9808.058481] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9808.058487] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9808.058489] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9808.058499] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9808.058503] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9808.058508] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9808.058511] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 9808.058513] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 9808.058516] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 9808.058518] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 9808.058522] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 9808.058525] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9808.058527] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 9808.058535] Call Trace: [ 9808.058539] [ 9808.058542] ? show_regs.cold.9+0x22/0x2f [ 9808.058550] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9808.058556] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9808.058560] ? watchdog+0x40/0x40 [ 9808.058564] ? hrtimer_interrupt+0x128/0x2c0 [ 9808.058432] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9808.058568] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9808.058577] ? apic_timer_interrupt+0xf/0x20 [ 9808.058582] [ 9808.058584] ? native_safe_halt+0x2e/0x50 [ 9808.058588] kvm_wait+0xcd/0xe0 [ 9808.058596] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9808.058602] do_raw_spin_lock+0x177/0x190 [ 9808.058606] _raw_spin_lock+0x21/0x30 [ 9808.058610] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9808.059452] ? do_raw_spin_unlock+0x75/0x190 [ 9808.059458] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9808.059653] ? do_wait_intr+0xf0/0xf0 [ 9808.059661] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9808.059825] mdc_reint+0x2d/0x110 [mdc] [ 9808.059851] mdc_unlink+0x658/0x870 [mdc] [ 9808.059868] lmv_unlink+0x34d/0xbd0 [lmv] [ 9808.059894] ll_unlink+0x24d/0x7f0 [lustre] [ 9808.060091] vfs_unlink+0x202/0x330 [ 9808.060097] do_unlinkat+0x2fa/0x460 [ 9808.060100] __x64_sys_unlink+0x37/0x50 [ 9808.060102] do_syscall_64+0xc1/0x440 [ 9808.060107] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9808.060111] RIP: 0033:0x7f151bdee10b [ 9808.060116] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9808.060119] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9808.060122] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 9808.060124] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 9808.060125] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9808.060127] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9808.060128] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9808.060179] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9808.061226] after_reply+0xe2c/0x1780 [ptlrpc] [ 9808.062278] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9808.062561] ? __next_timer_interrupt+0x160/0x160 [ 9808.062570] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9808.062770] ? do_wait_intr+0xf0/0xf0 [ 9808.062778] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9808.076028] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9808.076115] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9808.076123] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9808.076130] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9808.076413] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9808.076424] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9808.076427] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9808.076431] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9808.076433] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 9808.076435] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 9808.076437] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 9808.076440] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 9808.076443] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 9808.076446] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9808.076448] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9808.076457] Call Trace: [ 9808.076462] [ 9808.076468] ? show_regs.cold.9+0x22/0x2f [ 9808.076478] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9808.076484] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9808.076488] ? watchdog+0x40/0x40 [ 9808.076492] ? hrtimer_interrupt+0x128/0x2c0 [ 9808.076496] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9808.076502] ? apic_timer_interrupt+0xf/0x20 [ 9808.076507] [ 9808.076510] ? native_safe_halt+0x2e/0x50 [ 9808.076514] kvm_wait+0xcd/0xe0 [ 9808.076522] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9808.076530] do_raw_spin_lock+0x177/0x190 [ 9808.076534] _raw_spin_lock+0x21/0x30 [ 9808.076538] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9808.076753] process_one_work+0x2c8/0x700 [ 9808.076760] worker_thread+0x296/0x6e0 [ 9808.076764] ? rescuer_thread+0x570/0x570 [ 9808.076767] kthread+0x1d1/0x200 [ 9808.076771] ? set_kthread_struct+0x70/0x70 [ 9808.076775] ret_from_fork+0x1f/0x30 [ 9808.663654] kthread+0x1d1/0x200 [ 9808.664664] ? set_kthread_struct+0x70/0x70 [ 9808.666226] ret_from_fork+0x1f/0x30 [ 9812.965430] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7882s! [ 9812.977301] Showing busy workqueues and worker pools: [ 9812.981703] workqueue events: flags=0x0 [ 9812.995670] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9813.009643] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 9813.030250] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9813.034501] pending: kfree_rcu_monitor [ 9813.037368] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9813.041405] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 9813.049739] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9813.054535] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 9813.061089] workqueue events_long: flags=0x0 [ 9813.063470] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9813.068364] pending: lu_objects_flush [obdclass] [ 9813.073354] workqueue events_unbound: flags=0x2 [ 9813.077126] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9813.080687] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9813.086921] workqueue events_power_efficient: flags=0x80 [ 9813.090240] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9813.093968] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 9813.098099] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9813.100759] pending: neigh_periodic_work [ 9813.103424] workqueue mm_percpu_wq: flags=0x8 [ 9813.106172] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9813.109456] pending: vmstat_update, lru_add_drain_per_cpu [ 9813.112714] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9813.116349] pending: vmstat_update [ 9813.118074] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9813.122434] pending: lru_add_drain_per_cpu [ 9813.125303] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9813.128608] pending: lru_add_drain_per_cpu BAR(48) [ 9813.136570] workqueue writeback: flags=0x4a [ 9813.140749] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 9813.144300] pending: wb_workfn [ 9813.146651] workqueue xprtiod: flags=0xa [ 9813.147786] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9813.151063] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9813.156039] workqueue ipv6_addrconf: flags=0x40008 [ 9813.159840] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9813.163525] pending: addrconf_verify_work [ 9813.165568] workqueue knbd0-recv: flags=0x1a [ 9813.168110] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9813.175477] in-flight: 658:recv_work [ 9813.178030] workqueue ptlrpc_pinger: flags=0x2 [ 9813.181263] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9813.186533] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9813.193670] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7882s workers=4 idle: 32984 37 8 [ 9813.197981] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9797s workers=2 idle: 661 [ 9828.057063] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 9828.061075] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9828.088360] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9828.094007] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9828.099447] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9828.101674] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9828.114791] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9828.117620] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9828.125831] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 9828.131741] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9828.138629] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 9828.147958] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 9828.155969] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 9828.158992] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9828.162080] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 9828.164955] Call Trace: [ 9828.165862] [ 9828.166898] ? show_regs.cold.9+0x22/0x2f [ 9828.168574] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9828.170779] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9828.172935] ? watchdog+0x40/0x40 [ 9828.176871] ? hrtimer_interrupt+0x128/0x2c0 [ 9828.180663] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9828.183477] ? apic_timer_interrupt+0xf/0x20 [ 9828.185377] [ 9828.186377] ? native_safe_halt+0x2e/0x50 [ 9828.188853] kvm_wait+0xcd/0xe0 [ 9828.191066] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9828.198817] do_raw_spin_lock+0x177/0x190 [ 9828.200418] _raw_spin_lock+0x21/0x30 [ 9828.203337] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9828.207333] ? do_raw_spin_unlock+0x75/0x190 [ 9828.212633] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9828.216088] ? do_wait_intr+0xf0/0xf0 [ 9828.219274] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9828.227131] mdc_close+0x53b/0xd70 [mdc] [ 9828.232238] lmv_close+0x18d/0x490 [lmv] [ 9828.234047] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9828.237892] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9828.240868] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9828.243173] ll_md_close+0x274/0x800 [lustre] [ 9828.246302] ll_file_release+0x128/0xc50 [lustre] [ 9828.252279] __fput+0x10d/0x440 [ 9828.253258] ____fput+0x12/0x20 [ 9828.254334] task_work_run+0xc6/0x110 [ 9828.261558] exit_to_usermode_loop+0x1dd/0x1f0 [ 9828.269171] do_syscall_64+0x3d6/0x440 [ 9828.276294] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9828.280374] RIP: 0033:0x7f151bee3cd8 [ 9828.282505] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9828.292399] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9828.297584] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 9828.301021] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 9828.304284] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9828.313182] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9828.323398] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9836.057607] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 9836.059409] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9836.062201] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9836.062288] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9836.062296] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9836.062298] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9836.062312] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9836.062317] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9836.062322] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 9836.062324] RDX: 0000000000005ef8 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 9836.062326] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 9836.062329] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 9836.062331] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9836.062334] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 9836.062337] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9836.062340] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9836.062348] Call Trace: [ 9836.062353] [ 9836.062359] ? show_regs.cold.9+0x22/0x2f [ 9836.062367] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9836.062373] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9836.062377] ? watchdog+0x40/0x40 [ 9836.062381] ? hrtimer_interrupt+0x128/0x2c0 [ 9836.062385] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9836.062393] ? apic_timer_interrupt+0xf/0x20 [ 9836.062411] [ 9836.062414] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9836.062419] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9836.062423] do_raw_spin_lock+0x177/0x190 [ 9836.062427] _raw_spin_lock+0x21/0x30 [ 9836.062433] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9836.062934] obd_mod_free+0x46/0x140 [ptlrpc] [ 9836.063142] mdc_commit_open+0x91/0xb0 [mdc] [ 9836.063180] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9836.063387] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9836.063796] after_reply+0xe2c/0x1780 [ptlrpc] [ 9836.064019] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9836.064255] ? __next_timer_interrupt+0x160/0x160 [ 9836.064265] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9836.064579] ? do_wait_intr+0xf0/0xf0 [ 9836.064722] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9836.064973] kthread+0x1d1/0x200 [ 9836.064979] ? set_kthread_struct+0x70/0x70 [ 9836.064983] ret_from_fork+0x1f/0x30 [ 9836.484348] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9836.534294] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9836.539952] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9836.544467] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9836.547346] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9836.550261] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9836.556400] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9836.559879] RAX: 0000000000007f84 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9836.563064] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 9836.578219] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 9836.582246] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 9836.596890] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 9836.606376] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 9836.621374] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9836.633898] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9836.642508] Call Trace: [ 9836.644496] [ 9836.650944] ? show_regs.cold.9+0x22/0x2f [ 9836.655052] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9836.663487] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9836.671562] ? watchdog+0x40/0x40 [ 9836.675755] ? hrtimer_interrupt+0x128/0x2c0 [ 9836.677371] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9836.687779] ? apic_timer_interrupt+0xf/0x20 [ 9836.691962] [ 9836.694103] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9836.697516] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9836.704154] do_raw_spin_lock+0x177/0x190 [ 9836.709584] _raw_spin_lock+0x21/0x30 [ 9836.716532] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9836.722219] process_one_work+0x2c8/0x700 [ 9836.725166] worker_thread+0x296/0x6e0 [ 9836.732504] ? rescuer_thread+0x570/0x570 [ 9836.737648] kthread+0x1d1/0x200 [ 9836.738880] ? set_kthread_struct+0x70/0x70 [ 9836.740322] ret_from_fork+0x1f/0x30 [ 9840.052069] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 9840.056122] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9840.074340] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9840.079804] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9840.082856] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9840.084365] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9840.094224] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9840.097097] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9840.099754] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 9840.103892] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 9840.107571] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 9840.111458] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 9840.117832] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 9840.124223] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9840.127913] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 9840.130484] Call Trace: [ 9840.131813] [ 9840.132944] ? show_regs.cold.9+0x22/0x2f [ 9840.135478] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9840.140275] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9840.143240] ? watchdog+0x40/0x40 [ 9840.144920] ? hrtimer_interrupt+0x128/0x2c0 [ 9840.150454] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9840.153806] ? apic_timer_interrupt+0xf/0x20 [ 9840.155805] [ 9840.160443] ? native_safe_halt+0x2e/0x50 [ 9840.165496] kvm_wait+0xcd/0xe0 [ 9840.171142] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9840.174465] do_raw_spin_lock+0x177/0x190 [ 9840.177268] _raw_spin_lock+0x21/0x30 [ 9840.181180] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9840.185762] ? do_raw_spin_unlock+0x75/0x190 [ 9840.188484] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9840.192959] ? do_wait_intr+0xf0/0xf0 [ 9840.194814] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9840.198104] mdc_reint+0x2d/0x110 [mdc] [ 9840.200280] mdc_unlink+0x658/0x870 [mdc] [ 9840.204381] lmv_unlink+0x34d/0xbd0 [lmv] [ 9840.206998] ll_unlink+0x24d/0x7f0 [lustre] [ 9840.211514] vfs_unlink+0x202/0x330 [ 9840.213868] do_unlinkat+0x2fa/0x460 [ 9840.216416] __x64_sys_unlink+0x37/0x50 [ 9840.218321] do_syscall_64+0xc1/0x440 [ 9840.220425] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9840.224365] RIP: 0033:0x7f151bdee10b [ 9840.226254] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9840.236402] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9840.240901] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 9840.244318] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 9840.247371] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9840.251136] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9840.254381] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9843.680218] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7913s! [ 9843.683345] Showing busy workqueues and worker pools: [ 9843.685222] workqueue events: flags=0x0 [ 9843.695416] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9843.701990] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 9843.711304] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9843.717170] pending: kfree_rcu_monitor [ 9843.724374] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9843.728230] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 9843.747115] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9843.759384] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 9843.764820] workqueue events_long: flags=0x0 [ 9843.770892] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9843.780332] pending: lu_objects_flush [obdclass] [ 9843.788241] workqueue events_unbound: flags=0x2 [ 9843.793907] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9843.797309] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9843.803460] workqueue events_power_efficient: flags=0x80 [ 9843.806052] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9843.809267] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 9843.812516] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9843.815150] pending: neigh_periodic_work [ 9843.816432] workqueue mm_percpu_wq: flags=0x8 [ 9843.817778] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9843.821008] pending: vmstat_update, lru_add_drain_per_cpu [ 9843.824169] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9843.828323] pending: vmstat_update [ 9843.829984] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9843.832699] pending: lru_add_drain_per_cpu [ 9843.834442] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9843.837404] pending: lru_add_drain_per_cpu BAR(48) [ 9843.839481] workqueue writeback: flags=0x4a [ 9843.841382] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 9843.843852] pending: wb_workfn [ 9843.845204] workqueue xprtiod: flags=0xa [ 9843.847286] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9843.849994] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9843.854841] workqueue ipv6_addrconf: flags=0x40008 [ 9843.856909] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9843.860011] pending: addrconf_verify_work [ 9843.862162] workqueue knbd0-recv: flags=0x1a [ 9843.865633] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9843.868999] in-flight: 658:recv_work [ 9843.870450] workqueue ptlrpc_pinger: flags=0x2 [ 9843.872013] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9843.874586] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9843.876042] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7913s workers=4 idle: 32984 37 8 [ 9843.879798] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9828s workers=2 idle: 661 [ 9856.059084] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 9856.064980] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9856.114539] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9856.118794] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9856.122291] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9856.124529] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9856.148184] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9856.162317] RAX: 0000000000007514 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9856.174983] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 9856.184276] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9856.190936] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 9856.193139] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 9856.210788] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 9856.218402] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9856.229809] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 9856.232370] Call Trace: [ 9856.237281] [ 9856.238363] ? show_regs.cold.9+0x22/0x2f [ 9856.246038] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9856.253047] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9856.265163] ? watchdog+0x40/0x40 [ 9856.268378] ? hrtimer_interrupt+0x128/0x2c0 [ 9856.275730] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9856.278707] ? apic_timer_interrupt+0xf/0x20 [ 9856.281027] [ 9856.282387] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9856.285555] do_raw_spin_lock+0x177/0x190 [ 9856.288659] _raw_spin_lock+0x21/0x30 [ 9856.290793] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9856.296396] ? do_raw_spin_unlock+0x75/0x190 [ 9856.305319] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9856.307211] ? do_wait_intr+0xf0/0xf0 [ 9856.325227] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9856.336895] mdc_close+0x53b/0xd70 [mdc] [ 9856.338767] lmv_close+0x18d/0x490 [lmv] [ 9856.341147] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9856.349839] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9856.356508] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9856.361955] ll_md_close+0x274/0x800 [lustre] [ 9856.369178] ll_file_release+0x128/0xc50 [lustre] [ 9856.376754] __fput+0x10d/0x440 [ 9856.380328] ____fput+0x12/0x20 [ 9856.383938] task_work_run+0xc6/0x110 [ 9856.388636] exit_to_usermode_loop+0x1dd/0x1f0 [ 9856.395371] do_syscall_64+0x3d6/0x440 [ 9856.406282] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9856.413066] RIP: 0033:0x7f151bee3cd8 [ 9856.419810] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9856.453676] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9856.463996] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 9856.476511] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 9856.490221] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9856.503523] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9856.523395] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9864.055034] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 9864.058574] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9864.063301] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9864.082073] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9864.082093] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9864.082096] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9864.082123] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9864.082127] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9864.082136] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 9864.082141] RDX: 0000000000002e8d RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 9864.082143] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 9864.082145] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 9864.082148] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9864.082153] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 9864.082156] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9864.082160] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9864.082173] Call Trace: [ 9864.082185] [ 9864.082192] ? show_regs.cold.9+0x22/0x2f [ 9864.082205] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9864.082238] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9864.082244] ? watchdog+0x40/0x40 [ 9864.082332] ? hrtimer_interrupt+0x128/0x2c0 [ 9864.082336] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9864.082351] ? apic_timer_interrupt+0xf/0x20 [ 9864.082419] [ 9864.082425] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9864.082433] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9864.082441] do_raw_spin_lock+0x177/0x190 [ 9864.082448] _raw_spin_lock+0x21/0x30 [ 9864.082458] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9864.082847] obd_mod_free+0x46/0x140 [ptlrpc] [ 9864.083172] mdc_commit_open+0x91/0xb0 [mdc] [ 9864.083217] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9864.083426] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9864.083795] after_reply+0xe2c/0x1780 [ptlrpc] [ 9864.089899] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9864.089997] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9864.090004] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9864.090010] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9864.090355] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9864.090367] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9864.090372] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9864.090386] RAX: 000000000000690e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9864.090389] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 9864.090391] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 9864.090393] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 9864.090396] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 9864.090402] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 9864.090405] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9864.090407] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9864.090414] Call Trace: [ 9864.090420] [ 9864.090426] ? show_regs.cold.9+0x22/0x2f [ 9864.090440] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9864.090456] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9864.090461] ? watchdog+0x40/0x40 [ 9864.090465] ? hrtimer_interrupt+0x128/0x2c0 [ 9864.090471] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9864.090479] ? apic_timer_interrupt+0xf/0x20 [ 9864.090483] [ 9864.090486] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9864.090490] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9864.090506] do_raw_spin_lock+0x177/0x190 [ 9864.090510] _raw_spin_lock+0x21/0x30 [ 9864.090515] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9864.090762] process_one_work+0x2c8/0x700 [ 9864.090773] worker_thread+0x296/0x6e0 [ 9864.090781] ? rescuer_thread+0x570/0x570 [ 9864.090785] kthread+0x1d1/0x200 [ 9864.090792] ? set_kthread_struct+0x70/0x70 [ 9864.090796] ret_from_fork+0x1f/0x30 [ 9864.350129] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9864.361344] ? __next_timer_interrupt+0x160/0x160 [ 9864.362950] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9864.368499] ? do_wait_intr+0xf0/0xf0 [ 9864.372127] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9864.379332] kthread+0x1d1/0x200 [ 9864.380767] ? set_kthread_struct+0x70/0x70 [ 9864.382915] ret_from_fork+0x1f/0x30 [ 9868.056951] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 9868.064890] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9868.125502] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9868.141837] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9868.153117] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9868.155133] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9868.177230] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9868.181358] RAX: 0000000000007d90 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9868.201192] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 9868.208239] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 9868.215039] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 9868.220511] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 9868.229734] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 9868.238216] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9868.244440] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 9868.251294] Call Trace: [ 9868.253375] [ 9868.254414] ? show_regs.cold.9+0x22/0x2f [ 9868.261815] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9868.266811] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9868.270434] ? watchdog+0x40/0x40 [ 9868.282443] ? hrtimer_interrupt+0x128/0x2c0 [ 9868.285537] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9868.288230] ? apic_timer_interrupt+0xf/0x20 [ 9868.290180] [ 9868.291223] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9868.303638] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9868.308726] do_raw_spin_lock+0x177/0x190 [ 9868.310063] _raw_spin_lock+0x21/0x30 [ 9868.316721] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9868.320257] ? do_raw_spin_unlock+0x75/0x190 [ 9868.322787] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9868.325892] ? do_wait_intr+0xf0/0xf0 [ 9868.328482] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9868.333789] mdc_reint+0x2d/0x110 [mdc] [ 9868.336842] mdc_unlink+0x658/0x870 [mdc] [ 9868.340809] lmv_unlink+0x34d/0xbd0 [lmv] [ 9868.346684] ll_unlink+0x24d/0x7f0 [lustre] [ 9868.348177] vfs_unlink+0x202/0x330 [ 9868.349362] do_unlinkat+0x2fa/0x460 [ 9868.354349] __x64_sys_unlink+0x37/0x50 [ 9868.355678] do_syscall_64+0xc1/0x440 [ 9868.358066] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9868.360825] RIP: 0033:0x7f151bdee10b [ 9868.364464] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9868.403246] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9868.413248] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 9868.422030] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 9868.427105] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9868.433140] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9868.446096] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9873.892119] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7943s! [ 9873.904146] Showing busy workqueues and worker pools: [ 9873.913753] workqueue events: flags=0x0 [ 9873.922472] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9873.928807] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 9873.937158] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9873.942096] pending: kfree_rcu_monitor [ 9873.945467] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9873.950751] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 9873.961164] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9873.975741] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 9873.987377] workqueue events_long: flags=0x0 [ 9873.995645] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9874.004256] pending: lu_objects_flush [obdclass] [ 9874.010050] workqueue events_unbound: flags=0x2 [ 9874.015433] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9874.027609] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9874.035054] workqueue events_power_efficient: flags=0x80 [ 9874.046440] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9874.059297] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 9874.063138] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9874.074846] pending: neigh_periodic_work [ 9874.076358] workqueue mm_percpu_wq: flags=0x8 [ 9874.090022] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9874.095106] pending: vmstat_update, lru_add_drain_per_cpu [ 9874.100923] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9874.104624] pending: vmstat_update [ 9874.111142] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9874.114099] pending: lru_add_drain_per_cpu [ 9874.117148] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9874.123046] pending: lru_add_drain_per_cpu BAR(48) [ 9874.125990] workqueue writeback: flags=0x4a [ 9874.133244] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 9874.138955] pending: wb_workfn [ 9874.143490] workqueue xprtiod: flags=0xa [ 9874.148222] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9874.153589] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9874.157019] workqueue ipv6_addrconf: flags=0x40008 [ 9874.160155] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9874.163924] pending: addrconf_verify_work [ 9874.169018] workqueue knbd0-recv: flags=0x1a [ 9874.177150] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9874.183781] in-flight: 658:recv_work [ 9874.192138] workqueue ptlrpc_pinger: flags=0x2 [ 9874.199845] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9874.206198] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9874.210357] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7943s workers=4 idle: 32984 37 8 [ 9874.221448] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9858s workers=2 idle: 661 [ 9884.065499] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 9884.070007] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9884.130783] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9884.169826] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9884.184514] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9884.186590] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9884.221491] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 9884.233092] RAX: 0000000000005362 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9884.247777] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 9884.258674] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9884.275005] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 9884.285650] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 9884.294925] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 9884.311859] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9884.323032] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 9884.331093] Call Trace: [ 9884.332030] [ 9884.345914] ? show_regs.cold.9+0x22/0x2f [ 9884.347869] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9884.353866] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9884.361637] ? watchdog+0x40/0x40 [ 9884.370541] ? hrtimer_interrupt+0x128/0x2c0 [ 9884.386907] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9884.395852] ? apic_timer_interrupt+0xf/0x20 [ 9884.399886] [ 9884.402427] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9884.412578] do_raw_spin_lock+0x177/0x190 [ 9884.417649] _raw_spin_lock+0x21/0x30 [ 9884.426067] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9884.437975] ? do_raw_spin_unlock+0x75/0x190 [ 9884.448289] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9884.456684] ? do_wait_intr+0xf0/0xf0 [ 9884.462687] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9884.470633] mdc_close+0x53b/0xd70 [mdc] [ 9884.474174] lmv_close+0x18d/0x490 [lmv] [ 9884.476454] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9884.478940] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9884.482515] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9884.486385] ll_md_close+0x274/0x800 [lustre] [ 9884.490950] ll_file_release+0x128/0xc50 [lustre] [ 9884.495588] __fput+0x10d/0x440 [ 9884.497616] ____fput+0x12/0x20 [ 9884.502254] task_work_run+0xc6/0x110 [ 9884.503915] exit_to_usermode_loop+0x1dd/0x1f0 [ 9884.507953] do_syscall_64+0x3d6/0x440 [ 9884.511388] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9884.513868] RIP: 0033:0x7f151bee3cd8 [ 9884.517373] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9884.547492] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9884.554648] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 9884.560919] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 9884.567740] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9884.576982] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9884.583315] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9892.055951] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 9892.056020] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9892.063170] Modules linked in: [ 9892.073421] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9892.073507] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9892.073515] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9892.073522] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9892.079001] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9892.079082] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9892.079088] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9892.079091] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9892.079106] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9892.079111] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9892.079115] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 9892.079118] RDX: 0000000000000743 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 9892.079121] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 9892.079123] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 9892.079126] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9892.079129] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 9892.079132] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9892.079134] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9892.079145] Call Trace: [ 9892.079150] [ 9892.079155] ? show_regs.cold.9+0x22/0x2f [ 9892.079165] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9892.079171] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9892.079175] ? watchdog+0x40/0x40 [ 9892.079180] ? hrtimer_interrupt+0x128/0x2c0 [ 9892.079184] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9892.079191] ? apic_timer_interrupt+0xf/0x20 [ 9892.079196] [ 9892.079198] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9892.079203] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9892.079208] do_raw_spin_lock+0x177/0x190 [ 9892.079212] _raw_spin_lock+0x21/0x30 [ 9892.079218] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9892.085134] obd_mod_free+0x46/0x140 [ptlrpc] [ 9892.086144] mdc_commit_open+0x91/0xb0 [mdc] [ 9892.086191] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9892.088456] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9892.088693] after_reply+0xe2c/0x1780 [ptlrpc] [ 9892.134440] [ 9892.134450] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9892.134468] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9892.134472] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9892.134477] RAX: 0000000000005b3a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9892.134480] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 9892.134483] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 9892.134485] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 9892.134488] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 9892.134492] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 9892.134495] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9892.134497] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9892.134506] Call Trace: [ 9892.134510] [ 9892.134516] ? show_regs.cold.9+0x22/0x2f [ 9892.134525] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9892.134531] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9892.134536] ? watchdog+0x40/0x40 [ 9892.134541] ? hrtimer_interrupt+0x128/0x2c0 [ 9892.134546] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9892.134553] ? apic_timer_interrupt+0xf/0x20 [ 9892.134558] [ 9892.134561] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9892.134566] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9892.134574] do_raw_spin_lock+0x177/0x190 [ 9892.134579] _raw_spin_lock+0x21/0x30 [ 9892.134583] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9892.135349] process_one_work+0x2c8/0x700 [ 9892.135357] worker_thread+0x296/0x6e0 [ 9892.135362] ? rescuer_thread+0x570/0x570 [ 9892.135366] kthread+0x1d1/0x200 [ 9892.135370] ? set_kthread_struct+0x70/0x70 [ 9892.135374] ret_from_fork+0x1f/0x30 [ 9892.588069] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9892.603389] ? __next_timer_interrupt+0x160/0x160 [ 9892.606633] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9892.615647] ? do_wait_intr+0xf0/0xf0 [ 9892.625138] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9892.633865] kthread+0x1d1/0x200 [ 9892.648067] ? set_kthread_struct+0x70/0x70 [ 9892.654054] ret_from_fork+0x1f/0x30 [ 9900.052038] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [ 9900.055705] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9900.079410] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9900.085785] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9900.090948] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9900.094389] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9900.104311] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9900.108493] RAX: 000000000000528c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9900.112794] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 9900.115789] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 9900.119467] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 9900.123062] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 9900.126791] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 9900.130530] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9900.133527] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 9900.136954] Call Trace: [ 9900.138288] [ 9900.139894] ? show_regs.cold.9+0x22/0x2f [ 9900.141987] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9900.143943] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9900.145794] ? watchdog+0x40/0x40 [ 9900.148289] ? hrtimer_interrupt+0x128/0x2c0 [ 9900.149986] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9900.151925] ? apic_timer_interrupt+0xf/0x20 [ 9900.154338] [ 9900.155394] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9900.158076] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9900.160369] do_raw_spin_lock+0x177/0x190 [ 9900.161948] _raw_spin_lock+0x21/0x30 [ 9900.163750] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9900.166162] ? do_raw_spin_unlock+0x75/0x190 [ 9900.167982] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9900.170322] ? do_wait_intr+0xf0/0xf0 [ 9900.172044] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9900.174687] mdc_reint+0x2d/0x110 [mdc] [ 9900.176496] mdc_unlink+0x658/0x870 [mdc] [ 9900.178364] lmv_unlink+0x34d/0xbd0 [lmv] [ 9900.179890] ll_unlink+0x24d/0x7f0 [lustre] [ 9900.181797] vfs_unlink+0x202/0x330 [ 9900.183834] do_unlinkat+0x2fa/0x460 [ 9900.185507] __x64_sys_unlink+0x37/0x50 [ 9900.188503] do_syscall_64+0xc1/0x440 [ 9900.190849] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9900.193904] RIP: 0033:0x7f151bdee10b [ 9900.195605] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9900.205746] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9900.209173] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 9900.212334] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 9900.215287] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9900.218231] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9900.221384] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9904.625741] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7974s! [ 9904.638203] Showing busy workqueues and worker pools: [ 9904.643406] workqueue events: flags=0x0 [ 9904.646794] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9904.656376] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 9904.668662] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9904.674736] pending: kfree_rcu_monitor [ 9904.692381] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9904.699698] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 9904.714383] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9904.727366] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 9904.743483] workqueue events_long: flags=0x0 [ 9904.745096] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9904.759948] pending: lu_objects_flush [obdclass] [ 9904.769121] workqueue events_unbound: flags=0x2 [ 9904.770516] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9904.775816] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9904.782784] workqueue events_power_efficient: flags=0x80 [ 9904.791076] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9904.796489] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 9904.804768] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9904.813590] pending: neigh_periodic_work [ 9904.815710] workqueue mm_percpu_wq: flags=0x8 [ 9904.816145] rcu: INFO: rcu_sched self-detected stall on CPU [ 9904.819680] pwq 6: [ 9904.822494] rcu: 1-...!: (4542092 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [ 9904.822511] (t=7980169 jiffies g=249657 q=4710) [ 9904.822516] rcu: rcu_sched kthread starved for 7975121 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 9904.822520] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 9904.822522] rcu: RCU grace-period kthread stack dump: [ 9904.822524] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 9904.822532] Call Trace: [ 9904.822540] __schedule+0x351/0xcb0 [ 9904.822554] schedule+0xc0/0x180 [ 9904.822560] schedule_timeout+0xb4/0x190 [ 9904.822564] ? do_raw_spin_unlock+0x75/0x190 [ 9904.822572] ? __next_timer_interrupt+0x160/0x160 [ 9904.822579] ? prepare_to_swait_event+0xf3/0x140 [ 9904.822586] rcu_gp_kthread+0x8dd/0x15b0 [ 9904.822592] ? force_qs_rnp+0x2c0/0x2c0 [ 9904.822595] kthread+0x1d1/0x200 [ 9904.822599] ? set_kthread_struct+0x70/0x70 [ 9904.822603] ret_from_fork+0x1f/0x30 [ 9904.822610] rcu: Stack dump where RCU GP kthread last ran: [ 9904.822613] NMI backtrace for cpu 1 [ 9904.822616] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9904.822620] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9904.822622] Call Trace: [ 9904.822625] [ 9904.822626] ? dump_stack+0xbb/0x10e [ 9904.822632] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9904.822638] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9904.822643] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9904.822648] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9904.822653] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9904.822657] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 9904.822663] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 9904.822667] ? update_process_times+0x6d/0xc0 [ 9904.822672] ? tick_sched_handle.isra.18+0x27/0x90 [ 9904.822679] ? tick_sched_timer+0x53/0xb0 [ 9904.822683] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9904.822686] ? tick_sched_handle.isra.18+0x90/0x90 [ 9904.822691] ? hrtimer_interrupt+0x128/0x2c0 [ 9904.822695] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9904.822700] ? apic_timer_interrupt+0xf/0x20 [ 9904.822705] [ 9904.822708] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9904.822712] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9904.822717] ? do_raw_spin_lock+0x177/0x190 [ 9904.822721] ? _raw_spin_lock+0x21/0x30 [ 9904.822726] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9904.823341] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9904.823540] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9904.823576] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9904.823779] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9904.824527] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9904.824724] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9904.824917] ? __next_timer_interrupt+0x160/0x160 [ 9904.824925] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9904.825618] ? do_wait_intr+0xf0/0xf0 [ 9904.825627] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9904.825828] ? kthread+0x1d1/0x200 [ 9904.825833] ? set_kthread_struct+0x70/0x70 [ 9904.825836] ? ret_from_fork+0x1f/0x30 [ 9904.825851] NMI backtrace for cpu 1 [ 9904.825855] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9904.825861] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9904.825863] Call Trace: [ 9904.825868] [ 9904.825870] ? dump_stack+0xbb/0x10e [ 9904.825875] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9904.825881] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9904.825886] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9904.825889] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9904.825894] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9904.825897] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 9904.825903] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 9904.825908] ? update_process_times+0x6d/0xc0 [ 9904.825914] ? tick_sched_handle.isra.18+0x27/0x90 [ 9904.825919] ? tick_sched_timer+0x53/0xb0 [ 9904.825923] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9904.825927] ? tick_sched_handle.isra.18+0x90/0x90 [ 9904.825932] ? hrtimer_interrupt+0x128/0x2c0 [ 9904.825937] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9904.825942] ? apic_timer_interrupt+0xf/0x20 [ 9904.825946] [ 9904.826747] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9904.826761] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9904.826766] ? do_raw_spin_lock+0x177/0x190 [ 9904.826771] ? _raw_spin_lock+0x21/0x30 [ 9904.826775] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9904.827567] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9904.827767] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9904.827803] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9904.829177] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9904.829393] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9904.829584] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9904.829775] ? __next_timer_interrupt+0x160/0x160 [ 9904.829786] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9904.830690] ? do_wait_intr+0xf0/0xf0 [ 9904.830701] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9904.830906] ? kthread+0x1d1/0x200 [ 9904.830911] ? set_kthread_struct+0x70/0x70 [ 9904.830915] ? ret_from_fork+0x1f/0x30 [ 9905.341549] cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9905.349517] pending: vmstat_update, lru_add_drain_per_cpu [ 9905.353152] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9905.360618] pending: vmstat_update [ 9905.361989] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9905.369210] pending: lru_add_drain_per_cpu [ 9905.371725] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9905.378453] pending: lru_add_drain_per_cpu BAR(48) [ 9905.380404] workqueue writeback: flags=0x4a [ 9905.387757] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 9905.403022] pending: wb_workfn [ 9905.404300] workqueue xprtiod: flags=0xa [ 9905.405691] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9905.418715] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9905.424936] workqueue ipv6_addrconf: flags=0x40008 [ 9905.427571] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9905.438269] pending: addrconf_verify_work [ 9905.440673] workqueue knbd0-recv: flags=0x1a [ 9905.444659] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9905.453761] in-flight: 658:recv_work [ 9905.456767] workqueue ptlrpc_pinger: flags=0x2 [ 9905.461188] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9905.466690] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9905.469860] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7974s workers=4 idle: 32984 37 8 [ 9905.479237] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9890s workers=2 idle: 661 [ 9912.057020] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [ 9912.059303] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9912.075393] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9912.080116] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9912.083843] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9912.085504] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9912.096724] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9912.100366] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9912.104262] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 9912.108731] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9912.113846] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 9912.118016] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 9912.122729] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 9912.126841] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9912.129137] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 9912.133044] Call Trace: [ 9912.135020] [ 9912.136298] ? show_regs.cold.9+0x22/0x2f [ 9912.139216] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9912.142233] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9912.144456] ? watchdog+0x40/0x40 [ 9912.146644] ? hrtimer_interrupt+0x128/0x2c0 [ 9912.150479] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9912.153097] ? apic_timer_interrupt+0xf/0x20 [ 9912.156439] [ 9912.157971] ? native_safe_halt+0x2e/0x50 [ 9912.160281] kvm_wait+0xcd/0xe0 [ 9912.162445] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9912.165467] do_raw_spin_lock+0x177/0x190 [ 9912.168497] _raw_spin_lock+0x21/0x30 [ 9912.171823] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9912.176747] ? do_raw_spin_unlock+0x75/0x190 [ 9912.180058] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9912.183723] ? do_wait_intr+0xf0/0xf0 [ 9912.186003] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9912.189414] mdc_close+0x53b/0xd70 [mdc] [ 9912.191799] lmv_close+0x18d/0x490 [lmv] [ 9912.194050] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9912.196704] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9912.200321] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9912.203121] ll_md_close+0x274/0x800 [lustre] [ 9912.205723] ll_file_release+0x128/0xc50 [lustre] [ 9912.208916] __fput+0x10d/0x440 [ 9912.210704] ____fput+0x12/0x20 [ 9912.212361] task_work_run+0xc6/0x110 [ 9912.214427] exit_to_usermode_loop+0x1dd/0x1f0 [ 9912.216789] do_syscall_64+0x3d6/0x440 [ 9912.219259] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9912.222978] RIP: 0033:0x7f151bee3cd8 [ 9912.225799] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9912.236233] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9912.240418] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 9912.244413] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 9912.249323] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9912.253308] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9912.256990] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9920.058872] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9920.065586] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9920.090678] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9920.097839] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9920.103688] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9920.112068] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9920.115115] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9920.131629] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9920.136569] RAX: 0000000000007f58 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9920.140212] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 9920.143648] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 9920.147246] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 9920.151089] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 9920.154516] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 9920.159015] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9920.162008] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9920.165571] Call Trace: [ 9920.166494] [ 9920.167288] ? show_regs.cold.9+0x22/0x2f [ 9920.169283] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9920.171683] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9920.174029] ? watchdog+0x40/0x40 [ 9920.175590] ? hrtimer_interrupt+0x128/0x2c0 [ 9920.177125] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9920.179135] ? apic_timer_interrupt+0xf/0x20 [ 9920.180678] [ 9920.181352] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9920.184225] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9920.186374] do_raw_spin_lock+0x177/0x190 [ 9920.188327] _raw_spin_lock+0x21/0x30 [ 9920.190277] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9920.193311] process_one_work+0x2c8/0x700 [ 9920.195279] worker_thread+0x296/0x6e0 [ 9920.197310] ? rescuer_thread+0x570/0x570 [ 9920.199374] kthread+0x1d1/0x200 [ 9920.201127] ? set_kthread_struct+0x70/0x70 [ 9920.203254] ret_from_fork+0x1f/0x30 [ 9932.055039] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [ 9932.056355] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 9932.067964] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9932.068044] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9932.068052] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9932.068054] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9932.068068] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9932.068073] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9932.068078] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 9932.068081] RDX: 0000000000004edb RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 9932.068083] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 9932.068086] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 9932.068088] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9932.068092] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 9932.068095] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9932.068097] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9932.068106] Call Trace: [ 9932.068112] [ 9932.068118] ? show_regs.cold.9+0x22/0x2f [ 9932.068217] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9932.068224] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9932.068229] ? watchdog+0x40/0x40 [ 9932.068234] ? hrtimer_interrupt+0x128/0x2c0 [ 9932.068238] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9932.068247] ? apic_timer_interrupt+0xf/0x20 [ 9932.068261] [ 9932.068264] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9932.068269] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9932.068274] do_raw_spin_lock+0x177/0x190 [ 9932.068279] _raw_spin_lock+0x21/0x30 [ 9932.068285] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9932.068672] obd_mod_free+0x46/0x140 [ptlrpc] [ 9932.069064] mdc_commit_open+0x91/0xb0 [mdc] [ 9932.069104] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9932.070489] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9932.073280] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw [ 9932.102460] after_reply+0xe2c/0x1780 [ptlrpc] [ 9932.102998] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9932.103341] ? __next_timer_interrupt+0x160/0x160 [ 9932.103352] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9932.103578] ? do_wait_intr+0xf0/0xf0 [ 9932.103588] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9932.104003] kthread+0x1d1/0x200 [ 9932.104010] ? set_kthread_struct+0x70/0x70 [ 9932.104015] ret_from_fork+0x1f/0x30 [ 9932.323200] libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9932.331370] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9932.335746] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9932.349217] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9932.352972] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9932.370376] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9932.375033] RAX: 0000000000007d0f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9932.383574] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 9932.396053] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 9932.412206] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 9932.414461] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 9932.420478] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 9932.424791] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9932.430620] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 9932.438637] Call Trace: [ 9932.443549] [ 9932.444260] ? show_regs.cold.9+0x22/0x2f [ 9932.448256] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9932.454760] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9932.457679] ? watchdog+0x40/0x40 [ 9932.458714] ? hrtimer_interrupt+0x128/0x2c0 [ 9932.460800] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9932.464646] ? apic_timer_interrupt+0xf/0x20 [ 9932.466002] [ 9932.466699] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9932.471094] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9932.473314] do_raw_spin_lock+0x177/0x190 [ 9932.474754] _raw_spin_lock+0x21/0x30 [ 9932.475927] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9932.480992] ? do_raw_spin_unlock+0x75/0x190 [ 9932.483214] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9932.486988] ? do_wait_intr+0xf0/0xf0 [ 9932.488991] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9932.491671] mdc_reint+0x2d/0x110 [mdc] [ 9932.498138] mdc_unlink+0x658/0x870 [mdc] [ 9932.500615] lmv_unlink+0x34d/0xbd0 [lmv] [ 9932.504917] ll_unlink+0x24d/0x7f0 [lustre] [ 9932.509393] vfs_unlink+0x202/0x330 [ 9932.511129] do_unlinkat+0x2fa/0x460 [ 9932.512931] __x64_sys_unlink+0x37/0x50 [ 9932.518289] do_syscall_64+0xc1/0x440 [ 9932.519955] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9932.522199] RIP: 0033:0x7f151bdee10b [ 9932.523597] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9932.536015] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9932.546665] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 9932.560392] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 9932.569686] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9932.574479] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9932.581552] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9935.844691] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8005s! [ 9935.851062] Showing busy workqueues and worker pools: [ 9935.855158] workqueue events: flags=0x0 [ 9935.859368] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9935.869252] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 9935.883860] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9935.892345] pending: kfree_rcu_monitor [ 9935.898248] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9935.905108] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 9935.914964] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9935.921212] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 9935.930378] workqueue events_long: flags=0x0 [ 9935.936183] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9935.942945] pending: lu_objects_flush [obdclass] [ 9935.951103] workqueue events_unbound: flags=0x2 [ 9935.955475] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9935.965179] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9935.976814] workqueue events_power_efficient: flags=0x80 [ 9935.980541] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9935.991234] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 9935.993319] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9936.006832] pending: neigh_periodic_work [ 9936.016543] workqueue mm_percpu_wq: flags=0x8 [ 9936.021548] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9936.023572] pending: vmstat_update, lru_add_drain_per_cpu [ 9936.025608] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9936.028107] pending: vmstat_update [ 9936.029227] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9936.031376] pending: lru_add_drain_per_cpu [ 9936.033026] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9936.035478] pending: lru_add_drain_per_cpu BAR(48) [ 9936.037424] workqueue writeback: flags=0x4a [ 9936.039103] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 9936.041370] pending: wb_workfn [ 9936.042902] workqueue xprtiod: flags=0xa [ 9936.044371] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9936.049930] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9936.062200] workqueue ipv6_addrconf: flags=0x40008 [ 9936.068542] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9936.072660] pending: addrconf_verify_work [ 9936.080018] workqueue knbd0-recv: flags=0x1a [ 9936.084469] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9936.091335] in-flight: 658:recv_work [ 9936.095298] workqueue ptlrpc_pinger: flags=0x2 [ 9936.101964] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9936.107260] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9936.112054] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8005s workers=4 idle: 32984 37 8 [ 9936.121655] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9920s workers=2 idle: 661 [ 9940.057130] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 9940.062346] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9940.084992] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9940.090791] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9940.095117] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9940.096903] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9940.103595] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9940.106914] RAX: 000000000000530c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9940.109682] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 9940.112795] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9940.115046] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 9940.117805] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 9940.119802] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 9940.122377] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9940.124182] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 9940.126252] Call Trace: [ 9940.126974] [ 9940.127516] ? show_regs.cold.9+0x22/0x2f [ 9940.128604] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9940.130343] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9940.132721] ? watchdog+0x40/0x40 [ 9940.133800] ? hrtimer_interrupt+0x128/0x2c0 [ 9940.135075] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9940.136367] ? apic_timer_interrupt+0xf/0x20 [ 9940.137839] [ 9940.140256] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9940.143064] do_raw_spin_lock+0x177/0x190 [ 9940.145422] _raw_spin_lock+0x21/0x30 [ 9940.147267] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9940.149986] ? do_raw_spin_unlock+0x75/0x190 [ 9940.151401] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9940.155345] ? do_wait_intr+0xf0/0xf0 [ 9940.160492] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9940.164218] mdc_close+0x53b/0xd70 [mdc] [ 9940.166612] lmv_close+0x18d/0x490 [lmv] [ 9940.169564] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9940.175527] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9940.179491] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9940.181296] ll_md_close+0x274/0x800 [lustre] [ 9940.185997] ll_file_release+0x128/0xc50 [lustre] [ 9940.189194] __fput+0x10d/0x440 [ 9940.190316] ____fput+0x12/0x20 [ 9940.193023] task_work_run+0xc6/0x110 [ 9940.199145] exit_to_usermode_loop+0x1dd/0x1f0 [ 9940.200661] do_syscall_64+0x3d6/0x440 [ 9940.212157] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9940.214463] RIP: 0033:0x7f151bee3cd8 [ 9940.216698] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9940.237513] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9940.246492] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 9940.249287] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 9940.260198] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9940.266149] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9940.275269] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9948.056013] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9948.060425] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9948.096624] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9948.107498] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9948.115388] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9948.120087] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9948.124062] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9948.135221] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9948.147612] RAX: 0000000000004583 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9948.157453] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 9948.167291] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 9948.175116] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 9948.182607] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 9948.188580] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 9948.193188] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9948.197146] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9948.202444] Call Trace: [ 9948.205535] [ 9948.208048] ? show_regs.cold.9+0x22/0x2f [ 9948.212208] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9948.217272] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9948.224999] ? watchdog+0x40/0x40 [ 9948.229847] ? hrtimer_interrupt+0x128/0x2c0 [ 9948.237782] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9948.240836] ? apic_timer_interrupt+0xf/0x20 [ 9948.242857] [ 9948.246179] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9948.251470] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9948.254664] do_raw_spin_lock+0x177/0x190 [ 9948.257404] _raw_spin_lock+0x21/0x30 [ 9948.258826] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9948.262170] process_one_work+0x2c8/0x700 [ 9948.263667] worker_thread+0x296/0x6e0 [ 9948.266745] ? rescuer_thread+0x570/0x570 [ 9948.269024] kthread+0x1d1/0x200 [ 9948.270491] ? set_kthread_struct+0x70/0x70 [ 9948.276088] ret_from_fork+0x1f/0x30 [ 9960.056338] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 9960.062506] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9960.135018] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9960.144916] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9960.151356] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9960.163784] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9960.183829] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9960.196333] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 9960.203127] RDX: 0000000000007334 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 9960.208061] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 9960.212691] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 9960.215135] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9960.226751] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 9960.235220] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9960.237818] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9960.241195] Call Trace: [ 9960.242482] [ 9960.243669] ? show_regs.cold.9+0x22/0x2f [ 9960.245699] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9960.247867] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9960.249969] ? watchdog+0x40/0x40 [ 9960.252195] ? hrtimer_interrupt+0x128/0x2c0 [ 9960.255083] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9960.263430] ? apic_timer_interrupt+0xf/0x20 [ 9960.264512] [ 9960.265292] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9960.275171] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9960.278124] do_raw_spin_lock+0x177/0x190 [ 9960.280170] _raw_spin_lock+0x21/0x30 [ 9960.281913] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9960.284254] obd_mod_free+0x46/0x140 [ptlrpc] [ 9960.286721] mdc_commit_open+0x91/0xb0 [mdc] [ 9960.289056] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9960.291412] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9960.294090] after_reply+0xe2c/0x1780 [ptlrpc] [ 9960.296262] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9960.298751] ? __next_timer_interrupt+0x160/0x160 [ 9960.300686] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9960.302773] ? do_wait_intr+0xf0/0xf0 [ 9960.307394] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9960.314724] kthread+0x1d1/0x200 [ 9960.315798] ? set_kthread_struct+0x70/0x70 [ 9960.317017] ret_from_fork+0x1f/0x30 [ 9964.052057] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 9964.055930] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9964.084818] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9964.090253] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9964.095131] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9964.098206] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9964.105930] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9964.109454] RAX: 0000000000003e6d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9964.113016] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 9964.116682] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 9964.119412] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 9964.123047] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 9964.127222] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 9964.131400] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9964.133538] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 9964.137102] Call Trace: [ 9964.138258] [ 9964.139307] ? show_regs.cold.9+0x22/0x2f [ 9964.141494] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9964.143921] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9964.146158] ? watchdog+0x40/0x40 [ 9964.147535] ? hrtimer_interrupt+0x128/0x2c0 [ 9964.149141] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9964.151194] ? apic_timer_interrupt+0xf/0x20 [ 9964.152819] [ 9964.153443] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9964.156051] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9964.158929] do_raw_spin_lock+0x177/0x190 [ 9964.160592] _raw_spin_lock+0x21/0x30 [ 9964.162592] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9964.166184] ? do_raw_spin_unlock+0x75/0x190 [ 9964.168340] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9964.171580] ? do_wait_intr+0xf0/0xf0 [ 9964.173466] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9964.176324] mdc_reint+0x2d/0x110 [mdc] [ 9964.178338] mdc_unlink+0x658/0x870 [mdc] [ 9964.180411] lmv_unlink+0x34d/0xbd0 [lmv] [ 9964.182599] ll_unlink+0x24d/0x7f0 [lustre] [ 9964.184942] vfs_unlink+0x202/0x330 [ 9964.186324] do_unlinkat+0x2fa/0x460 [ 9964.188055] __x64_sys_unlink+0x37/0x50 [ 9964.189480] do_syscall_64+0xc1/0x440 [ 9964.191324] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9964.193561] RIP: 0033:0x7f151bdee10b [ 9964.195136] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9964.202730] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9964.206069] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 9964.209015] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 9964.211936] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9964.214518] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9964.217713] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9966.569926] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8035s! [ 9966.575692] Showing busy workqueues and worker pools: [ 9966.577353] workqueue events: flags=0x0 [ 9966.581818] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9966.587086] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 9966.598802] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9966.602317] pending: kfree_rcu_monitor [ 9966.609336] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9966.615898] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 9966.625008] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9966.633452] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 9966.640770] workqueue events_long: flags=0x0 [ 9966.642060] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9966.649622] pending: lu_objects_flush [obdclass] [ 9966.655073] workqueue events_unbound: flags=0x2 [ 9966.662125] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9966.669020] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9966.676273] workqueue events_power_efficient: flags=0x80 [ 9966.684661] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9966.692669] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 9966.701590] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9966.705089] pending: neigh_periodic_work [ 9966.708013] workqueue mm_percpu_wq: flags=0x8 [ 9966.711374] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9966.714865] pending: vmstat_update, lru_add_drain_per_cpu [ 9966.717243] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9966.720543] pending: vmstat_update [ 9966.722338] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9966.725059] pending: lru_add_drain_per_cpu [ 9966.726169] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9966.728214] pending: lru_add_drain_per_cpu BAR(48) [ 9966.729917] workqueue writeback: flags=0x4a [ 9966.731008] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 9966.733069] pending: wb_workfn [ 9966.735285] workqueue xprtiod: flags=0xa [ 9966.736821] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9966.738865] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9966.746118] workqueue ipv6_addrconf: flags=0x40008 [ 9966.754186] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9966.764038] pending: addrconf_verify_work [ 9966.769054] workqueue knbd0-recv: flags=0x1a [ 9966.771882] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9966.780199] in-flight: 658:recv_work [ 9966.782675] workqueue ptlrpc_pinger: flags=0x2 [ 9966.787024] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9966.790642] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9966.798909] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8036s workers=4 idle: 32984 37 8 [ 9966.809216] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9951s workers=2 idle: 661 [ 9968.060322] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 9968.066164] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9968.098555] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9968.106160] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9968.112500] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9968.116744] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9968.129717] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 9968.135664] RAX: 0000000000001a85 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9968.139315] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 9968.143091] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9968.148025] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 9968.155660] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 9968.159316] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 9968.163302] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9968.166188] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 9968.172495] Call Trace: [ 9968.173358] [ 9968.174321] ? show_regs.cold.9+0x22/0x2f [ 9968.176181] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9968.179706] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9968.182661] ? watchdog+0x40/0x40 [ 9968.184632] ? hrtimer_interrupt+0x128/0x2c0 [ 9968.187308] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9968.189526] ? apic_timer_interrupt+0xf/0x20 [ 9968.191321] [ 9968.192550] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9968.194942] do_raw_spin_lock+0x177/0x190 [ 9968.196828] _raw_spin_lock+0x21/0x30 [ 9968.200306] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9968.205274] ? do_raw_spin_unlock+0x75/0x190 [ 9968.210981] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9968.214072] ? do_wait_intr+0xf0/0xf0 [ 9968.218176] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9968.221869] mdc_close+0x53b/0xd70 [mdc] [ 9968.223798] lmv_close+0x18d/0x490 [lmv] [ 9968.225856] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9968.228785] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9968.231710] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9968.234426] ll_md_close+0x274/0x800 [lustre] [ 9968.236561] ll_file_release+0x128/0xc50 [lustre] [ 9968.238140] __fput+0x10d/0x440 [ 9968.240217] ____fput+0x12/0x20 [ 9968.241961] task_work_run+0xc6/0x110 [ 9968.243934] exit_to_usermode_loop+0x1dd/0x1f0 [ 9968.246541] do_syscall_64+0x3d6/0x440 [ 9968.248841] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9968.252612] RIP: 0033:0x7f151bee3cd8 [ 9968.254332] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9968.265133] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9968.269726] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 9968.279794] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 9968.289783] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9968.297705] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9968.303111] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9976.063258] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [ 9976.065964] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9976.118402] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9976.126596] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9976.132780] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9976.135533] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9976.138832] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9976.149860] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9976.152925] RAX: 000000000000750e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9976.156894] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [ 9976.160819] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [ 9976.165934] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [ 9976.169725] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [ 9976.173382] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [ 9976.177811] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9976.181014] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9976.185714] Call Trace: [ 9976.186721] [ 9976.187593] ? show_regs.cold.9+0x22/0x2f [ 9976.189661] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9976.192000] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9976.194650] ? watchdog+0x40/0x40 [ 9976.196463] ? hrtimer_interrupt+0x128/0x2c0 [ 9976.197913] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9976.201475] ? apic_timer_interrupt+0xf/0x20 [ 9976.204918] [ 9976.210623] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9976.215002] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9976.223506] do_raw_spin_lock+0x177/0x190 [ 9976.229527] _raw_spin_lock+0x21/0x30 [ 9976.236718] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9976.246558] process_one_work+0x2c8/0x700 [ 9976.252850] worker_thread+0x296/0x6e0 [ 9976.262903] ? rescuer_thread+0x570/0x570 [ 9976.264342] kthread+0x1d1/0x200 [ 9976.273964] ? set_kthread_struct+0x70/0x70 [ 9976.283012] ret_from_fork+0x1f/0x30 [ 9988.055283] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [ 9988.061099] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9988.115928] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9988.122931] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9988.128007] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9988.130490] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9988.140385] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9988.145082] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [ 9988.150750] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [ 9988.156845] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [ 9988.164433] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [ 9988.173968] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9988.181346] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [ 9988.190613] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9988.195654] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [ 9988.198117] Call Trace: [ 9988.199014] [ 9988.207284] ? show_regs.cold.9+0x22/0x2f [ 9988.210401] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9988.216312] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9988.219481] ? watchdog+0x40/0x40 [ 9988.221903] ? hrtimer_interrupt+0x128/0x2c0 [ 9988.227131] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9988.232074] ? apic_timer_interrupt+0xf/0x20 [ 9988.235616] [ 9988.237357] ? native_safe_halt+0x2e/0x50 [ 9988.244035] kvm_wait+0xcd/0xe0 [ 9988.245477] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9988.250853] do_raw_spin_lock+0x177/0x190 [ 9988.253645] _raw_spin_lock+0x21/0x30 [ 9988.256811] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9988.262089] obd_mod_free+0x46/0x140 [ptlrpc] [ 9988.267453] mdc_commit_open+0x91/0xb0 [mdc] [ 9988.269418] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9988.273098] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9988.278957] after_reply+0xe2c/0x1780 [ptlrpc] [ 9988.284649] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9988.292846] ? __next_timer_interrupt+0x160/0x160 [ 9988.298882] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9988.302729] ? do_wait_intr+0xf0/0xf0 [ 9988.305414] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9988.309964] kthread+0x1d1/0x200 [ 9988.311938] ? set_kthread_struct+0x70/0x70 [ 9988.317043] ret_from_fork+0x1f/0x30 [ 9992.052013] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [ 9992.056994] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9992.082086] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9992.087820] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9992.091837] RIP: 0010:__pv_queued_spin_lock_slowpath+0x318/0x6d0 [ 9992.095446] Code: b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 <85> d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 eb e0 48 83 05 06 [ 9992.107848] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9992.113914] RAX: 0000000000004e2d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9992.117595] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [ 9992.122068] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [ 9992.125912] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [ 9992.128870] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [ 9992.132681] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [ 9992.136844] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9992.139452] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [ 9992.143664] Call Trace: [ 9992.146212] [ 9992.147279] ? show_regs.cold.9+0x22/0x2f [ 9992.149986] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9992.152352] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9992.154597] ? watchdog+0x40/0x40 [ 9992.156403] ? hrtimer_interrupt+0x128/0x2c0 [ 9992.158315] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9992.161342] ? apic_timer_interrupt+0xf/0x20 [ 9992.163423] [ 9992.164307] ? __pv_queued_spin_lock_slowpath+0x318/0x6d0 [ 9992.168125] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9992.171766] do_raw_spin_lock+0x177/0x190 [ 9992.175436] _raw_spin_lock+0x21/0x30 [ 9992.177115] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9992.180375] ? do_raw_spin_unlock+0x75/0x190 [ 9992.183695] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9992.186017] ? do_wait_intr+0xf0/0xf0 [ 9992.188511] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9992.191075] mdc_reint+0x2d/0x110 [mdc] [ 9992.193248] mdc_unlink+0x658/0x870 [mdc] [ 9992.195559] lmv_unlink+0x34d/0xbd0 [lmv] [ 9992.198241] ll_unlink+0x24d/0x7f0 [lustre] [ 9992.200546] vfs_unlink+0x202/0x330 [ 9992.201819] do_unlinkat+0x2fa/0x460 [ 9992.203679] __x64_sys_unlink+0x37/0x50 [ 9992.206398] do_syscall_64+0xc1/0x440 [ 9992.208333] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9992.210991] RIP: 0033:0x7f151bdee10b [ 9992.212822] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9992.219479] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9992.224348] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [ 9992.226960] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [ 9992.230099] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9992.232617] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9992.235003] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9996.057134] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [ 9996.062615] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9996.085646] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9996.091873] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9996.104906] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9996.110542] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9996.131673] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9996.135455] RAX: 000000000000376b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [ 9996.141020] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [ 9996.147995] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9996.162027] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [ 9996.174921] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [ 9996.182934] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [ 9996.196890] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9996.209434] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [ 9996.216899] Call Trace: [ 9996.217639] [ 9996.218299] ? show_regs.cold.9+0x22/0x2f [ 9996.219616] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9996.225250] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9996.226950] ? watchdog+0x40/0x40 [ 9996.228025] ? hrtimer_interrupt+0x128/0x2c0 [ 9996.234571] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9996.236191] ? apic_timer_interrupt+0xf/0x20 [ 9996.242935] [ 9996.243661] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9996.247906] do_raw_spin_lock+0x177/0x190 [ 9996.249552] _raw_spin_lock+0x21/0x30 [ 9996.254204] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9996.261785] ? do_raw_spin_unlock+0x75/0x190 [ 9996.266555] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9996.269372] ? do_wait_intr+0xf0/0xf0 [ 9996.272784] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9996.274877] mdc_close+0x53b/0xd70 [mdc] [ 9996.279646] lmv_close+0x18d/0x490 [lmv] [ 9996.281850] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9996.291525] ll_close_inode_openhandle+0x493/0xea0 [lustre] [ 9996.294923] ll_md_real_close+0xa5/0x2a0 [lustre] [ 9996.297650] ll_md_close+0x274/0x800 [lustre] [ 9996.302499] ll_file_release+0x128/0xc50 [lustre] [ 9996.307419] __fput+0x10d/0x440 [ 9996.311985] ____fput+0x12/0x20 [ 9996.313104] task_work_run+0xc6/0x110 [ 9996.314561] exit_to_usermode_loop+0x1dd/0x1f0 [ 9996.319391] do_syscall_64+0x3d6/0x440 [ 9996.320667] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9996.325906] RIP: 0033:0x7f151bee3cd8 [ 9996.327058] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [ 9996.339683] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [ 9996.346612] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [ 9996.352927] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [ 9996.356337] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [ 9996.361048] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [ 9996.366921] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9997.287333] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8066s! [ 9997.291691] Showing busy workqueues and worker pools: [ 9997.300738] workqueue events: flags=0x0 [ 9997.306633] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9997.319457] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [ 9997.331079] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9997.335479] pending: kfree_rcu_monitor [ 9997.339218] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9997.343246] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [ 9997.355542] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9997.369361] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [ 9997.378010] workqueue events_long: flags=0x0 [ 9997.385092] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9997.389255] pending: lu_objects_flush [obdclass] [ 9997.396716] workqueue events_unbound: flags=0x2 [ 9997.407320] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [ 9997.411912] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [ 9997.420377] workqueue events_power_efficient: flags=0x80 [ 9997.441311] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9997.459932] pending: do_cache_clean, neigh_periodic_work, check_lifetime [ 9997.476344] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9997.483267] pending: neigh_periodic_work [ 9997.486328] workqueue mm_percpu_wq: flags=0x8 [ 9997.489488] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9997.498994] pending: vmstat_update, lru_add_drain_per_cpu [ 9997.503597] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9997.513578] pending: vmstat_update [ 9997.514781] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9997.521103] pending: lru_add_drain_per_cpu [ 9997.523102] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9997.527089] pending: lru_add_drain_per_cpu BAR(48) [ 9997.529977] workqueue writeback: flags=0x4a [ 9997.533626] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 9997.537463] pending: wb_workfn [ 9997.541409] workqueue xprtiod: flags=0xa [ 9997.548074] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [ 9997.552594] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [ 9997.558159] workqueue ipv6_addrconf: flags=0x40008 [ 9997.568569] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9997.573246] pending: addrconf_verify_work [ 9997.576047] workqueue knbd0-recv: flags=0x1a [ 9997.583223] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9997.588095] in-flight: 658:recv_work [ 9997.590328] workqueue ptlrpc_pinger: flags=0x2 [ 9997.591894] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9997.596308] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [ 9997.604343] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8067s workers=4 idle: 32984 37 8 [ 9997.614605] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9982s workers=2 idle: 661 [10004.058578] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [10004.063570] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10004.102751] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10004.115270] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10004.124484] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10004.132423] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10004.137532] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10004.152357] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10004.158429] RAX: 0000000000007912 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10004.164838] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [10004.170727] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [10004.178672] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [10004.182881] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [10004.188721] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [10004.195447] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10004.207583] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10004.214647] Call Trace: [10004.217971] [10004.219951] ? show_regs.cold.9+0x22/0x2f [10004.224862] ? watchdog_timer_fn.cold.6+0x62/0x10a [10004.231362] ? __hrtimer_run_queues+0x1c2/0x4d0 [10004.236737] ? watchdog+0x40/0x40 [10004.240935] ? hrtimer_interrupt+0x128/0x2c0 [10004.244691] ? smp_apic_timer_interrupt+0xa6/0x270 [10004.249557] ? apic_timer_interrupt+0xf/0x20 [10004.251127] [10004.257394] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10004.262826] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10004.268645] do_raw_spin_lock+0x177/0x190 [10004.273761] _raw_spin_lock+0x21/0x30 [10004.279447] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10004.284459] process_one_work+0x2c8/0x700 [10004.285889] worker_thread+0x296/0x6e0 [10004.296966] ? rescuer_thread+0x570/0x570 [10004.298325] kthread+0x1d1/0x200 [10004.304906] ? set_kthread_struct+0x70/0x70 [10004.310701] ret_from_fork+0x1f/0x30 [10016.055034] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [10016.058524] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10016.086975] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10016.099678] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10016.105799] RIP: 0010:native_safe_halt+0x2e/0x50 [10016.110780] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10016.125684] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10016.129413] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [10016.135143] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [10016.138787] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [10016.141342] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [10016.144293] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10016.151048] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [10016.159457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10016.163380] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10016.166557] Call Trace: [10016.172869] [10016.173822] ? show_regs.cold.9+0x22/0x2f [10016.179191] ? watchdog_timer_fn.cold.6+0x62/0x10a [10016.182137] ? __hrtimer_run_queues+0x1c2/0x4d0 [10016.187376] ? watchdog+0x40/0x40 [10016.188850] ? hrtimer_interrupt+0x128/0x2c0 [10016.190662] ? smp_apic_timer_interrupt+0xa6/0x270 [10016.193235] ? apic_timer_interrupt+0xf/0x20 [10016.195063] [10016.195760] ? native_safe_halt+0x2e/0x50 [10016.197342] kvm_wait+0xcd/0xe0 [10016.198551] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10016.200355] do_raw_spin_lock+0x177/0x190 [10016.202173] _raw_spin_lock+0x21/0x30 [10016.203932] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10016.206773] obd_mod_free+0x46/0x140 [ptlrpc] [10016.208447] mdc_commit_open+0x91/0xb0 [mdc] [10016.209937] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10016.212316] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10016.214354] after_reply+0xe2c/0x1780 [ptlrpc] [10016.216538] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10016.218693] ? __next_timer_interrupt+0x160/0x160 [10016.220235] ptlrpcd+0xa82/0xe40 [ptlrpc] [10016.222158] ? do_wait_intr+0xf0/0xf0 [10016.223172] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10016.225117] kthread+0x1d1/0x200 [10016.226192] ? set_kthread_struct+0x70/0x70 [10016.228007] ret_from_fork+0x1f/0x30 [10024.052963] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [10024.059457] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10024.064144] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [10024.078258] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10024.078278] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10024.078283] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10024.078301] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10024.078307] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [10024.078399] RAX: 0000000000007c8f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10024.078402] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [10024.078405] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [10024.078407] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [10024.078410] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [10024.078414] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [10024.078417] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10024.078420] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [10024.078428] Call Trace: [10024.078435] [10024.078442] ? show_regs.cold.9+0x22/0x2f [10024.078453] ? watchdog_timer_fn.cold.6+0x62/0x10a [10024.078461] ? __hrtimer_run_queues+0x1c2/0x4d0 [10024.078466] ? watchdog+0x40/0x40 [10024.078471] ? hrtimer_interrupt+0x128/0x2c0 [10024.078476] ? smp_apic_timer_interrupt+0xa6/0x270 [10024.078486] ? apic_timer_interrupt+0xf/0x20 [10024.078491] [10024.078495] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10024.078501] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10024.078506] do_raw_spin_lock+0x177/0x190 [10024.078511] _raw_spin_lock+0x21/0x30 [10024.078520] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10024.078995] ? do_raw_spin_unlock+0x75/0x190 [10024.079002] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10024.079226] ? do_wait_intr+0xf0/0xf0 [10024.079235] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10024.079533] mdc_reint+0x2d/0x110 [mdc] [10024.079580] mdc_unlink+0x658/0x870 [mdc] [10024.079608] lmv_unlink+0x34d/0xbd0 [lmv] [10024.079642] ll_unlink+0x24d/0x7f0 [lustre] [10024.079903] vfs_unlink+0x202/0x330 [10024.079913] do_unlinkat+0x2fa/0x460 [10024.079919] __x64_sys_unlink+0x37/0x50 [10024.079923] do_syscall_64+0xc1/0x440 [10024.079931] entry_SYSCALL_64_after_hwframe+0x49/0xae [10024.079949] RIP: 0033:0x7f151bdee10b [10024.079959] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10024.079964] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10024.079969] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [10024.079972] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [10024.079975] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10024.079978] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10024.079981] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10024.430142] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10024.477439] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10024.485967] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10024.497123] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10024.502474] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10024.511740] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10024.514448] RAX: 000000000000171e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10024.520570] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [10024.525570] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [10024.529699] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [10024.533747] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [10024.536395] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [10024.543817] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10024.547883] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [10024.556355] Call Trace: [10024.557740] [10024.558883] ? show_regs.cold.9+0x22/0x2f [10024.563916] ? watchdog_timer_fn.cold.6+0x62/0x10a [10024.565893] ? __hrtimer_run_queues+0x1c2/0x4d0 [10024.568550] ? watchdog+0x40/0x40 [10024.570044] ? hrtimer_interrupt+0x128/0x2c0 [10024.571530] ? smp_apic_timer_interrupt+0xa6/0x270 [10024.573314] ? apic_timer_interrupt+0xf/0x20 [10024.575766] [10024.578621] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10024.585364] do_raw_spin_lock+0x177/0x190 [10024.588737] _raw_spin_lock+0x21/0x30 [10024.590577] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10024.596596] ? do_raw_spin_unlock+0x75/0x190 [10024.598971] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10024.602784] ? do_wait_intr+0xf0/0xf0 [10024.604094] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10024.608126] mdc_close+0x53b/0xd70 [mdc] [10024.609471] lmv_close+0x18d/0x490 [lmv] [10024.611042] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10024.619508] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10024.626427] ll_md_real_close+0xa5/0x2a0 [lustre] [10024.631958] ll_md_close+0x274/0x800 [lustre] [10024.634415] ll_file_release+0x128/0xc50 [lustre] [10024.637720] __fput+0x10d/0x440 [10024.641580] ____fput+0x12/0x20 [10024.645359] task_work_run+0xc6/0x110 [10024.647755] exit_to_usermode_loop+0x1dd/0x1f0 [10024.651658] do_syscall_64+0x3d6/0x440 [10024.657981] entry_SYSCALL_64_after_hwframe+0x49/0xae [10024.665365] RIP: 0033:0x7f151bee3cd8 [10024.669792] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10024.689500] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10024.699241] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [10024.704585] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [10024.711343] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10024.719367] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10024.724474] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10028.001042] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8097s! [10028.005428] Showing busy workqueues and worker pools: [10028.008431] workqueue events: flags=0x0 [10028.011093] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10028.015597] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [10028.024034] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10028.028595] pending: kfree_rcu_monitor [10028.031619] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10028.041169] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [10028.049363] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10028.056111] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [10028.067316] workqueue events_long: flags=0x0 [10028.075316] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10028.086235] pending: lu_objects_flush [obdclass] [10028.090092] workqueue events_unbound: flags=0x2 [10028.093249] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10028.097614] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10028.104194] workqueue events_power_efficient: flags=0x80 [10028.108286] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10028.112655] pending: do_cache_clean, neigh_periodic_work, check_lifetime [10028.119654] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10028.124979] pending: neigh_periodic_work [10028.128500] workqueue mm_percpu_wq: flags=0x8 [10028.134016] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10028.138960] pending: vmstat_update, lru_add_drain_per_cpu [10028.142566] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10028.147465] pending: vmstat_update [10028.150527] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10028.156067] pending: lru_add_drain_per_cpu [10028.164626] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10028.174439] pending: lru_add_drain_per_cpu BAR(48) [10028.178613] workqueue writeback: flags=0x4a [10028.181488] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [10028.183759] pending: wb_workfn [10028.185525] workqueue xprtiod: flags=0xa [10028.187371] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10028.196491] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [10028.208021] workqueue ipv6_addrconf: flags=0x40008 [10028.216230] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10028.228772] pending: addrconf_verify_work [10028.237731] workqueue knbd0-recv: flags=0x1a [10028.242920] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10028.254789] in-flight: 658:recv_work [10028.263077] workqueue ptlrpc_pinger: flags=0x2 [10028.265398] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10028.277775] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10028.283298] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8097s workers=4 idle: 32984 37 8 [10028.293265] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10013s workers=2 idle: 661 [10032.061630] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [10032.071578] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10032.100212] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10032.109833] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10032.118310] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10032.124204] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10032.129205] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10032.144984] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10032.155189] RAX: 00000000000024fe RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10032.163723] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [10032.169922] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [10032.172225] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [10032.175975] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [10032.181255] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [10032.183927] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10032.185957] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10032.188532] Call Trace: [10032.190118] [10032.191024] ? show_regs.cold.9+0x22/0x2f [10032.193108] ? watchdog_timer_fn.cold.6+0x62/0x10a [10032.197209] ? __hrtimer_run_queues+0x1c2/0x4d0 [10032.201287] ? watchdog+0x40/0x40 [10032.203214] ? hrtimer_interrupt+0x128/0x2c0 [10032.205347] ? smp_apic_timer_interrupt+0xa6/0x270 [10032.209761] ? apic_timer_interrupt+0xf/0x20 [10032.214102] [10032.216154] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10032.220338] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10032.224421] do_raw_spin_lock+0x177/0x190 [10032.225458] _raw_spin_lock+0x21/0x30 [10032.242057] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10032.250048] process_one_work+0x2c8/0x700 [10032.255514] worker_thread+0x296/0x6e0 [10032.263709] ? rescuer_thread+0x570/0x570 [10032.268172] kthread+0x1d1/0x200 [10032.273178] ? set_kthread_struct+0x70/0x70 [10032.274738] ret_from_fork+0x1f/0x30 [10044.055056] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [10044.057858] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10044.089826] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10044.095889] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10044.100196] RIP: 0010:native_safe_halt+0x2e/0x50 [10044.102406] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10044.111478] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10044.114924] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [10044.118093] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [10044.121402] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [10044.125030] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [10044.129037] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10044.132720] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [10044.136478] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10044.139085] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10044.142985] Call Trace: [10044.147852] [10044.149353] ? show_regs.cold.9+0x22/0x2f [10044.154027] ? watchdog_timer_fn.cold.6+0x62/0x10a [10044.156693] ? __hrtimer_run_queues+0x1c2/0x4d0 [10044.158951] ? watchdog+0x40/0x40 [10044.160823] ? hrtimer_interrupt+0x128/0x2c0 [10044.162868] ? smp_apic_timer_interrupt+0xa6/0x270 [10044.165264] ? apic_timer_interrupt+0xf/0x20 [10044.167770] [10044.168962] ? native_safe_halt+0x2e/0x50 [10044.171187] kvm_wait+0xcd/0xe0 [10044.173109] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10044.176139] do_raw_spin_lock+0x177/0x190 [10044.177956] _raw_spin_lock+0x21/0x30 [10044.180309] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10044.182727] obd_mod_free+0x46/0x140 [ptlrpc] [10044.185119] mdc_commit_open+0x91/0xb0 [mdc] [10044.186985] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10044.189408] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10044.191561] after_reply+0xe2c/0x1780 [ptlrpc] [10044.193102] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10044.195392] ? __next_timer_interrupt+0x160/0x160 [10044.197694] ptlrpcd+0xa82/0xe40 [ptlrpc] [10044.199808] ? do_wait_intr+0xf0/0xf0 [10044.201567] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10044.204160] kthread+0x1d1/0x200 [10044.206034] ? set_kthread_struct+0x70/0x70 [10044.207766] ret_from_fork+0x1f/0x30 [10052.058923] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [10052.065706] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10052.110304] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10052.125136] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10052.132747] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10052.142639] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10052.164708] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10052.173503] RAX: 0000000000004e8e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10052.183754] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [10052.192705] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [10052.203251] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [10052.209836] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [10052.216347] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [10052.226132] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10052.231687] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [10052.238692] Call Trace: [10052.240623] [10052.247228] ? show_regs.cold.9+0x22/0x2f [10052.258191] ? watchdog_timer_fn.cold.6+0x62/0x10a [10052.262639] ? __hrtimer_run_queues+0x1c2/0x4d0 [10052.274307] ? watchdog+0x40/0x40 [10052.275838] ? hrtimer_interrupt+0x128/0x2c0 [10052.278464] ? smp_apic_timer_interrupt+0xa6/0x270 [10052.290309] ? apic_timer_interrupt+0xf/0x20 [10052.302392] [10052.309278] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10052.318178] do_raw_spin_lock+0x177/0x190 [10052.319567] _raw_spin_lock+0x21/0x30 [10052.330337] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10052.336354] ? do_raw_spin_unlock+0x75/0x190 [10052.340079] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10052.346031] ? do_wait_intr+0xf0/0xf0 [10052.347321] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10052.361435] mdc_close+0x53b/0xd70 [mdc] [10052.364945] lmv_close+0x18d/0x490 [lmv] [10052.376647] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10052.386851] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10052.391594] ll_md_real_close+0xa5/0x2a0 [lustre] [10052.394516] ll_md_close+0x274/0x800 [lustre] [10052.399479] ll_file_release+0x128/0xc50 [lustre] [10052.402946] __fput+0x10d/0x440 [10052.405343] ____fput+0x12/0x20 [10052.407300] task_work_run+0xc6/0x110 [10052.410259] exit_to_usermode_loop+0x1dd/0x1f0 [10052.413765] do_syscall_64+0x3d6/0x440 [10052.417015] entry_SYSCALL_64_after_hwframe+0x49/0xae [10052.420200] RIP: 0033:0x7f151bee3cd8 [10052.423966] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10052.438193] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10052.443023] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [10052.453257] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [10052.463136] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10052.466983] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10052.470473] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10056.052484] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [10056.062829] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10056.117006] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10056.123584] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10056.136832] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10056.144948] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10056.171377] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10056.181225] RAX: 0000000000002670 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10056.193713] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [10056.199375] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [10056.205981] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [10056.212445] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [10056.219801] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [10056.224983] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10056.228202] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [10056.238850] Call Trace: [10056.241167] [10056.242293] ? show_regs.cold.9+0x22/0x2f [10056.244353] ? watchdog_timer_fn.cold.6+0x62/0x10a [10056.256601] ? __hrtimer_run_queues+0x1c2/0x4d0 [10056.262881] ? watchdog+0x40/0x40 [10056.268402] ? hrtimer_interrupt+0x128/0x2c0 [10056.276420] ? smp_apic_timer_interrupt+0xa6/0x270 [10056.280760] ? apic_timer_interrupt+0xf/0x20 [10056.283951] [10056.285692] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10056.292924] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10056.298610] do_raw_spin_lock+0x177/0x190 [10056.300022] _raw_spin_lock+0x21/0x30 [10056.302177] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10056.305594] ? do_raw_spin_unlock+0x75/0x190 [10056.310420] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10056.316940] ? do_wait_intr+0xf0/0xf0 [10056.322683] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10056.327318] mdc_reint+0x2d/0x110 [mdc] [10056.330981] mdc_unlink+0x658/0x870 [mdc] [10056.336888] lmv_unlink+0x34d/0xbd0 [lmv] [10056.342528] ll_unlink+0x24d/0x7f0 [lustre] [10056.344183] vfs_unlink+0x202/0x330 [10056.348807] do_unlinkat+0x2fa/0x460 [10056.352894] __x64_sys_unlink+0x37/0x50 [10056.356581] do_syscall_64+0xc1/0x440 [10056.358266] entry_SYSCALL_64_after_hwframe+0x49/0xae [10056.361730] RIP: 0033:0x7f151bdee10b [10056.366414] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10056.384432] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10056.397336] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [10056.405927] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [10056.421746] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10056.428245] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10056.433560] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10058.721475] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8128s! [10058.729595] Showing busy workqueues and worker pools: [10058.733624] workqueue events: flags=0x0 [10058.735824] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10058.744428] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [10058.752514] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10058.755416] pending: kfree_rcu_monitor [10058.757384] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10058.760376] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [10058.764404] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10058.769804] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [10058.774428] workqueue events_long: flags=0x0 [10058.782795] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10058.790143] pending: lu_objects_flush [obdclass] [10058.793943] workqueue events_unbound: flags=0x2 [10058.797775] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10058.805749] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10058.817963] workqueue events_power_efficient: flags=0x80 [10058.822534] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10058.826753] pending: do_cache_clean, neigh_periodic_work, check_lifetime [10058.834191] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10058.840836] pending: neigh_periodic_work [10058.843661] workqueue mm_percpu_wq: flags=0x8 [10058.849658] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10058.856639] pending: vmstat_update, lru_add_drain_per_cpu [10058.863362] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10058.867495] pending: vmstat_update [10058.871758] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10058.879319] pending: lru_add_drain_per_cpu [10058.883095] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10058.887726] pending: lru_add_drain_per_cpu BAR(48) [10058.892683] workqueue writeback: flags=0x4a [10058.895940] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [10058.898828] pending: wb_workfn [10058.901450] workqueue xprtiod: flags=0xa [10058.904871] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10058.907543] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [10058.912903] workqueue ipv6_addrconf: flags=0x40008 [10058.914913] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10058.928859] pending: addrconf_verify_work [10058.932568] workqueue knbd0-recv: flags=0x1a [10058.934904] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10058.937921] in-flight: 658:recv_work [10058.939491] workqueue ptlrpc_pinger: flags=0x2 [10058.941835] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10058.944381] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10058.947571] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8128s workers=4 idle: 32984 37 8 [10058.951039] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10043s workers=2 idle: 661 [10060.056114] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [10060.063644] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10060.098124] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10060.109738] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10060.119115] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10060.125058] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10060.127188] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10060.144357] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10060.147218] RAX: 0000000000003188 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10060.153348] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [10060.159310] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [10060.165620] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [10060.171563] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [10060.177217] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [10060.182234] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10060.186910] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10060.190141] Call Trace: [10060.191186] [10060.192052] ? show_regs.cold.9+0x22/0x2f [10060.197398] ? watchdog_timer_fn.cold.6+0x62/0x10a [10060.199581] ? __hrtimer_run_queues+0x1c2/0x4d0 [10060.201399] ? watchdog+0x40/0x40 [10060.205818] ? hrtimer_interrupt+0x128/0x2c0 [10060.208164] ? smp_apic_timer_interrupt+0xa6/0x270 [10060.211207] ? apic_timer_interrupt+0xf/0x20 [10060.213289] [10060.215046] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10060.218310] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10060.220743] do_raw_spin_lock+0x177/0x190 [10060.223057] _raw_spin_lock+0x21/0x30 [10060.224588] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10060.230203] process_one_work+0x2c8/0x700 [10060.232422] worker_thread+0x296/0x6e0 [10060.236106] ? rescuer_thread+0x570/0x570 [10060.240329] kthread+0x1d1/0x200 [10060.244612] ? set_kthread_struct+0x70/0x70 [10060.246929] ret_from_fork+0x1f/0x30 [10072.058781] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [10072.071252] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10072.135535] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10072.146514] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10072.152300] RIP: 0010:native_safe_halt+0x2e/0x50 [10072.158464] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10072.170608] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10072.174215] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [10072.177778] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [10072.183871] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [10072.191751] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [10072.198141] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10072.206488] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [10072.212270] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10072.216111] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10072.221731] Call Trace: [10072.226446] [10072.228679] ? show_regs.cold.9+0x22/0x2f [10072.238308] ? watchdog_timer_fn.cold.6+0x62/0x10a [10072.241659] ? __hrtimer_run_queues+0x1c2/0x4d0 [10072.244594] ? watchdog+0x40/0x40 [10072.246668] ? hrtimer_interrupt+0x128/0x2c0 [10072.252820] ? smp_apic_timer_interrupt+0xa6/0x270 [10072.255798] ? apic_timer_interrupt+0xf/0x20 [10072.259204] [10072.260536] ? native_safe_halt+0x2e/0x50 [10072.264076] kvm_wait+0xcd/0xe0 [10072.266083] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10072.270124] do_raw_spin_lock+0x177/0x190 [10072.272988] _raw_spin_lock+0x21/0x30 [10072.275326] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10072.278507] obd_mod_free+0x46/0x140 [ptlrpc] [10072.281303] mdc_commit_open+0x91/0xb0 [mdc] [10072.284636] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10072.287761] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10072.292314] after_reply+0xe2c/0x1780 [ptlrpc] [10072.297081] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10072.300650] ? __next_timer_interrupt+0x160/0x160 [10072.303503] ptlrpcd+0xa82/0xe40 [ptlrpc] [10072.306348] ? do_wait_intr+0xf0/0xf0 [10072.308238] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10072.310752] kthread+0x1d1/0x200 [10072.312406] ? set_kthread_struct+0x70/0x70 [10072.314974] ret_from_fork+0x1f/0x30 [10080.062392] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [10080.073005] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10080.114490] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10080.126642] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10080.133852] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10080.138939] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10080.158389] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10080.167614] RAX: 0000000000005f42 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10080.174247] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [10080.180719] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [10080.184218] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [10080.190188] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [10080.195635] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [10080.202539] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10080.204599] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [10080.211648] Call Trace: [10080.212655] [10080.216165] ? show_regs.cold.9+0x22/0x2f [10080.220052] ? watchdog_timer_fn.cold.6+0x62/0x10a [10080.225127] ? __hrtimer_run_queues+0x1c2/0x4d0 [10080.228638] ? watchdog+0x40/0x40 [10080.233574] ? hrtimer_interrupt+0x128/0x2c0 [10080.236505] ? smp_apic_timer_interrupt+0xa6/0x270 [10080.242207] ? apic_timer_interrupt+0xf/0x20 [10080.245235] [10080.246777] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10080.251128] do_raw_spin_lock+0x177/0x190 [10080.255522] _raw_spin_lock+0x21/0x30 [10080.257512] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10080.265670] ? do_raw_spin_unlock+0x75/0x190 [10080.266992] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10080.273155] ? do_wait_intr+0xf0/0xf0 [10080.275559] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10080.282650] mdc_close+0x53b/0xd70 [mdc] [10080.287178] lmv_close+0x18d/0x490 [lmv] [10080.291829] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10080.299917] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10080.306449] ll_md_real_close+0xa5/0x2a0 [lustre] [10080.311403] ll_md_close+0x274/0x800 [lustre] [10080.314858] ll_file_release+0x128/0xc50 [lustre] [10080.325761] __fput+0x10d/0x440 [10080.328874] ____fput+0x12/0x20 [10080.332170] task_work_run+0xc6/0x110 [10080.336583] exit_to_usermode_loop+0x1dd/0x1f0 [10080.341131] do_syscall_64+0x3d6/0x440 [10080.342855] entry_SYSCALL_64_after_hwframe+0x49/0xae [10080.346158] RIP: 0033:0x7f151bee3cd8 [10080.349129] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10080.369102] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10080.375466] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [10080.384628] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [10080.391496] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10080.398061] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10080.401724] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10084.052147] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [10084.055407] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10084.077613] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10084.083625] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10084.087850] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10084.092628] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10084.104851] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10084.108887] RAX: 0000000000006488 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10084.113922] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [10084.118606] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [10084.122132] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [10084.126028] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [10084.130684] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [10084.135023] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10084.139836] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [10084.145435] Call Trace: [10084.146654] [10084.147881] ? show_regs.cold.9+0x22/0x2f [10084.150673] ? watchdog_timer_fn.cold.6+0x62/0x10a [10084.153215] ? __hrtimer_run_queues+0x1c2/0x4d0 [10084.155646] ? watchdog+0x40/0x40 [10084.158204] ? hrtimer_interrupt+0x128/0x2c0 [10084.160644] ? smp_apic_timer_interrupt+0xa6/0x270 [10084.163712] ? apic_timer_interrupt+0xf/0x20 [10084.165883] [10084.167126] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10084.169972] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10084.173071] do_raw_spin_lock+0x177/0x190 [10084.175635] _raw_spin_lock+0x21/0x30 [10084.177211] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10084.180180] ? do_raw_spin_unlock+0x75/0x190 [10084.182705] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10084.185889] ? do_wait_intr+0xf0/0xf0 [10084.187644] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10084.190798] mdc_reint+0x2d/0x110 [mdc] [10084.193508] mdc_unlink+0x658/0x870 [mdc] [10084.195695] lmv_unlink+0x34d/0xbd0 [lmv] [10084.198775] ll_unlink+0x24d/0x7f0 [lustre] [10084.201506] vfs_unlink+0x202/0x330 [10084.204164] do_unlinkat+0x2fa/0x460 [10084.206551] __x64_sys_unlink+0x37/0x50 [10084.209502] do_syscall_64+0xc1/0x440 [10084.211744] entry_SYSCALL_64_after_hwframe+0x49/0xae [10084.215490] RIP: 0033:0x7f151bdee10b [10084.217686] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10084.229484] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10084.233919] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [10084.237885] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [10084.243572] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10084.247991] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10084.252046] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10084.819475] rcu: INFO: rcu_sched self-detected stall on CPU [10084.827179] rcu: 1-...!: (4643131 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [10084.842646] (t=8160172 jiffies g=249657 q=4795) [10084.847829] rcu: rcu_sched kthread starved for 8155124 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [10084.864931] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [10084.874191] rcu: RCU grace-period kthread stack dump: [10084.878727] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [10084.883158] Call Trace: [10084.884385] __schedule+0x351/0xcb0 [10084.886664] schedule+0xc0/0x180 [10084.888982] schedule_timeout+0xb4/0x190 [10084.891588] ? do_raw_spin_unlock+0x75/0x190 [10084.894061] ? __next_timer_interrupt+0x160/0x160 [10084.907019] ? prepare_to_swait_event+0xf3/0x140 [10084.913253] rcu_gp_kthread+0x8dd/0x15b0 [10084.915430] ? force_qs_rnp+0x2c0/0x2c0 [10084.922560] kthread+0x1d1/0x200 [10084.924253] ? set_kthread_struct+0x70/0x70 [10084.929090] ret_from_fork+0x1f/0x30 [10084.931111] rcu: Stack dump where RCU GP kthread last ran: [10084.938474] Sending NMI from CPU 1 to CPUs 3: [10084.952470] NMI backtrace for cpu 3 [10084.952710] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10084.952713] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10084.952714] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10084.952716] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10084.952718] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 [10084.952720] RAX: 000000000000037f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10084.952722] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [10084.952723] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [10084.952724] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [10084.952725] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [10084.952726] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [10084.952727] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10084.952728] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [10084.952729] Call Trace: [10084.952729] [10084.952730] ? show_regs.cold.9+0x22/0x2f [10084.952731] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [10084.952732] ? nmi_cpu_backtrace_handler+0x15/0x40 [10084.952733] ? nmi_handle+0xbd/0x210 [10084.952733] ? default_do_nmi+0x4b/0x1b0 [10084.952734] ? do_nmi+0x280/0x380 [10084.952735] ? end_repeat_nmi+0x16/0x1f [10084.952736] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10084.952737] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10084.952737] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10084.952738] [10084.952739] do_raw_spin_lock+0x177/0x190 [10084.952740] _raw_spin_lock+0x21/0x30 [10084.952741] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10084.952742] ? do_raw_spin_unlock+0x75/0x190 [10084.952743] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10084.952744] ? do_wait_intr+0xf0/0xf0 [10084.952745] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10084.952746] mdc_close+0x53b/0xd70 [mdc] [10084.952747] lmv_close+0x18d/0x490 [lmv] [10084.952748] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10084.952749] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10084.952749] ll_md_real_close+0xa5/0x2a0 [lustre] [10084.952750] ll_md_close+0x274/0x800 [lustre] [10084.952751] ll_file_release+0x128/0xc50 [lustre] [10084.952752] __fput+0x10d/0x440 [10084.952753] ____fput+0x12/0x20 [10084.952753] task_work_run+0xc6/0x110 [10084.952754] exit_to_usermode_loop+0x1dd/0x1f0 [10084.952755] do_syscall_64+0x3d6/0x440 [10084.952756] entry_SYSCALL_64_after_hwframe+0x49/0xae [10084.952757] RIP: 0033:0x7f151bee3cd8 [10084.952758] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10084.952760] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10084.952762] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [10084.952763] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [10084.952764] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10084.952765] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10084.952767] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10084.952979] NMI backtrace for cpu 1 [10085.406662] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10085.418054] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10085.429489] Call Trace: [10085.430227] [10085.431655] ? dump_stack+0xbb/0x10e [10085.436616] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [10085.438808] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [10085.446270] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [10085.451936] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [10085.455503] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [10085.462845] ? rcu_dump_cpu_stacks+0x11b/0x171 [10085.467055] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [10085.470909] ? update_process_times+0x6d/0xc0 [10085.477306] ? tick_sched_handle.isra.18+0x27/0x90 [10085.482996] ? tick_sched_timer+0x53/0xb0 [10085.485206] ? __hrtimer_run_queues+0x1c2/0x4d0 [10085.489746] ? tick_sched_handle.isra.18+0x90/0x90 [10085.495593] ? hrtimer_interrupt+0x128/0x2c0 [10085.501475] ? smp_apic_timer_interrupt+0xa6/0x270 [10085.507273] ? apic_timer_interrupt+0xf/0x20 [10085.510600] [10085.512694] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10085.518416] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10085.524757] ? do_raw_spin_lock+0x177/0x190 [10085.527196] ? _raw_spin_lock+0x21/0x30 [10085.530699] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10085.535370] ? obd_mod_free+0x46/0x140 [ptlrpc] [10085.541726] ? mdc_commit_open+0x91/0xb0 [mdc] [10085.550380] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10085.560825] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10085.568600] ? after_reply+0xe2c/0x1780 [ptlrpc] [10085.572623] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10085.577341] ? __next_timer_interrupt+0x160/0x160 [10085.580672] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [10085.583681] ? do_wait_intr+0xf0/0xf0 [10085.586518] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10085.589673] ? kthread+0x1d1/0x200 [10085.591171] ? set_kthread_struct+0x70/0x70 [10085.594189] ? ret_from_fork+0x1f/0x30 [10088.058107] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [10088.066405] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10088.095611] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10088.105244] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10088.109692] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10088.117028] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10088.124669] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10088.143672] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10088.153250] RAX: 00000000000034cf RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10088.163288] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [10088.167089] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [10088.170527] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [10088.177141] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [10088.184914] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [10088.191850] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10088.194951] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10088.202782] Call Trace: [10088.207899] [10088.209209] ? show_regs.cold.9+0x22/0x2f [10088.215723] ? watchdog_timer_fn.cold.6+0x62/0x10a [10088.220729] ? __hrtimer_run_queues+0x1c2/0x4d0 [10088.226732] ? watchdog+0x40/0x40 [10088.228450] ? hrtimer_interrupt+0x128/0x2c0 [10088.233887] ? smp_apic_timer_interrupt+0xa6/0x270 [10088.238394] ? apic_timer_interrupt+0xf/0x20 [10088.240158] [10088.240725] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10088.243065] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10088.245116] do_raw_spin_lock+0x177/0x190 [10088.247056] _raw_spin_lock+0x21/0x30 [10088.248732] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10088.251413] process_one_work+0x2c8/0x700 [10088.253575] worker_thread+0x296/0x6e0 [10088.255029] ? rescuer_thread+0x570/0x570 [10088.256807] kthread+0x1d1/0x200 [10088.258467] ? set_kthread_struct+0x70/0x70 [10088.260458] ret_from_fork+0x1f/0x30 [10089.440099] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8158s! [10089.442711] Showing busy workqueues and worker pools: [10089.445007] workqueue events: flags=0x0 [10089.447802] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10089.453527] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [10089.461084] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10089.468943] pending: kfree_rcu_monitor [10089.472102] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10089.478664] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [10089.484345] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10089.489576] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [10089.495351] workqueue events_long: flags=0x0 [10089.496786] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10089.500393] pending: lu_objects_flush [obdclass] [10089.504590] workqueue events_unbound: flags=0x2 [10089.507144] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10089.512513] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10089.521087] workqueue events_power_efficient: flags=0x80 [10089.524868] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10089.532367] pending: do_cache_clean, neigh_periodic_work, check_lifetime [10089.544966] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10089.556551] pending: neigh_periodic_work [10089.563968] workqueue mm_percpu_wq: flags=0x8 [10089.568015] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10089.581852] pending: vmstat_update, lru_add_drain_per_cpu [10089.595320] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10089.605168] pending: vmstat_update [10089.610481] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10089.625027] pending: lru_add_drain_per_cpu [10089.635261] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10089.642803] pending: lru_add_drain_per_cpu BAR(48) [10089.647837] workqueue writeback: flags=0x4a [10089.657152] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [10089.664877] pending: wb_workfn [10089.667850] workqueue xprtiod: flags=0xa [10089.669932] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10089.673961] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [10089.682317] workqueue ipv6_addrconf: flags=0x40008 [10089.685070] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10089.689896] pending: addrconf_verify_work [10089.691952] workqueue knbd0-recv: flags=0x1a [10089.696895] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10089.702716] in-flight: 658:recv_work [10089.705113] workqueue ptlrpc_pinger: flags=0x2 [10089.716626] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10089.725509] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10089.736017] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8159s workers=4 idle: 32984 37 8 [10089.747935] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10074s workers=2 idle: 661 [10112.055475] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [10112.060451] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [10112.066721] Modules linked in: lustre(O) osp(O) [10112.072911] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10112.073451] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10112.073459] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10112.073462] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10112.073478] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10112.073481] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10112.073486] RAX: 0000000000000897 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10112.073488] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [10112.073490] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [10112.073492] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [10112.073495] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [10112.073498] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [10112.073500] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10112.073503] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [10112.073511] Call Trace: [10112.073516] [10112.073522] ? show_regs.cold.9+0x22/0x2f [10112.073534] ? watchdog_timer_fn.cold.6+0x62/0x10a [10112.073540] ? __hrtimer_run_queues+0x1c2/0x4d0 [10112.073544] ? watchdog+0x40/0x40 [10112.073548] ? hrtimer_interrupt+0x128/0x2c0 [10112.073552] ? smp_apic_timer_interrupt+0xa6/0x270 [10112.073560] ? apic_timer_interrupt+0xf/0x20 [10112.073564] [10112.073568] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10112.073575] do_raw_spin_lock+0x177/0x190 [10112.073580] _raw_spin_lock+0x21/0x30 [10112.073584] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10112.073874] ? do_raw_spin_unlock+0x75/0x190 [10112.073880] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10112.074517] ? do_wait_intr+0xf0/0xf0 [10112.074526] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10112.074734] mdc_close+0x53b/0xd70 [mdc] [10112.074771] lmv_close+0x18d/0x490 [lmv] [10112.074803] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10112.075241] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10112.075420] ll_md_real_close+0xa5/0x2a0 [lustre] [10112.080521] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10112.080596] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10112.080603] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10112.080606] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10112.080619] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10112.080623] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10112.080628] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [10112.080630] RDX: 0000000000006ca9 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [10112.080633] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [10112.080635] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [10112.080637] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10112.080640] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [10112.080643] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10112.080645] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10112.080655] Call Trace: [10112.080660] [10112.080665] ? show_regs.cold.9+0x22/0x2f [10112.080675] ? watchdog_timer_fn.cold.6+0x62/0x10a [10112.080682] ? __hrtimer_run_queues+0x1c2/0x4d0 [10112.080686] ? watchdog+0x40/0x40 [10112.080691] ? hrtimer_interrupt+0x128/0x2c0 [10112.080695] ? smp_apic_timer_interrupt+0xa6/0x270 [10112.080704] ? apic_timer_interrupt+0xf/0x20 [10112.080709] [10112.080712] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10112.080718] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10112.080722] do_raw_spin_lock+0x177/0x190 [10112.080727] _raw_spin_lock+0x21/0x30 [10112.080733] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10112.081007] obd_mod_free+0x46/0x140 [ptlrpc] [10112.122799] ll_md_close+0x274/0x800 [lustre] [10112.123116] ll_file_release+0x128/0xc50 [lustre] [10112.123203] __fput+0x10d/0x440 [10112.123214] ____fput+0x12/0x20 [10112.123219] task_work_run+0xc6/0x110 [10112.123227] exit_to_usermode_loop+0x1dd/0x1f0 [10112.123234] do_syscall_64+0x3d6/0x440 [10112.123239] entry_SYSCALL_64_after_hwframe+0x49/0xae [10112.123247] RIP: 0033:0x7f151bee3cd8 [10112.123254] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10112.123258] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10112.123263] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [10112.123266] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [10112.123268] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10112.123270] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10112.123272] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10112.492836] mdc_commit_open+0x91/0xb0 [mdc] [10112.495386] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10112.497780] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10112.500137] after_reply+0xe2c/0x1780 [ptlrpc] [10112.501573] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10112.506104] ? __next_timer_interrupt+0x160/0x160 [10112.508795] ptlrpcd+0xa82/0xe40 [ptlrpc] [10112.511744] ? do_wait_intr+0xf0/0xf0 [10112.517159] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10112.520821] kthread+0x1d1/0x200 [10112.522485] ? set_kthread_struct+0x70/0x70 [10112.524751] ret_from_fork+0x1f/0x30 [10116.052021] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [10116.055693] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10116.057235] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [10116.076126] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10116.076141] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10116.076144] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10116.079572] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10116.079648] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10116.079653] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10116.079714] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10116.080065] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10116.080075] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10116.080080] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10116.080084] RAX: 0000000000003dee RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10116.080086] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [10116.080089] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [10116.080094] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [10116.080097] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [10116.080100] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [10116.080103] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10116.080106] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10116.080114] Call Trace: [10116.080118] [10116.080123] ? show_regs.cold.9+0x22/0x2f [10116.080132] ? watchdog_timer_fn.cold.6+0x62/0x10a [10116.080139] ? __hrtimer_run_queues+0x1c2/0x4d0 [10116.080143] ? watchdog+0x40/0x40 [10116.080147] ? hrtimer_interrupt+0x128/0x2c0 [10116.080150] ? smp_apic_timer_interrupt+0xa6/0x270 [10116.080158] ? apic_timer_interrupt+0xf/0x20 [10116.080161] [10116.080164] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10116.080168] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10116.080172] do_raw_spin_lock+0x177/0x190 [10116.080176] _raw_spin_lock+0x21/0x30 [10116.080181] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10116.080433] process_one_work+0x2c8/0x700 [10116.080440] worker_thread+0x296/0x6e0 [10116.080443] ? rescuer_thread+0x570/0x570 [10116.080446] kthread+0x1d1/0x200 [10116.080450] ? set_kthread_struct+0x70/0x70 [10116.080453] ret_from_fork+0x1f/0x30 [10116.201226] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10116.207690] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10116.211414] RAX: 0000000000003987 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10116.214538] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [10116.218674] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [10116.222667] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [10116.225870] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [10116.230116] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [10116.233608] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10116.236854] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [10116.240191] Call Trace: [10116.241276] [10116.242462] ? show_regs.cold.9+0x22/0x2f [10116.244265] ? watchdog_timer_fn.cold.6+0x62/0x10a [10116.246541] ? __hrtimer_run_queues+0x1c2/0x4d0 [10116.248276] ? watchdog+0x40/0x40 [10116.249911] ? hrtimer_interrupt+0x128/0x2c0 [10116.252422] ? smp_apic_timer_interrupt+0xa6/0x270 [10116.254266] ? apic_timer_interrupt+0xf/0x20 [10116.256068] [10116.256830] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10116.259487] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10116.261975] do_raw_spin_lock+0x177/0x190 [10116.264430] _raw_spin_lock+0x21/0x30 [10116.266179] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10116.268629] ? do_raw_spin_unlock+0x75/0x190 [10116.271181] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10116.273812] ? do_wait_intr+0xf0/0xf0 [10116.275833] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10116.278739] mdc_reint+0x2d/0x110 [mdc] [10116.280208] mdc_unlink+0x658/0x870 [mdc] [10116.281744] lmv_unlink+0x34d/0xbd0 [lmv] [10116.283788] ll_unlink+0x24d/0x7f0 [lustre] [10116.285627] vfs_unlink+0x202/0x330 [10116.287021] do_unlinkat+0x2fa/0x460 [10116.288404] __x64_sys_unlink+0x37/0x50 [10116.289933] do_syscall_64+0xc1/0x440 [10116.291493] entry_SYSCALL_64_after_hwframe+0x49/0xae [10116.293624] RIP: 0033:0x7f151bdee10b [10116.295314] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10116.303769] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10116.307684] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [10116.310530] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [10116.313311] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10116.316187] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10116.318758] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10120.160847] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8189s! [10120.170511] Showing busy workqueues and worker pools: [10120.172213] workqueue events: flags=0x0 [10120.174971] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10120.178472] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [10120.184415] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10120.187721] pending: kfree_rcu_monitor [10120.189822] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10120.193314] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [10120.198028] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10120.200163] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [10120.203718] workqueue events_long: flags=0x0 [10120.205679] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10120.208604] pending: lu_objects_flush [obdclass] [10120.211203] workqueue events_unbound: flags=0x2 [10120.213118] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10120.215808] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10120.220536] workqueue events_power_efficient: flags=0x80 [10120.222780] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10120.225813] pending: do_cache_clean, neigh_periodic_work, check_lifetime [10120.228993] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10120.232198] pending: neigh_periodic_work [10120.234090] workqueue mm_percpu_wq: flags=0x8 [10120.237817] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10120.242359] pending: vmstat_update, lru_add_drain_per_cpu [10120.243808] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10120.253523] pending: vmstat_update [10120.256609] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10120.265572] pending: lru_add_drain_per_cpu [10120.271188] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10120.284187] pending: lru_add_drain_per_cpu BAR(48) [10120.298252] workqueue writeback: flags=0x4a [10120.299533] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [10120.304697] pending: wb_workfn [10120.307572] workqueue xprtiod: flags=0xa [10120.315375] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10120.321199] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [10120.326773] workqueue ipv6_addrconf: flags=0x40008 [10120.331854] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10120.336619] pending: addrconf_verify_work [10120.338230] workqueue knbd0-recv: flags=0x1a [10120.346760] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10120.353096] in-flight: 658:recv_work [10120.355556] workqueue ptlrpc_pinger: flags=0x2 [10120.357522] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10120.360206] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10120.362356] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8189s workers=4 idle: 32984 37 8 [10120.372379] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10105s workers=2 idle: 661 [10140.058237] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [10140.061791] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [10140.067509] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10140.067591] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10140.067599] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10140.067602] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10140.067615] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10140.067619] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10140.067624] RAX: 0000000000007083 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10140.067626] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [10140.067628] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [10140.067630] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [10140.067633] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [10140.067636] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [10140.067639] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10140.067642] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [10140.067650] Call Trace: [10140.067656] [10140.067661] ? show_regs.cold.9+0x22/0x2f [10140.067670] ? watchdog_timer_fn.cold.6+0x62/0x10a [10140.067680] ? __hrtimer_run_queues+0x1c2/0x4d0 [10140.067684] ? watchdog+0x40/0x40 [10140.067688] ? hrtimer_interrupt+0x128/0x2c0 [10140.067692] ? smp_apic_timer_interrupt+0xa6/0x270 [10140.067700] ? apic_timer_interrupt+0xf/0x20 [10140.067704] [10140.067707] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10140.067713] do_raw_spin_lock+0x177/0x190 [10140.067718] _raw_spin_lock+0x21/0x30 [10140.067722] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10140.068890] ? do_raw_spin_unlock+0x75/0x190 [10140.068898] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10140.069093] ? do_wait_intr+0xf0/0xf0 [10140.069101] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10140.082207] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10140.082286] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10140.082293] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10140.082296] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10140.082309] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10140.082313] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10140.082318] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [10140.082321] RDX: 000000000000556f RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [10140.082323] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [10140.082325] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [10140.082327] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10140.082330] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [10140.082333] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10140.082336] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10140.082345] Call Trace: [10140.082350] [10140.082355] ? show_regs.cold.9+0x22/0x2f [10140.082364] ? watchdog_timer_fn.cold.6+0x62/0x10a [10140.082370] ? __hrtimer_run_queues+0x1c2/0x4d0 [10140.082374] ? watchdog+0x40/0x40 [10140.082379] ? hrtimer_interrupt+0x128/0x2c0 [10140.082383] ? smp_apic_timer_interrupt+0xa6/0x270 [10140.082391] ? apic_timer_interrupt+0xf/0x20 [10140.082396] [10140.082399] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10140.082404] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10140.082408] do_raw_spin_lock+0x177/0x190 [10140.082413] _raw_spin_lock+0x21/0x30 [10140.082419] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10140.082683] obd_mod_free+0x46/0x140 [ptlrpc] [10140.082897] mdc_commit_open+0x91/0xb0 [mdc] [10140.082931] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10140.083140] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10140.083355] after_reply+0xe2c/0x1780 [ptlrpc] [10140.083571] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10140.083773] ? __next_timer_interrupt+0x160/0x160 [10140.083782] ptlrpcd+0xa82/0xe40 [ptlrpc] [10140.084016] ? do_wait_intr+0xf0/0xf0 [10140.084026] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10140.084233] kthread+0x1d1/0x200 [10140.084239] ? set_kthread_struct+0x70/0x70 [10140.084243] ret_from_fork+0x1f/0x30 [10140.486299] mdc_close+0x53b/0xd70 [mdc] [10140.487897] lmv_close+0x18d/0x490 [lmv] [10140.489603] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10140.492468] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10140.495239] ll_md_real_close+0xa5/0x2a0 [lustre] [10140.497379] ll_md_close+0x274/0x800 [lustre] [10140.499626] ll_file_release+0x128/0xc50 [lustre] [10140.501863] __fput+0x10d/0x440 [10140.503264] ____fput+0x12/0x20 [10140.504691] task_work_run+0xc6/0x110 [10140.506472] exit_to_usermode_loop+0x1dd/0x1f0 [10140.508827] do_syscall_64+0x3d6/0x440 [10140.510697] entry_SYSCALL_64_after_hwframe+0x49/0xae [10140.513648] RIP: 0033:0x7f151bee3cd8 [10140.515926] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10140.526093] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10140.530404] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [10140.533996] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [10140.537397] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10140.542789] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10140.546924] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10144.056025] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10144.060582] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10144.116420] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10144.130505] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10144.137691] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10144.141920] RIP: 0010:native_safe_halt+0x2e/0x50 [10144.148519] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10144.182518] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10144.187345] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10144.190618] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [10144.196192] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [10144.204118] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [10144.212918] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [10144.216469] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [10144.219959] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10144.224153] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10144.228181] Call Trace: [10144.229540] [10144.230300] ? show_regs.cold.9+0x22/0x2f [10144.231683] ? watchdog_timer_fn.cold.6+0x62/0x10a [10144.234165] ? __hrtimer_run_queues+0x1c2/0x4d0 [10144.236373] ? watchdog+0x40/0x40 [10144.238400] ? hrtimer_interrupt+0x128/0x2c0 [10144.240451] ? smp_apic_timer_interrupt+0xa6/0x270 [10144.242921] ? apic_timer_interrupt+0xf/0x20 [10144.245923] [10144.246708] ? native_safe_halt+0x2e/0x50 [10144.252041] kvm_wait+0xcd/0xe0 [10144.255951] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10144.262879] do_raw_spin_lock+0x177/0x190 [10144.264198] _raw_spin_lock+0x21/0x30 [10144.280134] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10144.295050] process_one_work+0x2c8/0x700 [10144.300537] worker_thread+0x296/0x6e0 [10144.306522] ? rescuer_thread+0x570/0x570 [10144.309558] kthread+0x1d1/0x200 [10144.311484] ? set_kthread_struct+0x70/0x70 [10144.317056] ret_from_fork+0x1f/0x30 [10148.052457] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [10148.058151] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10148.096055] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10148.103872] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10148.107959] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10148.110453] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10148.121858] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10148.126187] RAX: 0000000000004d0a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10148.130336] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [10148.134807] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [10148.142528] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [10148.151316] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [10148.168133] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [10148.182420] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10148.189380] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [10148.195151] Call Trace: [10148.201481] [10148.202271] ? show_regs.cold.9+0x22/0x2f [10148.203702] ? watchdog_timer_fn.cold.6+0x62/0x10a [10148.208250] ? __hrtimer_run_queues+0x1c2/0x4d0 [10148.212153] ? watchdog+0x40/0x40 [10148.215278] ? hrtimer_interrupt+0x128/0x2c0 [10148.220153] ? smp_apic_timer_interrupt+0xa6/0x270 [10148.222671] ? apic_timer_interrupt+0xf/0x20 [10148.229479] [10148.234582] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10148.240954] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10148.251087] do_raw_spin_lock+0x177/0x190 [10148.252303] _raw_spin_lock+0x21/0x30 [10148.259193] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10148.265677] ? do_raw_spin_unlock+0x75/0x190 [10148.267200] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10148.271410] ? do_wait_intr+0xf0/0xf0 [10148.274576] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10148.278461] mdc_reint+0x2d/0x110 [mdc] [10148.283383] mdc_unlink+0x658/0x870 [mdc] [10148.286423] lmv_unlink+0x34d/0xbd0 [lmv] [10148.291153] ll_unlink+0x24d/0x7f0 [lustre] [10148.294204] vfs_unlink+0x202/0x330 [10148.295981] do_unlinkat+0x2fa/0x460 [10148.300286] __x64_sys_unlink+0x37/0x50 [10148.309320] do_syscall_64+0xc1/0x440 [10148.312379] entry_SYSCALL_64_after_hwframe+0x49/0xae [10148.314593] RIP: 0033:0x7f151bdee10b [10148.322196] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10148.351719] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10148.357930] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [10148.362578] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [10148.370308] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10148.377551] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10148.381616] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10150.886611] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8220s! [10150.902048] Showing busy workqueues and worker pools: [10150.905070] workqueue events: flags=0x0 [10150.907639] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10150.911758] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [10150.922010] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10150.928511] pending: kfree_rcu_monitor [10150.930619] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10150.938367] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [10150.947778] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10150.952283] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [10150.968972] workqueue events_long: flags=0x0 [10150.971218] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10150.977936] pending: lu_objects_flush [obdclass] [10150.986057] workqueue events_unbound: flags=0x2 [10150.989287] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10150.996443] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10151.006836] workqueue events_power_efficient: flags=0x80 [10151.017157] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10151.026496] pending: do_cache_clean, neigh_periodic_work, check_lifetime [10151.037797] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10151.045467] pending: neigh_periodic_work [10151.053476] workqueue mm_percpu_wq: flags=0x8 [10151.056498] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10151.062874] pending: vmstat_update, lru_add_drain_per_cpu [10151.080372] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10151.091635] pending: vmstat_update [10151.097137] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10151.112826] pending: lru_add_drain_per_cpu [10151.119302] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10151.123766] pending: lru_add_drain_per_cpu BAR(48) [10151.246115] workqueue writeback: flags=0x4a [10151.272543] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [10151.284747] pending: wb_workfn [10151.312757] workqueue xprtiod: flags=0xa [10151.323493] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10151.358808] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [10151.372498] workqueue ipv6_addrconf: flags=0x40008 [10151.378748] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10151.387551] pending: addrconf_verify_work [10151.393537] workqueue knbd0-recv: flags=0x1a [10151.398760] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10151.406971] in-flight: 658:recv_work [10151.410439] workqueue ptlrpc_pinger: flags=0x2 [10151.415307] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10151.420503] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10151.428071] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8220s workers=4 idle: 32984 37 8 [10151.436528] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10136s workers=2 idle: 661 [10168.056535] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [10168.063067] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10168.065000] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [10168.116583] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10168.116597] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10168.116600] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10168.116612] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10168.116616] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10168.116623] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [10168.116625] RDX: 0000000000001f89 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [10168.116628] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [10168.116630] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [10168.116632] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10168.116636] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [10168.116639] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10168.116641] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10168.116650] Call Trace: [10168.126830] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10168.127114] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10168.127123] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10168.127126] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10168.127141] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10168.127145] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10168.127150] RAX: 0000000000007b94 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10168.127152] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [10168.127155] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [10168.127157] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [10168.127160] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [10168.127163] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [10168.127166] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10168.127169] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [10168.127178] Call Trace: [10168.127183] [10168.127188] ? show_regs.cold.9+0x22/0x2f [10168.127215] ? watchdog_timer_fn.cold.6+0x62/0x10a [10168.127222] ? __hrtimer_run_queues+0x1c2/0x4d0 [10168.127227] ? watchdog+0x40/0x40 [10168.127232] ? hrtimer_interrupt+0x128/0x2c0 [10168.127237] ? smp_apic_timer_interrupt+0xa6/0x270 [10168.127245] ? apic_timer_interrupt+0xf/0x20 [10168.127251] [10168.127254] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10168.127262] do_raw_spin_lock+0x177/0x190 [10168.127268] _raw_spin_lock+0x21/0x30 [10168.127272] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10168.128019] ? do_raw_spin_unlock+0x75/0x190 [10168.128027] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10168.128244] ? do_wait_intr+0xf0/0xf0 [10168.128253] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10168.128480] mdc_close+0x53b/0xd70 [mdc] [10168.141684] [10168.141698] ? show_regs.cold.9+0x22/0x2f [10168.141714] ? watchdog_timer_fn.cold.6+0x62/0x10a [10168.141721] ? __hrtimer_run_queues+0x1c2/0x4d0 [10168.141726] ? watchdog+0x40/0x40 [10168.141730] ? hrtimer_interrupt+0x128/0x2c0 [10168.141735] ? smp_apic_timer_interrupt+0xa6/0x270 [10168.141743] ? apic_timer_interrupt+0xf/0x20 [10168.141749] [10168.141752] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10168.141760] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10168.141765] do_raw_spin_lock+0x177/0x190 [10168.141770] _raw_spin_lock+0x21/0x30 [10168.141778] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10168.142592] obd_mod_free+0x46/0x140 [ptlrpc] [10168.143341] mdc_commit_open+0x91/0xb0 [mdc] [10168.143381] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10168.156778] lmv_close+0x18d/0x490 [lmv] [10168.160205] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10168.160478] after_reply+0xe2c/0x1780 [ptlrpc] [10168.161311] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10168.161594] ? __next_timer_interrupt+0x160/0x160 [10168.161605] ptlrpcd+0xa82/0xe40 [ptlrpc] [10168.162947] ? do_wait_intr+0xf0/0xf0 [10168.162958] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10168.184218] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10168.185079] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10168.185215] ll_md_real_close+0xa5/0x2a0 [lustre] [10168.185299] ll_md_close+0x274/0x800 [lustre] [10168.185387] ll_file_release+0x128/0xc50 [lustre] [10168.185875] __fput+0x10d/0x440 [10168.185885] ____fput+0x12/0x20 [10168.197696] kthread+0x1d1/0x200 [10168.197720] ? set_kthread_struct+0x70/0x70 [10168.197726] ret_from_fork+0x1f/0x30 [10168.841466] task_work_run+0xc6/0x110 [10168.851372] exit_to_usermode_loop+0x1dd/0x1f0 [10168.853120] do_syscall_64+0x3d6/0x440 [10168.854388] entry_SYSCALL_64_after_hwframe+0x49/0xae [10168.865723] RIP: 0033:0x7f151bee3cd8 [10168.876288] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10168.907704] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10168.916810] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [10168.934358] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [10168.946547] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10168.964224] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10168.976601] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10172.059072] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10172.062356] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10172.104739] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10172.115401] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10172.124025] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10172.128253] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10172.138893] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10172.156726] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10172.164932] RAX: 000000000000105d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10172.171267] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [10172.177716] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [10172.185821] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [10172.190810] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [10172.194647] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [10172.206030] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10172.211182] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10172.213678] Call Trace: [10172.217246] [10172.225075] ? show_regs.cold.9+0x22/0x2f [10172.229846] ? watchdog_timer_fn.cold.6+0x62/0x10a [10172.237267] ? __hrtimer_run_queues+0x1c2/0x4d0 [10172.238717] ? watchdog+0x40/0x40 [10172.247701] ? hrtimer_interrupt+0x128/0x2c0 [10172.251296] ? smp_apic_timer_interrupt+0xa6/0x270 [10172.255915] ? apic_timer_interrupt+0xf/0x20 [10172.257403] [10172.263115] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10172.266795] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10172.275778] do_raw_spin_lock+0x177/0x190 [10172.277080] _raw_spin_lock+0x21/0x30 [10172.279567] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10172.289243] process_one_work+0x2c8/0x700 [10172.291013] worker_thread+0x296/0x6e0 [10172.303269] ? rescuer_thread+0x570/0x570 [10172.307406] kthread+0x1d1/0x200 [10172.312551] ? set_kthread_struct+0x70/0x70 [10172.316629] ret_from_fork+0x1f/0x30 [10176.052527] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [10176.056978] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10176.084592] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10176.090605] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10176.095677] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10176.106640] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10176.130541] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10176.136122] RAX: 0000000000003c83 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10176.140526] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [10176.144205] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [10176.148209] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [10176.153208] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [10176.158702] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [10176.167400] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10176.174806] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [10176.179033] Call Trace: [10176.181437] [10176.182035] ? show_regs.cold.9+0x22/0x2f [10176.183583] ? watchdog_timer_fn.cold.6+0x62/0x10a [10176.186060] ? __hrtimer_run_queues+0x1c2/0x4d0 [10176.188129] ? watchdog+0x40/0x40 [10176.189892] ? hrtimer_interrupt+0x128/0x2c0 [10176.191911] ? smp_apic_timer_interrupt+0xa6/0x270 [10176.194799] ? apic_timer_interrupt+0xf/0x20 [10176.196996] [10176.197709] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10176.199794] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10176.202765] do_raw_spin_lock+0x177/0x190 [10176.204678] _raw_spin_lock+0x21/0x30 [10176.206163] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10176.209049] ? do_raw_spin_unlock+0x75/0x190 [10176.211752] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10176.215265] ? do_wait_intr+0xf0/0xf0 [10176.220214] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10176.225109] mdc_reint+0x2d/0x110 [mdc] [10176.233662] mdc_unlink+0x658/0x870 [mdc] [10176.235917] lmv_unlink+0x34d/0xbd0 [lmv] [10176.245460] ll_unlink+0x24d/0x7f0 [lustre] [10176.249660] vfs_unlink+0x202/0x330 [10176.257362] do_unlinkat+0x2fa/0x460 [10176.258601] __x64_sys_unlink+0x37/0x50 [10176.264039] do_syscall_64+0xc1/0x440 [10176.268301] entry_SYSCALL_64_after_hwframe+0x49/0xae [10176.273164] RIP: 0033:0x7f151bdee10b [10176.277013] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10176.301925] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10176.309269] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [10176.313630] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [10176.317933] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10176.322549] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10176.326097] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10181.601583] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8251s! [10181.611147] Showing busy workqueues and worker pools: [10181.615958] workqueue events: flags=0x0 [10181.617374] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10181.627906] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [10181.637757] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10181.643260] pending: kfree_rcu_monitor [10181.655452] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10181.660925] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [10181.673256] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10181.684220] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [10181.699968] workqueue events_long: flags=0x0 [10181.706984] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10181.714789] pending: lu_objects_flush [obdclass] [10181.726607] workqueue events_unbound: flags=0x2 [10181.728262] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10181.731405] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10181.736820] workqueue events_power_efficient: flags=0x80 [10181.739811] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10181.743918] pending: do_cache_clean, neigh_periodic_work, check_lifetime [10181.747719] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10181.751538] pending: neigh_periodic_work [10181.753166] workqueue mm_percpu_wq: flags=0x8 [10181.754749] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10181.758683] pending: vmstat_update, lru_add_drain_per_cpu [10181.761017] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10181.764628] pending: vmstat_update [10181.766550] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10181.769881] pending: lru_add_drain_per_cpu [10181.771849] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10181.774555] pending: lru_add_drain_per_cpu BAR(48) [10181.776904] workqueue writeback: flags=0x4a [10181.778827] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [10181.781712] pending: wb_workfn [10181.783408] workqueue xprtiod: flags=0xa [10181.784776] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10181.790233] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [10181.794081] workqueue ipv6_addrconf: flags=0x40008 [10181.795979] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10181.802966] pending: addrconf_verify_work [10181.805618] workqueue knbd0-recv: flags=0x1a [10181.807637] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10181.810425] in-flight: 658:recv_work [10181.812391] workqueue ptlrpc_pinger: flags=0x2 [10181.814515] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10181.817463] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10181.819664] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8251s workers=4 idle: 32984 37 8 [10181.822517] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10166s workers=2 idle: 661 [10196.055648] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [10196.058814] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [10196.065968] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10196.066090] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10196.066098] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10196.066101] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10196.066158] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10196.066164] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10196.066168] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [10196.066171] RDX: 0000000000005617 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [10196.066173] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [10196.066175] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [10196.066177] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10196.066180] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [10196.066182] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10196.066185] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10196.066931] Call Trace: [10196.066957] [10196.066963] ? show_regs.cold.9+0x22/0x2f [10196.066974] ? watchdog_timer_fn.cold.6+0x62/0x10a [10196.066982] ? __hrtimer_run_queues+0x1c2/0x4d0 [10196.066987] ? watchdog+0x40/0x40 [10196.066991] ? hrtimer_interrupt+0x128/0x2c0 [10196.066995] ? smp_apic_timer_interrupt+0xa6/0x270 [10196.067003] ? apic_timer_interrupt+0xf/0x20 [10196.067008] [10196.067011] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10196.067016] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10196.067022] do_raw_spin_lock+0x177/0x190 [10196.067027] _raw_spin_lock+0x21/0x30 [10196.067034] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10196.069106] obd_mod_free+0x46/0x140 [ptlrpc] [10196.073680] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10196.073762] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10196.073770] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10196.073773] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10196.073786] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10196.073791] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10196.073796] RAX: 0000000000005566 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10196.073798] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [10196.073801] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [10196.073803] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [10196.073806] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [10196.073809] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [10196.073812] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10196.073814] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [10196.073823] Call Trace: [10196.073828] [10196.073833] ? show_regs.cold.9+0x22/0x2f [10196.073844] ? watchdog_timer_fn.cold.6+0x62/0x10a [10196.073851] ? __hrtimer_run_queues+0x1c2/0x4d0 [10196.073856] ? watchdog+0x40/0x40 [10196.073860] ? hrtimer_interrupt+0x128/0x2c0 [10196.073864] ? smp_apic_timer_interrupt+0xa6/0x270 [10196.073872] ? apic_timer_interrupt+0xf/0x20 [10196.073877] [10196.073880] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10196.073887] do_raw_spin_lock+0x177/0x190 [10196.073893] _raw_spin_lock+0x21/0x30 [10196.073898] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10196.076485] ? do_raw_spin_unlock+0x75/0x190 [10196.076495] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10196.076703] ? do_wait_intr+0xf0/0xf0 [10196.076711] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10196.076898] mdc_close+0x53b/0xd70 [mdc] [10196.076934] lmv_close+0x18d/0x490 [lmv] [10196.076963] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10196.077705] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10196.077819] ll_md_real_close+0xa5/0x2a0 [lustre] [10196.077899] ll_md_close+0x274/0x800 [lustre] [10196.077988] ll_file_release+0x128/0xc50 [lustre] [10196.129742] mdc_commit_open+0x91/0xb0 [mdc] [10196.129793] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10196.130058] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10196.130516] after_reply+0xe2c/0x1780 [ptlrpc] [10196.130729] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10196.130931] ? __next_timer_interrupt+0x160/0x160 [10196.130942] ptlrpcd+0xa82/0xe40 [ptlrpc] [10196.132413] ? do_wait_intr+0xf0/0xf0 [10196.132426] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10196.132658] kthread+0x1d1/0x200 [10196.132664] ? set_kthread_struct+0x70/0x70 [10196.132669] ret_from_fork+0x1f/0x30 [10196.584685] __fput+0x10d/0x440 [10196.589567] ____fput+0x12/0x20 [10196.592400] task_work_run+0xc6/0x110 [10196.594035] exit_to_usermode_loop+0x1dd/0x1f0 [10196.596113] do_syscall_64+0x3d6/0x440 [10196.601545] entry_SYSCALL_64_after_hwframe+0x49/0xae [10196.606027] RIP: 0033:0x7f151bee3cd8 [10196.607907] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10196.632504] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10196.646732] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [10196.657277] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [10196.661912] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10196.666251] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10196.670661] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10200.066884] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10200.069501] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10200.133089] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10200.160761] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10200.174735] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10200.181384] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10200.190700] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10200.216764] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [10200.225192] RAX: 000000000000771c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10200.238435] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [10200.250266] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [10200.258541] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [10200.274406] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [10200.298196] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [10200.309269] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10200.325364] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10200.327751] Call Trace: [10200.341859] [10200.349147] ? show_regs.cold.9+0x22/0x2f [10200.350924] ? watchdog_timer_fn.cold.6+0x62/0x10a [10200.354228] ? __hrtimer_run_queues+0x1c2/0x4d0 [10200.356489] ? watchdog+0x40/0x40 [10200.358319] ? hrtimer_interrupt+0x128/0x2c0 [10200.360854] ? smp_apic_timer_interrupt+0xa6/0x270 [10200.363923] ? apic_timer_interrupt+0xf/0x20 [10200.366292] [10200.367296] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10200.373743] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10200.377746] do_raw_spin_lock+0x177/0x190 [10200.385078] _raw_spin_lock+0x21/0x30 [10200.390888] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10200.397433] process_one_work+0x2c8/0x700 [10200.402224] worker_thread+0x296/0x6e0 [10200.408250] ? rescuer_thread+0x570/0x570 [10200.410888] kthread+0x1d1/0x200 [10200.417504] ? set_kthread_struct+0x70/0x70 [10200.420053] ret_from_fork+0x1f/0x30 [10208.052066] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [10208.072631] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10208.133281] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10208.146605] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10208.150404] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10208.153825] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10208.170187] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10208.178389] RAX: 00000000000036ac RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10208.184416] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [10208.187781] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [10208.193331] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [10208.198789] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [10208.203606] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [10208.209014] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10208.211830] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [10208.217771] Call Trace: [10208.218982] [10208.222205] ? show_regs.cold.9+0x22/0x2f [10208.226538] ? watchdog_timer_fn.cold.6+0x62/0x10a [10208.229415] ? __hrtimer_run_queues+0x1c2/0x4d0 [10208.238376] ? watchdog+0x40/0x40 [10208.242806] ? hrtimer_interrupt+0x128/0x2c0 [10208.251402] ? smp_apic_timer_interrupt+0xa6/0x270 [10208.254862] ? apic_timer_interrupt+0xf/0x20 [10208.259415] [10208.265000] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10208.268406] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10208.274759] do_raw_spin_lock+0x177/0x190 [10208.283925] _raw_spin_lock+0x21/0x30 [10208.286184] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10208.295865] ? do_raw_spin_unlock+0x75/0x190 [10208.300425] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10208.307448] ? do_wait_intr+0xf0/0xf0 [10208.309485] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10208.316297] mdc_reint+0x2d/0x110 [mdc] [10208.321884] mdc_unlink+0x658/0x870 [mdc] [10208.324405] lmv_unlink+0x34d/0xbd0 [lmv] [10208.333351] ll_unlink+0x24d/0x7f0 [lustre] [10208.338446] vfs_unlink+0x202/0x330 [10208.343237] do_unlinkat+0x2fa/0x460 [10208.346247] __x64_sys_unlink+0x37/0x50 [10208.355404] do_syscall_64+0xc1/0x440 [10208.363228] entry_SYSCALL_64_after_hwframe+0x49/0xae [10208.371400] RIP: 0033:0x7f151bdee10b [10208.381434] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10208.407544] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10208.416484] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [10208.425326] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [10208.427625] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10208.435881] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10208.443473] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10212.327882] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8281s! [10212.332565] Showing busy workqueues and worker pools: [10212.334967] workqueue events: flags=0x0 [10212.339206] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10212.342444] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [10212.348112] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10212.356334] pending: kfree_rcu_monitor [10212.358644] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10212.363152] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [10212.373907] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10212.378447] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [10212.388311] workqueue events_long: flags=0x0 [10212.392865] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10212.402592] pending: lu_objects_flush [obdclass] [10212.413445] workqueue events_unbound: flags=0x2 [10212.421183] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10212.429818] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10212.436897] workqueue events_power_efficient: flags=0x80 [10212.440976] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10212.445910] pending: do_cache_clean, neigh_periodic_work, check_lifetime [10212.457512] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10212.460207] pending: neigh_periodic_work [10212.463539] workqueue mm_percpu_wq: flags=0x8 [10212.469974] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10212.475454] pending: vmstat_update, lru_add_drain_per_cpu [10212.479440] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10212.482885] pending: vmstat_update [10212.489655] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10212.491838] pending: lru_add_drain_per_cpu [10212.496538] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10212.504362] pending: lru_add_drain_per_cpu BAR(48) [10212.514559] workqueue writeback: flags=0x4a [10212.515612] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [10212.517522] pending: wb_workfn [10212.520938] workqueue xprtiod: flags=0xa [10212.530857] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10212.543893] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [10212.550261] workqueue ipv6_addrconf: flags=0x40008 [10212.551856] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10212.559551] pending: addrconf_verify_work [10212.568074] workqueue knbd0-recv: flags=0x1a [10212.574507] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10212.582101] in-flight: 658:recv_work [10212.583457] workqueue ptlrpc_pinger: flags=0x2 [10212.592859] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10212.599292] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10212.605846] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8282s workers=4 idle: 32984 37 8 [10212.615181] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10197s workers=2 idle: 661 [10224.055481] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [10224.058625] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [10224.063825] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10224.064416] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10224.064424] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10224.064427] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10224.064450] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10224.064454] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10224.064459] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [10224.064462] RDX: 0000000000004e8f RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [10224.064464] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [10224.064466] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [10224.064468] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10224.064471] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [10224.064474] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10224.064476] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10224.064484] Call Trace: [10224.064491] [10224.064612] ? show_regs.cold.9+0x22/0x2f [10224.064622] ? watchdog_timer_fn.cold.6+0x62/0x10a [10224.064628] ? __hrtimer_run_queues+0x1c2/0x4d0 [10224.064632] ? watchdog+0x40/0x40 [10224.064637] ? hrtimer_interrupt+0x128/0x2c0 [10224.064641] ? smp_apic_timer_interrupt+0xa6/0x270 [10224.064649] ? apic_timer_interrupt+0xf/0x20 [10224.064653] [10224.064656] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10224.064661] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10224.064666] do_raw_spin_lock+0x177/0x190 [10224.064671] _raw_spin_lock+0x21/0x30 [10224.064677] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10224.079432] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10224.079512] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10224.079519] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10224.079522] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10224.079535] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10224.079540] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10224.079545] RAX: 0000000000001717 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10224.079547] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [10224.079550] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [10224.079552] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [10224.079555] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [10224.079558] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [10224.079561] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10224.079562] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [10224.079571] Call Trace: [10224.079576] [10224.079582] ? show_regs.cold.9+0x22/0x2f [10224.079591] ? watchdog_timer_fn.cold.6+0x62/0x10a [10224.079598] ? __hrtimer_run_queues+0x1c2/0x4d0 [10224.079602] ? watchdog+0x40/0x40 [10224.079605] ? hrtimer_interrupt+0x128/0x2c0 [10224.079609] ? smp_apic_timer_interrupt+0xa6/0x270 [10224.079616] ? apic_timer_interrupt+0xf/0x20 [10224.079621] [10224.079624] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10224.079630] do_raw_spin_lock+0x177/0x190 [10224.079634] _raw_spin_lock+0x21/0x30 [10224.079638] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10224.080361] ? do_raw_spin_unlock+0x75/0x190 [10224.080368] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10224.128207] obd_mod_free+0x46/0x140 [ptlrpc] [10224.128478] mdc_commit_open+0x91/0xb0 [mdc] [10224.129427] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10224.129713] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10224.130165] after_reply+0xe2c/0x1780 [ptlrpc] [10224.130370] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10224.130662] ? __next_timer_interrupt+0x160/0x160 [10224.130673] ptlrpcd+0xa82/0xe40 [ptlrpc] [10224.131077] ? do_wait_intr+0xf0/0xf0 [10224.131088] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10224.131308] kthread+0x1d1/0x200 [10224.131315] ? set_kthread_struct+0x70/0x70 [10224.131319] ret_from_fork+0x1f/0x30 [10224.527771] ? do_wait_intr+0xf0/0xf0 [10224.533773] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10224.540047] mdc_close+0x53b/0xd70 [mdc] [10224.541472] lmv_close+0x18d/0x490 [lmv] [10224.545306] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10224.549344] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10224.552184] ll_md_real_close+0xa5/0x2a0 [lustre] [10224.554781] ll_md_close+0x274/0x800 [lustre] [10224.557455] ll_file_release+0x128/0xc50 [lustre] [10224.565495] __fput+0x10d/0x440 [10224.566631] ____fput+0x12/0x20 [10224.571157] task_work_run+0xc6/0x110 [10224.572415] exit_to_usermode_loop+0x1dd/0x1f0 [10224.577430] do_syscall_64+0x3d6/0x440 [10224.578980] entry_SYSCALL_64_after_hwframe+0x49/0xae [10224.581105] RIP: 0033:0x7f151bee3cd8 [10224.582753] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10224.590533] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10224.594071] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [10224.597172] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [10224.599543] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10224.602966] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10224.610987] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10228.056047] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10228.061317] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10228.090548] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10228.097525] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10228.103883] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10228.108286] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10228.111362] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10228.123223] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10228.127336] RAX: 0000000000006f07 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10228.131637] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [10228.135276] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [10228.139537] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [10228.144055] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [10228.148518] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [10228.153124] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10228.156238] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10228.160785] Call Trace: [10228.162578] [10228.163662] ? show_regs.cold.9+0x22/0x2f [10228.166149] ? watchdog_timer_fn.cold.6+0x62/0x10a [10228.169044] ? __hrtimer_run_queues+0x1c2/0x4d0 [10228.172108] ? watchdog+0x40/0x40 [10228.174121] ? hrtimer_interrupt+0x128/0x2c0 [10228.176606] ? smp_apic_timer_interrupt+0xa6/0x270 [10228.179633] ? apic_timer_interrupt+0xf/0x20 [10228.182601] [10228.183560] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10228.186321] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10228.188779] do_raw_spin_lock+0x177/0x190 [10228.190786] _raw_spin_lock+0x21/0x30 [10228.192739] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10228.195518] process_one_work+0x2c8/0x700 [10228.197245] worker_thread+0x296/0x6e0 [10228.198636] ? rescuer_thread+0x570/0x570 [10228.200696] kthread+0x1d1/0x200 [10228.202305] ? set_kthread_struct+0x70/0x70 [10228.204254] ret_from_fork+0x1f/0x30 [10240.055268] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [10240.068882] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10240.111273] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10240.115791] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10240.123543] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10240.130761] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10240.143723] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10240.150396] RAX: 00000000000059a7 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10240.155297] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [10240.157884] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [10240.165138] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [10240.170539] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [10240.175219] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [10240.179090] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10240.181512] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [10240.184692] Call Trace: [10240.186906] [10240.187645] ? show_regs.cold.9+0x22/0x2f [10240.190390] ? watchdog_timer_fn.cold.6+0x62/0x10a [10240.195074] ? __hrtimer_run_queues+0x1c2/0x4d0 [10240.197921] ? watchdog+0x40/0x40 [10240.199201] ? hrtimer_interrupt+0x128/0x2c0 [10240.200688] ? smp_apic_timer_interrupt+0xa6/0x270 [10240.202902] ? apic_timer_interrupt+0xf/0x20 [10240.205029] [10240.206174] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10240.208232] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10240.210278] do_raw_spin_lock+0x177/0x190 [10240.211796] _raw_spin_lock+0x21/0x30 [10240.213186] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10240.215429] ? do_raw_spin_unlock+0x75/0x190 [10240.217004] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10240.218957] ? do_wait_intr+0xf0/0xf0 [10240.220044] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10240.222667] mdc_reint+0x2d/0x110 [mdc] [10240.229263] mdc_unlink+0x658/0x870 [mdc] [10240.232197] lmv_unlink+0x34d/0xbd0 [lmv] [10240.233792] ll_unlink+0x24d/0x7f0 [lustre] [10240.236667] vfs_unlink+0x202/0x330 [10240.238564] do_unlinkat+0x2fa/0x460 [10240.240117] __x64_sys_unlink+0x37/0x50 [10240.242069] do_syscall_64+0xc1/0x440 [10240.243698] entry_SYSCALL_64_after_hwframe+0x49/0xae [10240.252690] RIP: 0033:0x7f151bdee10b [10240.254824] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10240.266147] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10240.276548] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [10240.283152] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [10240.294376] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10240.309963] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10240.322457] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10243.041979] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8312s! [10243.045779] Showing busy workqueues and worker pools: [10243.047300] workqueue events: flags=0x0 [10243.050412] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10243.054389] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [10243.060193] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10243.065022] pending: kfree_rcu_monitor [10243.066365] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10243.070358] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [10243.075947] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10243.079636] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [10243.084147] workqueue events_long: flags=0x0 [10243.086928] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10243.090970] pending: lu_objects_flush [obdclass] [10243.094110] workqueue events_unbound: flags=0x2 [10243.101285] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10243.105122] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10243.110984] workqueue events_power_efficient: flags=0x80 [10243.114094] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10243.117445] pending: do_cache_clean, neigh_periodic_work, check_lifetime [10243.124219] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10243.129723] pending: neigh_periodic_work [10243.132505] workqueue mm_percpu_wq: flags=0x8 [10243.136261] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10243.141347] pending: vmstat_update, lru_add_drain_per_cpu [10243.144314] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10243.147507] pending: vmstat_update [10243.150020] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10243.153158] pending: lru_add_drain_per_cpu [10243.155396] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10243.159327] pending: lru_add_drain_per_cpu BAR(48) [10243.162016] workqueue writeback: flags=0x4a [10243.164405] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [10243.166452] pending: wb_workfn [10243.168248] workqueue xprtiod: flags=0xa [10243.170252] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10243.173231] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [10243.177416] workqueue ipv6_addrconf: flags=0x40008 [10243.180043] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10243.182879] pending: addrconf_verify_work [10243.184711] workqueue knbd0-recv: flags=0x1a [10243.187293] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10243.190304] in-flight: 658:recv_work [10243.191825] workqueue ptlrpc_pinger: flags=0x2 [10243.194074] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10243.197160] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10243.200453] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8312s workers=4 idle: 32984 37 8 [10243.205295] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10227s workers=2 idle: 661 [10252.057089] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [10252.060464] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10252.060767] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [10252.083900] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10252.083914] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10252.083917] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10252.083932] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10252.083936] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10252.083942] RAX: 0000000000005a0b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10252.083945] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [10252.083947] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [10252.083949] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [10252.083952] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [10252.083955] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [10252.083958] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10252.083960] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [10252.083970] Call Trace: [10252.083975] [10252.083981] ? show_regs.cold.9+0x22/0x2f [10252.083991] ? watchdog_timer_fn.cold.6+0x62/0x10a [10252.083997] ? __hrtimer_run_queues+0x1c2/0x4d0 [10252.084002] ? watchdog+0x40/0x40 [10252.084006] ? hrtimer_interrupt+0x128/0x2c0 [10252.084011] ? smp_apic_timer_interrupt+0xa6/0x270 [10252.084019] ? apic_timer_interrupt+0xf/0x20 [10252.084033] [10252.084036] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10252.084049] do_raw_spin_lock+0x177/0x190 [10252.084055] _raw_spin_lock+0x21/0x30 [10252.084061] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10252.084394] ? do_raw_spin_unlock+0x75/0x190 [10252.084400] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10252.084671] ? do_wait_intr+0xf0/0xf0 [10252.084679] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10252.084897] mdc_close+0x53b/0xd70 [mdc] [10252.084935] lmv_close+0x18d/0x490 [lmv] [10252.084962] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10252.089226] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10252.089305] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10252.089313] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10252.089317] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10252.089330] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10252.089334] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10252.089341] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [10252.089343] RDX: 00000000000009ec RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [10252.089345] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [10252.089347] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [10252.089350] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10252.089353] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [10252.089356] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10252.089358] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10252.089366] Call Trace: [10252.089371] [10252.089377] ? show_regs.cold.9+0x22/0x2f [10252.089387] ? watchdog_timer_fn.cold.6+0x62/0x10a [10252.095334] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10252.095463] ll_md_real_close+0xa5/0x2a0 [lustre] [10252.095546] ll_md_close+0x274/0x800 [lustre] [10252.095627] ll_file_release+0x128/0xc50 [lustre] [10252.095710] __fput+0x10d/0x440 [10252.095721] ____fput+0x12/0x20 [10252.095726] task_work_run+0xc6/0x110 [10252.095979] exit_to_usermode_loop+0x1dd/0x1f0 [10252.095988] do_syscall_64+0x3d6/0x440 [10252.095992] entry_SYSCALL_64_after_hwframe+0x49/0xae [10252.096001] RIP: 0033:0x7f151bee3cd8 [10252.096010] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10252.096014] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10252.096019] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [10252.096021] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [10252.096024] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10252.096026] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10252.096028] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10252.467059] ? __hrtimer_run_queues+0x1c2/0x4d0 [10252.468250] ? watchdog+0x40/0x40 [10252.469205] ? hrtimer_interrupt+0x128/0x2c0 [10252.470878] ? smp_apic_timer_interrupt+0xa6/0x270 [10252.472135] ? apic_timer_interrupt+0xf/0x20 [10252.473313] [10252.474059] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10252.475530] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10252.477055] do_raw_spin_lock+0x177/0x190 [10252.478160] _raw_spin_lock+0x21/0x30 [10252.479180] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10252.480912] obd_mod_free+0x46/0x140 [ptlrpc] [10252.482285] mdc_commit_open+0x91/0xb0 [mdc] [10252.483734] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10252.485316] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10252.487106] after_reply+0xe2c/0x1780 [ptlrpc] [10252.488654] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10252.490268] ? __next_timer_interrupt+0x160/0x160 [10252.491813] ptlrpcd+0xa82/0xe40 [ptlrpc] [10252.493237] ? do_wait_intr+0xf0/0xf0 [10252.494258] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10252.495985] kthread+0x1d1/0x200 [10252.497756] ? set_kthread_struct+0x70/0x70 [10252.498942] ret_from_fork+0x1f/0x30 [10256.056162] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10256.063885] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10256.097939] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10256.104552] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10256.109529] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10256.113291] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10256.117417] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10256.130756] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10256.134948] RAX: 0000000000006883 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10256.141860] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [10256.148940] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [10256.156545] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [10256.161490] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [10256.171307] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [10256.174514] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10256.179498] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10256.195029] Call Trace: [10256.197218] [10256.199825] ? show_regs.cold.9+0x22/0x2f [10256.203107] ? watchdog_timer_fn.cold.6+0x62/0x10a [10256.207287] ? __hrtimer_run_queues+0x1c2/0x4d0 [10256.209641] ? watchdog+0x40/0x40 [10256.212787] ? hrtimer_interrupt+0x128/0x2c0 [10256.216058] ? smp_apic_timer_interrupt+0xa6/0x270 [10256.222064] ? apic_timer_interrupt+0xf/0x20 [10256.223609] [10256.230996] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10256.234216] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10256.240351] do_raw_spin_lock+0x177/0x190 [10256.249137] _raw_spin_lock+0x21/0x30 [10256.253074] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10256.258916] process_one_work+0x2c8/0x700 [10256.261451] worker_thread+0x296/0x6e0 [10256.264532] ? rescuer_thread+0x570/0x570 [10256.272799] kthread+0x1d1/0x200 [10256.275739] ? set_kthread_struct+0x70/0x70 [10256.277296] ret_from_fork+0x1f/0x30 [10264.822143] rcu: INFO: rcu_sched self-detected stall on CPU [10264.830083] rcu: 1-...!: (4738474 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [10264.843079] (t=8340175 jiffies g=249657 q=4907) [10264.848457] rcu: rcu_sched kthread starved for 8335127 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [10264.861057] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [10264.864974] rcu: RCU grace-period kthread stack dump: [10264.869919] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [10264.883667] Call Trace: [10264.884776] __schedule+0x351/0xcb0 [10264.889449] schedule+0xc0/0x180 [10264.890949] schedule_timeout+0xb4/0x190 [10264.893192] ? do_raw_spin_unlock+0x75/0x190 [10264.895192] ? __next_timer_interrupt+0x160/0x160 [10264.903154] ? prepare_to_swait_event+0xf3/0x140 [10264.905766] rcu_gp_kthread+0x8dd/0x15b0 [10264.908986] ? force_qs_rnp+0x2c0/0x2c0 [10264.911441] kthread+0x1d1/0x200 [10264.916126] ? set_kthread_struct+0x70/0x70 [10264.923408] ret_from_fork+0x1f/0x30 [10264.927463] rcu: Stack dump where RCU GP kthread last ran: [10264.934013] Sending NMI from CPU 1 to CPUs 3: [10264.941092] NMI backtrace for cpu 3 [10264.941099] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10264.941102] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10264.941103] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10264.941105] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10264.941107] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 [10264.941110] RAX: 00000000000076ef RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10264.941111] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [10264.941112] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [10264.941114] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [10264.941119] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [10264.941121] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [10264.941122] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10264.941123] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [10264.941124] Call Trace: [10264.941125] [10264.941126] ? show_regs.cold.9+0x22/0x2f [10264.941127] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [10264.941128] ? nmi_cpu_backtrace_handler+0x15/0x40 [10264.941129] ? nmi_handle+0xbd/0x210 [10264.941130] ? default_do_nmi+0x4b/0x1b0 [10264.941131] ? do_nmi+0x280/0x380 [10264.941132] ? end_repeat_nmi+0x16/0x1f [10264.941133] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10264.941134] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10264.941135] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10264.941136] [10264.941137] do_raw_spin_lock+0x177/0x190 [10264.941138] _raw_spin_lock+0x21/0x30 [10264.941139] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10264.941140] ? do_raw_spin_unlock+0x75/0x190 [10264.941140] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10264.941141] ? do_wait_intr+0xf0/0xf0 [10264.941142] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10264.941144] mdc_close+0x53b/0xd70 [mdc] [10264.941144] lmv_close+0x18d/0x490 [lmv] [10264.941145] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10264.941146] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10264.941147] ll_md_real_close+0xa5/0x2a0 [lustre] [10264.941148] ll_md_close+0x274/0x800 [lustre] [10264.941149] ll_file_release+0x128/0xc50 [lustre] [10264.941150] __fput+0x10d/0x440 [10264.941151] ____fput+0x12/0x20 [10264.941152] task_work_run+0xc6/0x110 [10264.941156] exit_to_usermode_loop+0x1dd/0x1f0 [10264.941157] do_syscall_64+0x3d6/0x440 [10264.941158] entry_SYSCALL_64_after_hwframe+0x49/0xae [10264.941159] RIP: 0033:0x7f151bee3cd8 [10264.941160] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10264.941162] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10264.941164] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [10264.941166] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [10264.941167] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10264.941168] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10264.941170] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10264.944493] NMI backtrace for cpu 1 [10265.137885] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10265.143953] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10265.152135] Call Trace: [10265.152959] [10265.154756] ? dump_stack+0xbb/0x10e [10265.158287] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [10265.159949] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [10265.161576] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [10265.167957] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [10265.170452] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [10265.177312] ? rcu_dump_cpu_stacks+0x11b/0x171 [10265.179200] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [10265.181925] ? update_process_times+0x6d/0xc0 [10265.188421] ? tick_sched_handle.isra.18+0x27/0x90 [10265.199246] ? tick_sched_timer+0x53/0xb0 [10265.206410] ? __hrtimer_run_queues+0x1c2/0x4d0 [10265.208213] ? tick_sched_handle.isra.18+0x90/0x90 [10265.212504] ? hrtimer_interrupt+0x128/0x2c0 [10265.217009] ? smp_apic_timer_interrupt+0xa6/0x270 [10265.221114] ? apic_timer_interrupt+0xf/0x20 [10265.224005] [10265.225692] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10265.228709] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10265.231003] ? do_raw_spin_lock+0x177/0x190 [10265.233183] ? _raw_spin_lock+0x21/0x30 [10265.235415] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10265.239266] ? obd_mod_free+0x46/0x140 [ptlrpc] [10265.242831] ? mdc_commit_open+0x91/0xb0 [mdc] [10265.245153] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10265.247352] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10265.250878] ? after_reply+0xe2c/0x1780 [ptlrpc] [10265.255492] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10265.259132] ? __next_timer_interrupt+0x160/0x160 [10265.262353] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [10265.264204] ? do_wait_intr+0xf0/0xf0 [10265.265851] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10265.268213] ? kthread+0x1d1/0x200 [10265.269765] ? set_kthread_struct+0x70/0x70 [10265.271459] ? ret_from_fork+0x1f/0x30 [10268.054540] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [10268.062811] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10268.108730] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10268.117666] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10268.127043] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10268.132700] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10268.157385] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10268.166680] RAX: 000000000000525c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10268.177895] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [10268.185143] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [10268.200188] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [10268.213586] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [10268.222935] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [10268.235076] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10268.245746] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [10268.252430] Call Trace: [10268.256142] [10268.257525] ? show_regs.cold.9+0x22/0x2f [10268.262201] ? watchdog_timer_fn.cold.6+0x62/0x10a [10268.264046] ? __hrtimer_run_queues+0x1c2/0x4d0 [10268.269043] ? watchdog+0x40/0x40 [10268.270974] ? hrtimer_interrupt+0x128/0x2c0 [10268.272521] ? smp_apic_timer_interrupt+0xa6/0x270 [10268.279414] ? apic_timer_interrupt+0xf/0x20 [10268.283480] [10268.284383] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10268.288165] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10268.294458] do_raw_spin_lock+0x177/0x190 [10268.295591] _raw_spin_lock+0x21/0x30 [10268.305220] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10268.309166] ? do_raw_spin_unlock+0x75/0x190 [10268.312754] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10268.316982] ? do_wait_intr+0xf0/0xf0 [10268.320158] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10268.324097] mdc_reint+0x2d/0x110 [mdc] [10268.326042] mdc_unlink+0x658/0x870 [mdc] [10268.328270] lmv_unlink+0x34d/0xbd0 [lmv] [10268.330183] ll_unlink+0x24d/0x7f0 [lustre] [10268.339032] vfs_unlink+0x202/0x330 [10268.343365] do_unlinkat+0x2fa/0x460 [10268.347515] __x64_sys_unlink+0x37/0x50 [10268.350990] do_syscall_64+0xc1/0x440 [10268.352381] entry_SYSCALL_64_after_hwframe+0x49/0xae [10268.357978] RIP: 0033:0x7f151bdee10b [10268.360012] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10268.371281] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10268.375427] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [10268.379008] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [10268.382597] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10268.386330] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10268.389861] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10273.249113] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8342s! [10273.254204] Showing busy workqueues and worker pools: [10273.257784] workqueue events: flags=0x0 [10273.260559] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10273.264727] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [10273.277812] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10273.289117] pending: kfree_rcu_monitor [10273.297916] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10273.308497] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [10273.318711] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10273.328820] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [10273.333657] workqueue events_long: flags=0x0 [10273.335863] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10273.341915] pending: lu_objects_flush [obdclass] [10273.345507] workqueue events_unbound: flags=0x2 [10273.349826] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10273.353759] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10273.358288] workqueue events_power_efficient: flags=0x80 [10273.362066] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10273.365618] pending: do_cache_clean, neigh_periodic_work, check_lifetime [10273.369325] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10273.372914] pending: neigh_periodic_work [10273.374969] workqueue mm_percpu_wq: flags=0x8 [10273.377229] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10273.380572] pending: vmstat_update, lru_add_drain_per_cpu [10273.383067] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10273.386895] pending: vmstat_update [10273.388862] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10273.392758] pending: lru_add_drain_per_cpu [10273.395227] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10273.398071] pending: lru_add_drain_per_cpu BAR(48) [10273.400873] workqueue writeback: flags=0x4a [10273.402901] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [10273.406590] pending: wb_workfn [10273.408131] workqueue xprtiod: flags=0xa [10273.409877] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10273.412995] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [10273.417778] workqueue ipv6_addrconf: flags=0x40008 [10273.420130] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10273.423599] pending: addrconf_verify_work [10273.425583] workqueue knbd0-recv: flags=0x1a [10273.427759] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10273.430874] in-flight: 658:recv_work [10273.432598] workqueue ptlrpc_pinger: flags=0x2 [10273.434619] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10273.436733] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10273.439824] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8342s workers=4 idle: 32984 37 8 [10273.443603] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10258s workers=2 idle: 661 [10284.062958] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10284.067755] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10284.094761] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10284.107033] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10284.121595] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10284.129765] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10284.134478] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10284.147615] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10284.153623] RAX: 0000000000005121 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10284.160967] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [10284.170965] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [10284.180609] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [10284.186914] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [10284.194992] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [10284.204107] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10284.207212] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10284.212789] Call Trace: [10284.214254] [10284.215228] ? show_regs.cold.9+0x22/0x2f [10284.217469] ? watchdog_timer_fn.cold.6+0x62/0x10a [10284.219932] ? __hrtimer_run_queues+0x1c2/0x4d0 [10284.221856] ? watchdog+0x40/0x40 [10284.223210] ? hrtimer_interrupt+0x128/0x2c0 [10284.225016] ? smp_apic_timer_interrupt+0xa6/0x270 [10284.227249] ? apic_timer_interrupt+0xf/0x20 [10284.229266] [10284.230177] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10284.232916] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10284.236025] do_raw_spin_lock+0x177/0x190 [10284.241753] _raw_spin_lock+0x21/0x30 [10284.244482] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10284.255914] process_one_work+0x2c8/0x700 [10284.257599] worker_thread+0x296/0x6e0 [10284.260009] ? rescuer_thread+0x570/0x570 [10284.268050] kthread+0x1d1/0x200 [10284.272007] ? set_kthread_struct+0x70/0x70 [10284.278898] ret_from_fork+0x1f/0x30 [10292.057024] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [10292.057216] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [10292.063216] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10292.063296] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10292.063305] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10292.063308] RIP: 0010:native_safe_halt+0x2e/0x50 [10292.063321] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10292.063325] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10292.063329] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10292.063332] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [10292.063333] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [10292.063336] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [10292.063338] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [10292.064031] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [10292.064036] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10292.064039] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [10292.064048] Call Trace: [10292.064055] [10292.064061] ? show_regs.cold.9+0x22/0x2f [10292.064072] ? watchdog_timer_fn.cold.6+0x62/0x10a [10292.064078] ? __hrtimer_run_queues+0x1c2/0x4d0 [10292.064082] ? watchdog+0x40/0x40 [10292.064086] ? hrtimer_interrupt+0x128/0x2c0 [10292.064091] ? smp_apic_timer_interrupt+0xa6/0x270 [10292.064099] ? apic_timer_interrupt+0xf/0x20 [10292.064104] [10292.064107] ? native_safe_halt+0x2e/0x50 [10292.064111] kvm_wait+0xcd/0xe0 [10292.064118] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10292.064127] do_raw_spin_lock+0x177/0x190 [10292.064131] _raw_spin_lock+0x21/0x30 [10292.064136] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10292.064539] ? do_raw_spin_unlock+0x75/0x190 [10292.064546] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10292.070072] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10292.070210] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10292.070216] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10292.070219] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10292.070233] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10292.070237] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10292.070241] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [10292.070243] RDX: 00000000000037a9 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [10292.070245] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [10292.070248] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [10292.070250] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10292.070253] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [10292.070256] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10292.070258] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10292.071057] Call Trace: [10292.071065] [10292.071070] ? show_regs.cold.9+0x22/0x2f [10292.071081] ? watchdog_timer_fn.cold.6+0x62/0x10a [10292.071087] ? __hrtimer_run_queues+0x1c2/0x4d0 [10292.071091] ? watchdog+0x40/0x40 [10292.071095] ? hrtimer_interrupt+0x128/0x2c0 [10292.071098] ? smp_apic_timer_interrupt+0xa6/0x270 [10292.071107] ? apic_timer_interrupt+0xf/0x20 [10292.071112] [10292.071114] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10292.071119] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10292.071185] do_raw_spin_lock+0x177/0x190 [10292.071191] _raw_spin_lock+0x21/0x30 [10292.071198] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10292.071470] obd_mod_free+0x46/0x140 [ptlrpc] [10292.072095] mdc_commit_open+0x91/0xb0 [mdc] [10292.072138] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10292.072333] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10292.072862] after_reply+0xe2c/0x1780 [ptlrpc] [10292.073065] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10292.073258] ? __next_timer_interrupt+0x160/0x160 [10292.073268] ptlrpcd+0xa82/0xe40 [ptlrpc] [10292.073478] ? do_wait_intr+0xf0/0xf0 [10292.073487] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10292.073918] kthread+0x1d1/0x200 [10292.073924] ? set_kthread_struct+0x70/0x70 [10292.073927] ret_from_fork+0x1f/0x30 [10292.423824] ? do_wait_intr+0xf0/0xf0 [10292.425394] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10292.429783] mdc_close+0x53b/0xd70 [mdc] [10292.432515] lmv_close+0x18d/0x490 [lmv] [10292.435502] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10292.438082] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10292.440587] ll_md_real_close+0xa5/0x2a0 [lustre] [10292.442174] ll_md_close+0x274/0x800 [lustre] [10292.443883] ll_file_release+0x128/0xc50 [lustre] [10292.445528] __fput+0x10d/0x440 [10292.447123] ____fput+0x12/0x20 [10292.448721] task_work_run+0xc6/0x110 [10292.451609] exit_to_usermode_loop+0x1dd/0x1f0 [10292.454089] do_syscall_64+0x3d6/0x440 [10292.458468] entry_SYSCALL_64_after_hwframe+0x49/0xae [10292.461607] RIP: 0033:0x7f151bee3cd8 [10292.463076] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10292.471831] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10292.476630] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [10292.479756] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [10292.485101] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10292.488550] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10292.491922] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10300.052675] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [10300.057033] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10300.093068] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10300.106522] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10300.112632] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10300.121301] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10300.142404] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10300.154135] RAX: 0000000000003e31 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10300.158250] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [10300.175425] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [10300.188636] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [10300.201404] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [10300.209380] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [10300.216618] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10300.219573] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [10300.223285] Call Trace: [10300.224769] [10300.225679] ? show_regs.cold.9+0x22/0x2f [10300.227049] ? watchdog_timer_fn.cold.6+0x62/0x10a [10300.229491] ? __hrtimer_run_queues+0x1c2/0x4d0 [10300.231572] ? watchdog+0x40/0x40 [10300.233027] ? hrtimer_interrupt+0x128/0x2c0 [10300.234930] ? smp_apic_timer_interrupt+0xa6/0x270 [10300.238457] ? apic_timer_interrupt+0xf/0x20 [10300.241077] [10300.242443] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10300.246619] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10300.250121] do_raw_spin_lock+0x177/0x190 [10300.252461] _raw_spin_lock+0x21/0x30 [10300.255864] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10300.258430] ? do_raw_spin_unlock+0x75/0x190 [10300.262824] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10300.267938] ? do_wait_intr+0xf0/0xf0 [10300.270034] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10300.275522] mdc_reint+0x2d/0x110 [mdc] [10300.277729] mdc_unlink+0x658/0x870 [mdc] [10300.279804] lmv_unlink+0x34d/0xbd0 [lmv] [10300.286228] ll_unlink+0x24d/0x7f0 [lustre] [10300.290597] vfs_unlink+0x202/0x330 [10300.295974] do_unlinkat+0x2fa/0x460 [10300.303964] __x64_sys_unlink+0x37/0x50 [10300.307766] do_syscall_64+0xc1/0x440 [10300.308925] entry_SYSCALL_64_after_hwframe+0x49/0xae [10300.319899] RIP: 0033:0x7f151bdee10b [10300.325541] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10300.339983] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10300.349053] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [10300.361428] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [10300.368940] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10300.377980] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10300.381969] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10303.456289] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8372s! [10303.463505] Showing busy workqueues and worker pools: [10303.466022] workqueue events: flags=0x0 [10303.468773] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10303.472546] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [10303.477718] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10303.481321] pending: kfree_rcu_monitor [10303.483441] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10303.487325] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [10303.494343] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10303.499801] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [10303.504657] workqueue events_long: flags=0x0 [10303.506817] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10303.510616] pending: lu_objects_flush [obdclass] [10303.512942] workqueue events_unbound: flags=0x2 [10303.515057] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10303.517884] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10303.522594] workqueue events_power_efficient: flags=0x80 [10303.525287] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10303.528507] pending: do_cache_clean, neigh_periodic_work, check_lifetime [10303.531738] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10303.534980] pending: neigh_periodic_work [10303.536886] workqueue mm_percpu_wq: flags=0x8 [10303.539220] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10303.542804] pending: vmstat_update, lru_add_drain_per_cpu [10303.545321] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10303.548432] pending: vmstat_update [10303.550252] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10303.553600] pending: lru_add_drain_per_cpu [10303.556075] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10303.559689] pending: lru_add_drain_per_cpu BAR(48) [10303.561763] workqueue writeback: flags=0x4a [10303.563653] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [10303.567454] pending: wb_workfn [10303.569263] workqueue xprtiod: flags=0xa [10303.571589] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10303.575641] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [10303.579316] workqueue ipv6_addrconf: flags=0x40008 [10303.580890] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10303.583862] pending: addrconf_verify_work [10303.586050] workqueue knbd0-recv: flags=0x1a [10303.588158] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10303.591619] in-flight: 658:recv_work [10303.593910] workqueue ptlrpc_pinger: flags=0x2 [10303.596421] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10303.599574] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10303.602932] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8373s workers=4 idle: 32984 37 8 [10303.606670] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10288s workers=2 idle: 661 [10312.056014] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10312.064747] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10312.100550] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10312.107310] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10312.114737] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10312.120098] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10312.125189] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10312.136956] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10312.142235] RAX: 0000000000004a8a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10312.146957] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [10312.151542] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [10312.154888] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [10312.161284] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [10312.167621] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [10312.179310] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10312.197219] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10312.203100] Call Trace: [10312.205282] [10312.205994] ? show_regs.cold.9+0x22/0x2f [10312.207186] ? watchdog_timer_fn.cold.6+0x62/0x10a [10312.209632] ? __hrtimer_run_queues+0x1c2/0x4d0 [10312.217484] ? watchdog+0x40/0x40 [10312.220636] ? hrtimer_interrupt+0x128/0x2c0 [10312.224180] ? smp_apic_timer_interrupt+0xa6/0x270 [10312.227760] ? apic_timer_interrupt+0xf/0x20 [10312.230788] [10312.231628] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10312.236769] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10312.239787] do_raw_spin_lock+0x177/0x190 [10312.241531] _raw_spin_lock+0x21/0x30 [10312.243315] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10312.249125] process_one_work+0x2c8/0x700 [10312.251779] worker_thread+0x296/0x6e0 [10312.256478] ? rescuer_thread+0x570/0x570 [10312.264001] kthread+0x1d1/0x200 [10312.265696] ? set_kthread_struct+0x70/0x70 [10312.270700] ret_from_fork+0x1f/0x30 [10320.055012] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [10320.058283] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [10320.059205] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10320.059475] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10320.059483] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10320.059487] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10320.059501] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10320.059506] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10320.059511] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [10320.059514] RDX: 0000000000001a31 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [10320.059516] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [10320.059518] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [10320.059521] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10320.059524] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [10320.059526] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10320.059529] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10320.059546] Call Trace: [10320.059552] [10320.059559] ? show_regs.cold.9+0x22/0x2f [10320.059570] ? watchdog_timer_fn.cold.6+0x62/0x10a [10320.059577] ? __hrtimer_run_queues+0x1c2/0x4d0 [10320.059582] ? watchdog+0x40/0x40 [10320.059586] ? hrtimer_interrupt+0x128/0x2c0 [10320.059589] ? smp_apic_timer_interrupt+0xa6/0x270 [10320.059597] ? apic_timer_interrupt+0xf/0x20 [10320.059601] [10320.059604] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10320.059608] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10320.059612] do_raw_spin_lock+0x177/0x190 [10320.059617] _raw_spin_lock+0x21/0x30 [10320.059622] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10320.059896] obd_mod_free+0x46/0x140 [ptlrpc] [10320.060114] mdc_commit_open+0x91/0xb0 [mdc] [10320.060148] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10320.060516] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10320.060753] after_reply+0xe2c/0x1780 [ptlrpc] [10320.060952] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10320.061232] ? __next_timer_interrupt+0x160/0x160 [10320.061242] ptlrpcd+0xa82/0xe40 [ptlrpc] [10320.070143] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10320.070222] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10320.070231] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10320.070234] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10320.070249] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10320.098374] ? do_wait_intr+0xf0/0xf0 [10320.098398] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10320.098741] kthread+0x1d1/0x200 [10320.098748] ? set_kthread_struct+0x70/0x70 [10320.098751] ret_from_fork+0x1f/0x30 [10320.360761] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10320.371526] RAX: 0000000000006c2e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10320.382979] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [10320.384669] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [10320.391115] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [10320.394785] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [10320.403204] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [10320.409551] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10320.415536] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [10320.425606] Call Trace: [10320.427117] [10320.428653] ? show_regs.cold.9+0x22/0x2f [10320.432986] ? watchdog_timer_fn.cold.6+0x62/0x10a [10320.438221] ? __hrtimer_run_queues+0x1c2/0x4d0 [10320.440374] ? watchdog+0x40/0x40 [10320.441932] ? hrtimer_interrupt+0x128/0x2c0 [10320.443619] ? smp_apic_timer_interrupt+0xa6/0x270 [10320.446752] ? apic_timer_interrupt+0xf/0x20 [10320.448252] [10320.449039] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10320.457554] do_raw_spin_lock+0x177/0x190 [10320.461546] _raw_spin_lock+0x21/0x30 [10320.463593] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10320.471458] ? do_raw_spin_unlock+0x75/0x190 [10320.473319] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10320.479585] ? do_wait_intr+0xf0/0xf0 [10320.486352] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10320.488877] mdc_close+0x53b/0xd70 [mdc] [10320.493421] lmv_close+0x18d/0x490 [lmv] [10320.494887] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10320.497935] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10320.499924] ll_md_real_close+0xa5/0x2a0 [lustre] [10320.508832] ll_md_close+0x274/0x800 [lustre] [10320.521830] ll_file_release+0x128/0xc50 [lustre] [10320.523663] __fput+0x10d/0x440 [10320.539614] ____fput+0x12/0x20 [10320.540768] task_work_run+0xc6/0x110 [10320.541969] exit_to_usermode_loop+0x1dd/0x1f0 [10320.561906] do_syscall_64+0x3d6/0x440 [10320.564903] entry_SYSCALL_64_after_hwframe+0x49/0xae [10320.569871] RIP: 0033:0x7f151bee3cd8 [10320.572902] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10320.591131] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10320.595364] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [10320.601361] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [10320.605552] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10320.613485] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10320.626251] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10328.056471] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [10328.062752] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10328.099381] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10328.110972] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10328.128179] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10328.135373] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10328.150776] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10328.166715] RAX: 0000000000007a1c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10328.185490] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [10328.190741] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [10328.199315] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [10328.204365] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [10328.207758] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [10328.211753] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10328.216250] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [10328.220939] Call Trace: [10328.222342] [10328.223480] ? show_regs.cold.9+0x22/0x2f [10328.230675] ? watchdog_timer_fn.cold.6+0x62/0x10a [10328.235648] ? __hrtimer_run_queues+0x1c2/0x4d0 [10328.240452] ? watchdog+0x40/0x40 [10328.242392] ? hrtimer_interrupt+0x128/0x2c0 [10328.245294] ? smp_apic_timer_interrupt+0xa6/0x270 [10328.251411] ? apic_timer_interrupt+0xf/0x20 [10328.253370] [10328.257277] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10328.262560] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10328.265573] do_raw_spin_lock+0x177/0x190 [10328.267547] _raw_spin_lock+0x21/0x30 [10328.269602] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10328.272604] ? do_raw_spin_unlock+0x75/0x190 [10328.276665] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10328.281819] ? do_wait_intr+0xf0/0xf0 [10328.283658] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10328.292394] mdc_reint+0x2d/0x110 [mdc] [10328.294287] mdc_unlink+0x658/0x870 [mdc] [10328.300434] lmv_unlink+0x34d/0xbd0 [lmv] [10328.303643] ll_unlink+0x24d/0x7f0 [lustre] [10328.315292] vfs_unlink+0x202/0x330 [10328.320552] do_unlinkat+0x2fa/0x460 [10328.323396] __x64_sys_unlink+0x37/0x50 [10328.329149] do_syscall_64+0xc1/0x440 [10328.332888] entry_SYSCALL_64_after_hwframe+0x49/0xae [10328.344170] RIP: 0033:0x7f151bdee10b [10328.345333] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10328.366244] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10328.375146] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [10328.381430] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [10328.387736] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10328.392572] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10328.396326] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10333.664221] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8403s! [10333.667442] Showing busy workqueues and worker pools: [10333.670505] workqueue events: flags=0x0 [10333.676178] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10333.680577] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [10333.695125] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10333.698585] pending: kfree_rcu_monitor [10333.700519] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10333.707300] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [10333.714458] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10333.724089] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [10333.734748] workqueue events_long: flags=0x0 [10333.740299] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10333.746924] pending: lu_objects_flush [obdclass] [10333.748923] workqueue events_unbound: flags=0x2 [10333.755596] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10333.760842] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10333.775596] workqueue events_power_efficient: flags=0x80 [10333.778608] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10333.791441] pending: do_cache_clean, neigh_periodic_work, check_lifetime [10333.796063] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10333.803676] pending: neigh_periodic_work [10333.806272] workqueue mm_percpu_wq: flags=0x8 [10333.809584] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10333.818408] pending: vmstat_update, lru_add_drain_per_cpu [10333.822534] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10333.824902] pending: vmstat_update [10333.826816] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10333.831124] pending: lru_add_drain_per_cpu [10333.834682] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10333.841104] pending: lru_add_drain_per_cpu BAR(48) [10333.843785] workqueue writeback: flags=0x4a [10333.845946] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [10333.853138] pending: wb_workfn [10333.856198] workqueue xprtiod: flags=0xa [10333.859136] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10333.870102] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [10333.874008] workqueue ipv6_addrconf: flags=0x40008 [10333.880352] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10333.885932] pending: addrconf_verify_work [10333.892899] workqueue knbd0-recv: flags=0x1a [10333.896347] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10333.900568] in-flight: 658:recv_work [10333.902722] workqueue ptlrpc_pinger: flags=0x2 [10333.906669] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10333.912329] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10333.923135] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8403s workers=4 idle: 32984 37 8 [10333.929865] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10318s workers=2 idle: 661 [10340.056708] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10340.061291] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10340.116632] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10340.127315] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10340.132347] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10340.138922] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10340.142898] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10340.155688] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10340.164495] RAX: 0000000000006f9d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10340.171561] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [10340.178609] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [10340.183575] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [10340.188506] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [10340.193996] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [10340.198194] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10340.213228] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10340.218720] Call Trace: [10340.227917] [10340.229603] ? show_regs.cold.9+0x22/0x2f [10340.239914] ? watchdog_timer_fn.cold.6+0x62/0x10a [10340.246909] ? __hrtimer_run_queues+0x1c2/0x4d0 [10340.253102] ? watchdog+0x40/0x40 [10340.256030] ? hrtimer_interrupt+0x128/0x2c0 [10340.259040] ? smp_apic_timer_interrupt+0xa6/0x270 [10340.262328] ? apic_timer_interrupt+0xf/0x20 [10340.270119] [10340.271273] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10340.274958] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10340.277631] do_raw_spin_lock+0x177/0x190 [10340.280279] _raw_spin_lock+0x21/0x30 [10340.287674] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10340.297592] process_one_work+0x2c8/0x700 [10340.304024] worker_thread+0x296/0x6e0 [10340.309369] ? rescuer_thread+0x570/0x570 [10340.314203] kthread+0x1d1/0x200 [10340.319295] ? set_kthread_struct+0x70/0x70 [10340.323814] ret_from_fork+0x1f/0x30 [10348.055208] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [10348.057172] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [10348.059002] Modules linked in: [10348.064323] Modules linked in: [10348.064326] lustre(O) osp(O) [10348.066244] lustre(O) [10348.067556] ofd(O) [10348.068337] osp(O) [10348.068866] lod(O) [10348.069746] ofd(O) [10348.070636] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10348.070712] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10348.070719] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10348.070722] RIP: 0010:native_safe_halt+0x2e/0x50 [10348.070735] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10348.070740] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10348.070744] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [10348.070746] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [10348.070748] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [10348.070750] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [10348.070753] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10348.070756] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [10348.070759] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10348.070761] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10348.070771] Call Trace: [10348.070776] [10348.070782] ? show_regs.cold.9+0x22/0x2f [10348.070793] ? watchdog_timer_fn.cold.6+0x62/0x10a [10348.070838] ? __hrtimer_run_queues+0x1c2/0x4d0 [10348.070843] ? watchdog+0x40/0x40 [10348.070847] ? hrtimer_interrupt+0x128/0x2c0 [10348.070850] ? smp_apic_timer_interrupt+0xa6/0x270 [10348.070856] ? apic_timer_interrupt+0xf/0x20 [10348.070861] [10348.070864] ? native_safe_halt+0x2e/0x50 [10348.070867] kvm_wait+0xcd/0xe0 [10348.070875] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10348.070882] do_raw_spin_lock+0x177/0x190 [10348.070886] _raw_spin_lock+0x21/0x30 [10348.070892] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10348.071171] obd_mod_free+0x46/0x140 [ptlrpc] [10348.071364] mdc_commit_open+0x91/0xb0 [mdc] [10348.071404] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10348.071595] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10348.072759] lod(O) [10348.073861] after_reply+0xe2c/0x1780 [ptlrpc] [10348.074127] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10348.095337] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10348.095407] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10348.095415] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10348.095418] RIP: 0010:native_safe_halt+0x2e/0x50 [10348.095432] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10348.095437] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10348.095441] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10348.095443] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [10348.095445] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [10348.095447] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [10348.095450] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [10348.095454] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [10348.095456] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10348.095459] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [10348.095468] Call Trace: [10348.095474] [10348.095479] ? show_regs.cold.9+0x22/0x2f [10348.095488] ? watchdog_timer_fn.cold.6+0x62/0x10a [10348.095494] ? __hrtimer_run_queues+0x1c2/0x4d0 [10348.095498] ? watchdog+0x40/0x40 [10348.095501] ? hrtimer_interrupt+0x128/0x2c0 [10348.095505] ? smp_apic_timer_interrupt+0xa6/0x270 [10348.095512] ? apic_timer_interrupt+0xf/0x20 [10348.095516] [10348.095518] ? native_safe_halt+0x2e/0x50 [10348.095522] kvm_wait+0xcd/0xe0 [10348.095546] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10348.095554] do_raw_spin_lock+0x177/0x190 [10348.095558] _raw_spin_lock+0x21/0x30 [10348.095562] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10348.100341] ? __next_timer_interrupt+0x160/0x160 [10348.104929] ? do_raw_spin_unlock+0x75/0x190 [10348.106887] ptlrpcd+0xa82/0xe40 [ptlrpc] [10348.129652] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10348.134901] ? do_wait_intr+0xf0/0xf0 [10348.139532] ? do_wait_intr+0xf0/0xf0 [10348.144923] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10348.153381] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10348.158522] kthread+0x1d1/0x200 [10348.158548] ? set_kthread_struct+0x70/0x70 [10348.158553] ret_from_fork+0x1f/0x30 [10348.403599] mdc_close+0x53b/0xd70 [mdc] [10348.405315] lmv_close+0x18d/0x490 [lmv] [10348.407519] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10348.409991] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10348.412697] ll_md_real_close+0xa5/0x2a0 [lustre] [10348.414930] ll_md_close+0x274/0x800 [lustre] [10348.417310] ll_file_release+0x128/0xc50 [lustre] [10348.419614] __fput+0x10d/0x440 [10348.420939] ____fput+0x12/0x20 [10348.422506] task_work_run+0xc6/0x110 [10348.424404] exit_to_usermode_loop+0x1dd/0x1f0 [10348.426714] do_syscall_64+0x3d6/0x440 [10348.428290] entry_SYSCALL_64_after_hwframe+0x49/0xae [10348.430986] RIP: 0033:0x7f151bee3cd8 [10348.432902] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10348.450364] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10348.458928] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [10348.465365] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [10348.475609] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10348.487592] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10348.500223] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10360.053267] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [10360.055552] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10360.086852] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10360.094020] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10360.099345] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10360.102843] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10360.113221] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10360.117167] RAX: 000000000000690d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10360.120758] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [10360.124820] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [10360.128276] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [10360.131942] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [10360.135133] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [10360.139245] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10360.142197] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [10360.145513] Call Trace: [10360.146492] [10360.147308] ? show_regs.cold.9+0x22/0x2f [10360.149127] ? watchdog_timer_fn.cold.6+0x62/0x10a [10360.152513] ? __hrtimer_run_queues+0x1c2/0x4d0 [10360.155399] ? watchdog+0x40/0x40 [10360.157051] ? hrtimer_interrupt+0x128/0x2c0 [10360.159916] ? smp_apic_timer_interrupt+0xa6/0x270 [10360.163098] ? apic_timer_interrupt+0xf/0x20 [10360.165792] [10360.167246] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10360.170879] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10360.173523] do_raw_spin_lock+0x177/0x190 [10360.176533] _raw_spin_lock+0x21/0x30 [10360.179223] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10360.182764] ? do_raw_spin_unlock+0x75/0x190 [10360.185185] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10360.189076] ? do_wait_intr+0xf0/0xf0 [10360.191133] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10360.194365] mdc_reint+0x2d/0x110 [mdc] [10360.196078] mdc_unlink+0x658/0x870 [mdc] [10360.198197] lmv_unlink+0x34d/0xbd0 [lmv] [10360.200914] ll_unlink+0x24d/0x7f0 [lustre] [10360.202579] vfs_unlink+0x202/0x330 [10360.204785] do_unlinkat+0x2fa/0x460 [10360.207258] __x64_sys_unlink+0x37/0x50 [10360.209537] do_syscall_64+0xc1/0x440 [10360.211716] entry_SYSCALL_64_after_hwframe+0x49/0xae [10360.214847] RIP: 0033:0x7f151bdee10b [10360.216821] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10360.227799] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10360.231438] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [10360.236028] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [10360.240922] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10360.246883] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10360.253247] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10364.389220] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8433s! [10364.403321] Showing busy workqueues and worker pools: [10364.410010] workqueue events: flags=0x0 [10364.413994] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10364.423294] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [10364.438029] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10364.456835] pending: kfree_rcu_monitor [10364.460781] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10364.469691] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [10364.488553] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10364.500980] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [10364.509476] workqueue events_long: flags=0x0 [10364.518486] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10364.521673] pending: lu_objects_flush [obdclass] [10364.523574] workqueue events_unbound: flags=0x2 [10364.528968] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10364.533025] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10364.540463] workqueue events_power_efficient: flags=0x80 [10364.546907] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10364.559494] pending: do_cache_clean, neigh_periodic_work, check_lifetime [10364.566366] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10364.577985] pending: neigh_periodic_work [10364.581962] workqueue mm_percpu_wq: flags=0x8 [10364.584808] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10364.592142] pending: vmstat_update, lru_add_drain_per_cpu [10364.597980] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10364.600167] pending: vmstat_update [10364.604010] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10364.607415] pending: lru_add_drain_per_cpu [10364.609877] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10364.621041] pending: lru_add_drain_per_cpu BAR(48) [10364.627052] workqueue writeback: flags=0x4a [10364.633376] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [10364.645824] pending: wb_workfn [10364.648204] workqueue xprtiod: flags=0xa [10364.654035] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10364.663925] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [10364.670254] workqueue ipv6_addrconf: flags=0x40008 [10364.673974] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10364.676248] pending: addrconf_verify_work [10364.678972] workqueue knbd0-recv: flags=0x1a [10364.684497] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10364.694189] in-flight: 658:recv_work [10364.699634] workqueue ptlrpc_pinger: flags=0x2 [10364.702917] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10364.716841] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10364.724031] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8434s workers=4 idle: 32984 37 8 [10364.728897] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10349s workers=2 idle: 661 [10368.056971] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [10368.066310] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10368.104710] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10368.111979] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10368.121155] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10368.133620] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10368.137178] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10368.155636] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10368.163691] RAX: 0000000000007882 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10368.171711] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [10368.181174] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [10368.187950] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [10368.194221] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [10368.201464] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [10368.210918] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10368.217166] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10368.228947] Call Trace: [10368.230311] [10368.232881] ? show_regs.cold.9+0x22/0x2f [10368.239253] ? watchdog_timer_fn.cold.6+0x62/0x10a [10368.246998] ? __hrtimer_run_queues+0x1c2/0x4d0 [10368.249071] ? watchdog+0x40/0x40 [10368.256220] ? hrtimer_interrupt+0x128/0x2c0 [10368.258277] ? smp_apic_timer_interrupt+0xa6/0x270 [10368.266117] ? apic_timer_interrupt+0xf/0x20 [10368.270040] [10368.270919] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10368.277126] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10368.281012] do_raw_spin_lock+0x177/0x190 [10368.285216] _raw_spin_lock+0x21/0x30 [10368.292649] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10368.295277] process_one_work+0x2c8/0x700 [10368.306645] worker_thread+0x296/0x6e0 [10368.312986] ? rescuer_thread+0x570/0x570 [10368.314915] kthread+0x1d1/0x200 [10368.320215] ? set_kthread_struct+0x70/0x70 [10368.322049] ret_from_fork+0x1f/0x30 [10376.055018] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [10376.057021] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [10376.058101] Modules linked in: [10376.060872] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 [10376.062103] lustre(O) [10376.062105] pcspkr squashfs ata_generic [10376.074250] osp(O) [10376.075037] crct10dif_pclmul [10376.076675] ofd(O) lod(O) mdt(O) mdd(O) [10376.077950] crc32_pclmul [10376.079170] mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul [10376.081265] crc32c_intel [10376.082140] crc32c_intel [10376.096754] ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10376.096779] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10376.096785] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10376.096788] RIP: 0010:native_safe_halt+0x2e/0x50 [10376.096801] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10376.096805] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10376.096809] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10376.096811] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [10376.096814] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [10376.096816] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [10376.096819] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [10376.096822] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [10376.096825] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10376.096827] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [10376.096837] Call Trace: [10376.096842] [10376.096847] ? show_regs.cold.9+0x22/0x2f [10376.096856] ? watchdog_timer_fn.cold.6+0x62/0x10a [10376.096864] ? __hrtimer_run_queues+0x1c2/0x4d0 [10376.096868] ? watchdog+0x40/0x40 [10376.096872] ? hrtimer_interrupt+0x128/0x2c0 [10376.096876] ? smp_apic_timer_interrupt+0xa6/0x270 [10376.096884] ? apic_timer_interrupt+0xf/0x20 [10376.096888] [10376.096891] ? native_safe_halt+0x2e/0x50 [10376.096895] kvm_wait+0xcd/0xe0 [10376.096902] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10376.096912] do_raw_spin_lock+0x177/0x190 [10376.096916] _raw_spin_lock+0x21/0x30 [10376.096920] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10376.097243] ? do_raw_spin_unlock+0x75/0x190 [10376.097253] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10376.097667] ? do_wait_intr+0xf0/0xf0 [10376.097675] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10376.099019] ata_piix [10376.100319] mdc_close+0x53b/0xd70 [mdc] [10376.110752] ghash_clmulni_intel [10376.115885] lmv_close+0x18d/0x490 [lmv] [10376.123009] serio_raw [10376.125337] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10376.142237] libata [10376.145619] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10376.157766] dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10376.157788] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10376.157795] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10376.157798] RIP: 0010:native_safe_halt+0x2e/0x50 [10376.157812] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10376.157816] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10376.157821] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [10376.157823] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [10376.157825] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [10376.157827] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [10376.157830] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10376.157834] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [10376.157837] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10376.157839] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10376.157848] Call Trace: [10376.157853] [10376.157859] ? show_regs.cold.9+0x22/0x2f [10376.157869] ? watchdog_timer_fn.cold.6+0x62/0x10a [10376.157876] ? __hrtimer_run_queues+0x1c2/0x4d0 [10376.157880] ? watchdog+0x40/0x40 [10376.157884] ? hrtimer_interrupt+0x128/0x2c0 [10376.157888] ? smp_apic_timer_interrupt+0xa6/0x270 [10376.157897] ? apic_timer_interrupt+0xf/0x20 [10376.157901] [10376.157904] ? native_safe_halt+0x2e/0x50 [10376.157908] kvm_wait+0xcd/0xe0 [10376.157916] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10376.157923] do_raw_spin_lock+0x177/0x190 [10376.157927] _raw_spin_lock+0x21/0x30 [10376.157933] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10376.158198] obd_mod_free+0x46/0x140 [ptlrpc] [10376.158406] mdc_commit_open+0x91/0xb0 [mdc] [10376.158444] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10376.158655] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10376.158855] after_reply+0xe2c/0x1780 [ptlrpc] [10376.159057] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10376.159259] ? __next_timer_interrupt+0x160/0x160 [10376.159269] ptlrpcd+0xa82/0xe40 [ptlrpc] [10376.159479] ? do_wait_intr+0xf0/0xf0 [10376.159487] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10376.166114] ll_md_real_close+0xa5/0x2a0 [lustre] [10376.166235] ll_md_close+0x274/0x800 [lustre] [10376.166327] ll_file_release+0x128/0xc50 [lustre] [10376.166407] __fput+0x10d/0x440 [10376.166416] ____fput+0x12/0x20 [10376.166420] task_work_run+0xc6/0x110 [10376.166428] exit_to_usermode_loop+0x1dd/0x1f0 [10376.166436] do_syscall_64+0x3d6/0x440 [10376.166440] entry_SYSCALL_64_after_hwframe+0x49/0xae [10376.166449] RIP: 0033:0x7f151bee3cd8 [10376.166457] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10376.166461] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10376.166466] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [10376.166468] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [10376.166470] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10376.166473] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10376.166475] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10376.424259] kthread+0x1d1/0x200 [10376.426095] ? set_kthread_struct+0x70/0x70 [10376.428313] ret_from_fork+0x1f/0x30 [10392.052154] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [10392.055929] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10392.098119] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10392.111919] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10392.121940] RIP: 0010:native_safe_halt+0x2e/0x50 [10392.126898] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10392.141121] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10392.145712] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10392.149542] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [10392.154185] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [10392.163758] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [10392.171882] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [10392.177952] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [10392.185031] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10392.194029] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [10392.208503] Call Trace: [10392.214098] [10392.214882] ? show_regs.cold.9+0x22/0x2f [10392.216286] ? watchdog_timer_fn.cold.6+0x62/0x10a [10392.230202] ? __hrtimer_run_queues+0x1c2/0x4d0 [10392.239952] ? watchdog+0x40/0x40 [10392.242496] ? hrtimer_interrupt+0x128/0x2c0 [10392.246026] ? smp_apic_timer_interrupt+0xa6/0x270 [10392.248886] ? apic_timer_interrupt+0xf/0x20 [10392.254109] [10392.257151] ? native_safe_halt+0x2e/0x50 [10392.260129] kvm_wait+0xcd/0xe0 [10392.262075] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10392.267570] do_raw_spin_lock+0x177/0x190 [10392.270676] _raw_spin_lock+0x21/0x30 [10392.274446] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10392.280173] ? do_raw_spin_unlock+0x75/0x190 [10392.287523] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10392.296174] ? do_wait_intr+0xf0/0xf0 [10392.298984] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10392.302889] mdc_reint+0x2d/0x110 [mdc] [10392.305665] mdc_unlink+0x658/0x870 [mdc] [10392.309764] lmv_unlink+0x34d/0xbd0 [lmv] [10392.312200] ll_unlink+0x24d/0x7f0 [lustre] [10392.314240] vfs_unlink+0x202/0x330 [10392.324192] do_unlinkat+0x2fa/0x460 [10392.328470] __x64_sys_unlink+0x37/0x50 [10392.333123] do_syscall_64+0xc1/0x440 [10392.338518] entry_SYSCALL_64_after_hwframe+0x49/0xae [10392.343825] RIP: 0033:0x7f151bdee10b [10392.345138] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10392.370946] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10392.383868] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [10392.393230] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [10392.400388] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10392.404832] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10392.411551] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10395.104042] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8464s! [10395.106656] Showing busy workqueues and worker pools: [10395.109329] workqueue events: flags=0x0 [10395.111338] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10395.116181] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [10395.120296] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10395.123397] pending: kfree_rcu_monitor [10395.125034] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10395.127958] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [10395.131808] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10395.134430] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [10395.137396] workqueue events_long: flags=0x0 [10395.139156] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10395.141744] pending: lu_objects_flush [obdclass] [10395.143967] workqueue events_unbound: flags=0x2 [10395.145514] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10395.148040] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10395.151828] workqueue events_power_efficient: flags=0x80 [10395.154047] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10395.156563] pending: do_cache_clean, neigh_periodic_work, check_lifetime [10395.159285] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10395.162290] pending: neigh_periodic_work [10395.163814] workqueue mm_percpu_wq: flags=0x8 [10395.165576] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10395.168237] pending: vmstat_update, lru_add_drain_per_cpu [10395.170865] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10395.173559] pending: vmstat_update [10395.174953] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10395.177779] pending: lru_add_drain_per_cpu [10395.179774] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10395.182194] pending: lru_add_drain_per_cpu BAR(48) [10395.184388] workqueue writeback: flags=0x4a [10395.186215] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [10395.188493] pending: wb_workfn [10395.189825] workqueue xprtiod: flags=0xa [10395.191689] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10395.194365] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [10395.197939] workqueue ipv6_addrconf: flags=0x40008 [10395.199875] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10395.202540] pending: addrconf_verify_work [10395.204198] workqueue knbd0-recv: flags=0x1a [10395.206066] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10395.208546] in-flight: 658:recv_work [10395.210324] workqueue ptlrpc_pinger: flags=0x2 [10395.212172] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10395.214981] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10395.217313] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8464s workers=4 idle: 32984 37 8 [10395.220390] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10379s workers=2 idle: 661 [10396.056127] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [10396.061466] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10396.092988] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10396.099890] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10396.106452] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10396.110477] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10396.114463] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10396.128323] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10396.133404] RAX: 000000000000448b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10396.138397] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [10396.146368] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [10396.151406] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [10396.156162] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [10396.159910] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [10396.163868] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10396.176524] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10396.180390] Call Trace: [10396.181410] [10396.182831] ? show_regs.cold.9+0x22/0x2f [10396.189739] ? watchdog_timer_fn.cold.6+0x62/0x10a [10396.192851] ? __hrtimer_run_queues+0x1c2/0x4d0 [10396.195161] ? watchdog+0x40/0x40 [10396.195983] ? hrtimer_interrupt+0x128/0x2c0 [10396.197670] ? smp_apic_timer_interrupt+0xa6/0x270 [10396.198921] ? apic_timer_interrupt+0xf/0x20 [10396.200443] [10396.201328] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10396.203444] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10396.206398] do_raw_spin_lock+0x177/0x190 [10396.207769] _raw_spin_lock+0x21/0x30 [10396.208748] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10396.210392] process_one_work+0x2c8/0x700 [10396.211381] worker_thread+0x296/0x6e0 [10396.212315] ? rescuer_thread+0x570/0x570 [10396.214099] kthread+0x1d1/0x200 [10396.215292] ? set_kthread_struct+0x70/0x70 [10396.216888] ret_from_fork+0x1f/0x30 [10404.055037] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [10404.057009] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [10404.057765] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) [10404.061004] Modules linked in: [10404.061007] osc(O) lmv(O) ec(O) [10404.067107] lustre(O) [10404.068736] fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10404.068798] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10404.068805] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10404.068808] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10404.068823] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10404.068827] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10404.068831] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [10404.068834] RDX: 0000000000001da2 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [10404.068837] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [10404.068843] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [10404.068845] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10404.068849] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [10404.068852] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10404.068854] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10404.068863] Call Trace: [10404.068869] [10404.068874] ? show_regs.cold.9+0x22/0x2f [10404.068883] ? watchdog_timer_fn.cold.6+0x62/0x10a [10404.068890] ? __hrtimer_run_queues+0x1c2/0x4d0 [10404.068894] ? watchdog+0x40/0x40 [10404.068909] ? hrtimer_interrupt+0x128/0x2c0 [10404.068914] ? smp_apic_timer_interrupt+0xa6/0x270 [10404.068922] ? apic_timer_interrupt+0xf/0x20 [10404.068926] [10404.068930] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10404.068935] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10404.068940] do_raw_spin_lock+0x177/0x190 [10404.068944] _raw_spin_lock+0x21/0x30 [10404.068950] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10404.069228] obd_mod_free+0x46/0x140 [ptlrpc] [10404.069437] mdc_commit_open+0x91/0xb0 [mdc] [10404.069474] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10404.069675] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10404.069872] after_reply+0xe2c/0x1780 [ptlrpc] [10404.070080] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10404.070284] ? __next_timer_interrupt+0x160/0x160 [10404.070294] ptlrpcd+0xa82/0xe40 [ptlrpc] [10404.070516] ? do_wait_intr+0xf0/0xf0 [10404.070526] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10404.070733] kthread+0x1d1/0x200 [10404.070738] ? set_kthread_struct+0x70/0x70 [10404.070741] ret_from_fork+0x1f/0x30 [10404.234908] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10404.253978] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10404.258852] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10404.262808] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10404.266666] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10404.277003] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10404.280633] RAX: 0000000000005cfe RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10404.283738] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [10404.287321] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [10404.290821] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [10404.294120] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [10404.297748] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [10404.301490] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10404.303886] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [10404.307371] Call Trace: [10404.308005] [10404.308937] ? show_regs.cold.9+0x22/0x2f [10404.311003] ? watchdog_timer_fn.cold.6+0x62/0x10a [10404.313383] ? __hrtimer_run_queues+0x1c2/0x4d0 [10404.315521] ? watchdog+0x40/0x40 [10404.316673] ? hrtimer_interrupt+0x128/0x2c0 [10404.318010] ? smp_apic_timer_interrupt+0xa6/0x270 [10404.320317] ? apic_timer_interrupt+0xf/0x20 [10404.322065] [10404.323237] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10404.325689] do_raw_spin_lock+0x177/0x190 [10404.327554] _raw_spin_lock+0x21/0x30 [10404.329397] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10404.332020] ? do_raw_spin_unlock+0x75/0x190 [10404.334417] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10404.337323] ? do_wait_intr+0xf0/0xf0 [10404.338984] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10404.341646] mdc_close+0x53b/0xd70 [mdc] [10404.343746] lmv_close+0x18d/0x490 [lmv] [10404.345608] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10404.348373] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10404.351329] ll_md_real_close+0xa5/0x2a0 [lustre] [10404.353622] ll_md_close+0x274/0x800 [lustre] [10404.355677] ll_file_release+0x128/0xc50 [lustre] [10404.358730] __fput+0x10d/0x440 [10404.360052] ____fput+0x12/0x20 [10404.361669] task_work_run+0xc6/0x110 [10404.363338] exit_to_usermode_loop+0x1dd/0x1f0 [10404.365439] do_syscall_64+0x3d6/0x440 [10404.367022] entry_SYSCALL_64_after_hwframe+0x49/0xae [10404.369459] RIP: 0033:0x7f151bee3cd8 [10404.370889] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10404.380123] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10404.383473] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [10404.386595] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [10404.389848] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10404.393230] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10404.396785] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10420.052612] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [10420.058289] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10420.092459] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10420.098612] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10420.106986] RIP: 0010:native_safe_halt+0x2e/0x50 [10420.109957] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10420.126435] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10420.131236] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10420.134435] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [10420.139954] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [10420.146877] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [10420.150941] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [10420.156889] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [10420.162043] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10420.163954] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [10420.171423] Call Trace: [10420.173286] [10420.174274] ? show_regs.cold.9+0x22/0x2f [10420.176423] ? watchdog_timer_fn.cold.6+0x62/0x10a [10420.178775] ? __hrtimer_run_queues+0x1c2/0x4d0 [10420.181297] ? watchdog+0x40/0x40 [10420.182667] ? hrtimer_interrupt+0x128/0x2c0 [10420.184467] ? smp_apic_timer_interrupt+0xa6/0x270 [10420.190201] ? apic_timer_interrupt+0xf/0x20 [10420.194272] [10420.196112] ? native_safe_halt+0x2e/0x50 [10420.197813] kvm_wait+0xcd/0xe0 [10420.199359] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10420.202267] do_raw_spin_lock+0x177/0x190 [10420.203853] _raw_spin_lock+0x21/0x30 [10420.205600] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10420.210389] ? do_raw_spin_unlock+0x75/0x190 [10420.211745] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10420.213405] ? do_wait_intr+0xf0/0xf0 [10420.218325] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10420.221441] mdc_reint+0x2d/0x110 [mdc] [10420.223456] mdc_unlink+0x658/0x870 [mdc] [10420.225668] lmv_unlink+0x34d/0xbd0 [lmv] [10420.227594] ll_unlink+0x24d/0x7f0 [lustre] [10420.229668] vfs_unlink+0x202/0x330 [10420.231501] do_unlinkat+0x2fa/0x460 [10420.234377] __x64_sys_unlink+0x37/0x50 [10420.238228] do_syscall_64+0xc1/0x440 [10420.242701] entry_SYSCALL_64_after_hwframe+0x49/0xae [10420.246901] RIP: 0033:0x7f151bdee10b [10420.248015] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10420.274742] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10420.277694] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [10420.289768] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [10420.295463] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10420.300423] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10420.304354] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10424.057529] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [10424.067188] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10424.153547] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10424.168796] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10424.179768] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10424.188153] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10424.190937] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10424.217759] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10424.225949] RAX: 0000000000002983 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10424.235727] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [10424.244663] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [10424.251423] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [10424.254973] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [10424.267158] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [10424.270009] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10424.273530] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10424.277055] Call Trace: [10424.278655] [10424.280174] ? show_regs.cold.9+0x22/0x2f [10424.281994] ? watchdog_timer_fn.cold.6+0x62/0x10a [10424.284219] ? __hrtimer_run_queues+0x1c2/0x4d0 [10424.286153] ? watchdog+0x40/0x40 [10424.287662] ? hrtimer_interrupt+0x128/0x2c0 [10424.289985] ? smp_apic_timer_interrupt+0xa6/0x270 [10424.292349] ? apic_timer_interrupt+0xf/0x20 [10424.294282] [10424.295137] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10424.297623] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10424.299775] do_raw_spin_lock+0x177/0x190 [10424.301402] _raw_spin_lock+0x21/0x30 [10424.304117] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10424.306505] process_one_work+0x2c8/0x700 [10424.308094] worker_thread+0x296/0x6e0 [10424.310270] ? rescuer_thread+0x570/0x570 [10424.312279] kthread+0x1d1/0x200 [10424.314415] ? set_kthread_struct+0x70/0x70 [10424.320852] ret_from_fork+0x1f/0x30 [10425.312150] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8494s! [10425.316489] Showing busy workqueues and worker pools: [10425.318627] workqueue events: flags=0x0 [10425.320752] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10425.323795] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [10425.327618] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10425.330536] pending: kfree_rcu_monitor [10425.331669] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10425.334972] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [10425.338901] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10425.341431] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [10425.344420] workqueue events_long: flags=0x0 [10425.346189] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10425.348183] pending: lu_objects_flush [obdclass] [10425.349646] workqueue events_unbound: flags=0x2 [10425.351228] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10425.353551] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10425.357762] workqueue events_power_efficient: flags=0x80 [10425.360046] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10425.362273] pending: do_cache_clean, neigh_periodic_work, check_lifetime [10425.364768] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10425.367180] pending: neigh_periodic_work [10425.368454] workqueue mm_percpu_wq: flags=0x8 [10425.370029] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10425.373408] pending: vmstat_update, lru_add_drain_per_cpu [10425.375312] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10425.377385] pending: vmstat_update [10425.378589] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10425.380521] pending: lru_add_drain_per_cpu [10425.381765] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10425.384364] pending: lru_add_drain_per_cpu BAR(48) [10425.386419] workqueue writeback: flags=0x4a [10425.387899] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [10425.390162] pending: wb_workfn [10425.391332] workqueue xprtiod: flags=0xa [10425.395804] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10425.398993] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [10425.402990] workqueue ipv6_addrconf: flags=0x40008 [10425.407196] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10425.410063] pending: addrconf_verify_work [10425.412439] workqueue knbd0-recv: flags=0x1a [10425.414446] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10425.417616] in-flight: 658:recv_work [10425.419493] workqueue ptlrpc_pinger: flags=0x2 [10425.421449] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10425.425719] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10425.428632] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8494s workers=4 idle: 32984 37 8 [10425.433563] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10410s workers=2 idle: 661 [10432.055097] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [10432.057269] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [10432.059470] Modules linked in: [10432.067041] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10432.067192] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10432.067206] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10432.067209] RIP: 0010:native_safe_halt+0x2e/0x50 [10432.067225] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10432.067235] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10432.067240] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10432.067242] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [10432.067244] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [10432.067247] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [10432.067249] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [10432.067252] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [10432.067255] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10432.067258] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [10432.067267] Call Trace: [10432.067272] [10432.067286] ? show_regs.cold.9+0x22/0x2f [10432.067296] ? watchdog_timer_fn.cold.6+0x62/0x10a [10432.067303] ? __hrtimer_run_queues+0x1c2/0x4d0 [10432.067313] ? watchdog+0x40/0x40 [10432.067318] ? hrtimer_interrupt+0x128/0x2c0 [10432.067323] ? smp_apic_timer_interrupt+0xa6/0x270 [10432.067338] ? apic_timer_interrupt+0xf/0x20 [10432.067343] [10432.067346] ? native_safe_halt+0x2e/0x50 [10432.067351] kvm_wait+0xcd/0xe0 [10432.067358] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10432.067367] do_raw_spin_lock+0x177/0x190 [10432.067371] _raw_spin_lock+0x21/0x30 [10432.067375] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10432.068893] ? do_raw_spin_unlock+0x75/0x190 [10432.068906] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10432.069211] ? do_wait_intr+0xf0/0xf0 [10432.069219] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10432.069462] mdc_close+0x53b/0xd70 [mdc] [10432.069509] lmv_close+0x18d/0x490 [lmv] [10432.069540] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10432.069742] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10432.069878] ll_md_real_close+0xa5/0x2a0 [lustre] [10432.069978] ll_md_close+0x274/0x800 [lustre] [10432.070085] ll_file_release+0x128/0xc50 [lustre] [10432.070179] __fput+0x10d/0x440 [10432.070188] ____fput+0x12/0x20 [10432.070192] task_work_run+0xc6/0x110 [10432.070199] exit_to_usermode_loop+0x1dd/0x1f0 [10432.070208] do_syscall_64+0x3d6/0x440 [10432.070212] entry_SYSCALL_64_after_hwframe+0x49/0xae [10432.070220] RIP: 0033:0x7f151bee3cd8 [10432.070226] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10432.070231] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10432.070236] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [10432.070243] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [10432.070245] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10432.070247] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10432.070249] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10432.232449] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10432.248362] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10432.252408] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10432.255752] RIP: 0010:native_safe_halt+0x2e/0x50 [10432.257538] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10432.265438] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10432.268177] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [10432.270507] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [10432.272820] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [10432.275509] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [10432.278215] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10432.280608] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [10432.283212] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10432.285136] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10432.287707] Call Trace: [10432.288754] [10432.289384] ? show_regs.cold.9+0x22/0x2f [10432.290800] ? watchdog_timer_fn.cold.6+0x62/0x10a [10432.292820] ? __hrtimer_run_queues+0x1c2/0x4d0 [10432.294321] ? watchdog+0x40/0x40 [10432.297312] ? hrtimer_interrupt+0x128/0x2c0 [10432.299361] ? smp_apic_timer_interrupt+0xa6/0x270 [10432.301127] ? apic_timer_interrupt+0xf/0x20 [10432.302436] [10432.303292] ? native_safe_halt+0x2e/0x50 [10432.304795] kvm_wait+0xcd/0xe0 [10432.306045] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10432.307789] do_raw_spin_lock+0x177/0x190 [10432.309421] _raw_spin_lock+0x21/0x30 [10432.310925] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10432.312624] obd_mod_free+0x46/0x140 [ptlrpc] [10432.314407] mdc_commit_open+0x91/0xb0 [mdc] [10432.315618] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10432.317433] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10432.319301] after_reply+0xe2c/0x1780 [ptlrpc] [10432.321108] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10432.323126] ? __next_timer_interrupt+0x160/0x160 [10432.324529] ptlrpcd+0xa82/0xe40 [ptlrpc] [10432.326264] ? do_wait_intr+0xf0/0xf0 [10432.327408] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10432.329386] kthread+0x1d1/0x200 [10432.330519] ? set_kthread_struct+0x70/0x70 [10432.331803] ret_from_fork+0x1f/0x30 [10444.825025] rcu: INFO: rcu_sched self-detected stall on CPU [10444.827939] rcu: 1-...!: (4857335 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [10444.834295] (t=8520178 jiffies g=249657 q=4907) [10444.836917] rcu: rcu_sched kthread starved for 8515130 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [10444.843134] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [10444.850482] rcu: RCU grace-period kthread stack dump: [10444.853151] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [10444.859300] Call Trace: [10444.862871] __schedule+0x351/0xcb0 [10444.864898] schedule+0xc0/0x180 [10444.866696] schedule_timeout+0xb4/0x190 [10444.869319] ? do_raw_spin_unlock+0x75/0x190 [10444.874109] ? __next_timer_interrupt+0x160/0x160 [10444.877231] ? prepare_to_swait_event+0xf3/0x140 [10444.879294] rcu_gp_kthread+0x8dd/0x15b0 [10444.881087] ? force_qs_rnp+0x2c0/0x2c0 [10444.882706] kthread+0x1d1/0x200 [10444.884472] ? set_kthread_struct+0x70/0x70 [10444.886544] ret_from_fork+0x1f/0x30 [10444.888394] rcu: Stack dump where RCU GP kthread last ran: [10444.891573] Sending NMI from CPU 1 to CPUs 0: [10444.893693] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [10444.896808] NMI backtrace for cpu 1 [10444.898691] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10444.904562] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10444.909320] Call Trace: [10444.913151] [10444.914610] ? dump_stack+0xbb/0x10e [10444.917109] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [10444.919699] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [10444.922565] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [10444.925144] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [10444.927654] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [10444.930232] ? rcu_dump_cpu_stacks+0x11b/0x171 [10444.932616] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [10444.935199] ? update_process_times+0x6d/0xc0 [10444.937382] ? tick_sched_handle.isra.18+0x27/0x90 [10444.940267] ? tick_sched_timer+0x53/0xb0 [10444.943514] ? __hrtimer_run_queues+0x1c2/0x4d0 [10444.946098] ? tick_sched_handle.isra.18+0x90/0x90 [10444.948670] ? hrtimer_interrupt+0x128/0x2c0 [10444.951284] ? smp_apic_timer_interrupt+0xa6/0x270 [10444.955085] ? apic_timer_interrupt+0xf/0x20 [10444.957392] [10444.958639] ? native_safe_halt+0x2e/0x50 [10444.962203] ? kvm_wait+0xcd/0xe0 [10444.964286] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10444.967108] ? do_raw_spin_lock+0x177/0x190 [10444.971780] ? _raw_spin_lock+0x21/0x30 [10444.975167] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10444.978967] ? obd_mod_free+0x46/0x140 [ptlrpc] [10444.982425] ? mdc_commit_open+0x91/0xb0 [mdc] [10444.985198] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10444.988774] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10444.992893] ? after_reply+0xe2c/0x1780 [ptlrpc] [10445.000889] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10445.005993] ? __next_timer_interrupt+0x160/0x160 [10445.008567] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [10445.010881] ? do_wait_intr+0xf0/0xf0 [10445.013349] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10445.015635] ? kthread+0x1d1/0x200 [10445.016802] ? set_kthread_struct+0x70/0x70 [10445.018158] ? ret_from_fork+0x1f/0x30 [10452.056927] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [10452.059832] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10452.118650] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10452.128337] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10452.138748] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10452.149114] RIP: 0010:native_safe_halt+0x2e/0x50 [10452.152772] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10452.169175] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10452.184843] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10452.196065] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [10452.202534] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [10452.210151] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [10452.218518] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [10452.223256] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [10452.228400] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10452.232955] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10452.237242] Call Trace: [10452.240062] [10452.241642] ? show_regs.cold.9+0x22/0x2f [10452.244265] ? watchdog_timer_fn.cold.6+0x62/0x10a [10452.247668] ? __hrtimer_run_queues+0x1c2/0x4d0 [10452.249892] ? watchdog+0x40/0x40 [10452.253733] ? hrtimer_interrupt+0x128/0x2c0 [10452.257338] ? smp_apic_timer_interrupt+0xa6/0x270 [10452.260602] ? apic_timer_interrupt+0xf/0x20 [10452.267745] [10452.270265] ? native_safe_halt+0x2e/0x50 [10452.275349] kvm_wait+0xcd/0xe0 [10452.278377] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10452.284431] do_raw_spin_lock+0x177/0x190 [10452.287381] _raw_spin_lock+0x21/0x30 [10452.289976] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10452.295229] process_one_work+0x2c8/0x700 [10452.299312] worker_thread+0x296/0x6e0 [10452.302362] ? rescuer_thread+0x570/0x570 [10452.303857] kthread+0x1d1/0x200 [10452.306889] ? set_kthread_struct+0x70/0x70 [10452.309188] ret_from_fork+0x1f/0x30 [10455.522217] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8524s! [10455.525564] Showing busy workqueues and worker pools: [10455.528318] workqueue events: flags=0x0 [10455.530802] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10455.533766] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [10455.538098] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10455.541248] pending: kfree_rcu_monitor [10455.543091] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10455.546200] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [10455.552580] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10455.555942] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [10455.559972] workqueue events_long: flags=0x0 [10455.563240] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10455.566471] pending: lu_objects_flush [obdclass] [10455.568465] workqueue events_unbound: flags=0x2 [10455.570226] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10455.572176] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10455.576135] workqueue events_power_efficient: flags=0x80 [10455.578252] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10455.581469] pending: do_cache_clean, neigh_periodic_work, check_lifetime [10455.585052] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10455.588433] pending: neigh_periodic_work [10455.591332] workqueue mm_percpu_wq: flags=0x8 [10455.593718] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10455.596972] pending: vmstat_update, lru_add_drain_per_cpu [10455.599434] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10455.603226] pending: vmstat_update [10455.604922] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10455.608726] pending: lru_add_drain_per_cpu [10455.610738] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10455.613405] pending: lru_add_drain_per_cpu BAR(48) [10455.615412] workqueue writeback: flags=0x4a [10455.617257] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [10455.619401] pending: wb_workfn [10455.621027] workqueue xprtiod: flags=0xa [10455.622902] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10455.625444] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [10455.629026] workqueue ipv6_addrconf: flags=0x40008 [10455.631976] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10455.634883] pending: addrconf_verify_work [10455.636821] workqueue knbd0-recv: flags=0x1a [10455.638762] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10455.641915] in-flight: 658:recv_work [10455.643639] workqueue ptlrpc_pinger: flags=0x2 [10455.645474] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10455.648129] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10455.650383] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8525s workers=4 idle: 32984 37 8 [10455.654014] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10440s workers=2 idle: 661 [10460.060267] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [10460.070694] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10460.130494] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10460.157083] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10460.163553] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10460.167795] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10460.178848] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10460.188409] RAX: 0000000000007e1b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10460.195609] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [10460.207415] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [10460.224229] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [10460.237041] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [10460.247223] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [10460.253129] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10460.260065] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [10460.268130] Call Trace: [10460.269288] [10460.272079] ? show_regs.cold.9+0x22/0x2f [10460.276028] ? watchdog_timer_fn.cold.6+0x62/0x10a [10460.279952] ? __hrtimer_run_queues+0x1c2/0x4d0 [10460.283228] ? watchdog+0x40/0x40 [10460.289134] ? hrtimer_interrupt+0x128/0x2c0 [10460.295150] ? smp_apic_timer_interrupt+0xa6/0x270 [10460.305050] ? apic_timer_interrupt+0xf/0x20 [10460.310332] [10460.313724] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10460.317062] do_raw_spin_lock+0x177/0x190 [10460.319230] _raw_spin_lock+0x21/0x30 [10460.321731] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10460.325304] ? do_raw_spin_unlock+0x75/0x190 [10460.327927] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10460.331031] ? do_wait_intr+0xf0/0xf0 [10460.332963] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10460.336009] mdc_close+0x53b/0xd70 [mdc] [10460.338367] lmv_close+0x18d/0x490 [lmv] [10460.340922] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10460.345773] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10460.348601] ll_md_real_close+0xa5/0x2a0 [lustre] [10460.350946] ll_md_close+0x274/0x800 [lustre] [10460.353016] ll_file_release+0x128/0xc50 [lustre] [10460.355479] __fput+0x10d/0x440 [10460.357009] ____fput+0x12/0x20 [10460.358513] task_work_run+0xc6/0x110 [10460.362035] exit_to_usermode_loop+0x1dd/0x1f0 [10460.364424] do_syscall_64+0x3d6/0x440 [10460.368623] entry_SYSCALL_64_after_hwframe+0x49/0xae [10460.371216] RIP: 0033:0x7f151bee3cd8 [10460.372884] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10460.389138] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10460.396121] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [10460.400277] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [10460.404235] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10460.406899] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10460.410223] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10472.057851] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [10472.063195] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10472.097283] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10472.102862] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10472.113172] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10472.120185] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10472.134770] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10472.141078] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [10472.146461] RDX: 0000000000003d2f RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [10472.151313] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [10472.156401] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [10472.162418] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10472.168336] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [10472.177840] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10472.181264] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10472.184361] Call Trace: [10472.185418] [10472.186443] ? show_regs.cold.9+0x22/0x2f [10472.189304] ? watchdog_timer_fn.cold.6+0x62/0x10a [10472.192384] ? __hrtimer_run_queues+0x1c2/0x4d0 [10472.193677] ? watchdog+0x40/0x40 [10472.194507] ? hrtimer_interrupt+0x128/0x2c0 [10472.195560] ? smp_apic_timer_interrupt+0xa6/0x270 [10472.200907] ? apic_timer_interrupt+0xf/0x20 [10472.204164] [10472.204996] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10472.216176] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10472.219769] do_raw_spin_lock+0x177/0x190 [10472.221956] _raw_spin_lock+0x21/0x30 [10472.225825] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10472.227513] obd_mod_free+0x46/0x140 [ptlrpc] [10472.230580] mdc_commit_open+0x91/0xb0 [mdc] [10472.233911] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10472.240173] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10472.243539] after_reply+0xe2c/0x1780 [ptlrpc] [10472.246555] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10472.250024] ? __next_timer_interrupt+0x160/0x160 [10472.253445] ptlrpcd+0xa82/0xe40 [ptlrpc] [10472.257724] ? do_wait_intr+0xf0/0xf0 [10472.259776] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10472.265409] kthread+0x1d1/0x200 [10472.267864] ? set_kthread_struct+0x70/0x70 [10472.269662] ret_from_fork+0x1f/0x30 [10480.054179] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [10480.056035] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [10480.062936] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10480.063019] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10480.063027] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10480.063030] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10480.063043] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10480.063047] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10480.063053] RAX: 0000000000007d85 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10480.070503] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl [10480.113766] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [10480.113777] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [10480.113780] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [10480.113782] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [10480.113786] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [10480.113789] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10480.113791] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [10480.113798] Call Trace: [10480.113806] [10480.113813] ? show_regs.cold.9+0x22/0x2f [10480.113825] ? watchdog_timer_fn.cold.6+0x62/0x10a [10480.113832] ? __hrtimer_run_queues+0x1c2/0x4d0 [10480.113835] ? watchdog+0x40/0x40 [10480.113839] ? hrtimer_interrupt+0x128/0x2c0 [10480.113843] ? smp_apic_timer_interrupt+0xa6/0x270 [10480.113850] ? apic_timer_interrupt+0xf/0x20 [10480.113854] [10480.113857] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10480.113863] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10480.113867] do_raw_spin_lock+0x177/0x190 [10480.113872] _raw_spin_lock+0x21/0x30 [10480.113879] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10480.129107] i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10480.129142] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10480.129155] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10480.129161] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10480.130259] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10480.130277] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10480.130281] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10480.130286] RAX: 000000000000290e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10480.130288] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [10480.130290] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [10480.130292] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [10480.130295] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [10480.130298] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [10480.130301] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10480.130303] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10480.130312] Call Trace: [10480.130318] [10480.130324] ? show_regs.cold.9+0x22/0x2f [10480.130334] ? watchdog_timer_fn.cold.6+0x62/0x10a [10480.130340] ? __hrtimer_run_queues+0x1c2/0x4d0 [10480.130344] ? watchdog+0x40/0x40 [10480.130348] ? hrtimer_interrupt+0x128/0x2c0 [10480.130352] ? smp_apic_timer_interrupt+0xa6/0x270 [10480.130360] ? apic_timer_interrupt+0xf/0x20 [10480.130365] [10480.130368] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10480.130374] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10480.130379] do_raw_spin_lock+0x177/0x190 [10480.130397] _raw_spin_lock+0x21/0x30 [10480.130402] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10480.135200] ? do_raw_spin_unlock+0x75/0x190 [10480.135220] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10480.137305] ? do_wait_intr+0xf0/0xf0 [10480.137329] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10480.143687] process_one_work+0x2c8/0x700 [10480.143712] worker_thread+0x296/0x6e0 [10480.143717] ? rescuer_thread+0x570/0x570 [10480.143722] kthread+0x1d1/0x200 [10480.143726] ? set_kthread_struct+0x70/0x70 [10480.143730] ret_from_fork+0x1f/0x30 [10480.636022] mdc_reint+0x2d/0x110 [mdc] [10480.640679] mdc_unlink+0x658/0x870 [mdc] [10480.645839] lmv_unlink+0x34d/0xbd0 [lmv] [10480.650338] ll_unlink+0x24d/0x7f0 [lustre] [10480.657318] vfs_unlink+0x202/0x330 [10480.658628] do_unlinkat+0x2fa/0x460 [10480.665687] __x64_sys_unlink+0x37/0x50 [10480.674749] do_syscall_64+0xc1/0x440 [10480.679945] entry_SYSCALL_64_after_hwframe+0x49/0xae [10480.686651] RIP: 0033:0x7f151bdee10b [10480.694277] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10480.714436] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10480.721354] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [10480.727003] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [10480.741989] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10480.750972] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10480.757966] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10485.729038] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8555s! [10485.731660] Showing busy workqueues and worker pools: [10485.736133] workqueue events: flags=0x0 [10485.739209] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10485.744089] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [10485.749416] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10485.753893] pending: kfree_rcu_monitor [10485.756832] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10485.761673] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [10485.773140] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10485.777714] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [10485.781627] workqueue events_long: flags=0x0 [10485.785222] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10485.790697] pending: lu_objects_flush [obdclass] [10485.796739] workqueue events_unbound: flags=0x2 [10485.801305] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10485.804937] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10485.811958] workqueue events_power_efficient: flags=0x80 [10485.818863] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10485.826753] pending: do_cache_clean, neigh_periodic_work, check_lifetime [10485.830865] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10485.836687] pending: neigh_periodic_work [10485.838715] workqueue mm_percpu_wq: flags=0x8 [10485.842724] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10485.846285] pending: vmstat_update, lru_add_drain_per_cpu [10485.848808] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10485.851606] pending: vmstat_update [10485.853554] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10485.857230] pending: lru_add_drain_per_cpu [10485.859042] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10485.864163] pending: lru_add_drain_per_cpu BAR(48) [10485.868216] workqueue writeback: flags=0x4a [10485.870757] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [10485.873674] pending: wb_workfn [10485.876040] workqueue xprtiod: flags=0xa [10485.877777] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10485.881152] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [10485.885543] workqueue ipv6_addrconf: flags=0x40008 [10485.887289] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10485.890812] pending: addrconf_verify_work [10485.892668] workqueue knbd0-recv: flags=0x1a [10485.894613] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10485.896176] in-flight: 658:recv_work [10485.897344] workqueue ptlrpc_pinger: flags=0x2 [10485.900253] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10485.903664] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10485.906570] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8555s workers=4 idle: 32984 37 8 [10485.911201] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10470s workers=2 idle: 661 [10488.057014] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [10488.060041] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10488.080822] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10488.092211] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10488.099850] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10488.102587] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10488.121075] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10488.126483] RAX: 0000000000006a54 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10488.136171] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [10488.148555] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [10488.153276] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [10488.157137] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [10488.163259] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [10488.169630] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10488.171436] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [10488.185139] Call Trace: [10488.187542] [10488.188275] ? show_regs.cold.9+0x22/0x2f [10488.189430] ? watchdog_timer_fn.cold.6+0x62/0x10a [10488.196413] ? __hrtimer_run_queues+0x1c2/0x4d0 [10488.197914] ? watchdog+0x40/0x40 [10488.199847] ? hrtimer_interrupt+0x128/0x2c0 [10488.206027] ? smp_apic_timer_interrupt+0xa6/0x270 [10488.215114] ? apic_timer_interrupt+0xf/0x20 [10488.220821] [10488.222751] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10488.224473] do_raw_spin_lock+0x177/0x190 [10488.226534] _raw_spin_lock+0x21/0x30 [10488.228114] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10488.232020] ? do_raw_spin_unlock+0x75/0x190 [10488.233432] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10488.237242] ? do_wait_intr+0xf0/0xf0 [10488.238505] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10488.250351] mdc_close+0x53b/0xd70 [mdc] [10488.251472] lmv_close+0x18d/0x490 [lmv] [10488.257376] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10488.259994] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10488.267908] ll_md_real_close+0xa5/0x2a0 [lustre] [10488.270202] ll_md_close+0x274/0x800 [lustre] [10488.272159] ll_file_release+0x128/0xc50 [lustre] [10488.279908] __fput+0x10d/0x440 [10488.287116] ____fput+0x12/0x20 [10488.288450] task_work_run+0xc6/0x110 [10488.291847] exit_to_usermode_loop+0x1dd/0x1f0 [10488.293886] do_syscall_64+0x3d6/0x440 [10488.297450] entry_SYSCALL_64_after_hwframe+0x49/0xae [10488.302295] RIP: 0033:0x7f151bee3cd8 [10488.307137] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10488.328852] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10488.337097] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [10488.340845] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [10488.347249] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10488.354197] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10488.362993] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10500.058271] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [10500.064573] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10500.119340] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10500.135948] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10500.149486] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10500.166843] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10500.188394] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10500.205491] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [10500.213531] RDX: 00000000000008cb RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [10500.220090] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [10500.230814] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [10500.233295] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10500.265040] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [10500.267905] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10500.281773] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10500.302105] Call Trace: [10500.302941] [10500.303604] ? show_regs.cold.9+0x22/0x2f [10500.321941] ? watchdog_timer_fn.cold.6+0x62/0x10a [10500.330898] ? __hrtimer_run_queues+0x1c2/0x4d0 [10500.338079] ? watchdog+0x40/0x40 [10500.342072] ? hrtimer_interrupt+0x128/0x2c0 [10500.344144] ? smp_apic_timer_interrupt+0xa6/0x270 [10500.349966] ? apic_timer_interrupt+0xf/0x20 [10500.360501] [10500.366584] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10500.376800] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10500.387670] do_raw_spin_lock+0x177/0x190 [10500.398455] _raw_spin_lock+0x21/0x30 [10500.409492] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10500.420228] obd_mod_free+0x46/0x140 [ptlrpc] [10500.428696] mdc_commit_open+0x91/0xb0 [mdc] [10500.433145] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10500.446058] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10500.471142] after_reply+0xe2c/0x1780 [ptlrpc] [10500.476672] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10500.479542] ? __next_timer_interrupt+0x160/0x160 [10500.481770] ptlrpcd+0xa82/0xe40 [ptlrpc] [10500.483852] ? do_wait_intr+0xf0/0xf0 [10500.485342] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10500.493495] kthread+0x1d1/0x200 [10500.496896] ? set_kthread_struct+0x70/0x70 [10500.504169] ret_from_fork+0x1f/0x30 [10508.056047] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10508.059304] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10508.083393] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10508.098365] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10508.102442] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10508.106173] RIP: 0010:native_safe_halt+0x2e/0x50 [10508.107936] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10508.122874] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [10508.133509] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10508.144314] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [10508.157924] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [10508.160338] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [10508.169199] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [10508.184924] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [10508.192886] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10508.197066] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10508.210175] Call Trace: [10508.211075] [10508.211800] ? show_regs.cold.9+0x22/0x2f [10508.213238] ? watchdog_timer_fn.cold.6+0x62/0x10a [10508.223686] ? __hrtimer_run_queues+0x1c2/0x4d0 [10508.229311] ? watchdog+0x40/0x40 [10508.231392] ? hrtimer_interrupt+0x128/0x2c0 [10508.236941] ? smp_apic_timer_interrupt+0xa6/0x270 [10508.240435] ? apic_timer_interrupt+0xf/0x20 [10508.243330] [10508.244216] ? native_safe_halt+0x2e/0x50 [10508.247353] kvm_wait+0xcd/0xe0 [10508.251851] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10508.255554] do_raw_spin_lock+0x177/0x190 [10508.258456] _raw_spin_lock+0x21/0x30 [10508.262840] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10508.266450] process_one_work+0x2c8/0x700 [10508.270380] worker_thread+0x296/0x6e0 [10508.276579] ? rescuer_thread+0x570/0x570 [10508.280715] kthread+0x1d1/0x200 [10508.282309] ? set_kthread_struct+0x70/0x70 [10508.285083] ret_from_fork+0x1f/0x30 [10512.052085] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [10512.054784] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10512.086775] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10512.097882] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10512.103004] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10512.106191] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10512.114684] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10512.118060] RAX: 0000000000002adf RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10512.121368] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [10512.124473] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [10512.127805] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [10512.130659] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [10512.133637] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [10512.137425] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10512.139948] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [10512.143373] Call Trace: [10512.144069] [10512.144778] ? show_regs.cold.9+0x22/0x2f [10512.147922] ? watchdog_timer_fn.cold.6+0x62/0x10a [10512.153009] ? __hrtimer_run_queues+0x1c2/0x4d0 [10512.155725] ? watchdog+0x40/0x40 [10512.159441] ? hrtimer_interrupt+0x128/0x2c0 [10512.161038] ? smp_apic_timer_interrupt+0xa6/0x270 [10512.166414] ? apic_timer_interrupt+0xf/0x20 [10512.168672] [10512.170028] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10512.172761] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10512.175848] do_raw_spin_lock+0x177/0x190 [10512.178342] _raw_spin_lock+0x21/0x30 [10512.180396] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10512.183505] ? do_raw_spin_unlock+0x75/0x190 [10512.185512] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10512.188451] ? do_wait_intr+0xf0/0xf0 [10512.189643] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10512.191370] mdc_reint+0x2d/0x110 [mdc] [10512.192723] mdc_unlink+0x658/0x870 [mdc] [10512.194367] lmv_unlink+0x34d/0xbd0 [lmv] [10512.196369] ll_unlink+0x24d/0x7f0 [lustre] [10512.199798] vfs_unlink+0x202/0x330 [10512.202363] do_unlinkat+0x2fa/0x460 [10512.204775] __x64_sys_unlink+0x37/0x50 [10512.205832] do_syscall_64+0xc1/0x440 [10512.206972] entry_SYSCALL_64_after_hwframe+0x49/0xae [10512.209848] RIP: 0033:0x7f151bdee10b [10512.211463] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10512.223764] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10512.228198] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [10512.230502] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [10512.234857] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10512.237845] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10512.247357] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10515.941676] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8585s! [10515.952837] Showing busy workqueues and worker pools: [10515.956371] workqueue events: flags=0x0 [10515.958265] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10515.964555] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [10515.969354] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10515.972633] pending: kfree_rcu_monitor [10515.974770] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10515.979443] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [10515.986859] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10515.992185] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [10515.996859] workqueue events_long: flags=0x0 [10515.998895] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10516.001811] pending: lu_objects_flush [obdclass] [10516.004811] workqueue events_unbound: flags=0x2 [10516.006972] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10516.010447] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10516.014805] workqueue events_power_efficient: flags=0x80 [10516.016943] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10516.019682] pending: do_cache_clean, neigh_periodic_work, check_lifetime [10516.022830] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10516.025520] pending: neigh_periodic_work [10516.027318] workqueue mm_percpu_wq: flags=0x8 [10516.031553] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10516.035953] pending: vmstat_update, lru_add_drain_per_cpu [10516.038824] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10516.043418] pending: vmstat_update [10516.046988] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10516.051558] pending: lru_add_drain_per_cpu [10516.056833] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10516.060384] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [10516.061814] pending: lru_add_drain_per_cpu BAR(48) [10516.061864] workqueue writeback: flags=0x4a [10516.061869] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [10516.061879] pending: wb_workfn [10516.061893] workqueue xprtiod: flags=0xa [10516.061895] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10516.061900] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [10516.061920] workqueue ipv6_addrconf: flags=0x40008 [10516.061923] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10516.061933] pending: addrconf_verify_work [10516.061946] workqueue knbd0-recv: flags=0x1a [10516.061948] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10516.061955] in-flight: 658:recv_work [10516.061965] workqueue ptlrpc_pinger: flags=0x2 [10516.061968] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10516.061972] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10516.062284] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8585s workers=4 idle: 32984 37 8 [10516.062297] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10500s workers=2 idle: 661 [10516.167850] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10516.201711] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10516.207057] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10516.228776] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10516.241619] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10516.266905] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10516.269959] RAX: 0000000000001a08 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10516.274315] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [10516.276599] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [10516.278670] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [10516.285193] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [10516.290697] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [10516.292533] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10516.305523] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [10516.307222] Call Trace: [10516.307845] [10516.308868] ? show_regs.cold.9+0x22/0x2f [10516.310468] ? watchdog_timer_fn.cold.6+0x62/0x10a [10516.316846] ? __hrtimer_run_queues+0x1c2/0x4d0 [10516.319183] ? watchdog+0x40/0x40 [10516.320208] ? hrtimer_interrupt+0x128/0x2c0 [10516.323595] ? smp_apic_timer_interrupt+0xa6/0x270 [10516.328488] ? apic_timer_interrupt+0xf/0x20 [10516.330738] [10516.331790] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10516.333865] do_raw_spin_lock+0x177/0x190 [10516.335744] _raw_spin_lock+0x21/0x30 [10516.337554] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10516.340962] ? do_raw_spin_unlock+0x75/0x190 [10516.343758] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10516.353297] ? do_wait_intr+0xf0/0xf0 [10516.359151] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10516.368230] mdc_close+0x53b/0xd70 [mdc] [10516.371530] lmv_close+0x18d/0x490 [lmv] [10516.372848] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10516.374959] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10516.376919] ll_md_real_close+0xa5/0x2a0 [lustre] [10516.378959] ll_md_close+0x274/0x800 [lustre] [10516.382176] ll_file_release+0x128/0xc50 [lustre] [10516.388790] __fput+0x10d/0x440 [10516.390727] ____fput+0x12/0x20 [10516.391552] task_work_run+0xc6/0x110 [10516.392471] exit_to_usermode_loop+0x1dd/0x1f0 [10516.395232] do_syscall_64+0x3d6/0x440 [10516.397522] entry_SYSCALL_64_after_hwframe+0x49/0xae [10516.398861] RIP: 0033:0x7f151bee3cd8 [10516.400699] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10516.410343] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10516.413869] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [10516.416676] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [10516.420649] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10516.424529] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10516.428294] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10528.055105] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [10528.060777] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10528.104705] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10528.115319] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10528.120451] RIP: 0010:native_safe_halt+0x2e/0x50 [10528.131282] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10528.148968] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [10528.161150] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [10528.167257] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [10528.172164] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [10528.176617] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [10528.180870] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10528.192737] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [10528.210069] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10528.223190] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10528.230508] Call Trace: [10528.234977] [10528.236543] ? show_regs.cold.9+0x22/0x2f [10528.241413] ? watchdog_timer_fn.cold.6+0x62/0x10a [10528.244614] ? __hrtimer_run_queues+0x1c2/0x4d0 [10528.248404] ? watchdog+0x40/0x40 [10528.251299] ? hrtimer_interrupt+0x128/0x2c0 [10528.254271] ? smp_apic_timer_interrupt+0xa6/0x270 [10528.256959] ? apic_timer_interrupt+0xf/0x20 [10528.262301] [10528.263422] ? native_safe_halt+0x2e/0x50 [10528.266068] kvm_wait+0xcd/0xe0 [10528.269513] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10528.272673] do_raw_spin_lock+0x177/0x190 [10528.275715] _raw_spin_lock+0x21/0x30 [10528.279600] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10528.282645] obd_mod_free+0x46/0x140 [ptlrpc] [10528.286035] mdc_commit_open+0x91/0xb0 [mdc] [10528.287863] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10528.293687] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10528.296544] after_reply+0xe2c/0x1780 [ptlrpc] [10528.301647] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10528.304713] ? __next_timer_interrupt+0x160/0x160 [10528.307576] ptlrpcd+0xa82/0xe40 [ptlrpc] [10528.311158] ? do_wait_intr+0xf0/0xf0 [10528.313535] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10528.316520] kthread+0x1d1/0x200 [10528.318227] ? set_kthread_struct+0x70/0x70 [10528.320038] ret_from_fork+0x1f/0x30 [10536.056631] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10536.059265] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10536.079163] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10536.083799] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10536.087329] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10536.090015] RIP: 0010:native_safe_halt+0x2e/0x50 [10536.092145] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10536.100411] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10536.103988] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10536.106946] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [10536.109935] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [10536.113499] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [10536.116422] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [10536.119498] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [10536.123010] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10536.125529] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10536.128882] Call Trace: [10536.130045] [10536.130855] ? show_regs.cold.9+0x22/0x2f [10536.132632] ? watchdog_timer_fn.cold.6+0x62/0x10a [10536.135525] ? __hrtimer_run_queues+0x1c2/0x4d0 [10536.137434] ? watchdog+0x40/0x40 [10536.138618] ? hrtimer_interrupt+0x128/0x2c0 [10536.140695] ? smp_apic_timer_interrupt+0xa6/0x270 [10536.142723] ? apic_timer_interrupt+0xf/0x20 [10536.144782] [10536.145693] ? native_safe_halt+0x2e/0x50 [10536.147466] kvm_wait+0xcd/0xe0 [10536.148906] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10536.151647] do_raw_spin_lock+0x177/0x190 [10536.153439] _raw_spin_lock+0x21/0x30 [10536.155529] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10536.159505] process_one_work+0x2c8/0x700 [10536.161218] worker_thread+0x296/0x6e0 [10536.162837] ? rescuer_thread+0x570/0x570 [10536.164805] kthread+0x1d1/0x200 [10536.165965] ? set_kthread_struct+0x70/0x70 [10536.167942] ret_from_fork+0x1f/0x30 [10544.052372] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [10544.059828] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [10544.069347] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10544.069436] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10544.069444] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10544.069447] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10544.069462] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10544.069466] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10544.069470] RAX: 0000000000007d0b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10544.069472] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [10544.069475] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [10544.069477] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [10544.069479] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [10544.069483] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [10544.069486] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10544.069488] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [10544.069497] Call Trace: [10544.069502] [10544.069518] ? show_regs.cold.9+0x22/0x2f [10544.069528] ? watchdog_timer_fn.cold.6+0x62/0x10a [10544.069534] ? __hrtimer_run_queues+0x1c2/0x4d0 [10544.069538] ? watchdog+0x40/0x40 [10544.069542] ? hrtimer_interrupt+0x128/0x2c0 [10544.069546] ? smp_apic_timer_interrupt+0xa6/0x270 [10544.069554] ? apic_timer_interrupt+0xf/0x20 [10544.069559] [10544.069562] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10544.069567] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10544.069571] do_raw_spin_lock+0x177/0x190 [10544.069576] _raw_spin_lock+0x21/0x30 [10544.069582] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10544.070100] ? do_raw_spin_unlock+0x75/0x190 [10544.070107] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10544.070370] ? do_wait_intr+0xf0/0xf0 [10544.070379] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10544.070880] mdc_reint+0x2d/0x110 [mdc] [10544.070953] mdc_unlink+0x658/0x870 [mdc] [10544.070982] lmv_unlink+0x34d/0xbd0 [lmv] [10544.071015] ll_unlink+0x24d/0x7f0 [lustre] [10544.071131] vfs_unlink+0x202/0x330 [10544.071138] do_unlinkat+0x2fa/0x460 [10544.071143] __x64_sys_unlink+0x37/0x50 [10544.071146] do_syscall_64+0xc1/0x440 [10544.071153] entry_SYSCALL_64_after_hwframe+0x49/0xae [10544.071160] RIP: 0033:0x7f151bdee10b [10544.071167] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10544.071170] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10544.071175] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [10544.071178] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [10544.071180] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10544.071182] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10544.071184] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10544.331260] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10544.355395] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10544.361456] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10544.367566] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10544.370476] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10544.386262] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 [10544.394162] RAX: 0000000000004600 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10544.399872] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [10544.410378] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [10544.418970] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [10544.424207] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [10544.427716] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [10544.436016] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10544.439078] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [10544.442233] Call Trace: [10544.443749] [10544.444712] ? show_regs.cold.9+0x22/0x2f [10544.446843] ? watchdog_timer_fn.cold.6+0x62/0x10a [10544.451409] ? __hrtimer_run_queues+0x1c2/0x4d0 [10544.453150] ? watchdog+0x40/0x40 [10544.457275] ? hrtimer_interrupt+0x128/0x2c0 [10544.461035] ? smp_apic_timer_interrupt+0xa6/0x270 [10544.463970] ? apic_timer_interrupt+0xf/0x20 [10544.465934] [10544.467116] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10544.474079] do_raw_spin_lock+0x177/0x190 [10544.477624] _raw_spin_lock+0x21/0x30 [10544.479007] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10544.483752] ? do_raw_spin_unlock+0x75/0x190 [10544.485462] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10544.488277] ? do_wait_intr+0xf0/0xf0 [10544.489638] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10544.495750] mdc_close+0x53b/0xd70 [mdc] [10544.498737] lmv_close+0x18d/0x490 [lmv] [10544.500928] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10544.503991] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10544.514290] ll_md_real_close+0xa5/0x2a0 [lustre] [10544.518058] ll_md_close+0x274/0x800 [lustre] [10544.519383] ll_file_release+0x128/0xc50 [lustre] [10544.532284] __fput+0x10d/0x440 [10544.533241] ____fput+0x12/0x20 [10544.534569] task_work_run+0xc6/0x110 [10544.539841] exit_to_usermode_loop+0x1dd/0x1f0 [10544.542039] do_syscall_64+0x3d6/0x440 [10544.544689] entry_SYSCALL_64_after_hwframe+0x49/0xae [10544.547198] RIP: 0033:0x7f151bee3cd8 [10544.550025] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10544.559039] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10544.562440] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [10544.565272] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [10544.569597] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10544.572877] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10544.577086] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10546.148147] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8615s! [10546.159606] Showing busy workqueues and worker pools: [10546.163714] workqueue events: flags=0x0 [10546.166553] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10546.177364] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [10546.191504] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10546.204772] pending: kfree_rcu_monitor [10546.212332] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10546.221784] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [10546.232398] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10546.242939] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [10546.260466] workqueue events_long: flags=0x0 [10546.265204] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10546.270526] pending: lu_objects_flush [obdclass] [10546.273413] workqueue events_unbound: flags=0x2 [10546.275705] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10546.280395] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10546.285990] workqueue events_power_efficient: flags=0x80 [10546.289099] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10546.292659] pending: do_cache_clean, neigh_periodic_work, check_lifetime [10546.295870] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10546.298504] pending: neigh_periodic_work [10546.300327] workqueue mm_percpu_wq: flags=0x8 [10546.301914] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10546.304668] pending: vmstat_update, lru_add_drain_per_cpu [10546.307186] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10546.310231] pending: vmstat_update [10546.312465] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10546.316374] pending: lru_add_drain_per_cpu [10546.318656] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10546.321303] pending: lru_add_drain_per_cpu BAR(48) [10546.323543] workqueue writeback: flags=0x4a [10546.325364] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [10546.327960] pending: wb_workfn [10546.329419] workqueue xprtiod: flags=0xa [10546.331124] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10546.333862] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [10546.337486] workqueue ipv6_addrconf: flags=0x40008 [10546.339909] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10546.342464] pending: addrconf_verify_work [10546.344133] workqueue knbd0-recv: flags=0x1a [10546.345981] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10546.349548] in-flight: 658:recv_work [10546.350796] workqueue ptlrpc_pinger: flags=0x2 [10546.353103] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10546.356381] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10546.359205] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8615s workers=4 idle: 32984 37 8 [10546.363647] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10531s workers=2 idle: 661 [10556.055082] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [10556.058160] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10556.077667] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10556.082469] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10556.086179] RIP: 0010:native_safe_halt+0x2e/0x50 [10556.088352] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10556.095980] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10556.099082] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [10556.102105] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [10556.105172] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [10556.108166] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [10556.111338] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10556.114176] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [10556.117364] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10556.119611] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10556.122330] Call Trace: [10556.123449] [10556.124170] ? show_regs.cold.9+0x22/0x2f [10556.125695] ? watchdog_timer_fn.cold.6+0x62/0x10a [10556.127636] ? __hrtimer_run_queues+0x1c2/0x4d0 [10556.129715] ? watchdog+0x40/0x40 [10556.130743] ? hrtimer_interrupt+0x128/0x2c0 [10556.132312] ? smp_apic_timer_interrupt+0xa6/0x270 [10556.134078] ? apic_timer_interrupt+0xf/0x20 [10556.135724] [10556.136388] ? native_safe_halt+0x2e/0x50 [10556.138069] kvm_wait+0xcd/0xe0 [10556.139162] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10556.141268] do_raw_spin_lock+0x177/0x190 [10556.142766] _raw_spin_lock+0x21/0x30 [10556.144134] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10556.146345] obd_mod_free+0x46/0x140 [ptlrpc] [10556.148183] mdc_commit_open+0x91/0xb0 [mdc] [10556.149944] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10556.152281] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10556.154899] after_reply+0xe2c/0x1780 [ptlrpc] [10556.156710] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10556.159160] ? __next_timer_interrupt+0x160/0x160 [10556.160711] ptlrpcd+0xa82/0xe40 [ptlrpc] [10556.162878] ? do_wait_intr+0xf0/0xf0 [10556.164683] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10556.166861] kthread+0x1d1/0x200 [10556.168091] ? set_kthread_struct+0x70/0x70 [10556.169332] ret_from_fork+0x1f/0x30 [10564.058396] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10564.067581] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10564.130284] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10564.136793] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10564.142830] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10564.149132] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10564.154457] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10564.167994] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10564.171916] RAX: 000000000000769c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10564.185052] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [10564.186911] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [10564.192548] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [10564.196447] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [10564.203521] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [10564.212891] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10564.219469] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10564.223050] Call Trace: [10564.224673] [10564.226077] ? show_regs.cold.9+0x22/0x2f [10564.227858] ? watchdog_timer_fn.cold.6+0x62/0x10a [10564.231290] ? __hrtimer_run_queues+0x1c2/0x4d0 [10564.233845] ? watchdog+0x40/0x40 [10564.235507] ? hrtimer_interrupt+0x128/0x2c0 [10564.241057] ? smp_apic_timer_interrupt+0xa6/0x270 [10564.243434] ? apic_timer_interrupt+0xf/0x20 [10564.246098] [10564.247133] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10564.255199] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10564.258786] do_raw_spin_lock+0x177/0x190 [10564.260353] _raw_spin_lock+0x21/0x30 [10564.263539] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10564.268146] process_one_work+0x2c8/0x700 [10564.269161] worker_thread+0x296/0x6e0 [10564.280047] ? rescuer_thread+0x570/0x570 [10564.281077] kthread+0x1d1/0x200 [10564.281875] ? set_kthread_struct+0x70/0x70 [10564.285828] ret_from_fork+0x1f/0x30 [10572.052034] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [10572.055880] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10572.060498] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [10572.086687] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10572.091477] Modules linked in: [10572.100179] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10572.100190] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10572.100207] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10572.100211] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10572.100217] RAX: 0000000000006422 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10572.100219] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [10572.100222] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [10572.100224] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [10572.100227] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [10572.100230] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [10572.100234] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10572.100236] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [10572.100245] Call Trace: [10572.100250] [10572.100255] ? show_regs.cold.9+0x22/0x2f [10572.100265] ? watchdog_timer_fn.cold.6+0x62/0x10a [10572.100272] ? __hrtimer_run_queues+0x1c2/0x4d0 [10572.100276] ? watchdog+0x40/0x40 [10572.100281] ? hrtimer_interrupt+0x128/0x2c0 [10572.100286] ? smp_apic_timer_interrupt+0xa6/0x270 [10572.100305] ? apic_timer_interrupt+0xf/0x20 [10572.100310] [10572.100313] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10572.100319] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10572.100324] do_raw_spin_lock+0x177/0x190 [10572.100329] _raw_spin_lock+0x21/0x30 [10572.100337] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10572.100605] ? do_raw_spin_unlock+0x75/0x190 [10572.100611] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10572.100826] ? do_wait_intr+0xf0/0xf0 [10572.100834] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10572.101029] mdc_reint+0x2d/0x110 [mdc] [10572.101069] mdc_unlink+0x658/0x870 [mdc] [10572.101098] lmv_unlink+0x34d/0xbd0 [lmv] [10572.101130] ll_unlink+0x24d/0x7f0 [lustre] [10572.101248] vfs_unlink+0x202/0x330 [10572.101257] do_unlinkat+0x2fa/0x460 [10572.101261] __x64_sys_unlink+0x37/0x50 [10572.101265] do_syscall_64+0xc1/0x440 [10572.101272] entry_SYSCALL_64_after_hwframe+0x49/0xae [10572.101278] RIP: 0033:0x7f151bdee10b [10572.101284] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10572.101288] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10572.101293] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [10572.101296] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [10572.101298] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10572.101300] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10572.101302] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10572.285851] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10572.301000] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10572.305029] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10572.308478] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10572.310405] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10572.316013] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10572.318412] RAX: 00000000000049a8 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10572.320696] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [10572.322981] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [10572.325243] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [10572.328114] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [10572.330592] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [10572.333332] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10572.336594] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [10572.339286] Call Trace: [10572.340194] [10572.340794] ? show_regs.cold.9+0x22/0x2f [10572.342381] ? watchdog_timer_fn.cold.6+0x62/0x10a [10572.344017] ? __hrtimer_run_queues+0x1c2/0x4d0 [10572.345321] ? watchdog+0x40/0x40 [10572.346345] ? hrtimer_interrupt+0x128/0x2c0 [10572.347571] ? smp_apic_timer_interrupt+0xa6/0x270 [10572.349362] ? apic_timer_interrupt+0xf/0x20 [10572.351261] [10572.352132] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10572.354385] do_raw_spin_lock+0x177/0x190 [10572.356249] _raw_spin_lock+0x21/0x30 [10572.357759] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10572.360407] ? do_raw_spin_unlock+0x75/0x190 [10572.362328] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10572.364806] ? do_wait_intr+0xf0/0xf0 [10572.369006] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10572.372728] mdc_close+0x53b/0xd70 [mdc] [10572.375770] lmv_close+0x18d/0x490 [lmv] [10572.377084] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10572.379756] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10572.382125] ll_md_real_close+0xa5/0x2a0 [lustre] [10572.384229] ll_md_close+0x274/0x800 [lustre] [10572.386103] ll_file_release+0x128/0xc50 [lustre] [10572.388417] __fput+0x10d/0x440 [10572.389552] ____fput+0x12/0x20 [10572.390695] task_work_run+0xc6/0x110 [10572.392124] exit_to_usermode_loop+0x1dd/0x1f0 [10572.394132] do_syscall_64+0x3d6/0x440 [10572.395643] entry_SYSCALL_64_after_hwframe+0x49/0xae [10572.397667] RIP: 0033:0x7f151bee3cd8 [10572.399063] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10572.406796] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10572.409472] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [10572.412559] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [10572.415408] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10572.417753] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10572.419727] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10576.865080] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8646s! [10576.867754] Showing busy workqueues and worker pools: [10576.869344] workqueue events: flags=0x0 [10576.870891] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10576.874197] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [10576.877834] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10576.879936] pending: kfree_rcu_monitor [10576.881124] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10576.884337] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [10576.890053] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10576.892815] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [10576.897878] workqueue events_long: flags=0x0 [10576.899352] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10576.902720] pending: lu_objects_flush [obdclass] [10576.904699] workqueue events_unbound: flags=0x2 [10576.906070] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10576.909091] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10576.913237] workqueue events_power_efficient: flags=0x80 [10576.916986] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10576.919096] pending: do_cache_clean, neigh_periodic_work, check_lifetime [10576.921944] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10576.925324] pending: neigh_periodic_work [10576.926933] workqueue mm_percpu_wq: flags=0x8 [10576.928251] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10576.931360] pending: vmstat_update, lru_add_drain_per_cpu [10576.934636] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10576.936533] pending: vmstat_update [10576.938837] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10576.941928] pending: lru_add_drain_per_cpu [10576.943936] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10576.946011] pending: lru_add_drain_per_cpu BAR(48) [10576.947749] workqueue writeback: flags=0x4a [10576.949332] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [10576.953479] pending: wb_workfn [10576.954698] workqueue xprtiod: flags=0xa [10576.955924] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10576.959207] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [10576.962766] workqueue ipv6_addrconf: flags=0x40008 [10576.964366] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10576.966960] pending: addrconf_verify_work [10576.968252] workqueue knbd0-recv: flags=0x1a [10576.969872] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10576.972075] in-flight: 658:recv_work [10576.973525] workqueue ptlrpc_pinger: flags=0x2 [10576.975904] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10576.978961] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10576.981361] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8646s workers=4 idle: 32984 37 8 [10576.984442] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10561s workers=2 idle: 661 [10584.064427] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [10584.071938] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10584.125362] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10584.149797] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10584.166819] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10584.168949] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10584.216259] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10584.226933] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [10584.232604] RDX: 0000000000007a72 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [10584.244690] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [10584.254681] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [10584.257153] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10584.271513] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [10584.281951] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10584.286189] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10584.296351] Call Trace: [10584.303990] [10584.309907] ? show_regs.cold.9+0x22/0x2f [10584.317241] ? watchdog_timer_fn.cold.6+0x62/0x10a [10584.320118] ? __hrtimer_run_queues+0x1c2/0x4d0 [10584.322963] ? watchdog+0x40/0x40 [10584.324736] ? hrtimer_interrupt+0x128/0x2c0 [10584.327536] ? smp_apic_timer_interrupt+0xa6/0x270 [10584.330539] ? apic_timer_interrupt+0xf/0x20 [10584.335939] [10584.336985] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10584.342313] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10584.358309] do_raw_spin_lock+0x177/0x190 [10584.359733] _raw_spin_lock+0x21/0x30 [10584.361003] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10584.367862] obd_mod_free+0x46/0x140 [ptlrpc] [10584.370416] mdc_commit_open+0x91/0xb0 [mdc] [10584.373758] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10584.379120] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10584.381288] after_reply+0xe2c/0x1780 [ptlrpc] [10584.397285] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10584.410140] ? __next_timer_interrupt+0x160/0x160 [10584.421520] ptlrpcd+0xa82/0xe40 [ptlrpc] [10584.427550] ? do_wait_intr+0xf0/0xf0 [10584.436764] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10584.438614] kthread+0x1d1/0x200 [10584.446305] ? set_kthread_struct+0x70/0x70 [10584.451273] ret_from_fork+0x1f/0x30 [10592.062454] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10592.076151] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10592.107192] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10592.118154] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10592.125466] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10592.127931] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10592.134558] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10592.151315] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10592.165336] RAX: 0000000000007f84 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10592.178620] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [10592.187267] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [10592.200006] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [10592.202536] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [10592.216726] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [10592.230883] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10592.238216] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10592.242952] Call Trace: [10592.243901] [10592.246257] ? show_regs.cold.9+0x22/0x2f [10592.248631] ? watchdog_timer_fn.cold.6+0x62/0x10a [10592.255150] ? __hrtimer_run_queues+0x1c2/0x4d0 [10592.262710] ? watchdog+0x40/0x40 [10592.268180] ? hrtimer_interrupt+0x128/0x2c0 [10592.269580] ? smp_apic_timer_interrupt+0xa6/0x270 [10592.276285] ? apic_timer_interrupt+0xf/0x20 [10592.280100] [10592.282441] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10592.286562] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10592.290241] do_raw_spin_lock+0x177/0x190 [10592.295549] _raw_spin_lock+0x21/0x30 [10592.300594] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10592.311972] process_one_work+0x2c8/0x700 [10592.313506] worker_thread+0x296/0x6e0 [10592.316227] ? rescuer_thread+0x570/0x570 [10592.320177] kthread+0x1d1/0x200 [10592.325713] ? set_kthread_struct+0x70/0x70 [10592.327992] ret_from_fork+0x1f/0x30 [10600.057011] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [10600.066247] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10600.104161] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10600.111874] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10600.119628] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10600.122993] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10600.138843] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10600.143550] RAX: 000000000000275b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10600.146649] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [10600.150389] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [10600.153579] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [10600.158407] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [10600.162316] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [10600.169089] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10600.172438] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [10600.176177] Call Trace: [10600.177694] [10600.179523] ? show_regs.cold.9+0x22/0x2f [10600.181636] ? watchdog_timer_fn.cold.6+0x62/0x10a [10600.186828] ? __hrtimer_run_queues+0x1c2/0x4d0 [10600.189577] ? watchdog+0x40/0x40 [10600.191502] ? hrtimer_interrupt+0x128/0x2c0 [10600.193435] ? smp_apic_timer_interrupt+0xa6/0x270 [10600.198417] ? apic_timer_interrupt+0xf/0x20 [10600.200657] [10600.201656] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10600.204486] do_raw_spin_lock+0x177/0x190 [10600.206321] _raw_spin_lock+0x21/0x30 [10600.207912] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10600.211791] ? do_raw_spin_unlock+0x75/0x190 [10600.213202] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10600.216342] ? do_wait_intr+0xf0/0xf0 [10600.219110] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10600.221116] mdc_close+0x53b/0xd70 [mdc] [10600.223387] lmv_close+0x18d/0x490 [lmv] [10600.225111] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10600.230523] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10600.236907] ll_md_real_close+0xa5/0x2a0 [lustre] [10600.239287] ll_md_close+0x274/0x800 [lustre] [10600.241592] ll_file_release+0x128/0xc50 [lustre] [10600.246311] __fput+0x10d/0x440 [10600.250427] ____fput+0x12/0x20 [10600.253932] task_work_run+0xc6/0x110 [10600.255249] exit_to_usermode_loop+0x1dd/0x1f0 [10600.257610] do_syscall_64+0x3d6/0x440 [10600.259563] entry_SYSCALL_64_after_hwframe+0x49/0xae [10600.263464] RIP: 0033:0x7f151bee3cd8 [10600.267170] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10600.277909] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10600.282390] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [10600.285253] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [10600.287481] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10600.291957] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10600.297650] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10604.053732] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [10604.057451] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10604.087618] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10604.092561] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10604.096621] RIP: 0010:native_safe_halt+0x2e/0x50 [10604.098767] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10604.111654] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10604.124346] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10604.132769] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [10604.135031] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [10604.140392] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [10604.144500] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [10604.150839] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [10604.157270] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10604.160556] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [10604.168642] Call Trace: [10604.172307] [10604.174370] ? show_regs.cold.9+0x22/0x2f [10604.179647] ? watchdog_timer_fn.cold.6+0x62/0x10a [10604.187388] ? __hrtimer_run_queues+0x1c2/0x4d0 [10604.191102] ? watchdog+0x40/0x40 [10604.193540] ? hrtimer_interrupt+0x128/0x2c0 [10604.198598] ? smp_apic_timer_interrupt+0xa6/0x270 [10604.201663] ? apic_timer_interrupt+0xf/0x20 [10604.204004] [10604.205293] ? native_safe_halt+0x2e/0x50 [10604.207590] kvm_wait+0xcd/0xe0 [10604.211392] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10604.213274] do_raw_spin_lock+0x177/0x190 [10604.214240] _raw_spin_lock+0x21/0x30 [10604.219638] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10604.225532] ? do_raw_spin_unlock+0x75/0x190 [10604.228978] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10604.231622] ? do_wait_intr+0xf0/0xf0 [10604.236543] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10604.247495] mdc_reint+0x2d/0x110 [mdc] [10604.248762] mdc_unlink+0x658/0x870 [mdc] [10604.250132] lmv_unlink+0x34d/0xbd0 [lmv] [10604.253976] ll_unlink+0x24d/0x7f0 [lustre] [10604.255404] vfs_unlink+0x202/0x330 [10604.257342] do_unlinkat+0x2fa/0x460 [10604.261522] __x64_sys_unlink+0x37/0x50 [10604.262808] do_syscall_64+0xc1/0x440 [10604.268797] entry_SYSCALL_64_after_hwframe+0x49/0xae [10604.272489] RIP: 0033:0x7f151bdee10b [10604.274248] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10604.285068] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10604.289595] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [10604.293280] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [10604.296429] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10604.299074] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10604.303871] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10607.076764] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8676s! [10607.086859] Showing busy workqueues and worker pools: [10607.091091] workqueue events: flags=0x0 [10607.099089] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10607.112229] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [10607.122119] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10607.131381] pending: kfree_rcu_monitor [10607.134606] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10607.141826] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [10607.148110] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10607.152431] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [10607.160951] workqueue events_long: flags=0x0 [10607.165086] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10607.169581] pending: lu_objects_flush [obdclass] [10607.172815] workqueue events_unbound: flags=0x2 [10607.175310] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10607.182780] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10607.196669] workqueue events_power_efficient: flags=0x80 [10607.203308] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10607.209902] pending: do_cache_clean, neigh_periodic_work, check_lifetime [10607.212331] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10607.216240] pending: neigh_periodic_work [10607.219232] workqueue mm_percpu_wq: flags=0x8 [10607.221553] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10607.225215] pending: vmstat_update, lru_add_drain_per_cpu [10607.228661] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10607.232552] pending: vmstat_update [10607.234427] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10607.237717] pending: lru_add_drain_per_cpu [10607.242971] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10607.250395] pending: lru_add_drain_per_cpu BAR(48) [10607.256628] workqueue writeback: flags=0x4a [10607.258184] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [10607.269926] pending: wb_workfn [10607.273062] workqueue xprtiod: flags=0xa [10607.278233] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10607.282663] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [10607.287157] workqueue ipv6_addrconf: flags=0x40008 [10607.290455] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10607.294449] pending: addrconf_verify_work [10607.297991] workqueue knbd0-recv: flags=0x1a [10607.300118] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10607.303267] in-flight: 658:recv_work [10607.305559] workqueue ptlrpc_pinger: flags=0x2 [10607.307731] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10607.310383] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10607.312928] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8676s workers=4 idle: 32984 37 8 [10607.316075] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10592s workers=2 idle: 661 [10612.055501] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [10612.063457] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10612.087186] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10612.092508] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10612.098132] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10612.100736] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10612.110044] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10612.114279] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [10612.119334] RDX: 0000000000000557 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [10612.124942] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [10612.129048] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [10612.133934] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10612.138598] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [10612.142280] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10612.145183] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10612.149220] Call Trace: [10612.150315] [10612.151278] ? show_regs.cold.9+0x22/0x2f [10612.153208] ? watchdog_timer_fn.cold.6+0x62/0x10a [10612.154966] ? __hrtimer_run_queues+0x1c2/0x4d0 [10612.156514] ? watchdog+0x40/0x40 [10612.158016] ? hrtimer_interrupt+0x128/0x2c0 [10612.159534] ? smp_apic_timer_interrupt+0xa6/0x270 [10612.161451] ? apic_timer_interrupt+0xf/0x20 [10612.162927] [10612.163809] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10612.165851] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10612.167864] do_raw_spin_lock+0x177/0x190 [10612.169333] _raw_spin_lock+0x21/0x30 [10612.170672] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10612.172675] obd_mod_free+0x46/0x140 [ptlrpc] [10612.174755] mdc_commit_open+0x91/0xb0 [mdc] [10612.176488] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10612.178953] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10612.181206] after_reply+0xe2c/0x1780 [ptlrpc] [10612.182808] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10612.185016] ? __next_timer_interrupt+0x160/0x160 [10612.186971] ptlrpcd+0xa82/0xe40 [ptlrpc] [10612.188625] ? do_wait_intr+0xf0/0xf0 [10612.190076] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10612.192174] kthread+0x1d1/0x200 [10612.193207] ? set_kthread_struct+0x70/0x70 [10612.194711] ret_from_fork+0x1f/0x30 [10620.057361] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10620.067524] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10620.102932] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10620.108797] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10620.113301] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10620.116270] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10620.119485] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10620.127947] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10620.131614] RAX: 0000000000006c8d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10620.134711] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [10620.138094] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [10620.142198] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [10620.146532] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [10620.150394] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [10620.153562] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10620.156942] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10620.159254] Call Trace: [10620.160178] [10620.161193] ? show_regs.cold.9+0x22/0x2f [10620.162910] ? watchdog_timer_fn.cold.6+0x62/0x10a [10620.164847] ? __hrtimer_run_queues+0x1c2/0x4d0 [10620.167121] ? watchdog+0x40/0x40 [10620.169249] ? hrtimer_interrupt+0x128/0x2c0 [10620.171981] ? smp_apic_timer_interrupt+0xa6/0x270 [10620.174944] ? apic_timer_interrupt+0xf/0x20 [10620.181722] [10620.182494] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10620.185983] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10620.189461] do_raw_spin_lock+0x177/0x190 [10620.193155] _raw_spin_lock+0x21/0x30 [10620.196170] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10620.199831] process_one_work+0x2c8/0x700 [10620.202015] worker_thread+0x296/0x6e0 [10620.205135] ? rescuer_thread+0x570/0x570 [10620.206946] kthread+0x1d1/0x200 [10620.209778] ? set_kthread_struct+0x70/0x70 [10620.212971] ret_from_fork+0x1f/0x30 [10624.828013] rcu: INFO: rcu_sched self-detected stall on CPU [10624.830710] rcu: 1-...!: (4976762 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [10624.835230] (t=8700181 jiffies g=249657 q=4936) [10624.837170] rcu: rcu_sched kthread starved for 8695133 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [10624.841780] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [10624.844444] rcu: RCU grace-period kthread stack dump: [10624.845917] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [10624.849745] Call Trace: [10624.850803] __schedule+0x351/0xcb0 [10624.852431] schedule+0xc0/0x180 [10624.853745] schedule_timeout+0xb4/0x190 [10624.855401] ? do_raw_spin_unlock+0x75/0x190 [10624.858183] ? __next_timer_interrupt+0x160/0x160 [10624.861533] ? prepare_to_swait_event+0xf3/0x140 [10624.863742] rcu_gp_kthread+0x8dd/0x15b0 [10624.871803] ? force_qs_rnp+0x2c0/0x2c0 [10624.879500] kthread+0x1d1/0x200 [10624.881588] ? set_kthread_struct+0x70/0x70 [10624.883175] ret_from_fork+0x1f/0x30 [10624.886213] rcu: Stack dump where RCU GP kthread last ran: [10624.890941] NMI backtrace for cpu 1 [10624.892923] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10624.903888] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10624.912153] Call Trace: [10624.916700] [10624.920863] ? dump_stack+0xbb/0x10e [10624.925764] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [10624.928659] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [10624.934786] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [10624.941720] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [10624.944354] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [10624.946859] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [10624.953686] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [10624.959265] ? update_process_times+0x6d/0xc0 [10624.960962] ? tick_sched_handle.isra.18+0x27/0x90 [10624.969279] ? tick_sched_timer+0x53/0xb0 [10624.974389] ? __hrtimer_run_queues+0x1c2/0x4d0 [10624.976745] ? tick_sched_handle.isra.18+0x90/0x90 [10624.979930] ? hrtimer_interrupt+0x128/0x2c0 [10624.985838] ? smp_apic_timer_interrupt+0xa6/0x270 [10624.990644] ? apic_timer_interrupt+0xf/0x20 [10624.997438] [10625.000881] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10625.008595] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10625.012763] ? do_raw_spin_lock+0x177/0x190 [10625.016677] ? _raw_spin_lock+0x21/0x30 [10625.020066] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10625.027238] ? obd_mod_free+0x46/0x140 [ptlrpc] [10625.034698] ? mdc_commit_open+0x91/0xb0 [mdc] [10625.038565] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10625.044898] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10625.048494] ? after_reply+0xe2c/0x1780 [ptlrpc] [10625.051488] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10625.053949] ? __next_timer_interrupt+0x160/0x160 [10625.061276] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [10625.063969] ? do_wait_intr+0xf0/0xf0 [10625.065918] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10625.070650] ? kthread+0x1d1/0x200 [10625.075324] ? set_kthread_struct+0x70/0x70 [10625.077655] ? ret_from_fork+0x1f/0x30 [10625.083547] NMI backtrace for cpu 1 [10625.085841] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10625.093583] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10625.101534] Call Trace: [10625.103176] [10625.104629] ? dump_stack+0xbb/0x10e [10625.111951] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [10625.113670] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [10625.130993] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [10625.135717] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [10625.141836] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [10625.144852] ? rcu_dump_cpu_stacks+0x11b/0x171 [10625.151844] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [10625.155001] ? update_process_times+0x6d/0xc0 [10625.163873] ? tick_sched_handle.isra.18+0x27/0x90 [10625.169831] ? tick_sched_timer+0x53/0xb0 [10625.176579] ? __hrtimer_run_queues+0x1c2/0x4d0 [10625.186635] ? tick_sched_handle.isra.18+0x90/0x90 [10625.191277] ? hrtimer_interrupt+0x128/0x2c0 [10625.199905] ? smp_apic_timer_interrupt+0xa6/0x270 [10625.201795] ? apic_timer_interrupt+0xf/0x20 [10625.214009] [10625.214832] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10625.220759] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10625.227216] ? do_raw_spin_lock+0x177/0x190 [10625.230194] ? _raw_spin_lock+0x21/0x30 [10625.231647] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10625.235595] ? obd_mod_free+0x46/0x140 [ptlrpc] [10625.238494] ? mdc_commit_open+0x91/0xb0 [mdc] [10625.241676] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10625.253971] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10625.262327] ? after_reply+0xe2c/0x1780 [ptlrpc] [10625.268033] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10625.277430] ? __next_timer_interrupt+0x160/0x160 [10625.282576] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [10625.286943] ? do_wait_intr+0xf0/0xf0 [10625.290015] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10625.293776] ? kthread+0x1d1/0x200 [10625.297602] ? set_kthread_struct+0x70/0x70 [10625.299065] ? ret_from_fork+0x1f/0x30 [10628.057040] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [10628.061114] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10628.080900] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10628.085803] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10628.091154] RIP: 0010:native_safe_halt+0x2e/0x50 [10628.093722] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10628.103495] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10628.107632] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10628.111518] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [10628.114991] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [10628.117973] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [10628.120798] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [10628.123722] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [10628.126492] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10628.128638] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [10628.131536] Call Trace: [10628.132721] [10628.133488] ? show_regs.cold.9+0x22/0x2f [10628.135058] ? watchdog_timer_fn.cold.6+0x62/0x10a [10628.137278] ? __hrtimer_run_queues+0x1c2/0x4d0 [10628.139132] ? watchdog+0x40/0x40 [10628.140715] ? hrtimer_interrupt+0x128/0x2c0 [10628.142482] ? smp_apic_timer_interrupt+0xa6/0x270 [10628.144537] ? apic_timer_interrupt+0xf/0x20 [10628.146221] [10628.147191] ? native_safe_halt+0x2e/0x50 [10628.148664] kvm_wait+0xcd/0xe0 [10628.149879] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10628.151851] do_raw_spin_lock+0x177/0x190 [10628.153930] _raw_spin_lock+0x21/0x30 [10628.155647] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10628.158550] ? do_raw_spin_unlock+0x75/0x190 [10628.160938] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10628.162898] ? do_wait_intr+0xf0/0xf0 [10628.164235] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10628.166753] mdc_close+0x53b/0xd70 [mdc] [10628.168842] lmv_close+0x18d/0x490 [lmv] [10628.171328] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10628.175189] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10628.177776] ll_md_real_close+0xa5/0x2a0 [lustre] [10628.179772] ll_md_close+0x274/0x800 [lustre] [10628.181804] ll_file_release+0x128/0xc50 [lustre] [10628.183693] __fput+0x10d/0x440 [10628.184673] ____fput+0x12/0x20 [10628.185564] task_work_run+0xc6/0x110 [10628.186649] exit_to_usermode_loop+0x1dd/0x1f0 [10628.188922] do_syscall_64+0x3d6/0x440 [10628.190511] entry_SYSCALL_64_after_hwframe+0x49/0xae [10628.192777] RIP: 0033:0x7f151bee3cd8 [10628.194225] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10628.202435] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10628.205658] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [10628.208590] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [10628.211481] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10628.214483] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10628.216669] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10632.055115] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [10632.064263] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10632.102695] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10632.114753] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10632.124727] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10632.131484] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10632.146298] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10632.151257] RAX: 0000000000007f85 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10632.156710] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [10632.165206] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [10632.177924] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [10632.194743] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [10632.201664] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [10632.210750] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10632.213672] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [10632.217929] Call Trace: [10632.219236] [10632.220855] ? show_regs.cold.9+0x22/0x2f [10632.224990] ? watchdog_timer_fn.cold.6+0x62/0x10a [10632.226429] ? __hrtimer_run_queues+0x1c2/0x4d0 [10632.231403] ? watchdog+0x40/0x40 [10632.232812] ? hrtimer_interrupt+0x128/0x2c0 [10632.234795] ? smp_apic_timer_interrupt+0xa6/0x270 [10632.237225] ? apic_timer_interrupt+0xf/0x20 [10632.246564] [10632.247317] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10632.253267] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10632.260688] do_raw_spin_lock+0x177/0x190 [10632.265201] _raw_spin_lock+0x21/0x30 [10632.270094] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10632.272574] ? do_raw_spin_unlock+0x75/0x190 [10632.281346] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10632.287281] ? do_wait_intr+0xf0/0xf0 [10632.290061] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10632.298288] mdc_reint+0x2d/0x110 [mdc] [10632.300088] mdc_unlink+0x658/0x870 [mdc] [10632.302387] lmv_unlink+0x34d/0xbd0 [lmv] [10632.304259] ll_unlink+0x24d/0x7f0 [lustre] [10632.306180] vfs_unlink+0x202/0x330 [10632.307831] do_unlinkat+0x2fa/0x460 [10632.309708] __x64_sys_unlink+0x37/0x50 [10632.311624] do_syscall_64+0xc1/0x440 [10632.321214] entry_SYSCALL_64_after_hwframe+0x49/0xae [10632.325714] RIP: 0033:0x7f151bdee10b [10632.328254] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10632.336362] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10632.342258] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [10632.348255] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [10632.352681] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10632.355728] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10632.357976] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10637.793462] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8707s! [10637.797517] Showing busy workqueues and worker pools: [10637.800515] workqueue events: flags=0x0 [10637.803291] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10637.808006] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [10637.813502] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10637.818494] pending: kfree_rcu_monitor [10637.821166] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10637.825556] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [10637.836428] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10637.844527] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [10637.852467] workqueue events_long: flags=0x0 [10637.854745] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10637.858523] pending: lu_objects_flush [obdclass] [10637.862924] workqueue events_unbound: flags=0x2 [10637.864676] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10637.868352] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10637.875479] workqueue events_power_efficient: flags=0x80 [10637.878139] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10637.884083] pending: do_cache_clean, neigh_periodic_work, check_lifetime [10637.888046] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10637.892520] pending: neigh_periodic_work [10637.896747] workqueue mm_percpu_wq: flags=0x8 [10637.899137] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10637.908975] pending: vmstat_update, lru_add_drain_per_cpu [10637.913587] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10637.919361] pending: vmstat_update [10637.921832] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10637.927098] pending: lru_add_drain_per_cpu [10637.931051] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10637.937456] pending: lru_add_drain_per_cpu BAR(48) [10637.941369] workqueue writeback: flags=0x4a [10637.942846] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [10637.948706] pending: wb_workfn [10637.950733] workqueue xprtiod: flags=0xa [10637.955829] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10637.958680] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [10637.965458] workqueue ipv6_addrconf: flags=0x40008 [10637.967434] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10637.970169] pending: addrconf_verify_work [10637.971834] workqueue knbd0-recv: flags=0x1a [10637.973772] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10637.976615] in-flight: 658:recv_work [10637.978332] workqueue ptlrpc_pinger: flags=0x2 [10637.980053] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10637.987143] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10637.992559] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8707s workers=4 idle: 32984 37 8 [10637.998024] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10622s workers=2 idle: 661 [10648.057991] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10648.062515] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10648.115681] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10648.132522] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10648.140549] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10648.151475] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10648.153413] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10648.175262] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10648.180229] RAX: 0000000000002516 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10648.188968] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [10648.196858] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [10648.200567] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [10648.209338] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [10648.213728] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [10648.217573] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10648.220127] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10648.224932] Call Trace: [10648.226130] [10648.227097] ? show_regs.cold.9+0x22/0x2f [10648.233358] ? watchdog_timer_fn.cold.6+0x62/0x10a [10648.235377] ? __hrtimer_run_queues+0x1c2/0x4d0 [10648.242262] ? watchdog+0x40/0x40 [10648.244115] ? hrtimer_interrupt+0x128/0x2c0 [10648.248462] ? smp_apic_timer_interrupt+0xa6/0x270 [10648.251647] ? apic_timer_interrupt+0xf/0x20 [10648.253861] [10648.259332] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10648.262917] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10648.267557] do_raw_spin_lock+0x177/0x190 [10648.272708] _raw_spin_lock+0x21/0x30 [10648.276503] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10648.278568] process_one_work+0x2c8/0x700 [10648.281720] worker_thread+0x296/0x6e0 [10648.287832] ? rescuer_thread+0x570/0x570 [10648.292389] kthread+0x1d1/0x200 [10648.294214] ? set_kthread_struct+0x70/0x70 [10648.296222] ret_from_fork+0x1f/0x30 [10652.055012] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [10652.064929] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10652.107017] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10652.112890] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10652.122155] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10652.125522] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10652.132999] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10652.136955] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [10652.140210] RDX: 0000000000005d54 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [10652.144285] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [10652.150423] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [10652.156387] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10652.160703] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [10652.165613] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10652.172504] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10652.179083] Call Trace: [10652.179908] [10652.180656] ? show_regs.cold.9+0x22/0x2f [10652.187372] ? watchdog_timer_fn.cold.6+0x62/0x10a [10652.190805] ? __hrtimer_run_queues+0x1c2/0x4d0 [10652.195989] ? watchdog+0x40/0x40 [10652.200472] ? hrtimer_interrupt+0x128/0x2c0 [10652.203987] ? smp_apic_timer_interrupt+0xa6/0x270 [10652.205782] ? apic_timer_interrupt+0xf/0x20 [10652.220612] [10652.227411] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10652.232829] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10652.236332] do_raw_spin_lock+0x177/0x190 [10652.238586] _raw_spin_lock+0x21/0x30 [10652.241689] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10652.248401] obd_mod_free+0x46/0x140 [ptlrpc] [10652.251365] mdc_commit_open+0x91/0xb0 [mdc] [10652.253642] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10652.260120] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10652.263316] after_reply+0xe2c/0x1780 [ptlrpc] [10652.269638] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10652.279413] ? __next_timer_interrupt+0x160/0x160 [10652.285831] ptlrpcd+0xa82/0xe40 [ptlrpc] [10652.288552] ? do_wait_intr+0xf0/0xf0 [10652.299550] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10652.310672] kthread+0x1d1/0x200 [10652.311692] ? set_kthread_struct+0x70/0x70 [10652.317415] ret_from_fork+0x1f/0x30 [10656.058596] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [10656.072605] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10656.122243] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10656.130688] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10656.136054] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10656.138685] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10656.147595] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10656.158155] RAX: 000000000000510d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10656.163081] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [10656.166950] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [10656.171040] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [10656.174078] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [10656.176971] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [10656.182681] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10656.185257] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [10656.189644] Call Trace: [10656.192120] [10656.193028] ? show_regs.cold.9+0x22/0x2f [10656.195183] ? watchdog_timer_fn.cold.6+0x62/0x10a [10656.203131] ? __hrtimer_run_queues+0x1c2/0x4d0 [10656.205793] ? watchdog+0x40/0x40 [10656.211570] ? hrtimer_interrupt+0x128/0x2c0 [10656.214366] ? smp_apic_timer_interrupt+0xa6/0x270 [10656.225217] ? apic_timer_interrupt+0xf/0x20 [10656.227551] [10656.228227] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10656.231673] do_raw_spin_lock+0x177/0x190 [10656.235463] _raw_spin_lock+0x21/0x30 [10656.237508] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10656.240402] ? do_raw_spin_unlock+0x75/0x190 [10656.244909] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10656.246723] ? do_wait_intr+0xf0/0xf0 [10656.250161] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10656.255320] mdc_close+0x53b/0xd70 [mdc] [10656.257587] lmv_close+0x18d/0x490 [lmv] [10656.259654] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10656.268331] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10656.271962] ll_md_real_close+0xa5/0x2a0 [lustre] [10656.277787] ll_md_close+0x274/0x800 [lustre] [10656.279382] ll_file_release+0x128/0xc50 [lustre] [10656.292598] __fput+0x10d/0x440 [10656.295403] ____fput+0x12/0x20 [10656.299657] task_work_run+0xc6/0x110 [10656.304498] exit_to_usermode_loop+0x1dd/0x1f0 [10656.306141] do_syscall_64+0x3d6/0x440 [10656.312521] entry_SYSCALL_64_after_hwframe+0x49/0xae [10656.315431] RIP: 0033:0x7f151bee3cd8 [10656.316835] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10656.333093] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10656.336149] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [10656.347206] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [10656.349844] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10656.355125] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10656.360217] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10664.057842] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [10664.066058] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10664.108410] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10664.120077] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10664.132049] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10664.136002] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10664.153363] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10664.159749] RAX: 0000000000002e14 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10664.164184] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [10664.168814] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [10664.176234] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [10664.179412] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [10664.186713] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [10664.191883] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10664.195246] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [10664.206810] Call Trace: [10664.208923] [10664.212056] ? show_regs.cold.9+0x22/0x2f [10664.215263] ? watchdog_timer_fn.cold.6+0x62/0x10a [10664.218564] ? __hrtimer_run_queues+0x1c2/0x4d0 [10664.222843] ? watchdog+0x40/0x40 [10664.225385] ? hrtimer_interrupt+0x128/0x2c0 [10664.229020] ? smp_apic_timer_interrupt+0xa6/0x270 [10664.237161] ? apic_timer_interrupt+0xf/0x20 [10664.240248] [10664.241472] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10664.245156] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10664.249170] do_raw_spin_lock+0x177/0x190 [10664.253068] _raw_spin_lock+0x21/0x30 [10664.255881] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10664.259553] ? do_raw_spin_unlock+0x75/0x190 [10664.263854] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10664.269612] ? do_wait_intr+0xf0/0xf0 [10664.272038] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10664.276293] mdc_reint+0x2d/0x110 [mdc] [10664.281225] mdc_unlink+0x658/0x870 [mdc] [10664.287238] lmv_unlink+0x34d/0xbd0 [lmv] [10664.292610] ll_unlink+0x24d/0x7f0 [lustre] [10664.294069] vfs_unlink+0x202/0x330 [10664.299502] do_unlinkat+0x2fa/0x460 [10664.301908] __x64_sys_unlink+0x37/0x50 [10664.307864] do_syscall_64+0xc1/0x440 [10664.318404] entry_SYSCALL_64_after_hwframe+0x49/0xae [10664.322169] RIP: 0033:0x7f151bdee10b [10664.327184] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10664.351259] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10664.356329] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [10664.360921] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [10664.369147] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10664.376912] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10664.385223] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10668.515065] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8737s! [10668.520549] Showing busy workqueues and worker pools: [10668.522906] workqueue events: flags=0x0 [10668.527296] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10668.530722] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [10668.537898] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10668.543269] pending: kfree_rcu_monitor [10668.545015] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10668.549483] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [10668.556019] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10668.559287] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [10668.563641] workqueue events_long: flags=0x0 [10668.565825] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10668.570764] pending: lu_objects_flush [obdclass] [10668.573825] workqueue events_unbound: flags=0x2 [10668.576491] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10668.581055] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10668.593303] workqueue events_power_efficient: flags=0x80 [10668.600229] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10668.607774] pending: do_cache_clean, neigh_periodic_work, check_lifetime [10668.612186] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10668.621362] pending: neigh_periodic_work [10668.624476] workqueue mm_percpu_wq: flags=0x8 [10668.626225] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10668.631417] pending: vmstat_update, lru_add_drain_per_cpu [10668.636337] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10668.645709] pending: vmstat_update [10668.647132] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10668.651403] pending: lru_add_drain_per_cpu [10668.654343] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10668.657178] pending: lru_add_drain_per_cpu BAR(48) [10668.659486] workqueue writeback: flags=0x4a [10668.664597] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [10668.669177] pending: wb_workfn [10668.670791] workqueue xprtiod: flags=0xa [10668.672407] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10668.674556] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [10668.680062] workqueue ipv6_addrconf: flags=0x40008 [10668.683269] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10668.687296] pending: addrconf_verify_work [10668.694543] workqueue knbd0-recv: flags=0x1a [10668.696170] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10668.702190] in-flight: 658:recv_work [10668.704510] workqueue ptlrpc_pinger: flags=0x2 [10668.708490] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10668.712404] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10668.717314] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8738s workers=4 idle: 32984 37 8 [10668.723310] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10653s workers=2 idle: 661 [10676.064078] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10676.071263] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10676.130892] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10676.139765] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10676.146442] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10676.150130] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10676.152097] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10676.157672] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10676.160429] RAX: 0000000000007604 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10676.163125] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [10676.165994] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [10676.168800] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [10676.171692] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [10676.173938] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [10676.176945] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10676.179221] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10676.181852] Call Trace: [10676.182841] [10676.183656] ? show_regs.cold.9+0x22/0x2f [10676.184997] ? watchdog_timer_fn.cold.6+0x62/0x10a [10676.186948] ? __hrtimer_run_queues+0x1c2/0x4d0 [10676.188648] ? watchdog+0x40/0x40 [10676.189864] ? hrtimer_interrupt+0x128/0x2c0 [10676.191660] ? smp_apic_timer_interrupt+0xa6/0x270 [10676.193571] ? apic_timer_interrupt+0xf/0x20 [10676.195120] [10676.196025] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10676.197980] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10676.200144] do_raw_spin_lock+0x177/0x190 [10676.201841] _raw_spin_lock+0x21/0x30 [10676.203130] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10676.205433] process_one_work+0x2c8/0x700 [10676.207072] worker_thread+0x296/0x6e0 [10676.208562] ? rescuer_thread+0x570/0x570 [10676.210070] kthread+0x1d1/0x200 [10676.211319] ? set_kthread_struct+0x70/0x70 [10676.213000] ret_from_fork+0x1f/0x30 [10680.059478] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [10680.066120] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10680.107227] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10680.119319] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10680.129147] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10680.136236] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10680.161641] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10680.167516] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [10680.176537] RDX: 00000000000018a4 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [10680.193920] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [10680.209155] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [10680.215668] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10680.234084] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [10680.248142] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10680.253321] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10680.255835] Call Trace: [10680.263391] [10680.264995] ? show_regs.cold.9+0x22/0x2f [10680.272201] ? watchdog_timer_fn.cold.6+0x62/0x10a [10680.277547] ? __hrtimer_run_queues+0x1c2/0x4d0 [10680.286319] ? watchdog+0x40/0x40 [10680.287533] ? hrtimer_interrupt+0x128/0x2c0 [10680.289056] ? smp_apic_timer_interrupt+0xa6/0x270 [10680.293921] ? apic_timer_interrupt+0xf/0x20 [10680.301353] [10680.303563] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10680.313113] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10680.315693] do_raw_spin_lock+0x177/0x190 [10680.320135] _raw_spin_lock+0x21/0x30 [10680.323340] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10680.326956] obd_mod_free+0x46/0x140 [ptlrpc] [10680.328761] mdc_commit_open+0x91/0xb0 [mdc] [10680.333349] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10680.343392] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10680.347742] after_reply+0xe2c/0x1780 [ptlrpc] [10680.349667] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10680.361296] ? __next_timer_interrupt+0x160/0x160 [10680.362948] ptlrpcd+0xa82/0xe40 [ptlrpc] [10680.364492] ? do_wait_intr+0xf0/0xf0 [10680.372289] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10680.380282] kthread+0x1d1/0x200 [10680.384921] ? set_kthread_struct+0x70/0x70 [10680.387852] ret_from_fork+0x1f/0x30 [10684.057011] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [10684.059929] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10684.078177] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10684.082900] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10684.086690] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10684.089312] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10684.098247] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10684.101322] RAX: 0000000000007710 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10684.104246] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [10684.107189] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [10684.110357] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [10684.113107] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [10684.115895] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [10684.119152] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10684.121394] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [10684.124342] Call Trace: [10684.125238] [10684.126152] ? show_regs.cold.9+0x22/0x2f [10684.127873] ? watchdog_timer_fn.cold.6+0x62/0x10a [10684.129947] ? __hrtimer_run_queues+0x1c2/0x4d0 [10684.131689] ? watchdog+0x40/0x40 [10684.132881] ? hrtimer_interrupt+0x128/0x2c0 [10684.134917] ? smp_apic_timer_interrupt+0xa6/0x270 [10684.136781] ? apic_timer_interrupt+0xf/0x20 [10684.138722] [10684.139369] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10684.141216] do_raw_spin_lock+0x177/0x190 [10684.142488] _raw_spin_lock+0x21/0x30 [10684.143862] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10684.146699] ? do_raw_spin_unlock+0x75/0x190 [10684.148186] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10684.149772] ? do_wait_intr+0xf0/0xf0 [10684.151050] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10684.153064] mdc_close+0x53b/0xd70 [mdc] [10684.154430] lmv_close+0x18d/0x490 [lmv] [10684.156034] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10684.158194] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10684.159896] ll_md_real_close+0xa5/0x2a0 [lustre] [10684.161817] ll_md_close+0x274/0x800 [lustre] [10684.163503] ll_file_release+0x128/0xc50 [lustre] [10684.165590] __fput+0x10d/0x440 [10684.166547] ____fput+0x12/0x20 [10684.167454] task_work_run+0xc6/0x110 [10684.168517] exit_to_usermode_loop+0x1dd/0x1f0 [10684.170132] do_syscall_64+0x3d6/0x440 [10684.171206] entry_SYSCALL_64_after_hwframe+0x49/0xae [10684.173430] RIP: 0033:0x7f151bee3cd8 [10684.174803] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10684.181967] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10684.184843] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [10684.186939] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [10684.189980] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10684.192178] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10684.194704] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10696.052024] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [10696.056596] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10696.083754] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10696.090557] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10696.095475] RIP: 0010:native_safe_halt+0x2e/0x50 [10696.097810] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10696.113044] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10696.118132] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10696.124435] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [10696.128185] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [10696.131589] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [10696.144810] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [10696.153201] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [10696.161081] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10696.169840] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [10696.181132] Call Trace: [10696.183810] [10696.184570] ? show_regs.cold.9+0x22/0x2f [10696.190278] ? watchdog_timer_fn.cold.6+0x62/0x10a [10696.196198] ? __hrtimer_run_queues+0x1c2/0x4d0 [10696.201121] ? watchdog+0x40/0x40 [10696.206083] ? hrtimer_interrupt+0x128/0x2c0 [10696.208483] ? smp_apic_timer_interrupt+0xa6/0x270 [10696.215529] ? apic_timer_interrupt+0xf/0x20 [10696.223192] [10696.224546] ? native_safe_halt+0x2e/0x50 [10696.231342] kvm_wait+0xcd/0xe0 [10696.232796] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10696.244231] do_raw_spin_lock+0x177/0x190 [10696.249701] _raw_spin_lock+0x21/0x30 [10696.255652] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10696.262713] ? do_raw_spin_unlock+0x75/0x190 [10696.274350] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10696.282827] ? do_wait_intr+0xf0/0xf0 [10696.284488] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10696.288249] mdc_reint+0x2d/0x110 [mdc] [10696.294316] mdc_unlink+0x658/0x870 [mdc] [10696.295766] lmv_unlink+0x34d/0xbd0 [lmv] [10696.297097] ll_unlink+0x24d/0x7f0 [lustre] [10696.308561] vfs_unlink+0x202/0x330 [10696.310956] do_unlinkat+0x2fa/0x460 [10696.312603] __x64_sys_unlink+0x37/0x50 [10696.315449] do_syscall_64+0xc1/0x440 [10696.317313] entry_SYSCALL_64_after_hwframe+0x49/0xae [10696.319908] RIP: 0033:0x7f151bdee10b [10696.321092] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10696.329476] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10696.333132] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [10696.336150] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [10696.339480] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10696.342508] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10696.345426] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10699.233031] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8768s! [10699.237201] Showing busy workqueues and worker pools: [10699.240029] workqueue events: flags=0x0 [10699.241787] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10699.244427] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [10699.248952] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10699.251251] pending: kfree_rcu_monitor [10699.253008] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10699.256677] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [10699.261674] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10699.264851] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [10699.268647] workqueue events_long: flags=0x0 [10699.270813] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10699.274614] pending: lu_objects_flush [obdclass] [10699.277575] workqueue events_unbound: flags=0x2 [10699.282968] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10699.285990] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10699.298960] workqueue events_power_efficient: flags=0x80 [10699.308526] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10699.318589] pending: do_cache_clean, neigh_periodic_work, check_lifetime [10699.326753] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10699.336647] pending: neigh_periodic_work [10699.343541] workqueue mm_percpu_wq: flags=0x8 [10699.348774] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10699.357493] pending: vmstat_update, lru_add_drain_per_cpu [10699.372213] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10699.380074] pending: vmstat_update [10699.381989] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10699.386281] pending: lru_add_drain_per_cpu [10699.388687] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10699.400666] pending: lru_add_drain_per_cpu BAR(48) [10699.407231] workqueue writeback: flags=0x4a [10699.409588] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [10699.414374] pending: wb_workfn [10699.420687] workqueue xprtiod: flags=0xa [10699.422135] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10699.432681] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [10699.442554] workqueue ipv6_addrconf: flags=0x40008 [10699.446755] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10699.454236] pending: addrconf_verify_work [10699.458816] workqueue knbd0-recv: flags=0x1a [10699.461164] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10699.467788] in-flight: 658:recv_work [10699.469227] workqueue ptlrpc_pinger: flags=0x2 [10699.471620] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10699.474755] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10699.477432] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8768s workers=4 idle: 32984 37 8 [10699.483061] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10684s workers=2 idle: 661 [10704.056023] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10704.067093] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10704.117961] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10704.128414] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10704.136312] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10704.143589] RIP: 0010:native_safe_halt+0x2e/0x50 [10704.145295] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10704.155952] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10704.159425] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10704.170025] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [10704.178070] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [10704.184135] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [10704.192872] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [10704.199317] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [10704.203947] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10704.208468] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10704.216126] Call Trace: [10704.221188] [10704.229015] ? show_regs.cold.9+0x22/0x2f [10704.235196] ? watchdog_timer_fn.cold.6+0x62/0x10a [10704.244958] ? __hrtimer_run_queues+0x1c2/0x4d0 [10704.247917] ? watchdog+0x40/0x40 [10704.249472] ? hrtimer_interrupt+0x128/0x2c0 [10704.254001] ? smp_apic_timer_interrupt+0xa6/0x270 [10704.256956] ? apic_timer_interrupt+0xf/0x20 [10704.259282] [10704.260061] ? native_safe_halt+0x2e/0x50 [10704.262127] kvm_wait+0xcd/0xe0 [10704.263925] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10704.267063] do_raw_spin_lock+0x177/0x190 [10704.269555] _raw_spin_lock+0x21/0x30 [10704.276989] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10704.282630] process_one_work+0x2c8/0x700 [10704.287023] worker_thread+0x296/0x6e0 [10704.289178] ? rescuer_thread+0x570/0x570 [10704.292267] kthread+0x1d1/0x200 [10704.294193] ? set_kthread_struct+0x70/0x70 [10704.296036] ret_from_fork+0x1f/0x30 [10708.058221] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [10708.068085] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10708.127032] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10708.138120] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10708.145664] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10708.153231] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10708.170979] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10708.175291] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [10708.180395] RDX: 00000000000022d1 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [10708.187065] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [10708.194058] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [10708.203098] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10708.206500] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [10708.218157] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10708.224866] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10708.238056] Call Trace: [10708.238923] [10708.244378] ? show_regs.cold.9+0x22/0x2f [10708.248422] ? watchdog_timer_fn.cold.6+0x62/0x10a [10708.256269] ? __hrtimer_run_queues+0x1c2/0x4d0 [10708.260013] ? watchdog+0x40/0x40 [10708.262391] ? hrtimer_interrupt+0x128/0x2c0 [10708.266101] ? smp_apic_timer_interrupt+0xa6/0x270 [10708.269708] ? apic_timer_interrupt+0xf/0x20 [10708.275095] [10708.276189] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10708.281466] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10708.289313] do_raw_spin_lock+0x177/0x190 [10708.292967] _raw_spin_lock+0x21/0x30 [10708.300461] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10708.316100] obd_mod_free+0x46/0x140 [ptlrpc] [10708.318027] mdc_commit_open+0x91/0xb0 [mdc] [10708.322398] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10708.324487] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10708.331313] after_reply+0xe2c/0x1780 [ptlrpc] [10708.334160] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10708.337133] ? __next_timer_interrupt+0x160/0x160 [10708.340005] ptlrpcd+0xa82/0xe40 [ptlrpc] [10708.342328] ? do_wait_intr+0xf0/0xf0 [10708.344395] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10708.352923] kthread+0x1d1/0x200 [10708.356573] ? set_kthread_struct+0x70/0x70 [10708.361041] ret_from_fork+0x1f/0x30 [10712.057280] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [10712.065115] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10712.106446] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10712.116745] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10712.121219] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10712.123761] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10712.141810] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10712.152126] RAX: 000000000000429d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10712.156520] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [10712.163057] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [10712.167965] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [10712.175203] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [10712.182479] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [10712.188658] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10712.191179] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [10712.196788] Call Trace: [10712.197579] [10712.198299] ? show_regs.cold.9+0x22/0x2f [10712.199901] ? watchdog_timer_fn.cold.6+0x62/0x10a [10712.201721] ? __hrtimer_run_queues+0x1c2/0x4d0 [10712.203221] ? watchdog+0x40/0x40 [10712.209105] ? hrtimer_interrupt+0x128/0x2c0 [10712.211032] ? smp_apic_timer_interrupt+0xa6/0x270 [10712.213469] ? apic_timer_interrupt+0xf/0x20 [10712.216083] [10712.219700] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10712.222649] do_raw_spin_lock+0x177/0x190 [10712.224052] _raw_spin_lock+0x21/0x30 [10712.227790] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10712.231411] ? do_raw_spin_unlock+0x75/0x190 [10712.233713] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10712.236562] ? do_wait_intr+0xf0/0xf0 [10712.238008] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10712.240498] mdc_close+0x53b/0xd70 [mdc] [10712.242449] lmv_close+0x18d/0x490 [lmv] [10712.244404] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10712.247201] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10712.254487] ll_md_real_close+0xa5/0x2a0 [lustre] [10712.257697] ll_md_close+0x274/0x800 [lustre] [10712.262647] ll_file_release+0x128/0xc50 [lustre] [10712.266200] __fput+0x10d/0x440 [10712.267700] ____fput+0x12/0x20 [10712.269420] task_work_run+0xc6/0x110 [10712.271078] exit_to_usermode_loop+0x1dd/0x1f0 [10712.273610] do_syscall_64+0x3d6/0x440 [10712.275528] entry_SYSCALL_64_after_hwframe+0x49/0xae [10712.277672] RIP: 0033:0x7f151bee3cd8 [10712.279561] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10712.288756] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10712.292805] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [10712.295888] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [10712.300985] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10712.304814] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10712.307836] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10724.052020] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [10724.055664] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10724.099599] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10724.116126] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10724.128527] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10724.130912] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10724.152580] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10724.160038] RAX: 0000000000003b88 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10724.170012] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [10724.180815] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [10724.192101] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [10724.194935] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [10724.204382] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [10724.218783] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10724.226493] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [10724.232180] Call Trace: [10724.233083] [10724.233860] ? show_regs.cold.9+0x22/0x2f [10724.235608] ? watchdog_timer_fn.cold.6+0x62/0x10a [10724.237394] ? __hrtimer_run_queues+0x1c2/0x4d0 [10724.239418] ? watchdog+0x40/0x40 [10724.243171] ? hrtimer_interrupt+0x128/0x2c0 [10724.247601] ? smp_apic_timer_interrupt+0xa6/0x270 [10724.254142] ? apic_timer_interrupt+0xf/0x20 [10724.256247] [10724.257493] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10724.261063] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10724.266981] do_raw_spin_lock+0x177/0x190 [10724.268661] _raw_spin_lock+0x21/0x30 [10724.270068] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10724.284373] ? do_raw_spin_unlock+0x75/0x190 [10724.292579] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10724.304497] ? do_wait_intr+0xf0/0xf0 [10724.305866] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10724.309994] mdc_reint+0x2d/0x110 [mdc] [10724.311605] mdc_unlink+0x658/0x870 [mdc] [10724.313713] lmv_unlink+0x34d/0xbd0 [lmv] [10724.316692] ll_unlink+0x24d/0x7f0 [lustre] [10724.319481] vfs_unlink+0x202/0x330 [10724.323721] do_unlinkat+0x2fa/0x460 [10724.326903] __x64_sys_unlink+0x37/0x50 [10724.332218] do_syscall_64+0xc1/0x440 [10724.333571] entry_SYSCALL_64_after_hwframe+0x49/0xae [10724.340525] RIP: 0033:0x7f151bdee10b [10724.343343] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10724.354551] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10724.357711] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [10724.363338] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [10724.371778] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10724.376984] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10724.384322] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10729.954473] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8799s! [10729.957617] Showing busy workqueues and worker pools: [10729.959083] workqueue events: flags=0x0 [10729.969370] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10729.976692] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [10729.989487] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10729.994403] pending: kfree_rcu_monitor [10729.996405] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10729.999350] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [10730.009329] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10730.015557] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [10730.019442] workqueue events_long: flags=0x0 [10730.027264] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10730.029595] pending: lu_objects_flush [obdclass] [10730.035421] workqueue events_unbound: flags=0x2 [10730.038889] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10730.044292] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10730.077933] workqueue events_power_efficient: flags=0x80 [10730.084777] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10730.094018] pending: do_cache_clean, neigh_periodic_work, check_lifetime [10730.096055] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10730.106874] pending: neigh_periodic_work [10730.109664] workqueue mm_percpu_wq: flags=0x8 [10730.112268] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10730.125537] pending: vmstat_update, lru_add_drain_per_cpu [10730.131625] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10730.140494] pending: vmstat_update [10730.144746] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10730.155524] pending: lru_add_drain_per_cpu [10730.157331] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10730.160813] pending: lru_add_drain_per_cpu BAR(48) [10730.164333] workqueue writeback: flags=0x4a [10730.168943] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [10730.180307] pending: wb_workfn [10730.184078] workqueue xprtiod: flags=0xa [10730.187826] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10730.195480] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [10730.204507] workqueue ipv6_addrconf: flags=0x40008 [10730.210056] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10730.219432] pending: addrconf_verify_work [10730.221817] workqueue knbd0-recv: flags=0x1a [10730.227496] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10730.234899] in-flight: 658:recv_work [10730.238702] workqueue ptlrpc_pinger: flags=0x2 [10730.241890] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10730.251614] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10730.254953] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8799s workers=4 idle: 32984 37 8 [10730.268631] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10715s workers=2 idle: 661 [10732.056028] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [10732.060984] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10732.087736] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10732.095362] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10732.099758] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10732.103444] RIP: 0010:native_safe_halt+0x2e/0x50 [10732.111352] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10732.133314] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10732.138226] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10732.147680] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [10732.153734] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [10732.163953] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [10732.184121] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [10732.194247] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [10732.199477] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10732.207339] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10732.211377] Call Trace: [10732.211993] [10732.214733] ? show_regs.cold.9+0x22/0x2f [10732.217747] ? watchdog_timer_fn.cold.6+0x62/0x10a [10732.219483] ? __hrtimer_run_queues+0x1c2/0x4d0 [10732.220730] ? watchdog+0x40/0x40 [10732.221806] ? hrtimer_interrupt+0x128/0x2c0 [10732.225121] ? smp_apic_timer_interrupt+0xa6/0x270 [10732.234280] ? apic_timer_interrupt+0xf/0x20 [10732.235735] [10732.236600] ? native_safe_halt+0x2e/0x50 [10732.237708] kvm_wait+0xcd/0xe0 [10732.241472] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10732.243437] do_raw_spin_lock+0x177/0x190 [10732.250297] _raw_spin_lock+0x21/0x30 [10732.253242] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10732.259281] process_one_work+0x2c8/0x700 [10732.263918] worker_thread+0x296/0x6e0 [10732.265494] ? rescuer_thread+0x570/0x570 [10732.267515] kthread+0x1d1/0x200 [10732.269054] ? set_kthread_struct+0x70/0x70 [10732.271151] ret_from_fork+0x1f/0x30 [10736.055140] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [10736.061732] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10736.089914] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10736.097113] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10736.101456] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10736.104307] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10736.116465] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10736.123640] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [10736.132517] RDX: 0000000000001669 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [10736.139548] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [10736.144905] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [10736.149401] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10736.159242] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [10736.166455] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10736.174154] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10736.182058] Call Trace: [10736.183258] [10736.184176] ? show_regs.cold.9+0x22/0x2f [10736.195067] ? watchdog_timer_fn.cold.6+0x62/0x10a [10736.199223] ? __hrtimer_run_queues+0x1c2/0x4d0 [10736.204575] ? watchdog+0x40/0x40 [10736.206468] ? hrtimer_interrupt+0x128/0x2c0 [10736.209702] ? smp_apic_timer_interrupt+0xa6/0x270 [10736.214611] ? apic_timer_interrupt+0xf/0x20 [10736.219549] [10736.226817] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10736.228922] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10736.236978] do_raw_spin_lock+0x177/0x190 [10736.240179] _raw_spin_lock+0x21/0x30 [10736.244104] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10736.248767] obd_mod_free+0x46/0x140 [ptlrpc] [10736.251286] mdc_commit_open+0x91/0xb0 [mdc] [10736.259771] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10736.266799] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10736.275253] after_reply+0xe2c/0x1780 [ptlrpc] [10736.282208] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10736.289649] ? __next_timer_interrupt+0x160/0x160 [10736.293802] ptlrpcd+0xa82/0xe40 [ptlrpc] [10736.296736] ? do_wait_intr+0xf0/0xf0 [10736.298550] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10736.302587] kthread+0x1d1/0x200 [10736.304309] ? set_kthread_struct+0x70/0x70 [10736.306170] ret_from_fork+0x1f/0x30 [10740.057464] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [10740.065764] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10740.111170] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10740.118653] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10740.125098] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10740.130017] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10740.145642] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10740.153904] RAX: 0000000000000449 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10740.162037] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [10740.171187] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [10740.177657] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [10740.180806] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [10740.190698] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [10740.202071] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10740.209809] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [10740.213696] Call Trace: [10740.215088] [10740.216647] ? show_regs.cold.9+0x22/0x2f [10740.220087] ? watchdog_timer_fn.cold.6+0x62/0x10a [10740.228049] ? __hrtimer_run_queues+0x1c2/0x4d0 [10740.229493] ? watchdog+0x40/0x40 [10740.231561] ? hrtimer_interrupt+0x128/0x2c0 [10740.234693] ? smp_apic_timer_interrupt+0xa6/0x270 [10740.239495] ? apic_timer_interrupt+0xf/0x20 [10740.241437] [10740.242529] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10740.244403] do_raw_spin_lock+0x177/0x190 [10740.251798] _raw_spin_lock+0x21/0x30 [10740.257069] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10740.262124] ? do_raw_spin_unlock+0x75/0x190 [10740.266075] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10740.268627] ? do_wait_intr+0xf0/0xf0 [10740.273524] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10740.278170] mdc_close+0x53b/0xd70 [mdc] [10740.280110] lmv_close+0x18d/0x490 [lmv] [10740.281493] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10740.288542] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10740.294084] ll_md_real_close+0xa5/0x2a0 [lustre] [10740.297577] ll_md_close+0x274/0x800 [lustre] [10740.301558] ll_file_release+0x128/0xc50 [lustre] [10740.305882] __fput+0x10d/0x440 [10740.308523] ____fput+0x12/0x20 [10740.311535] task_work_run+0xc6/0x110 [10740.314501] exit_to_usermode_loop+0x1dd/0x1f0 [10740.316533] do_syscall_64+0x3d6/0x440 [10740.319696] entry_SYSCALL_64_after_hwframe+0x49/0xae [10740.323805] RIP: 0033:0x7f151bee3cd8 [10740.331149] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10740.344825] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10740.352576] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [10740.357847] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [10740.364787] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10740.367780] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10740.380532] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10756.053033] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [10756.062573] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10756.087112] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10756.092052] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10756.096251] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10756.102033] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10756.110774] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10756.114596] RAX: 00000000000011c0 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10756.117517] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [10756.120649] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [10756.123896] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [10756.126914] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [10756.129696] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [10756.132565] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10756.134893] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [10756.137618] Call Trace: [10756.139075] [10756.139952] ? show_regs.cold.9+0x22/0x2f [10756.141050] ? watchdog_timer_fn.cold.6+0x62/0x10a [10756.142429] ? __hrtimer_run_queues+0x1c2/0x4d0 [10756.144391] ? watchdog+0x40/0x40 [10756.145691] ? hrtimer_interrupt+0x128/0x2c0 [10756.147480] ? smp_apic_timer_interrupt+0xa6/0x270 [10756.149565] ? apic_timer_interrupt+0xf/0x20 [10756.151360] [10756.152186] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10756.154388] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10756.156576] do_raw_spin_lock+0x177/0x190 [10756.157684] _raw_spin_lock+0x21/0x30 [10756.158632] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10756.165532] ? do_raw_spin_unlock+0x75/0x190 [10756.171107] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10756.177972] ? do_wait_intr+0xf0/0xf0 [10756.180999] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10756.186584] mdc_reint+0x2d/0x110 [mdc] [10756.188924] mdc_unlink+0x658/0x870 [mdc] [10756.190931] lmv_unlink+0x34d/0xbd0 [lmv] [10756.194271] ll_unlink+0x24d/0x7f0 [lustre] [10756.197395] vfs_unlink+0x202/0x330 [10756.199156] do_unlinkat+0x2fa/0x460 [10756.202190] __x64_sys_unlink+0x37/0x50 [10756.205475] do_syscall_64+0xc1/0x440 [10756.207846] entry_SYSCALL_64_after_hwframe+0x49/0xae [10756.213567] RIP: 0033:0x7f151bdee10b [10756.214886] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10756.229140] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10756.232440] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [10756.235171] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [10756.238833] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10756.246767] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10756.253863] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10760.056022] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [10760.058344] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10760.098895] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10760.112027] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10760.116130] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10760.120960] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10760.124478] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10760.139491] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10760.145769] RAX: 0000000000003203 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10760.149747] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [10760.159104] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [10760.164037] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [10760.168766] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [10760.173902] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [10760.180485] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10760.188650] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10760.194345] Call Trace: [10760.195133] [10760.195749] ? show_regs.cold.9+0x22/0x2f [10760.200136] ? watchdog_timer_fn.cold.6+0x62/0x10a [10760.202660] ? __hrtimer_run_queues+0x1c2/0x4d0 [10760.207328] ? watchdog+0x40/0x40 [10760.209072] ? hrtimer_interrupt+0x128/0x2c0 [10760.213797] ? smp_apic_timer_interrupt+0xa6/0x270 [10760.222095] ? apic_timer_interrupt+0xf/0x20 [10760.223644] [10760.224335] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10760.231069] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10760.236220] do_raw_spin_lock+0x177/0x190 [10760.238138] _raw_spin_lock+0x21/0x30 [10760.245239] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10760.247030] process_one_work+0x2c8/0x700 [10760.250900] worker_thread+0x296/0x6e0 [10760.253125] ? rescuer_thread+0x570/0x570 [10760.265262] kthread+0x1d1/0x200 [10760.266327] ? set_kthread_struct+0x70/0x70 [10760.267462] ret_from_fork+0x1f/0x30 [10760.672095] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8830s! [10760.678384] Showing busy workqueues and worker pools: [10760.683518] workqueue events: flags=0x0 [10760.685349] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10760.690084] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [10760.694543] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10760.699228] pending: kfree_rcu_monitor [10760.702479] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10760.706224] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [10760.710646] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10760.719503] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [10760.728305] workqueue events_long: flags=0x0 [10760.735345] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10760.739542] pending: lu_objects_flush [obdclass] [10760.742023] workqueue events_unbound: flags=0x2 [10760.744244] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10760.747384] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10760.751481] workqueue events_power_efficient: flags=0x80 [10760.753759] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10760.756577] pending: do_cache_clean, neigh_periodic_work, check_lifetime [10760.759309] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10760.762135] pending: neigh_periodic_work [10760.764050] workqueue mm_percpu_wq: flags=0x8 [10760.765943] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10760.768514] pending: vmstat_update, lru_add_drain_per_cpu [10760.771091] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10760.774081] pending: vmstat_update [10760.775992] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10760.778413] pending: lru_add_drain_per_cpu [10760.781078] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10760.783897] pending: lru_add_drain_per_cpu BAR(48) [10760.786045] workqueue writeback: flags=0x4a [10760.787499] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [10760.790021] pending: wb_workfn [10760.791503] workqueue xprtiod: flags=0xa [10760.793027] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10760.795607] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [10760.799632] workqueue ipv6_addrconf: flags=0x40008 [10760.801992] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10760.804829] pending: addrconf_verify_work [10760.806083] workqueue knbd0-recv: flags=0x1a [10760.807248] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10760.810338] in-flight: 658:recv_work [10760.812125] workqueue ptlrpc_pinger: flags=0x2 [10760.813862] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10760.816349] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10760.820737] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8830s workers=4 idle: 32984 37 8 [10760.825303] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10745s workers=2 idle: 661 [10764.055136] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [10764.059134] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10764.074630] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10764.077907] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10764.081898] RIP: 0010:native_safe_halt+0x2e/0x50 [10764.083832] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10764.091725] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10764.094883] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [10764.097923] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [10764.100972] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [10764.103924] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [10764.106850] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10764.109679] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [10764.113206] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10764.115718] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10764.118740] Call Trace: [10764.119911] [10764.120786] ? show_regs.cold.9+0x22/0x2f [10764.122463] ? watchdog_timer_fn.cold.6+0x62/0x10a [10764.124579] ? __hrtimer_run_queues+0x1c2/0x4d0 [10764.126529] ? watchdog+0x40/0x40 [10764.128072] ? hrtimer_interrupt+0x128/0x2c0 [10764.129920] ? smp_apic_timer_interrupt+0xa6/0x270 [10764.132171] ? apic_timer_interrupt+0xf/0x20 [10764.134184] [10764.135139] ? native_safe_halt+0x2e/0x50 [10764.136936] kvm_wait+0xcd/0xe0 [10764.138211] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10764.140403] do_raw_spin_lock+0x177/0x190 [10764.142283] _raw_spin_lock+0x21/0x30 [10764.143873] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10764.146380] obd_mod_free+0x46/0x140 [ptlrpc] [10764.148595] mdc_commit_open+0x91/0xb0 [mdc] [10764.150587] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10764.153301] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10764.155903] after_reply+0xe2c/0x1780 [ptlrpc] [10764.158320] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10764.161309] ? __next_timer_interrupt+0x160/0x160 [10764.162822] ptlrpcd+0xa82/0xe40 [ptlrpc] [10764.164547] ? do_wait_intr+0xf0/0xf0 [10764.165774] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10764.167482] kthread+0x1d1/0x200 [10764.168486] ? set_kthread_struct+0x70/0x70 [10764.169913] ret_from_fork+0x1f/0x30 [10768.057364] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [10768.062438] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10768.100240] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10768.108428] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10768.114980] RIP: 0010:native_safe_halt+0x2e/0x50 [10768.122665] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10768.144049] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10768.150494] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10768.158042] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [10768.166525] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [10768.174240] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [10768.186765] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [10768.191313] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [10768.195898] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10768.201587] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [10768.208224] Call Trace: [10768.213179] [10768.214331] ? show_regs.cold.9+0x22/0x2f [10768.220349] ? watchdog_timer_fn.cold.6+0x62/0x10a [10768.224812] ? __hrtimer_run_queues+0x1c2/0x4d0 [10768.227980] ? watchdog+0x40/0x40 [10768.232075] ? hrtimer_interrupt+0x128/0x2c0 [10768.242235] ? smp_apic_timer_interrupt+0xa6/0x270 [10768.247935] ? apic_timer_interrupt+0xf/0x20 [10768.249920] [10768.252390] ? native_safe_halt+0x2e/0x50 [10768.254367] kvm_wait+0xcd/0xe0 [10768.260775] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10768.266329] do_raw_spin_lock+0x177/0x190 [10768.268281] _raw_spin_lock+0x21/0x30 [10768.275138] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10768.281151] ? do_raw_spin_unlock+0x75/0x190 [10768.283985] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10768.290241] ? do_wait_intr+0xf0/0xf0 [10768.292981] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10768.296458] mdc_close+0x53b/0xd70 [mdc] [10768.299431] lmv_close+0x18d/0x490 [lmv] [10768.302197] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10768.305109] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10768.310972] ll_md_real_close+0xa5/0x2a0 [lustre] [10768.316541] ll_md_close+0x274/0x800 [lustre] [10768.321718] ll_file_release+0x128/0xc50 [lustre] [10768.324265] __fput+0x10d/0x440 [10768.326785] ____fput+0x12/0x20 [10768.328372] task_work_run+0xc6/0x110 [10768.329988] exit_to_usermode_loop+0x1dd/0x1f0 [10768.334950] do_syscall_64+0x3d6/0x440 [10768.338939] entry_SYSCALL_64_after_hwframe+0x49/0xae [10768.343945] RIP: 0033:0x7f151bee3cd8 [10768.345383] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10768.357563] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10768.361869] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [10768.369968] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [10768.375871] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10768.381494] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10768.386098] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10788.052070] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [10788.055937] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10788.056025] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [10788.082222] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10788.082281] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10788.082284] RIP: 0010:native_safe_halt+0x2e/0x50 [10788.082293] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10788.082296] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [10788.082301] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10788.082304] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [10788.082306] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [10788.082309] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [10788.082312] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [10788.082315] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [10788.082318] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10788.082320] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [10788.082329] Call Trace: [10788.082334] [10788.082339] ? show_regs.cold.9+0x22/0x2f [10788.082348] ? watchdog_timer_fn.cold.6+0x62/0x10a [10788.082353] ? __hrtimer_run_queues+0x1c2/0x4d0 [10788.082358] ? watchdog+0x40/0x40 [10788.082362] ? hrtimer_interrupt+0x128/0x2c0 [10788.082366] ? smp_apic_timer_interrupt+0xa6/0x270 [10788.082373] ? apic_timer_interrupt+0xf/0x20 [10788.082377] [10788.082380] ? native_safe_halt+0x2e/0x50 [10788.082384] kvm_wait+0xcd/0xe0 [10788.082391] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10788.082399] do_raw_spin_lock+0x177/0x190 [10788.082403] _raw_spin_lock+0x21/0x30 [10788.082410] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10788.086563] Modules linked in: [10788.091406] ? do_raw_spin_unlock+0x75/0x190 [10788.097380] lustre(O) [10788.099698] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10788.110318] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10788.110395] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10788.110403] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10788.110409] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10788.110709] RIP: 0010:native_safe_halt+0x2e/0x50 [10788.110720] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10788.110723] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10788.110728] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10788.110730] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [10788.110733] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [10788.114477] ? do_wait_intr+0xf0/0xf0 [10788.117929] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [10788.124639] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10788.127749] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [10788.133586] mdc_reint+0x2d/0x110 [mdc] [10788.136778] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [10788.136789] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10788.136791] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10788.136800] Call Trace: [10788.136808] [10788.136813] ? show_regs.cold.9+0x22/0x2f [10788.136828] ? watchdog_timer_fn.cold.6+0x62/0x10a [10788.136834] ? __hrtimer_run_queues+0x1c2/0x4d0 [10788.136839] ? watchdog+0x40/0x40 [10788.136843] ? hrtimer_interrupt+0x128/0x2c0 [10788.136848] ? smp_apic_timer_interrupt+0xa6/0x270 [10788.136857] ? apic_timer_interrupt+0xf/0x20 [10788.136862] [10788.136866] ? native_safe_halt+0x2e/0x50 [10788.136870] kvm_wait+0xcd/0xe0 [10788.136878] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10788.136888] do_raw_spin_lock+0x177/0x190 [10788.136893] _raw_spin_lock+0x21/0x30 [10788.136898] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10788.137266] process_one_work+0x2c8/0x700 [10788.137273] worker_thread+0x296/0x6e0 [10788.137278] ? rescuer_thread+0x570/0x570 [10788.137282] kthread+0x1d1/0x200 [10788.137287] ? set_kthread_struct+0x70/0x70 [10788.137291] ret_from_fork+0x1f/0x30 [10788.320242] mdc_unlink+0x658/0x870 [mdc] [10788.322954] lmv_unlink+0x34d/0xbd0 [lmv] [10788.324839] ll_unlink+0x24d/0x7f0 [lustre] [10788.326889] vfs_unlink+0x202/0x330 [10788.327775] do_unlinkat+0x2fa/0x460 [10788.329263] __x64_sys_unlink+0x37/0x50 [10788.330994] do_syscall_64+0xc1/0x440 [10788.332819] entry_SYSCALL_64_after_hwframe+0x49/0xae [10788.334939] RIP: 0033:0x7f151bdee10b [10788.336912] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10788.345634] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10788.349806] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [10788.353120] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [10788.356672] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10788.361376] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10788.368380] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10790.881041] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8860s! [10790.885794] Showing busy workqueues and worker pools: [10790.893122] workqueue events: flags=0x0 [10790.897922] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10790.900247] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [10790.917625] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10790.922012] pending: kfree_rcu_monitor [10790.927969] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10790.929826] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [10790.937009] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10790.940785] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [10790.945018] workqueue events_long: flags=0x0 [10790.948017] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10790.956585] pending: lu_objects_flush [obdclass] [10790.958555] workqueue events_unbound: flags=0x2 [10790.967132] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10790.976759] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10790.986668] workqueue events_power_efficient: flags=0x80 [10790.988568] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10790.994560] pending: do_cache_clean, neigh_periodic_work, check_lifetime [10791.005411] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10791.010444] pending: neigh_periodic_work [10791.014243] workqueue mm_percpu_wq: flags=0x8 [10791.016333] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10791.023097] pending: vmstat_update, lru_add_drain_per_cpu [10791.032222] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10791.039127] pending: vmstat_update [10791.042863] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10791.046691] pending: lru_add_drain_per_cpu [10791.048377] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10791.054280] pending: lru_add_drain_per_cpu BAR(48) [10791.058953] workqueue writeback: flags=0x4a [10791.064068] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [10791.068917] pending: wb_workfn [10791.070033] workqueue xprtiod: flags=0xa [10791.072926] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10791.075669] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [10791.078945] workqueue ipv6_addrconf: flags=0x40008 [10791.080917] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10791.083623] pending: addrconf_verify_work [10791.087817] workqueue knbd0-recv: flags=0x1a [10791.089882] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10791.093840] in-flight: 658:recv_work [10791.095829] workqueue ptlrpc_pinger: flags=0x2 [10791.097760] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10791.100344] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10791.102935] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8860s workers=4 idle: 32984 37 8 [10791.106709] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10775s workers=2 idle: 661 [10792.055010] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [10792.059698] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10792.096173] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10792.108290] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10792.115188] RIP: 0010:__pv_queued_spin_lock_slowpath+0x17c/0x6d0 [10792.119256] Code: 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 48 83 05 5c d6 dc 02 01 <83> ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 75 e6 48 83 05 2c d6 dc [10792.130076] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10792.135166] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [10792.140193] RDX: 00000000000075da RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [10792.145121] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [10792.150623] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [10792.155216] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10792.160901] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [10792.167663] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10792.170330] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10792.177257] Call Trace: [10792.178236] [10792.179045] ? show_regs.cold.9+0x22/0x2f [10792.181304] ? watchdog_timer_fn.cold.6+0x62/0x10a [10792.183470] ? __hrtimer_run_queues+0x1c2/0x4d0 [10792.189068] ? watchdog+0x40/0x40 [10792.191385] ? hrtimer_interrupt+0x128/0x2c0 [10792.194464] ? smp_apic_timer_interrupt+0xa6/0x270 [10792.199443] ? apic_timer_interrupt+0xf/0x20 [10792.201578] [10792.202732] ? __pv_queued_spin_lock_slowpath+0x17c/0x6d0 [10792.209707] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10792.213003] do_raw_spin_lock+0x177/0x190 [10792.216979] _raw_spin_lock+0x21/0x30 [10792.219707] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10792.224771] obd_mod_free+0x46/0x140 [ptlrpc] [10792.229152] mdc_commit_open+0x91/0xb0 [mdc] [10792.231611] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10792.236895] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10792.240935] after_reply+0xe2c/0x1780 [ptlrpc] [10792.244955] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10792.250211] ? __next_timer_interrupt+0x160/0x160 [10792.252276] ptlrpcd+0xa82/0xe40 [ptlrpc] [10792.258177] ? do_wait_intr+0xf0/0xf0 [10792.262628] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10792.266106] kthread+0x1d1/0x200 [10792.268724] ? set_kthread_struct+0x70/0x70 [10792.271755] ret_from_fork+0x1f/0x30 [10796.057009] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [10796.066809] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10796.109914] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10796.119132] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10796.124621] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10796.133224] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10796.143159] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10796.147269] RAX: 0000000000004903 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10796.158484] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [10796.163598] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [10796.167648] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [10796.174124] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [10796.182030] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [10796.189148] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10796.192649] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [10796.199519] Call Trace: [10796.200950] [10796.201988] ? show_regs.cold.9+0x22/0x2f [10796.204121] ? watchdog_timer_fn.cold.6+0x62/0x10a [10796.206264] ? __hrtimer_run_queues+0x1c2/0x4d0 [10796.209466] ? watchdog+0x40/0x40 [10796.211516] ? hrtimer_interrupt+0x128/0x2c0 [10796.213895] ? smp_apic_timer_interrupt+0xa6/0x270 [10796.216256] ? apic_timer_interrupt+0xf/0x20 [10796.218448] [10796.222225] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10796.225622] do_raw_spin_lock+0x177/0x190 [10796.228790] _raw_spin_lock+0x21/0x30 [10796.231630] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10796.236134] ? do_raw_spin_unlock+0x75/0x190 [10796.240277] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10796.247527] ? do_wait_intr+0xf0/0xf0 [10796.251300] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10796.257251] mdc_close+0x53b/0xd70 [mdc] [10796.260028] lmv_close+0x18d/0x490 [lmv] [10796.262860] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10796.266136] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10796.270693] ll_md_real_close+0xa5/0x2a0 [lustre] [10796.277347] ll_md_close+0x274/0x800 [lustre] [10796.281699] ll_file_release+0x128/0xc50 [lustre] [10796.283582] __fput+0x10d/0x440 [10796.286549] ____fput+0x12/0x20 [10796.288225] task_work_run+0xc6/0x110 [10796.290170] exit_to_usermode_loop+0x1dd/0x1f0 [10796.293516] do_syscall_64+0x3d6/0x440 [10796.300288] entry_SYSCALL_64_after_hwframe+0x49/0xae [10796.304128] RIP: 0033:0x7f151bee3cd8 [10796.306131] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10796.314234] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10796.321852] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [10796.327834] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [10796.340675] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10796.346754] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10796.351234] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10804.831668] rcu: INFO: rcu_sched self-detected stall on CPU [10804.835012] rcu: 1-...!: (5087897 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [10804.847939] (t=8880184 jiffies g=249657 q=4970) [10804.853405] rcu: rcu_sched kthread starved for 8875136 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [10804.861954] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [10804.867784] rcu: RCU grace-period kthread stack dump: [10804.871437] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [10804.881002] Call Trace: [10804.882333] __schedule+0x351/0xcb0 [10804.887731] schedule+0xc0/0x180 [10804.888984] schedule_timeout+0xb4/0x190 [10804.891155] ? do_raw_spin_unlock+0x75/0x190 [10804.895466] ? __next_timer_interrupt+0x160/0x160 [10804.899443] ? prepare_to_swait_event+0xf3/0x140 [10804.903318] rcu_gp_kthread+0x8dd/0x15b0 [10804.907480] ? force_qs_rnp+0x2c0/0x2c0 [10804.911542] kthread+0x1d1/0x200 [10804.916030] ? set_kthread_struct+0x70/0x70 [10804.917395] ret_from_fork+0x1f/0x30 [10804.918449] rcu: Stack dump where RCU GP kthread last ran: [10804.920782] Sending NMI from CPU 1 to CPUs 2: [10804.922838] NMI backtrace for cpu 2 skipped: idling at native_safe_halt+0x2e/0x50 [10804.925893] NMI backtrace for cpu 1 [10804.927606] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10804.932931] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10804.936965] Call Trace: [10804.938508] [10804.939802] ? dump_stack+0xbb/0x10e [10804.940960] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [10804.943154] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [10804.945575] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [10804.947824] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [10804.950421] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [10804.952604] ? rcu_dump_cpu_stacks+0x11b/0x171 [10804.954484] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [10804.956698] ? update_process_times+0x6d/0xc0 [10804.958811] ? tick_sched_handle.isra.18+0x27/0x90 [10804.960912] ? tick_sched_timer+0x53/0xb0 [10804.962749] ? __hrtimer_run_queues+0x1c2/0x4d0 [10804.964770] ? tick_sched_handle.isra.18+0x90/0x90 [10804.967476] ? hrtimer_interrupt+0x128/0x2c0 [10804.969304] ? smp_apic_timer_interrupt+0xa6/0x270 [10804.971094] ? apic_timer_interrupt+0xf/0x20 [10804.972853] [10804.973763] ? native_safe_halt+0x2e/0x50 [10804.975570] ? kvm_wait+0xcd/0xe0 [10804.976901] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10804.979105] ? do_raw_spin_lock+0x177/0x190 [10804.980930] ? _raw_spin_lock+0x21/0x30 [10804.982625] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10804.984961] ? obd_mod_free+0x46/0x140 [ptlrpc] [10804.987135] ? mdc_commit_open+0x91/0xb0 [mdc] [10804.988826] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10804.991544] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10804.994333] ? after_reply+0xe2c/0x1780 [ptlrpc] [10804.996490] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10804.998720] ? __next_timer_interrupt+0x160/0x160 [10805.000648] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [10805.002829] ? do_wait_intr+0xf0/0xf0 [10805.004304] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10805.006781] ? kthread+0x1d1/0x200 [10805.008100] ? set_kthread_struct+0x70/0x70 [10805.011797] ? ret_from_fork+0x1f/0x30 [10816.052484] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [10816.069375] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10816.125020] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10816.134904] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10816.136950] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10816.149416] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10816.166678] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10816.169934] RAX: 0000000000003e8b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10816.178164] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [10816.185633] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [10816.190936] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [10816.198240] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [10816.201233] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [10816.208022] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10816.210085] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [10816.218238] Call Trace: [10816.219114] [10816.220863] ? show_regs.cold.9+0x22/0x2f [10816.222540] ? watchdog_timer_fn.cold.6+0x62/0x10a [10816.225526] ? __hrtimer_run_queues+0x1c2/0x4d0 [10816.227618] ? watchdog+0x40/0x40 [10816.229755] ? hrtimer_interrupt+0x128/0x2c0 [10816.231985] ? smp_apic_timer_interrupt+0xa6/0x270 [10816.234103] ? apic_timer_interrupt+0xf/0x20 [10816.235676] [10816.239227] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10816.244317] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10816.249415] do_raw_spin_lock+0x177/0x190 [10816.250717] _raw_spin_lock+0x21/0x30 [10816.263995] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10816.267360] ? do_raw_spin_unlock+0x75/0x190 [10816.270838] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10816.273313] ? do_wait_intr+0xf0/0xf0 [10816.274878] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10816.277336] mdc_reint+0x2d/0x110 [mdc] [10816.280112] mdc_unlink+0x658/0x870 [mdc] [10816.286965] lmv_unlink+0x34d/0xbd0 [lmv] [10816.289656] ll_unlink+0x24d/0x7f0 [lustre] [10816.291719] vfs_unlink+0x202/0x330 [10816.295089] do_unlinkat+0x2fa/0x460 [10816.297836] __x64_sys_unlink+0x37/0x50 [10816.302616] do_syscall_64+0xc1/0x440 [10816.306751] entry_SYSCALL_64_after_hwframe+0x49/0xae [10816.309565] RIP: 0033:0x7f151bdee10b [10816.314826] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10816.325660] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10816.328912] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [10816.336182] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [10816.348917] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10816.359791] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10816.374010] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10821.604558] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8891s! [10821.612173] Showing busy workqueues and worker pools: [10821.614082] workqueue events: flags=0x0 [10821.617022] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10821.620645] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [10821.626509] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10821.628715] pending: kfree_rcu_monitor [10821.630522] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10821.635477] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [10821.640185] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10821.644524] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [10821.648445] workqueue events_long: flags=0x0 [10821.654571] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10821.658556] pending: lu_objects_flush [obdclass] [10821.660885] workqueue events_unbound: flags=0x2 [10821.663349] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10821.670489] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10821.677668] workqueue events_power_efficient: flags=0x80 [10821.680726] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10821.690054] pending: do_cache_clean, neigh_periodic_work, check_lifetime [10821.695665] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10821.700610] pending: neigh_periodic_work [10821.703444] workqueue mm_percpu_wq: flags=0x8 [10821.706509] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10821.712753] pending: vmstat_update, lru_add_drain_per_cpu [10821.714730] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10821.724638] pending: vmstat_update [10821.730658] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10821.739643] pending: lru_add_drain_per_cpu [10821.744953] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10821.748381] pending: lru_add_drain_per_cpu BAR(48) [10821.751601] workqueue writeback: flags=0x4a [10821.758019] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [10821.762996] pending: wb_workfn [10821.764670] workqueue xprtiod: flags=0xa [10821.770383] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10821.774999] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [10821.784696] workqueue ipv6_addrconf: flags=0x40008 [10821.789505] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10821.795288] pending: addrconf_verify_work [10821.797956] workqueue knbd0-recv: flags=0x1a [10821.801886] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10821.805991] in-flight: 658:recv_work [10821.807800] workqueue ptlrpc_pinger: flags=0x2 [10821.810394] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10821.813995] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10821.818031] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8891s workers=4 idle: 32984 37 8 [10821.824813] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10806s workers=2 idle: 661 [10824.057141] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [10824.062371] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10824.096764] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10824.108117] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10824.111905] RIP: 0010:native_safe_halt+0x2e/0x50 [10824.113733] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10824.125705] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10824.131554] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10824.135638] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [10824.140423] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [10824.147334] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [10824.154946] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [10824.162646] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [10824.174624] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10824.180708] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [10824.193727] Call Trace: [10824.198679] [10824.202654] ? show_regs.cold.9+0x22/0x2f [10824.210219] ? watchdog_timer_fn.cold.6+0x62/0x10a [10824.214845] ? __hrtimer_run_queues+0x1c2/0x4d0 [10824.223567] ? watchdog+0x40/0x40 [10824.225514] ? hrtimer_interrupt+0x128/0x2c0 [10824.229063] ? smp_apic_timer_interrupt+0xa6/0x270 [10824.231845] ? apic_timer_interrupt+0xf/0x20 [10824.235276] [10824.235905] ? native_safe_halt+0x2e/0x50 [10824.240959] kvm_wait+0xcd/0xe0 [10824.244953] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10824.255437] do_raw_spin_lock+0x177/0x190 [10824.257857] _raw_spin_lock+0x21/0x30 [10824.269966] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10824.273585] ? do_raw_spin_unlock+0x75/0x190 [10824.275024] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10824.282396] ? do_wait_intr+0xf0/0xf0 [10824.286856] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10824.297829] mdc_close+0x53b/0xd70 [mdc] [10824.303930] lmv_close+0x18d/0x490 [lmv] [10824.311813] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10824.316639] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10824.325902] ll_md_real_close+0xa5/0x2a0 [lustre] [10824.327355] ll_md_close+0x274/0x800 [lustre] [10824.329548] ll_file_release+0x128/0xc50 [lustre] [10824.332527] __fput+0x10d/0x440 [10824.337816] ____fput+0x12/0x20 [10824.338892] task_work_run+0xc6/0x110 [10824.342334] exit_to_usermode_loop+0x1dd/0x1f0 [10824.346943] do_syscall_64+0x3d6/0x440 [10824.356586] entry_SYSCALL_64_after_hwframe+0x49/0xae [10824.358050] RIP: 0033:0x7f151bee3cd8 [10824.365279] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10824.384461] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10824.394000] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [10824.404555] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [10824.420744] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10824.436835] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10824.451052] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10832.055015] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [10832.057099] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [10832.058480] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10832.058559] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10832.058566] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10832.058569] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10832.058584] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10832.058589] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10832.058593] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [10832.058596] RDX: 0000000000001c90 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [10832.058599] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [10832.058601] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [10832.058604] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10832.058607] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [10832.058610] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10832.058613] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10832.058621] Call Trace: [10832.058627] [10832.058633] ? show_regs.cold.9+0x22/0x2f [10832.058643] ? watchdog_timer_fn.cold.6+0x62/0x10a [10832.058651] ? __hrtimer_run_queues+0x1c2/0x4d0 [10832.058656] ? watchdog+0x40/0x40 [10832.058660] ? hrtimer_interrupt+0x128/0x2c0 [10832.058664] ? smp_apic_timer_interrupt+0xa6/0x270 [10832.058673] ? apic_timer_interrupt+0xf/0x20 [10832.058678] [10832.058681] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10832.058694] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10832.058817] do_raw_spin_lock+0x177/0x190 [10832.058822] _raw_spin_lock+0x21/0x30 [10832.058829] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10832.059295] obd_mod_free+0x46/0x140 [ptlrpc] [10832.059582] mdc_commit_open+0x91/0xb0 [mdc] [10832.059622] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10832.059837] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10832.060047] after_reply+0xe2c/0x1780 [ptlrpc] [10832.060257] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10832.063859] Modules linked in: lustre(O) osp(O) ofd(O) [10832.119839] ? __next_timer_interrupt+0x160/0x160 [10832.119861] ptlrpcd+0xa82/0xe40 [ptlrpc] [10832.120177] ? do_wait_intr+0xf0/0xf0 [10832.120187] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10832.122041] kthread+0x1d1/0x200 [10832.130599] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10832.130670] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10832.130678] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10832.130684] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10832.130957] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10832.130970] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10832.130974] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10832.130979] RAX: 0000000000007715 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10832.130981] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [10832.130984] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [10832.130986] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [10832.130988] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [10832.130991] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [10832.130994] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10832.130996] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10832.131006] Call Trace: [10832.131011] [10832.131016] ? show_regs.cold.9+0x22/0x2f [10832.131027] ? watchdog_timer_fn.cold.6+0x62/0x10a [10832.131035] ? __hrtimer_run_queues+0x1c2/0x4d0 [10832.131039] ? watchdog+0x40/0x40 [10832.131043] ? hrtimer_interrupt+0x128/0x2c0 [10832.131047] ? smp_apic_timer_interrupt+0xa6/0x270 [10832.131055] ? apic_timer_interrupt+0xf/0x20 [10832.131059] [10832.131062] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10832.131067] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10832.131072] do_raw_spin_lock+0x177/0x190 [10832.131077] _raw_spin_lock+0x21/0x30 [10832.131082] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10832.131317] process_one_work+0x2c8/0x700 [10832.131324] worker_thread+0x296/0x6e0 [10832.131328] ? rescuer_thread+0x570/0x570 [10832.131332] kthread+0x1d1/0x200 [10832.131336] ? set_kthread_struct+0x70/0x70 [10832.131339] ret_from_fork+0x1f/0x30 [10832.564465] ? set_kthread_struct+0x70/0x70 [10832.567600] ret_from_fork+0x1f/0x30 [10848.062538] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [10848.077423] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10848.140957] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10848.151277] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10848.161218] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10848.165312] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10848.175216] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10848.179159] RAX: 0000000000007f0e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10848.183957] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [10848.187083] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [10848.191046] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [10848.194193] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [10848.197650] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [10848.202330] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10848.205561] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [10848.213311] Call Trace: [10848.215656] [10848.219082] ? show_regs.cold.9+0x22/0x2f [10848.223026] ? watchdog_timer_fn.cold.6+0x62/0x10a [10848.228029] ? __hrtimer_run_queues+0x1c2/0x4d0 [10848.236060] ? watchdog+0x40/0x40 [10848.238704] ? hrtimer_interrupt+0x128/0x2c0 [10848.242147] ? smp_apic_timer_interrupt+0xa6/0x270 [10848.246159] ? apic_timer_interrupt+0xf/0x20 [10848.248673] [10848.250304] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10848.253551] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10848.256766] do_raw_spin_lock+0x177/0x190 [10848.259939] _raw_spin_lock+0x21/0x30 [10848.261544] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10848.265577] ? do_raw_spin_unlock+0x75/0x190 [10848.268010] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10848.270246] ? do_wait_intr+0xf0/0xf0 [10848.271540] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10848.273574] mdc_reint+0x2d/0x110 [mdc] [10848.275140] mdc_unlink+0x658/0x870 [mdc] [10848.277160] lmv_unlink+0x34d/0xbd0 [lmv] [10848.278671] ll_unlink+0x24d/0x7f0 [lustre] [10848.280254] vfs_unlink+0x202/0x330 [10848.281337] do_unlinkat+0x2fa/0x460 [10848.282978] __x64_sys_unlink+0x37/0x50 [10848.284250] do_syscall_64+0xc1/0x440 [10848.285508] entry_SYSCALL_64_after_hwframe+0x49/0xae [10848.287318] RIP: 0033:0x7f151bdee10b [10848.288579] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10848.295089] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10848.297732] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [10848.300594] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [10848.303726] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10848.306053] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10848.308548] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10852.057021] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [10852.059272] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10852.077674] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10852.081917] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10852.086245] RIP: 0010:native_safe_halt+0x2e/0x50 [10852.088281] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10852.095653] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10852.099026] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10852.102791] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [10852.105619] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [10852.108322] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [10852.111022] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [10852.113632] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [10852.117293] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10852.119356] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [10852.122382] Call Trace: [10852.123364] [10852.124274] ? show_regs.cold.9+0x22/0x2f [10852.126281] ? watchdog_timer_fn.cold.6+0x62/0x10a [10852.128241] ? __hrtimer_run_queues+0x1c2/0x4d0 [10852.129827] ? watchdog+0x40/0x40 [10852.131092] ? hrtimer_interrupt+0x128/0x2c0 [10852.132752] ? smp_apic_timer_interrupt+0xa6/0x270 [10852.134671] ? apic_timer_interrupt+0xf/0x20 [10852.137114] [10852.137997] ? native_safe_halt+0x2e/0x50 [10852.140072] kvm_wait+0xcd/0xe0 [10852.141479] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10852.143274] do_raw_spin_lock+0x177/0x190 [10852.145223] _raw_spin_lock+0x21/0x30 [10852.146474] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10852.148569] ? do_raw_spin_unlock+0x75/0x190 [10852.150024] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10852.152353] ? do_wait_intr+0xf0/0xf0 [10852.153767] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10852.155718] mdc_close+0x53b/0xd70 [mdc] [10852.157462] lmv_close+0x18d/0x490 [lmv] [10852.159064] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10852.160950] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10852.163045] ll_md_real_close+0xa5/0x2a0 [lustre] [10852.164832] ll_md_close+0x274/0x800 [lustre] [10852.166553] ll_file_release+0x128/0xc50 [lustre] [10852.170254] __fput+0x10d/0x440 [10852.171860] ____fput+0x12/0x20 [10852.172964] task_work_run+0xc6/0x110 [10852.174389] exit_to_usermode_loop+0x1dd/0x1f0 [10852.176390] do_syscall_64+0x3d6/0x440 [10852.177671] entry_SYSCALL_64_after_hwframe+0x49/0xae [10852.180037] RIP: 0033:0x7f151bee3cd8 [10852.181356] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10852.189325] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10852.191864] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [10852.194362] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [10852.200348] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10852.203709] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10852.206468] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10852.325280] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8921s! [10852.328481] Showing busy workqueues and worker pools: [10852.330942] workqueue events: flags=0x0 [10852.333096] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10852.336123] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [10852.340670] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10852.344304] pending: kfree_rcu_monitor [10852.346387] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10852.349353] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [10852.354267] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10852.356873] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [10852.360639] workqueue events_long: flags=0x0 [10852.362112] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10852.365359] pending: lu_objects_flush [obdclass] [10852.367386] workqueue events_unbound: flags=0x2 [10852.369370] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10852.371853] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10852.376293] workqueue events_power_efficient: flags=0x80 [10852.378417] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10852.380788] pending: do_cache_clean, neigh_periodic_work, check_lifetime [10852.383666] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10852.386307] pending: neigh_periodic_work [10852.388273] workqueue mm_percpu_wq: flags=0x8 [10852.390028] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10852.393079] pending: vmstat_update, lru_add_drain_per_cpu [10852.395331] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10852.398319] pending: vmstat_update [10852.400103] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10852.402924] pending: lru_add_drain_per_cpu [10852.404855] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10852.407102] pending: lru_add_drain_per_cpu BAR(48) [10852.409802] workqueue writeback: flags=0x4a [10852.411284] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [10852.413734] pending: wb_workfn [10852.415107] workqueue xprtiod: flags=0xa [10852.416651] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10852.418949] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [10852.421713] workqueue ipv6_addrconf: flags=0x40008 [10852.423880] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10852.426956] pending: addrconf_verify_work [10852.428488] workqueue knbd0-recv: flags=0x1a [10852.430184] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10852.432496] in-flight: 658:recv_work [10852.434072] workqueue ptlrpc_pinger: flags=0x2 [10852.435700] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10852.438744] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10852.442126] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8921s workers=4 idle: 32984 37 8 [10852.446365] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10837s workers=2 idle: 661 [10860.058359] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [10860.060966] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [10860.064568] Modules linked in: [10860.069797] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common [10860.071370] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver [10860.071423] sb_edac [10860.086464] intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10860.086503] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10860.086509] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10860.086512] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10860.086527] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10860.086531] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10860.086535] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [10860.086537] RDX: 000000000000775a RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [10860.086539] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [10860.086542] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [10860.086544] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10860.086547] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [10860.086549] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10860.086551] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10860.086560] Call Trace: [10860.086565] [10860.086572] ? show_regs.cold.9+0x22/0x2f [10860.086581] ? watchdog_timer_fn.cold.6+0x62/0x10a [10860.086587] ? __hrtimer_run_queues+0x1c2/0x4d0 [10860.086590] ? watchdog+0x40/0x40 [10860.086594] ? hrtimer_interrupt+0x128/0x2c0 [10860.086598] ? smp_apic_timer_interrupt+0xa6/0x270 [10860.086606] ? apic_timer_interrupt+0xf/0x20 [10860.086611] [10860.086613] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10860.086619] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10860.086623] do_raw_spin_lock+0x177/0x190 [10860.086627] _raw_spin_lock+0x21/0x30 [10860.086633] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10860.087346] obd_mod_free+0x46/0x140 [ptlrpc] [10860.087689] mdc_commit_open+0x91/0xb0 [mdc] [10860.087787] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10860.089378] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10860.089599] after_reply+0xe2c/0x1780 [ptlrpc] [10860.090668] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10860.121587] rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10860.121622] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10860.121628] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10860.121634] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10860.122319] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10860.122331] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10860.122336] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10860.122340] RAX: 0000000000005388 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10860.122342] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [10860.122345] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [10860.122347] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [10860.122349] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [10860.122352] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [10860.122354] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10860.122357] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10860.122366] Call Trace: [10860.122372] [10860.122376] ? show_regs.cold.9+0x22/0x2f [10860.122385] ? watchdog_timer_fn.cold.6+0x62/0x10a [10860.122391] ? __hrtimer_run_queues+0x1c2/0x4d0 [10860.122396] ? watchdog+0x40/0x40 [10860.122401] ? hrtimer_interrupt+0x128/0x2c0 [10860.122405] ? smp_apic_timer_interrupt+0xa6/0x270 [10860.122414] ? apic_timer_interrupt+0xf/0x20 [10860.122418] [10860.122421] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10860.122425] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10860.122431] do_raw_spin_lock+0x177/0x190 [10860.122436] _raw_spin_lock+0x21/0x30 [10860.122441] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10860.122666] process_one_work+0x2c8/0x700 [10860.122673] worker_thread+0x296/0x6e0 [10860.122678] ? rescuer_thread+0x570/0x570 [10860.122681] kthread+0x1d1/0x200 [10860.122686] ? set_kthread_struct+0x70/0x70 [10860.122690] ret_from_fork+0x1f/0x30 [10860.128886] ? __next_timer_interrupt+0x160/0x160 [10860.128907] ptlrpcd+0xa82/0xe40 [ptlrpc] [10860.129467] ? do_wait_intr+0xf0/0xf0 [10860.129477] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10860.129678] kthread+0x1d1/0x200 [10860.129684] ? set_kthread_struct+0x70/0x70 [10860.129687] ret_from_fork+0x1f/0x30 [10880.053284] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [10880.057018] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [10880.060899] Modules linked in: [10880.061362] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10880.061438] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10880.061444] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10880.061447] RIP: 0010:native_safe_halt+0x2e/0x50 [10880.061459] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10880.061463] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10880.061467] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10880.061469] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [10880.061471] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [10880.061473] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [10880.061475] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [10880.061478] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [10880.061480] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10880.061482] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [10880.061490] Call Trace: [10880.061496] [10880.061503] ? show_regs.cold.9+0x22/0x2f [10880.061511] ? watchdog_timer_fn.cold.6+0x62/0x10a [10880.061516] ? __hrtimer_run_queues+0x1c2/0x4d0 [10880.061520] ? watchdog+0x40/0x40 [10880.061523] ? hrtimer_interrupt+0x128/0x2c0 [10880.061526] ? smp_apic_timer_interrupt+0xa6/0x270 [10880.061532] ? apic_timer_interrupt+0xf/0x20 [10880.061536] [10880.061539] ? native_safe_halt+0x2e/0x50 [10880.061542] kvm_wait+0xcd/0xe0 [10880.061548] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10880.061556] do_raw_spin_lock+0x177/0x190 [10880.061559] _raw_spin_lock+0x21/0x30 [10880.061562] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10880.061990] ? do_raw_spin_unlock+0x75/0x190 [10880.064170] lustre(O) [10880.122941] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10880.123902] ? do_wait_intr+0xf0/0xf0 [10880.123913] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10880.126842] mdc_close+0x53b/0xd70 [mdc] [10880.126884] lmv_close+0x18d/0x490 [lmv] [10880.126911] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10880.127069] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10880.127182] ll_md_real_close+0xa5/0x2a0 [lustre] [10880.127913] ll_md_close+0x274/0x800 [lustre] [10880.127999] ll_file_release+0x128/0xc50 [lustre] [10880.128178] __fput+0x10d/0x440 [10880.128188] ____fput+0x12/0x20 [10880.128193] task_work_run+0xc6/0x110 [10880.128727] exit_to_usermode_loop+0x1dd/0x1f0 [10880.140819] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10880.140895] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10880.140902] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10880.140905] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10880.140918] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10880.140922] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10880.140927] RAX: 000000000000570a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10880.140929] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [10880.140931] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [10880.140934] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [10880.140936] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [10880.140940] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [10880.140943] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10880.140945] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [10880.140953] Call Trace: [10880.140958] [10880.140963] ? show_regs.cold.9+0x22/0x2f [10880.140972] ? watchdog_timer_fn.cold.6+0x62/0x10a [10880.140979] ? __hrtimer_run_queues+0x1c2/0x4d0 [10880.140983] ? watchdog+0x40/0x40 [10880.140987] ? hrtimer_interrupt+0x128/0x2c0 [10880.140992] ? smp_apic_timer_interrupt+0xa6/0x270 [10880.141000] ? apic_timer_interrupt+0xf/0x20 [10880.141004] [10880.141007] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10880.141012] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10880.141017] do_raw_spin_lock+0x177/0x190 [10880.141022] _raw_spin_lock+0x21/0x30 [10880.141028] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10880.142911] ? do_raw_spin_unlock+0x75/0x190 [10880.142920] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10880.145765] ? do_wait_intr+0xf0/0xf0 [10880.145777] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10880.145994] mdc_reint+0x2d/0x110 [mdc] [10880.146036] mdc_unlink+0x658/0x870 [mdc] [10880.146073] lmv_unlink+0x34d/0xbd0 [lmv] [10880.146600] ll_unlink+0x24d/0x7f0 [lustre] [10880.156623] do_syscall_64+0x3d6/0x440 [10880.156708] entry_SYSCALL_64_after_hwframe+0x49/0xae [10880.156719] RIP: 0033:0x7f151bee3cd8 [10880.156727] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10880.156731] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10880.156736] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [10880.156739] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [10880.156741] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10880.156743] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10880.156745] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10880.720371] vfs_unlink+0x202/0x330 [10880.731221] do_unlinkat+0x2fa/0x460 [10880.734888] __x64_sys_unlink+0x37/0x50 [10880.743265] do_syscall_64+0xc1/0x440 [10880.747693] entry_SYSCALL_64_after_hwframe+0x49/0xae [10880.750910] RIP: 0033:0x7f151bdee10b [10880.753003] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10880.767807] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10880.771964] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [10880.789110] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [10880.793000] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10880.801346] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10880.807372] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10882.528028] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8951s! [10882.531009] Showing busy workqueues and worker pools: [10882.534779] workqueue events: flags=0x0 [10882.537679] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10882.545986] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [10882.564642] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10882.577526] pending: kfree_rcu_monitor [10882.581243] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10882.586297] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [10882.594964] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10882.601262] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [10882.608648] workqueue events_long: flags=0x0 [10882.614554] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10882.619106] pending: lu_objects_flush [obdclass] [10882.622389] workqueue events_unbound: flags=0x2 [10882.625804] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10882.629169] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10882.643031] workqueue events_power_efficient: flags=0x80 [10882.647278] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10882.653891] pending: do_cache_clean, neigh_periodic_work, check_lifetime [10882.658112] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10882.671138] pending: neigh_periodic_work [10882.676661] workqueue mm_percpu_wq: flags=0x8 [10882.680030] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10882.684167] pending: vmstat_update, lru_add_drain_per_cpu [10882.688063] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10882.693166] pending: vmstat_update [10882.695153] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10882.700141] pending: lru_add_drain_per_cpu [10882.703253] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10882.710730] pending: lru_add_drain_per_cpu BAR(48) [10882.720934] workqueue writeback: flags=0x4a [10882.727247] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [10882.732127] pending: wb_workfn [10882.734625] workqueue xprtiod: flags=0xa [10882.736752] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10882.741183] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [10882.745429] workqueue ipv6_addrconf: flags=0x40008 [10882.748100] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10882.751440] pending: addrconf_verify_work [10882.753935] workqueue knbd0-recv: flags=0x1a [10882.756757] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10882.759561] in-flight: 658:recv_work [10882.761423] workqueue ptlrpc_pinger: flags=0x2 [10882.763686] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10882.766882] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10882.770444] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8952s workers=4 idle: 32984 37 8 [10882.774448] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10867s workers=2 idle: 661 [10888.055041] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [10888.059659] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10888.060514] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [10888.081342] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10888.081357] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10888.081360] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10888.081373] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10888.081377] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10888.081382] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [10888.081384] RDX: 00000000000046a0 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [10888.081386] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [10888.081389] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [10888.081391] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10888.081394] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [10888.081396] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10888.081398] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10888.081407] Call Trace: [10888.081413] [10888.081419] ? show_regs.cold.9+0x22/0x2f [10888.081492] ? watchdog_timer_fn.cold.6+0x62/0x10a [10888.081499] ? __hrtimer_run_queues+0x1c2/0x4d0 [10888.081504] ? watchdog+0x40/0x40 [10888.081509] ? hrtimer_interrupt+0x128/0x2c0 [10888.081513] ? smp_apic_timer_interrupt+0xa6/0x270 [10888.081522] ? apic_timer_interrupt+0xf/0x20 [10888.081527] [10888.081529] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10888.081533] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10888.081538] do_raw_spin_lock+0x177/0x190 [10888.081542] _raw_spin_lock+0x21/0x30 [10888.081551] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10888.081828] obd_mod_free+0x46/0x140 [ptlrpc] [10888.082015] mdc_commit_open+0x91/0xb0 [mdc] [10888.082057] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10888.082345] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10888.082628] after_reply+0xe2c/0x1780 [ptlrpc] [10888.082821] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10888.083014] ? __next_timer_interrupt+0x160/0x160 [10888.083024] ptlrpcd+0xa82/0xe40 [ptlrpc] [10888.083373] ? do_wait_intr+0xf0/0xf0 [10888.083382] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10888.083596] kthread+0x1d1/0x200 [10888.083603] ? set_kthread_struct+0x70/0x70 [10888.083606] ret_from_fork+0x1f/0x30 [10888.257110] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10888.276394] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10888.279667] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10888.283597] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10888.286213] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10888.289221] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10888.296939] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10888.300225] RAX: 0000000000002202 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10888.303586] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [10888.306808] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [10888.309780] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [10888.316068] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [10888.321335] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [10888.326710] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10888.328957] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10888.332009] Call Trace: [10888.333196] [10888.334331] ? show_regs.cold.9+0x22/0x2f [10888.336069] ? watchdog_timer_fn.cold.6+0x62/0x10a [10888.337744] ? __hrtimer_run_queues+0x1c2/0x4d0 [10888.340008] ? watchdog+0x40/0x40 [10888.341760] ? hrtimer_interrupt+0x128/0x2c0 [10888.344737] ? smp_apic_timer_interrupt+0xa6/0x270 [10888.347187] ? apic_timer_interrupt+0xf/0x20 [10888.349907] [10888.350640] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10888.354217] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10888.355848] do_raw_spin_lock+0x177/0x190 [10888.358382] _raw_spin_lock+0x21/0x30 [10888.362752] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10888.369171] process_one_work+0x2c8/0x700 [10888.370392] worker_thread+0x296/0x6e0 [10888.371278] ? rescuer_thread+0x570/0x570 [10888.372784] kthread+0x1d1/0x200 [10888.374742] ? set_kthread_struct+0x70/0x70 [10888.376716] ret_from_fork+0x1f/0x30 [10908.052159] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [10908.057030] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [10908.058844] Modules linked in: lustre(O) [10908.063572] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10908.063650] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10908.063658] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10908.063661] RIP: 0010:native_safe_halt+0x2e/0x50 [10908.063675] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10908.063679] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10908.063684] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10908.063687] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [10908.063689] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [10908.063691] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [10908.063693] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [10908.063697] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [10908.063699] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10908.063702] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [10908.063710] Call Trace: [10908.063716] [10908.063722] ? show_regs.cold.9+0x22/0x2f [10908.063732] ? watchdog_timer_fn.cold.6+0x62/0x10a [10908.063739] ? __hrtimer_run_queues+0x1c2/0x4d0 [10908.063743] ? watchdog+0x40/0x40 [10908.063748] ? hrtimer_interrupt+0x128/0x2c0 [10908.063752] ? smp_apic_timer_interrupt+0xa6/0x270 [10908.063759] ? apic_timer_interrupt+0xf/0x20 [10908.063763] [10908.063765] ? native_safe_halt+0x2e/0x50 [10908.063769] kvm_wait+0xcd/0xe0 [10908.063776] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10908.063784] do_raw_spin_lock+0x177/0x190 [10908.063789] _raw_spin_lock+0x21/0x30 [10908.063794] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10908.068768] osp(O) [10908.101707] ? do_raw_spin_unlock+0x75/0x190 [10908.101725] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10908.101982] ? do_wait_intr+0xf0/0xf0 [10908.101988] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10908.102116] mdc_close+0x53b/0xd70 [mdc] [10908.102148] lmv_close+0x18d/0x490 [lmv] [10908.102177] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10908.102661] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10908.110204] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10908.110337] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10908.110345] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10908.110348] RIP: 0010:native_safe_halt+0x2e/0x50 [10908.110360] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10908.110372] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10908.110377] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10908.110379] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [10908.110382] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [10908.110384] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [10908.110386] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [10908.110390] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [10908.110393] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10908.110395] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [10908.110403] Call Trace: [10908.110407] [10908.110412] ? show_regs.cold.9+0x22/0x2f [10908.110421] ? watchdog_timer_fn.cold.6+0x62/0x10a [10908.110428] ? __hrtimer_run_queues+0x1c2/0x4d0 [10908.110431] ? watchdog+0x40/0x40 [10908.110436] ? hrtimer_interrupt+0x128/0x2c0 [10908.110440] ? smp_apic_timer_interrupt+0xa6/0x270 [10908.110447] ? apic_timer_interrupt+0xf/0x20 [10908.110451] [10908.110454] ? native_safe_halt+0x2e/0x50 [10908.110458] kvm_wait+0xcd/0xe0 [10908.110463] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10908.110470] do_raw_spin_lock+0x177/0x190 [10908.110474] _raw_spin_lock+0x21/0x30 [10908.110478] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10908.117760] ll_md_real_close+0xa5/0x2a0 [lustre] [10908.117938] ll_md_close+0x274/0x800 [lustre] [10908.118023] ll_file_release+0x128/0xc50 [lustre] [10908.118105] __fput+0x10d/0x440 [10908.118114] ____fput+0x12/0x20 [10908.118120] task_work_run+0xc6/0x110 [10908.118127] exit_to_usermode_loop+0x1dd/0x1f0 [10908.118134] do_syscall_64+0x3d6/0x440 [10908.118138] entry_SYSCALL_64_after_hwframe+0x49/0xae [10908.118147] RIP: 0033:0x7f151bee3cd8 [10908.118155] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10908.118159] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10908.118217] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [10908.118220] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [10908.118222] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10908.118224] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10908.118226] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10908.545509] ? do_raw_spin_unlock+0x75/0x190 [10908.549186] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10908.552622] ? do_wait_intr+0xf0/0xf0 [10908.555902] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10908.559300] mdc_reint+0x2d/0x110 [mdc] [10908.564017] mdc_unlink+0x658/0x870 [mdc] [10908.569381] lmv_unlink+0x34d/0xbd0 [lmv] [10908.571594] ll_unlink+0x24d/0x7f0 [lustre] [10908.574297] vfs_unlink+0x202/0x330 [10908.577640] do_unlinkat+0x2fa/0x460 [10908.579435] __x64_sys_unlink+0x37/0x50 [10908.583353] do_syscall_64+0xc1/0x440 [10908.585189] entry_SYSCALL_64_after_hwframe+0x49/0xae [10908.595398] RIP: 0033:0x7f151bdee10b [10908.601000] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10908.616970] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10908.623214] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [10908.628548] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [10908.632686] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10908.637867] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10908.653371] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10913.250181] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8982s! [10913.252581] Showing busy workqueues and worker pools: [10913.254876] workqueue events: flags=0x0 [10913.258814] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10913.263519] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [10913.269389] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10913.278533] pending: kfree_rcu_monitor [10913.280046] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10913.283528] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [10913.292644] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10913.294988] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [10913.301669] workqueue events_long: flags=0x0 [10913.302932] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10913.314293] pending: lu_objects_flush [obdclass] [10913.315933] workqueue events_unbound: flags=0x2 [10913.320707] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10913.326834] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10913.334350] workqueue events_power_efficient: flags=0x80 [10913.338269] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10913.343647] pending: do_cache_clean, neigh_periodic_work, check_lifetime [10913.347330] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10913.351058] pending: neigh_periodic_work [10913.356369] workqueue mm_percpu_wq: flags=0x8 [10913.358572] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10913.365804] pending: vmstat_update, lru_add_drain_per_cpu [10913.369515] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10913.373563] pending: vmstat_update [10913.377251] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10913.381898] pending: lru_add_drain_per_cpu [10913.384338] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10913.386094] pending: lru_add_drain_per_cpu BAR(48) [10913.393631] workqueue writeback: flags=0x4a [10913.396768] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [10913.398669] pending: wb_workfn [10913.399639] workqueue xprtiod: flags=0xa [10913.404064] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10913.405836] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [10913.412900] workqueue ipv6_addrconf: flags=0x40008 [10913.414686] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10913.419718] pending: addrconf_verify_work [10913.422513] workqueue knbd0-recv: flags=0x1a [10913.427630] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10913.429363] in-flight: 658:recv_work [10913.433342] workqueue ptlrpc_pinger: flags=0x2 [10913.435865] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10913.440021] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10913.446528] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8982s workers=4 idle: 32984 37 8 [10913.451064] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10898s workers=2 idle: 661 [10916.055018] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [10916.056900] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10916.059859] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10916.059943] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10916.059950] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10916.059953] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10916.059967] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10916.059971] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10916.059976] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [10916.059978] RDX: 0000000000002d77 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [10916.059980] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [10916.059983] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [10916.059985] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10916.059988] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [10916.059991] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10916.059994] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10916.060003] Call Trace: [10916.060009] [10916.060015] ? show_regs.cold.9+0x22/0x2f [10916.060025] ? watchdog_timer_fn.cold.6+0x62/0x10a [10916.060031] ? __hrtimer_run_queues+0x1c2/0x4d0 [10916.060035] ? watchdog+0x40/0x40 [10916.060039] ? hrtimer_interrupt+0x128/0x2c0 [10916.060044] ? smp_apic_timer_interrupt+0xa6/0x270 [10916.060052] ? apic_timer_interrupt+0xf/0x20 [10916.060056] [10916.060059] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10916.060064] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10916.060069] do_raw_spin_lock+0x177/0x190 [10916.060073] _raw_spin_lock+0x21/0x30 [10916.060079] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10916.060502] obd_mod_free+0x46/0x140 [ptlrpc] [10916.060880] mdc_commit_open+0x91/0xb0 [mdc] [10916.060919] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10916.061147] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10916.061637] after_reply+0xe2c/0x1780 [ptlrpc] [10916.061954] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10916.062186] ? __next_timer_interrupt+0x160/0x160 [10916.062196] ptlrpcd+0xa82/0xe40 [ptlrpc] [10916.062636] ? do_wait_intr+0xf0/0xf0 [10916.062646] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10916.062896] kthread+0x1d1/0x200 [10916.062902] ? set_kthread_struct+0x70/0x70 [10916.062906] ret_from_fork+0x1f/0x30 [10916.225583] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10916.262927] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10916.272384] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10916.287670] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10916.296067] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10916.302819] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10916.315183] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10916.324974] RAX: 0000000000002e7e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10916.333031] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [10916.340082] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [10916.343950] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [10916.348344] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [10916.352332] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [10916.356906] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10916.360105] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10916.363414] Call Trace: [10916.364389] [10916.365322] ? show_regs.cold.9+0x22/0x2f [10916.366880] ? watchdog_timer_fn.cold.6+0x62/0x10a [10916.370258] ? __hrtimer_run_queues+0x1c2/0x4d0 [10916.374966] ? watchdog+0x40/0x40 [10916.377135] ? hrtimer_interrupt+0x128/0x2c0 [10916.381925] ? smp_apic_timer_interrupt+0xa6/0x270 [10916.384999] ? apic_timer_interrupt+0xf/0x20 [10916.387766] [10916.389222] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10916.393501] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10916.396996] do_raw_spin_lock+0x177/0x190 [10916.402854] _raw_spin_lock+0x21/0x30 [10916.407341] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10916.410335] process_one_work+0x2c8/0x700 [10916.414317] worker_thread+0x296/0x6e0 [10916.416229] ? rescuer_thread+0x570/0x570 [10916.421713] kthread+0x1d1/0x200 [10916.423123] ? set_kthread_struct+0x70/0x70 [10916.425807] ret_from_fork+0x1f/0x30 [10936.058911] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [10936.063249] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10936.081194] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10936.084899] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10936.089712] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10936.093353] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10936.101442] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10936.104005] RAX: 000000000000569a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10936.105872] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [10936.109217] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [10936.113542] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [10936.117524] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [10936.119757] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [10936.123520] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10936.125681] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [10936.129487] Call Trace: [10936.131151] [10936.134657] ? show_regs.cold.9+0x22/0x2f [10936.136233] ? watchdog_timer_fn.cold.6+0x62/0x10a [10936.138723] ? __hrtimer_run_queues+0x1c2/0x4d0 [10936.140409] ? watchdog+0x40/0x40 [10936.142201] ? hrtimer_interrupt+0x128/0x2c0 [10936.144221] ? smp_apic_timer_interrupt+0xa6/0x270 [10936.146183] ? apic_timer_interrupt+0xf/0x20 [10936.147598] [10936.148251] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10936.151910] do_raw_spin_lock+0x177/0x190 [10936.155387] _raw_spin_lock+0x21/0x30 [10936.157216] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10936.160465] ? do_raw_spin_unlock+0x75/0x190 [10936.162698] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10936.167292] ? do_wait_intr+0xf0/0xf0 [10936.176387] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10936.183886] mdc_close+0x53b/0xd70 [mdc] [10936.189485] lmv_close+0x18d/0x490 [lmv] [10936.192211] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10936.197477] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10936.207891] ll_md_real_close+0xa5/0x2a0 [lustre] [10936.210636] ll_md_close+0x274/0x800 [lustre] [10936.216885] ll_file_release+0x128/0xc50 [lustre] [10936.224284] __fput+0x10d/0x440 [10936.225910] ____fput+0x12/0x20 [10936.233218] task_work_run+0xc6/0x110 [10936.235195] exit_to_usermode_loop+0x1dd/0x1f0 [10936.243685] do_syscall_64+0x3d6/0x440 [10936.249641] entry_SYSCALL_64_after_hwframe+0x49/0xae [10936.253218] RIP: 0033:0x7f151bee3cd8 [10936.257234] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10936.281719] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10936.292381] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [10936.301424] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [10936.313402] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10936.318655] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10936.339748] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10940.053503] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [10940.059780] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10940.098824] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10940.108922] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10940.116171] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10940.118832] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10940.124875] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10940.126978] RAX: 0000000000007db1 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10940.129012] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [10940.131024] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [10940.133121] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [10940.137719] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [10940.141132] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [10940.144763] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10940.146916] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [10940.152646] Call Trace: [10940.153566] [10940.154592] ? show_regs.cold.9+0x22/0x2f [10940.156803] ? watchdog_timer_fn.cold.6+0x62/0x10a [10940.158407] ? __hrtimer_run_queues+0x1c2/0x4d0 [10940.159892] ? watchdog+0x40/0x40 [10940.160886] ? hrtimer_interrupt+0x128/0x2c0 [10940.162169] ? smp_apic_timer_interrupt+0xa6/0x270 [10940.166122] ? apic_timer_interrupt+0xf/0x20 [10940.168120] [10940.168810] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10940.171417] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10940.173496] do_raw_spin_lock+0x177/0x190 [10940.175279] _raw_spin_lock+0x21/0x30 [10940.178482] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10940.183508] ? do_raw_spin_unlock+0x75/0x190 [10940.185456] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10940.189093] ? do_wait_intr+0xf0/0xf0 [10940.194571] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10940.197414] mdc_reint+0x2d/0x110 [mdc] [10940.199146] mdc_unlink+0x658/0x870 [mdc] [10940.205807] lmv_unlink+0x34d/0xbd0 [lmv] [10940.208273] ll_unlink+0x24d/0x7f0 [lustre] [10940.210608] vfs_unlink+0x202/0x330 [10940.212279] do_unlinkat+0x2fa/0x460 [10940.219629] __x64_sys_unlink+0x37/0x50 [10940.221477] do_syscall_64+0xc1/0x440 [10940.223566] entry_SYSCALL_64_after_hwframe+0x49/0xae [10940.225520] RIP: 0033:0x7f151bdee10b [10940.227390] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10940.239682] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10940.243413] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [10940.251514] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [10940.255316] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10940.258516] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10940.266811] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10943.469599] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9012s! [10943.477034] Showing busy workqueues and worker pools: [10943.479878] workqueue events: flags=0x0 [10943.486495] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10943.490678] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [10943.503549] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10943.505979] pending: kfree_rcu_monitor [10943.513530] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10943.531366] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [10943.540150] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10943.548239] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [10943.556311] workqueue events_long: flags=0x0 [10943.559846] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10943.564398] pending: lu_objects_flush [obdclass] [10943.570553] workqueue events_unbound: flags=0x2 [10943.573102] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10943.578273] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10943.595326] workqueue events_power_efficient: flags=0x80 [10943.600544] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10943.612664] pending: do_cache_clean, neigh_periodic_work, check_lifetime [10943.621256] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10943.627597] pending: neigh_periodic_work [10943.629477] workqueue mm_percpu_wq: flags=0x8 [10943.637410] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10943.649883] pending: vmstat_update, lru_add_drain_per_cpu [10943.657694] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10943.672276] pending: vmstat_update [10943.676186] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10943.680135] pending: lru_add_drain_per_cpu [10943.686216] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10943.691308] pending: lru_add_drain_per_cpu BAR(48) [10943.694133] workqueue writeback: flags=0x4a [10943.698328] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [10943.703803] pending: wb_workfn [10943.705742] workqueue xprtiod: flags=0xa [10943.710704] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10943.722060] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [10943.724605] workqueue ipv6_addrconf: flags=0x40008 [10943.729443] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10943.738677] pending: addrconf_verify_work [10943.742599] workqueue knbd0-recv: flags=0x1a [10943.744706] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10943.759001] in-flight: 658:recv_work [10943.760606] workqueue ptlrpc_pinger: flags=0x2 [10943.762213] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10943.773361] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10943.779014] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9013s workers=4 idle: 32984 37 8 [10943.788255] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10928s workers=2 idle: 661 [10944.056028] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10944.057065] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [10944.059699] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10944.069236] Modules linked in: lustre(O) [10944.091396] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10944.105628] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10944.105706] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10944.105714] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10944.105718] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10944.105734] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10944.105739] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10944.105743] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [10944.105746] RDX: 000000000000561b RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [10944.105749] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [10944.105751] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [10944.105754] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10944.105758] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [10944.105761] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10944.105764] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10944.105773] Call Trace: [10944.105779] [10944.105785] ? show_regs.cold.9+0x22/0x2f [10944.105795] ? watchdog_timer_fn.cold.6+0x62/0x10a [10944.105802] ? __hrtimer_run_queues+0x1c2/0x4d0 [10944.105806] ? watchdog+0x40/0x40 [10944.105811] ? hrtimer_interrupt+0x128/0x2c0 [10944.105816] ? smp_apic_timer_interrupt+0xa6/0x270 [10944.105833] ? apic_timer_interrupt+0xf/0x20 [10944.105839] [10944.105843] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10944.105849] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10944.105854] do_raw_spin_lock+0x177/0x190 [10944.105858] _raw_spin_lock+0x21/0x30 [10944.105865] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10944.111643] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10944.111659] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10944.159120] obd_mod_free+0x46/0x140 [ptlrpc] [10944.166006] [10944.166026] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10944.166047] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10944.166052] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10944.166057] RAX: 0000000000004a05 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10944.166060] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [10944.166062] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [10944.166064] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [10944.166066] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [10944.166069] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [10944.166072] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10944.166075] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10944.166084] Call Trace: [10944.166090] [10944.166101] ? show_regs.cold.9+0x22/0x2f [10944.166111] ? watchdog_timer_fn.cold.6+0x62/0x10a [10944.166117] ? __hrtimer_run_queues+0x1c2/0x4d0 [10944.166121] ? watchdog+0x40/0x40 [10944.166125] ? hrtimer_interrupt+0x128/0x2c0 [10944.166128] ? smp_apic_timer_interrupt+0xa6/0x270 [10944.166136] ? apic_timer_interrupt+0xf/0x20 [10944.166141] [10944.166143] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10944.166147] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10944.166154] do_raw_spin_lock+0x177/0x190 [10944.166159] _raw_spin_lock+0x21/0x30 [10944.166164] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10944.166459] process_one_work+0x2c8/0x700 [10944.166467] worker_thread+0x296/0x6e0 [10944.166472] ? rescuer_thread+0x570/0x570 [10944.166475] kthread+0x1d1/0x200 [10944.166479] ? set_kthread_struct+0x70/0x70 [10944.166484] ret_from_fork+0x1f/0x30 [10944.388667] mdc_commit_open+0x91/0xb0 [mdc] [10944.391210] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10944.395307] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10944.400360] after_reply+0xe2c/0x1780 [ptlrpc] [10944.403261] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10944.406987] ? __next_timer_interrupt+0x160/0x160 [10944.410238] ptlrpcd+0xa82/0xe40 [ptlrpc] [10944.414446] ? do_wait_intr+0xf0/0xf0 [10944.416961] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10944.421474] kthread+0x1d1/0x200 [10944.425406] ? set_kthread_struct+0x70/0x70 [10944.427863] ret_from_fork+0x1f/0x30 [10964.057067] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [10964.060213] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10964.120970] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10964.130350] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10964.141798] RIP: 0010:native_safe_halt+0x2e/0x50 [10964.143072] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10964.167908] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10964.178432] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10964.190429] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [10964.199334] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [10964.205359] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [10964.212212] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [10964.221350] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [10964.231386] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10964.236937] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [10964.241403] Call Trace: [10964.242876] [10964.244386] ? show_regs.cold.9+0x22/0x2f [10964.248426] ? watchdog_timer_fn.cold.6+0x62/0x10a [10964.252109] ? __hrtimer_run_queues+0x1c2/0x4d0 [10964.257034] ? watchdog+0x40/0x40 [10964.259492] ? hrtimer_interrupt+0x128/0x2c0 [10964.263400] ? smp_apic_timer_interrupt+0xa6/0x270 [10964.266584] ? apic_timer_interrupt+0xf/0x20 [10964.268421] [10964.270046] ? native_safe_halt+0x2e/0x50 [10964.278428] kvm_wait+0xcd/0xe0 [10964.285690] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10964.287699] do_raw_spin_lock+0x177/0x190 [10964.289214] _raw_spin_lock+0x21/0x30 [10964.293187] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10964.300108] ? do_raw_spin_unlock+0x75/0x190 [10964.302264] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10964.304967] ? do_wait_intr+0xf0/0xf0 [10964.306338] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10964.311427] mdc_close+0x53b/0xd70 [mdc] [10964.314322] lmv_close+0x18d/0x490 [lmv] [10964.317326] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10964.319315] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10964.326224] ll_md_real_close+0xa5/0x2a0 [lustre] [10964.328258] ll_md_close+0x274/0x800 [lustre] [10964.329972] ll_file_release+0x128/0xc50 [lustre] [10964.331344] __fput+0x10d/0x440 [10964.336447] ____fput+0x12/0x20 [10964.340637] task_work_run+0xc6/0x110 [10964.345197] exit_to_usermode_loop+0x1dd/0x1f0 [10964.350213] do_syscall_64+0x3d6/0x440 [10964.352303] entry_SYSCALL_64_after_hwframe+0x49/0xae [10964.354327] RIP: 0033:0x7f151bee3cd8 [10964.358276] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10964.377678] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10964.385122] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [10964.389990] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [10964.394162] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10964.400247] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10964.403229] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10968.052137] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [10968.055715] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10968.099481] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10968.102694] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10968.121161] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10968.123394] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10968.133929] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10968.139848] RAX: 0000000000006936 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10968.144855] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [10968.151346] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [10968.163413] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [10968.166111] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [10968.180275] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [10968.191866] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10968.197190] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [10968.206504] Call Trace: [10968.207977] [10968.208943] ? show_regs.cold.9+0x22/0x2f [10968.214795] ? watchdog_timer_fn.cold.6+0x62/0x10a [10968.221523] ? __hrtimer_run_queues+0x1c2/0x4d0 [10968.228012] ? watchdog+0x40/0x40 [10968.233368] ? hrtimer_interrupt+0x128/0x2c0 [10968.239334] ? smp_apic_timer_interrupt+0xa6/0x270 [10968.245828] ? apic_timer_interrupt+0xf/0x20 [10968.250465] [10968.253451] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10968.266571] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10968.269184] do_raw_spin_lock+0x177/0x190 [10968.272969] _raw_spin_lock+0x21/0x30 [10968.275814] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10968.277993] ? do_raw_spin_unlock+0x75/0x190 [10968.279926] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10968.287590] ? do_wait_intr+0xf0/0xf0 [10968.293025] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10968.299002] mdc_reint+0x2d/0x110 [mdc] [10968.300539] mdc_unlink+0x658/0x870 [mdc] [10968.308349] lmv_unlink+0x34d/0xbd0 [lmv] [10968.313447] ll_unlink+0x24d/0x7f0 [lustre] [10968.315937] vfs_unlink+0x202/0x330 [10968.318477] do_unlinkat+0x2fa/0x460 [10968.322429] __x64_sys_unlink+0x37/0x50 [10968.329326] do_syscall_64+0xc1/0x440 [10968.332529] entry_SYSCALL_64_after_hwframe+0x49/0xae [10968.339481] RIP: 0033:0x7f151bdee10b [10968.345277] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10968.368024] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10968.375337] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [10968.377976] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [10968.383060] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10968.389189] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10968.395107] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10972.055067] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [10972.058132] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10972.061331] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [10972.118809] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10972.118823] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10972.118826] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10972.118841] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10972.118845] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10972.118851] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [10972.118853] RDX: 0000000000007c05 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [10972.118856] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [10972.118858] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [10972.118861] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10972.118865] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [10972.118868] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10972.118870] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10972.118879] Call Trace: [10972.118884] [10972.118890] ? show_regs.cold.9+0x22/0x2f [10972.118899] ? watchdog_timer_fn.cold.6+0x62/0x10a [10972.118905] ? __hrtimer_run_queues+0x1c2/0x4d0 [10972.118909] ? watchdog+0x40/0x40 [10972.118914] ? hrtimer_interrupt+0x128/0x2c0 [10972.118917] ? smp_apic_timer_interrupt+0xa6/0x270 [10972.118925] ? apic_timer_interrupt+0xf/0x20 [10972.118930] [10972.118932] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10972.118937] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10972.118941] do_raw_spin_lock+0x177/0x190 [10972.118945] _raw_spin_lock+0x21/0x30 [10972.118952] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10972.119363] obd_mod_free+0x46/0x140 [ptlrpc] [10972.120189] mdc_commit_open+0x91/0xb0 [mdc] [10972.120228] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10972.120430] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10972.120654] after_reply+0xe2c/0x1780 [ptlrpc] [10972.120880] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10972.121224] ? __next_timer_interrupt+0x160/0x160 [10972.121234] ptlrpcd+0xa82/0xe40 [ptlrpc] [10972.121443] ? do_wait_intr+0xf0/0xf0 [10972.121452] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10972.121795] kthread+0x1d1/0x200 [10972.121802] ? set_kthread_struct+0x70/0x70 [10972.121806] ret_from_fork+0x1f/0x30 [10972.316571] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10972.337866] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10972.342455] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10972.357815] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10972.368335] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10972.376118] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10972.396214] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10972.403096] RAX: 000000000000278b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10972.408756] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [10972.412280] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [10972.416368] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [10972.419688] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [10972.425369] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [10972.433475] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10972.438340] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10972.442824] Call Trace: [10972.444174] [10972.448460] ? show_regs.cold.9+0x22/0x2f [10972.454419] ? watchdog_timer_fn.cold.6+0x62/0x10a [10972.459572] ? __hrtimer_run_queues+0x1c2/0x4d0 [10972.464770] ? watchdog+0x40/0x40 [10972.467152] ? hrtimer_interrupt+0x128/0x2c0 [10972.471792] ? smp_apic_timer_interrupt+0xa6/0x270 [10972.474815] ? apic_timer_interrupt+0xf/0x20 [10972.478582] [10972.483895] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10972.485393] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10972.490442] do_raw_spin_lock+0x177/0x190 [10972.493024] _raw_spin_lock+0x21/0x30 [10972.498631] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10972.502271] process_one_work+0x2c8/0x700 [10972.504167] worker_thread+0x296/0x6e0 [10972.513366] ? rescuer_thread+0x570/0x570 [10972.516409] kthread+0x1d1/0x200 [10972.519488] ? set_kthread_struct+0x70/0x70 [10972.521949] ret_from_fork+0x1f/0x30 [10974.184031] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9043s! [10974.188450] Showing busy workqueues and worker pools: [10974.190850] workqueue events: flags=0x0 [10974.192301] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10974.195946] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [10974.199511] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10974.202028] pending: kfree_rcu_monitor [10974.203368] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10974.205923] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [10974.209130] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10974.211292] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [10974.214238] workqueue events_long: flags=0x0 [10974.215762] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10974.217968] pending: lu_objects_flush [obdclass] [10974.219914] workqueue events_unbound: flags=0x2 [10974.221213] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [10974.222977] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [10974.226030] workqueue events_power_efficient: flags=0x80 [10974.228978] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10974.231940] pending: do_cache_clean, neigh_periodic_work, check_lifetime [10974.234787] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10974.238162] pending: neigh_periodic_work [10974.239942] workqueue mm_percpu_wq: flags=0x8 [10974.242034] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10974.245102] pending: vmstat_update, lru_add_drain_per_cpu [10974.248133] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10974.251437] pending: vmstat_update [10974.253158] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10974.256487] pending: lru_add_drain_per_cpu [10974.258803] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10974.262295] pending: lru_add_drain_per_cpu BAR(48) [10974.264978] workqueue writeback: flags=0x4a [10974.266991] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [10974.270537] pending: wb_workfn [10974.272254] workqueue xprtiod: flags=0xa [10974.273887] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [10974.276689] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [10974.280339] workqueue ipv6_addrconf: flags=0x40008 [10974.282198] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10974.284618] pending: addrconf_verify_work [10974.286508] workqueue knbd0-recv: flags=0x1a [10974.288814] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10974.291158] in-flight: 658:recv_work [10974.292875] workqueue ptlrpc_pinger: flags=0x2 [10974.295141] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10974.298082] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [10974.300864] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9043s workers=4 idle: 32984 37 8 [10974.303708] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10959s workers=2 idle: 661 [10984.834022] rcu: INFO: rcu_sched self-detected stall on CPU [10984.837911] rcu: 1-...!: (5203810 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [10984.847817] (t=9060187 jiffies g=249657 q=4987) [10984.852909] rcu: rcu_sched kthread starved for 9055139 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [10984.860226] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [10984.868220] rcu: RCU grace-period kthread stack dump: [10984.870995] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [10984.876416] Call Trace: [10984.878055] __schedule+0x351/0xcb0 [10984.879712] schedule+0xc0/0x180 [10984.881898] schedule_timeout+0xb4/0x190 [10984.884082] ? do_raw_spin_unlock+0x75/0x190 [10984.886925] ? __next_timer_interrupt+0x160/0x160 [10984.889910] ? prepare_to_swait_event+0xf3/0x140 [10984.892394] rcu_gp_kthread+0x8dd/0x15b0 [10984.895862] ? force_qs_rnp+0x2c0/0x2c0 [10984.898165] kthread+0x1d1/0x200 [10984.899742] ? set_kthread_struct+0x70/0x70 [10984.902230] ret_from_fork+0x1f/0x30 [10984.907825] rcu: Stack dump where RCU GP kthread last ran: [10984.917866] Sending NMI from CPU 1 to CPUs 2: [10984.925358] NMI backtrace for cpu 2 [10984.925365] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10984.925367] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10984.925369] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10984.925372] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10984.925373] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10984.925375] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 [10984.925377] RAX: 0000000000007f8c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10984.925378] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [10984.925380] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [10984.925381] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [10984.925382] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [10984.925384] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [10984.925386] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10984.925387] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [10984.925389] Call Trace: [10984.925390] [10984.925391] ? show_regs.cold.9+0x22/0x2f [10984.925392] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [10984.925393] ? nmi_cpu_backtrace_handler+0x15/0x40 [10984.925394] ? nmi_handle+0xbd/0x210 [10984.925395] ? default_do_nmi+0x4b/0x1b0 [10984.925396] ? do_nmi+0x280/0x380 [10984.925398] ? end_repeat_nmi+0x16/0x1f [10984.925399] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10984.925400] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10984.925401] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10984.925402] [10984.925403] do_raw_spin_lock+0x177/0x190 [10984.925404] _raw_spin_lock+0x21/0x30 [10984.925405] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10984.925406] process_one_work+0x2c8/0x700 [10984.925407] worker_thread+0x296/0x6e0 [10984.925408] ? rescuer_thread+0x570/0x570 [10984.925409] kthread+0x1d1/0x200 [10984.925410] ? set_kthread_struct+0x70/0x70 [10984.925411] ret_from_fork+0x1f/0x30 [10984.933098] NMI backtrace for cpu 1 [10985.097757] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10985.103351] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10985.112355] Call Trace: [10985.113788] [10985.115648] ? dump_stack+0xbb/0x10e [10985.117091] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [10985.119993] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [10985.124259] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [10985.129034] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [10985.137682] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [10985.141224] ? rcu_dump_cpu_stacks+0x11b/0x171 [10985.142725] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [10985.147888] ? update_process_times+0x6d/0xc0 [10985.151011] ? tick_sched_handle.isra.18+0x27/0x90 [10985.154669] ? tick_sched_timer+0x53/0xb0 [10985.158088] ? __hrtimer_run_queues+0x1c2/0x4d0 [10985.165660] ? tick_sched_handle.isra.18+0x90/0x90 [10985.167264] ? hrtimer_interrupt+0x128/0x2c0 [10985.177858] ? smp_apic_timer_interrupt+0xa6/0x270 [10985.185779] ? apic_timer_interrupt+0xf/0x20 [10985.189436] [10985.190855] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10985.195967] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10985.205178] ? do_raw_spin_lock+0x177/0x190 [10985.214806] ? _raw_spin_lock+0x21/0x30 [10985.219428] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10985.224952] ? obd_mod_free+0x46/0x140 [ptlrpc] [10985.226896] ? mdc_commit_open+0x91/0xb0 [mdc] [10985.233813] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10985.238686] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10985.244170] ? after_reply+0xe2c/0x1780 [ptlrpc] [10985.248862] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10985.252129] ? __next_timer_interrupt+0x160/0x160 [10985.255966] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [10985.265103] ? do_wait_intr+0xf0/0xf0 [10985.266509] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10985.275741] ? kthread+0x1d1/0x200 [10985.278005] ? set_kthread_struct+0x70/0x70 [10985.281941] ? ret_from_fork+0x1f/0x30 [10992.057447] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [10992.062038] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10992.085761] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10992.096927] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10992.106950] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10992.112727] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10992.133587] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10992.139472] RAX: 0000000000006c42 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [10992.144205] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [10992.154020] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [10992.155955] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [10992.158656] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [10992.164309] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [10992.168125] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10992.171273] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [10992.178657] Call Trace: [10992.179523] [10992.180270] ? show_regs.cold.9+0x22/0x2f [10992.183112] ? watchdog_timer_fn.cold.6+0x62/0x10a [10992.185227] ? __hrtimer_run_queues+0x1c2/0x4d0 [10992.187100] ? watchdog+0x40/0x40 [10992.188688] ? hrtimer_interrupt+0x128/0x2c0 [10992.192114] ? smp_apic_timer_interrupt+0xa6/0x270 [10992.193711] ? apic_timer_interrupt+0xf/0x20 [10992.196360] [10992.198118] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10992.202856] do_raw_spin_lock+0x177/0x190 [10992.204136] _raw_spin_lock+0x21/0x30 [10992.205087] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10992.209081] ? do_raw_spin_unlock+0x75/0x190 [10992.210340] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10992.213293] ? do_wait_intr+0xf0/0xf0 [10992.218174] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10992.221821] mdc_close+0x53b/0xd70 [mdc] [10992.223168] lmv_close+0x18d/0x490 [lmv] [10992.226704] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10992.229539] ll_close_inode_openhandle+0x493/0xea0 [lustre] [10992.236284] ll_md_real_close+0xa5/0x2a0 [lustre] [10992.238888] ll_md_close+0x274/0x800 [lustre] [10992.242888] ll_file_release+0x128/0xc50 [lustre] [10992.244637] __fput+0x10d/0x440 [10992.245887] ____fput+0x12/0x20 [10992.251089] task_work_run+0xc6/0x110 [10992.253324] exit_to_usermode_loop+0x1dd/0x1f0 [10992.256481] do_syscall_64+0x3d6/0x440 [10992.258090] entry_SYSCALL_64_after_hwframe+0x49/0xae [10992.259744] RIP: 0033:0x7f151bee3cd8 [10992.261191] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [10992.269683] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [10992.272516] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [10992.274742] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [10992.277352] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [10992.279363] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [10992.281246] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11000.053703] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [11000.074827] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11000.125435] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11000.138650] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11000.154268] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11000.157748] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11000.168571] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11000.178963] RAX: 0000000000004d07 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11000.182606] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [11000.186019] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [11000.189545] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [11000.193014] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [11000.201053] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [11000.205131] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11000.207341] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [11000.212217] Call Trace: [11000.214647] [11000.215617] ? show_regs.cold.9+0x22/0x2f [11000.228876] ? watchdog_timer_fn.cold.6+0x62/0x10a [11000.231611] ? __hrtimer_run_queues+0x1c2/0x4d0 [11000.234128] ? watchdog+0x40/0x40 [11000.237572] ? hrtimer_interrupt+0x128/0x2c0 [11000.239736] ? smp_apic_timer_interrupt+0xa6/0x270 [11000.242775] ? apic_timer_interrupt+0xf/0x20 [11000.244891] [11000.250742] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11000.254919] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11000.270541] do_raw_spin_lock+0x177/0x190 [11000.274577] _raw_spin_lock+0x21/0x30 [11000.278983] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11000.283252] ? do_raw_spin_unlock+0x75/0x190 [11000.285314] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11000.290619] ? do_wait_intr+0xf0/0xf0 [11000.293315] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11000.298026] mdc_reint+0x2d/0x110 [mdc] [11000.300018] mdc_unlink+0x658/0x870 [mdc] [11000.302103] lmv_unlink+0x34d/0xbd0 [lmv] [11000.303987] ll_unlink+0x24d/0x7f0 [lustre] [11000.306783] vfs_unlink+0x202/0x330 [11000.314449] do_unlinkat+0x2fa/0x460 [11000.320695] __x64_sys_unlink+0x37/0x50 [11000.323724] do_syscall_64+0xc1/0x440 [11000.330815] entry_SYSCALL_64_after_hwframe+0x49/0xae [11000.337027] RIP: 0033:0x7f151bdee10b [11000.338619] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [11000.360065] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [11000.364093] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [11000.367307] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [11000.369607] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11000.372219] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11000.374850] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11004.389440] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9073s! [11004.391841] Showing busy workqueues and worker pools: [11004.393330] workqueue events: flags=0x0 [11004.403268] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11004.417458] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [11004.427295] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11004.435409] pending: kfree_rcu_monitor [11004.438714] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11004.445239] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [11004.450216] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11004.458931] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [11004.462939] workqueue events_long: flags=0x0 [11004.471838] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11004.484076] pending: lu_objects_flush [obdclass] [11004.487335] workqueue events_unbound: flags=0x2 [11004.490544] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11004.494802] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11004.503729] workqueue events_power_efficient: flags=0x80 [11004.508594] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11004.512224] pending: do_cache_clean, neigh_periodic_work, check_lifetime [11004.516946] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11004.522028] pending: neigh_periodic_work [11004.524029] workqueue mm_percpu_wq: flags=0x8 [11004.527636] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11004.534112] pending: vmstat_update, lru_add_drain_per_cpu [11004.538264] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11004.542943] pending: vmstat_update [11004.549776] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11004.555252] pending: lru_add_drain_per_cpu [11004.558087] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [11004.566199] pending: lru_add_drain_per_cpu BAR(48) [11004.579073] workqueue writeback: flags=0x4a [11004.583606] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [11004.595014] pending: wb_workfn [11004.597060] workqueue xprtiod: flags=0xa [11004.599931] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11004.604265] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [11004.611339] workqueue ipv6_addrconf: flags=0x40008 [11004.615123] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11004.624056] pending: addrconf_verify_work [11004.625705] workqueue knbd0-recv: flags=0x1a [11004.627426] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11004.631256] in-flight: 658:recv_work [11004.637856] workqueue ptlrpc_pinger: flags=0x2 [11004.641624] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11004.651613] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11004.660520] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9074s workers=4 idle: 32984 37 8 [11004.669098] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10989s workers=2 idle: 661 [11012.055950] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [11012.056276] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11012.060880] Modules linked in: [11012.064446] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11012.064970] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11012.064978] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11012.064984] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11012.065311] RIP: 0010:native_safe_halt+0x2e/0x50 [11012.065322] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11012.065326] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11012.065331] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11012.065333] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [11012.065335] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [11012.065337] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [11012.065339] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [11012.065342] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [11012.065345] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11012.065347] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11012.065371] Call Trace: [11012.065378] [11012.065383] ? show_regs.cold.9+0x22/0x2f [11012.066461] lustre(O) osp(O) ofd(O) lod(O) [11012.098425] ? watchdog_timer_fn.cold.6+0x62/0x10a [11012.098446] ? __hrtimer_run_queues+0x1c2/0x4d0 [11012.098450] ? watchdog+0x40/0x40 [11012.098455] ? hrtimer_interrupt+0x128/0x2c0 [11012.098459] ? smp_apic_timer_interrupt+0xa6/0x270 [11012.098468] ? apic_timer_interrupt+0xf/0x20 [11012.098476] [11012.098481] ? native_safe_halt+0x2e/0x50 [11012.098486] kvm_wait+0xcd/0xe0 [11012.098495] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11012.098505] do_raw_spin_lock+0x177/0x190 [11012.098510] _raw_spin_lock+0x21/0x30 [11012.098515] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11012.098982] process_one_work+0x2c8/0x700 [11012.098995] worker_thread+0x296/0x6e0 [11012.098999] ? rescuer_thread+0x570/0x570 [11012.099003] kthread+0x1d1/0x200 [11012.099007] ? set_kthread_struct+0x70/0x70 [11012.099011] ret_from_fork+0x1f/0x30 [11012.232626] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11012.248809] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11012.252988] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11012.256315] RIP: 0010:native_safe_halt+0x2e/0x50 [11012.258051] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11012.267475] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11012.272237] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [11012.274307] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [11012.285783] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [11012.290867] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [11012.295190] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11012.298859] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [11012.303183] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11012.306316] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11012.309975] Call Trace: [11012.311430] [11012.312383] ? show_regs.cold.9+0x22/0x2f [11012.314440] ? watchdog_timer_fn.cold.6+0x62/0x10a [11012.317473] ? __hrtimer_run_queues+0x1c2/0x4d0 [11012.319958] ? watchdog+0x40/0x40 [11012.321691] ? hrtimer_interrupt+0x128/0x2c0 [11012.324323] ? smp_apic_timer_interrupt+0xa6/0x270 [11012.326893] ? apic_timer_interrupt+0xf/0x20 [11012.329295] [11012.330336] ? native_safe_halt+0x2e/0x50 [11012.332789] kvm_wait+0xcd/0xe0 [11012.334193] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11012.337196] do_raw_spin_lock+0x177/0x190 [11012.339165] _raw_spin_lock+0x21/0x30 [11012.341518] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11012.344290] obd_mod_free+0x46/0x140 [ptlrpc] [11012.347071] mdc_commit_open+0x91/0xb0 [mdc] [11012.350160] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11012.354458] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11012.358385] after_reply+0xe2c/0x1780 [ptlrpc] [11012.362384] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11012.366384] ? __next_timer_interrupt+0x160/0x160 [11012.369557] ptlrpcd+0xa82/0xe40 [ptlrpc] [11012.372385] ? do_wait_intr+0xf0/0xf0 [11012.374545] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11012.377633] kthread+0x1d1/0x200 [11012.379295] ? set_kthread_struct+0x70/0x70 [11012.383391] ret_from_fork+0x1f/0x30 [11020.058259] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [11020.068701] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11020.146256] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11020.153063] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11020.165036] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11020.170915] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11020.195001] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11020.203687] RAX: 0000000000004cda RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11020.209809] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [11020.215188] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [11020.218658] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [11020.228056] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [11020.237548] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [11020.249891] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11020.255637] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [11020.268954] Call Trace: [11020.269945] [11020.272914] ? show_regs.cold.9+0x22/0x2f [11020.275987] ? watchdog_timer_fn.cold.6+0x62/0x10a [11020.278844] ? __hrtimer_run_queues+0x1c2/0x4d0 [11020.282126] ? watchdog+0x40/0x40 [11020.283483] ? hrtimer_interrupt+0x128/0x2c0 [11020.284613] ? smp_apic_timer_interrupt+0xa6/0x270 [11020.285949] ? apic_timer_interrupt+0xf/0x20 [11020.287528] [11020.287990] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11020.289739] do_raw_spin_lock+0x177/0x190 [11020.290635] _raw_spin_lock+0x21/0x30 [11020.291682] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11020.293454] ? do_raw_spin_unlock+0x75/0x190 [11020.294818] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11020.300684] ? do_wait_intr+0xf0/0xf0 [11020.301620] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11020.304410] mdc_close+0x53b/0xd70 [mdc] [11020.305965] lmv_close+0x18d/0x490 [lmv] [11020.307604] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11020.309900] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11020.312427] ll_md_real_close+0xa5/0x2a0 [lustre] [11020.315546] ll_md_close+0x274/0x800 [lustre] [11020.317074] ll_file_release+0x128/0xc50 [lustre] [11020.323002] __fput+0x10d/0x440 [11020.327850] ____fput+0x12/0x20 [11020.330955] task_work_run+0xc6/0x110 [11020.333316] exit_to_usermode_loop+0x1dd/0x1f0 [11020.336684] do_syscall_64+0x3d6/0x440 [11020.338041] entry_SYSCALL_64_after_hwframe+0x49/0xae [11020.339800] RIP: 0033:0x7f151bee3cd8 [11020.353855] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11020.366890] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11020.376970] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [11020.389496] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [11020.396037] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11020.400888] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11020.406816] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11032.057054] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [11032.066262] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11032.169748] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11032.187416] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11032.222468] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11032.231479] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11032.264571] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11032.284735] RAX: 000000000000733f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11032.301853] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [11032.315369] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [11032.333193] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [11032.346910] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [11032.355974] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [11032.374673] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11032.394530] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [11032.412507] Call Trace: [11032.419132] [11032.438679] ? show_regs.cold.9+0x22/0x2f [11032.442496] ? watchdog_timer_fn.cold.6+0x62/0x10a [11032.446137] ? __hrtimer_run_queues+0x1c2/0x4d0 [11032.449553] ? watchdog+0x40/0x40 [11032.451566] ? hrtimer_interrupt+0x128/0x2c0 [11032.454105] ? smp_apic_timer_interrupt+0xa6/0x270 [11032.456939] ? apic_timer_interrupt+0xf/0x20 [11032.459610] [11032.460714] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11032.463559] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11032.466460] do_raw_spin_lock+0x177/0x190 [11032.468352] _raw_spin_lock+0x21/0x30 [11032.470293] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11032.473017] ? do_raw_spin_unlock+0x75/0x190 [11032.475269] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11032.477881] ? do_wait_intr+0xf0/0xf0 [11032.479728] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11032.482475] mdc_reint+0x2d/0x110 [mdc] [11032.484479] mdc_unlink+0x658/0x870 [mdc] [11032.486574] lmv_unlink+0x34d/0xbd0 [lmv] [11032.488674] ll_unlink+0x24d/0x7f0 [lustre] [11032.490869] vfs_unlink+0x202/0x330 [11032.492678] do_unlinkat+0x2fa/0x460 [11032.494644] __x64_sys_unlink+0x37/0x50 [11032.496608] do_syscall_64+0xc1/0x440 [11032.498619] entry_SYSCALL_64_after_hwframe+0x49/0xae [11032.501115] RIP: 0033:0x7f151bdee10b [11032.502940] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [11032.512201] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [11032.516617] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [11032.520921] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [11032.524933] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11032.529005] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11032.532577] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11035.109842] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9104s! [11035.125683] Showing busy workqueues and worker pools: [11035.128872] workqueue events: flags=0x0 [11035.135434] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11035.138345] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [11035.145538] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11035.154347] pending: kfree_rcu_monitor [11035.156481] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11035.160320] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [11035.166992] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11035.169939] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [11035.173843] workqueue events_long: flags=0x0 [11035.175160] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11035.178292] pending: lu_objects_flush [obdclass] [11035.180320] workqueue events_unbound: flags=0x2 [11035.181828] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11035.187940] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11035.204435] workqueue events_power_efficient: flags=0x80 [11035.208114] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11035.212694] pending: do_cache_clean, neigh_periodic_work, check_lifetime [11035.217031] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11035.221348] pending: neigh_periodic_work [11035.224048] workqueue mm_percpu_wq: flags=0x8 [11035.227423] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11035.230182] pending: vmstat_update, lru_add_drain_per_cpu [11035.231898] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11035.234644] pending: vmstat_update [11035.237715] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11035.244635] pending: lru_add_drain_per_cpu [11035.261805] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [11035.264264] pending: lru_add_drain_per_cpu BAR(48) [11035.270608] workqueue writeback: flags=0x4a [11035.276161] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [11035.282319] pending: wb_workfn [11035.287686] workqueue xprtiod: flags=0xa [11035.290088] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11035.295663] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [11035.311439] workqueue ipv6_addrconf: flags=0x40008 [11035.321953] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11035.337714] pending: addrconf_verify_work [11035.346500] workqueue knbd0-recv: flags=0x1a [11035.349310] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11035.358503] in-flight: 658:recv_work [11035.360589] workqueue ptlrpc_pinger: flags=0x2 [11035.369266] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11035.373411] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11035.380891] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9104s workers=4 idle: 32984 37 8 [11035.399438] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11020s workers=2 idle: 661 [11040.056522] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11040.058418] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [11040.080713] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11040.080810] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11040.080818] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11040.080827] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11040.081872] RIP: 0010:native_safe_halt+0x2e/0x50 [11040.081892] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11040.081896] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11040.081900] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11040.082060] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [11040.082063] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [11040.082066] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [11040.082068] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [11040.082072] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [11040.082075] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11040.082139] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11040.082149] Call Trace: [11040.082153] [11040.082158] ? show_regs.cold.9+0x22/0x2f [11040.082172] ? watchdog_timer_fn.cold.6+0x62/0x10a [11040.082179] ? __hrtimer_run_queues+0x1c2/0x4d0 [11040.082183] ? watchdog+0x40/0x40 [11040.082188] ? hrtimer_interrupt+0x128/0x2c0 [11040.082196] ? smp_apic_timer_interrupt+0xa6/0x270 [11040.082203] ? apic_timer_interrupt+0xf/0x20 [11040.082208] [11040.082211] ? native_safe_halt+0x2e/0x50 [11040.082216] kvm_wait+0xcd/0xe0 [11040.082223] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11040.082445] do_raw_spin_lock+0x177/0x190 [11040.082452] _raw_spin_lock+0x21/0x30 [11040.082457] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11040.085980] process_one_work+0x2c8/0x700 [11040.085991] worker_thread+0x296/0x6e0 [11040.086394] ? rescuer_thread+0x570/0x570 [11040.086398] kthread+0x1d1/0x200 [11040.086402] ? set_kthread_struct+0x70/0x70 [11040.086406] ret_from_fork+0x1f/0x30 [11040.516575] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11040.568457] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11040.579523] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11040.590283] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11040.609255] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11040.640444] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11040.651732] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [11040.662889] RDX: 0000000000007b8f RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [11040.670438] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [11040.682355] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [11040.688450] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11040.701282] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [11040.717166] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11040.723416] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11040.725831] Call Trace: [11040.743207] [11040.743944] ? show_regs.cold.9+0x22/0x2f [11040.745391] ? watchdog_timer_fn.cold.6+0x62/0x10a [11040.752049] ? __hrtimer_run_queues+0x1c2/0x4d0 [11040.755303] ? watchdog+0x40/0x40 [11040.760806] ? hrtimer_interrupt+0x128/0x2c0 [11040.768393] ? smp_apic_timer_interrupt+0xa6/0x270 [11040.773539] ? apic_timer_interrupt+0xf/0x20 [11040.775019] [11040.794344] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11040.800617] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11040.807111] do_raw_spin_lock+0x177/0x190 [11040.815302] _raw_spin_lock+0x21/0x30 [11040.818275] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11040.834146] obd_mod_free+0x46/0x140 [ptlrpc] [11040.835925] mdc_commit_open+0x91/0xb0 [mdc] [11040.840154] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11040.844244] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11040.853603] after_reply+0xe2c/0x1780 [ptlrpc] [11040.861909] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11040.866821] ? __next_timer_interrupt+0x160/0x160 [11040.879854] ptlrpcd+0xa82/0xe40 [ptlrpc] [11040.893509] ? do_wait_intr+0xf0/0xf0 [11040.894882] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11040.907454] kthread+0x1d1/0x200 [11040.910886] ? set_kthread_struct+0x70/0x70 [11040.913341] ret_from_fork+0x1f/0x30 [11048.057277] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [11048.076512] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11048.167977] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11048.201622] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11048.217444] RIP: 0010:native_safe_halt+0x2e/0x50 [11048.228157] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11048.272001] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11048.273749] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11048.298612] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [11048.325311] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [11048.349232] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [11048.363616] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [11048.376145] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [11048.402074] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11048.412076] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [11048.426925] Call Trace: [11048.427717] [11048.442149] ? show_regs.cold.9+0x22/0x2f [11048.457815] ? watchdog_timer_fn.cold.6+0x62/0x10a [11048.471135] ? __hrtimer_run_queues+0x1c2/0x4d0 [11048.472622] ? watchdog+0x40/0x40 [11048.473817] ? hrtimer_interrupt+0x128/0x2c0 [11048.509448] ? smp_apic_timer_interrupt+0xa6/0x270 [11048.523086] ? apic_timer_interrupt+0xf/0x20 [11048.539762] [11048.541784] ? native_safe_halt+0x2e/0x50 [11048.562775] kvm_wait+0xcd/0xe0 [11048.579170] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11048.581088] do_raw_spin_lock+0x177/0x190 [11048.598994] _raw_spin_lock+0x21/0x30 [11048.614585] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11048.635026] ? do_raw_spin_unlock+0x75/0x190 [11048.640171] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11048.659524] ? do_wait_intr+0xf0/0xf0 [11048.667671] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11048.682044] mdc_close+0x53b/0xd70 [mdc] [11048.691389] lmv_close+0x18d/0x490 [lmv] [11048.696774] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11048.701719] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11048.720260] ll_md_real_close+0xa5/0x2a0 [lustre] [11048.733610] ll_md_close+0x274/0x800 [lustre] [11048.747579] ll_file_release+0x128/0xc50 [lustre] [11048.760627] __fput+0x10d/0x440 [11048.765636] ____fput+0x12/0x20 [11048.770601] task_work_run+0xc6/0x110 [11048.779027] exit_to_usermode_loop+0x1dd/0x1f0 [11048.790522] do_syscall_64+0x3d6/0x440 [11048.794275] entry_SYSCALL_64_after_hwframe+0x49/0xae [11048.808793] RIP: 0033:0x7f151bee3cd8 [11048.818431] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11048.860462] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11048.873737] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [11048.886053] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [11048.909781] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11048.923655] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11048.940109] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11060.053107] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [11060.058926] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11060.087106] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11060.093371] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11060.104324] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11060.110460] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11060.171511] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11060.189853] RAX: 000000000000250f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11060.216742] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [11060.226366] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [11060.238668] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [11060.244769] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [11060.254510] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [11060.261216] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11060.268680] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [11060.273565] Call Trace: [11060.276334] [11060.278675] ? show_regs.cold.9+0x22/0x2f [11060.282291] ? watchdog_timer_fn.cold.6+0x62/0x10a [11060.286565] ? __hrtimer_run_queues+0x1c2/0x4d0 [11060.292192] ? watchdog+0x40/0x40 [11060.297157] ? hrtimer_interrupt+0x128/0x2c0 [11060.299436] ? smp_apic_timer_interrupt+0xa6/0x270 [11060.303200] ? apic_timer_interrupt+0xf/0x20 [11060.308422] [11060.309521] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11060.315838] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11060.321806] do_raw_spin_lock+0x177/0x190 [11060.323980] _raw_spin_lock+0x21/0x30 [11060.329414] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11060.333669] ? do_raw_spin_unlock+0x75/0x190 [11060.338464] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11060.342719] ? do_wait_intr+0xf0/0xf0 [11060.351230] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11060.354334] mdc_reint+0x2d/0x110 [mdc] [11060.360064] mdc_unlink+0x658/0x870 [mdc] [11060.362241] lmv_unlink+0x34d/0xbd0 [lmv] [11060.366397] ll_unlink+0x24d/0x7f0 [lustre] [11060.371051] vfs_unlink+0x202/0x330 [11060.375175] do_unlinkat+0x2fa/0x460 [11060.380243] __x64_sys_unlink+0x37/0x50 [11060.384318] do_syscall_64+0xc1/0x440 [11060.389036] entry_SYSCALL_64_after_hwframe+0x49/0xae [11060.393556] RIP: 0033:0x7f151bdee10b [11060.398459] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [11060.418562] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [11060.428932] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [11060.437105] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [11060.446202] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11060.454332] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11060.462037] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11065.831348] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9135s! [11065.835256] Showing busy workqueues and worker pools: [11065.837587] workqueue events: flags=0x0 [11065.839594] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11065.843525] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [11065.864732] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11065.874685] pending: kfree_rcu_monitor [11065.882785] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11065.888987] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [11065.903691] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11065.912696] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [11065.923447] workqueue events_long: flags=0x0 [11065.925059] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11065.936392] pending: lu_objects_flush [obdclass] [11065.938325] workqueue events_unbound: flags=0x2 [11065.957026] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11065.965508] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11065.987624] workqueue events_power_efficient: flags=0x80 [11065.991898] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11066.003997] pending: do_cache_clean, neigh_periodic_work, check_lifetime [11066.022785] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11066.039047] pending: neigh_periodic_work [11066.049114] workqueue mm_percpu_wq: flags=0x8 [11066.054580] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11066.067780] pending: vmstat_update, lru_add_drain_per_cpu [11066.076120] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11066.084129] pending: vmstat_update [11066.090815] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11066.095674] pending: lru_add_drain_per_cpu [11066.104538] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [11066.116516] pending: lru_add_drain_per_cpu BAR(48) [11066.125407] workqueue writeback: flags=0x4a [11066.128597] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [11066.137666] pending: wb_workfn [11066.142829] workqueue xprtiod: flags=0xa [11066.148097] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11066.153439] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [11066.161068] workqueue ipv6_addrconf: flags=0x40008 [11066.183588] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11066.185936] pending: addrconf_verify_work [11066.200089] workqueue knbd0-recv: flags=0x1a [11066.209595] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11066.225689] in-flight: 658:recv_work [11066.236116] workqueue ptlrpc_pinger: flags=0x2 [11066.257776] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11066.271112] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11066.296072] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9135s workers=4 idle: 32984 37 8 [11066.304741] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11051s workers=2 idle: 661 [11068.056080] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11068.056198] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [11068.058963] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11068.059036] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11068.059076] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11068.059082] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11068.059656] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11068.059668] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11068.059672] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11068.059682] RAX: 0000000000006e4d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11068.059684] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [11068.059686] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [11068.059688] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [11068.059690] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [11068.059694] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [11068.059697] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11068.059700] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11068.059707] Call Trace: [11068.059713] [11068.059717] ? show_regs.cold.9+0x22/0x2f [11068.059727] ? watchdog_timer_fn.cold.6+0x62/0x10a [11068.059734] ? __hrtimer_run_queues+0x1c2/0x4d0 [11068.059738] ? watchdog+0x40/0x40 [11068.059743] ? hrtimer_interrupt+0x128/0x2c0 [11068.059746] ? smp_apic_timer_interrupt+0xa6/0x270 [11068.059754] ? apic_timer_interrupt+0xf/0x20 [11068.059758] [11068.059760] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11068.059765] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11068.059769] do_raw_spin_lock+0x177/0x190 [11068.059773] _raw_spin_lock+0x21/0x30 [11068.059778] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11068.059985] process_one_work+0x2c8/0x700 [11068.059992] worker_thread+0x296/0x6e0 [11068.059997] ? rescuer_thread+0x570/0x570 [11068.060001] kthread+0x1d1/0x200 [11068.060006] ? set_kthread_struct+0x70/0x70 [11068.060010] ret_from_fork+0x1f/0x30 [11068.334004] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11068.384062] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11068.394048] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11068.402907] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11068.412990] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11068.436387] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11068.449104] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [11068.467325] RDX: 0000000000007f6b RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [11068.480009] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [11068.491957] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [11068.494540] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11068.512306] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [11068.528614] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11068.533024] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11068.538219] Call Trace: [11068.539927] [11068.543990] ? show_regs.cold.9+0x22/0x2f [11068.550991] ? watchdog_timer_fn.cold.6+0x62/0x10a [11068.552713] ? __hrtimer_run_queues+0x1c2/0x4d0 [11068.558986] ? watchdog+0x40/0x40 [11068.560230] ? hrtimer_interrupt+0x128/0x2c0 [11068.569625] ? smp_apic_timer_interrupt+0xa6/0x270 [11068.577056] ? apic_timer_interrupt+0xf/0x20 [11068.579285] [11068.580050] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11068.598260] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11068.599521] do_raw_spin_lock+0x177/0x190 [11068.600481] _raw_spin_lock+0x21/0x30 [11068.611108] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11068.622136] obd_mod_free+0x46/0x140 [ptlrpc] [11068.628019] mdc_commit_open+0x91/0xb0 [mdc] [11068.632344] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11068.638125] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11068.646098] after_reply+0xe2c/0x1780 [ptlrpc] [11068.652320] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11068.655379] ? __next_timer_interrupt+0x160/0x160 [11068.661886] ptlrpcd+0xa82/0xe40 [ptlrpc] [11068.666095] ? do_wait_intr+0xf0/0xf0 [11068.670284] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11068.686224] kthread+0x1d1/0x200 [11068.687664] ? set_kthread_struct+0x70/0x70 [11068.691591] ret_from_fork+0x1f/0x30 [11076.064664] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [11076.080771] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11076.136460] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11076.154026] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11076.161382] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11076.165701] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11076.181214] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11076.192542] RAX: 0000000000000bef RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11076.199366] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [11076.204052] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [11076.207329] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [11076.209166] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [11076.217799] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [11076.223065] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11076.226269] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [11076.230782] Call Trace: [11076.231845] [11076.233394] ? show_regs.cold.9+0x22/0x2f [11076.235495] ? watchdog_timer_fn.cold.6+0x62/0x10a [11076.237321] ? __hrtimer_run_queues+0x1c2/0x4d0 [11076.239650] ? watchdog+0x40/0x40 [11076.243518] ? hrtimer_interrupt+0x128/0x2c0 [11076.246078] ? smp_apic_timer_interrupt+0xa6/0x270 [11076.251580] ? apic_timer_interrupt+0xf/0x20 [11076.253084] [11076.253841] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11076.262250] do_raw_spin_lock+0x177/0x190 [11076.263585] _raw_spin_lock+0x21/0x30 [11076.270809] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11076.276371] ? do_raw_spin_unlock+0x75/0x190 [11076.283674] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11076.291947] ? do_wait_intr+0xf0/0xf0 [11076.295591] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11076.304165] mdc_close+0x53b/0xd70 [mdc] [11076.307558] lmv_close+0x18d/0x490 [lmv] [11076.309997] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11076.313663] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11076.317354] ll_md_real_close+0xa5/0x2a0 [lustre] [11076.319856] ll_md_close+0x274/0x800 [lustre] [11076.322263] ll_file_release+0x128/0xc50 [lustre] [11076.324319] __fput+0x10d/0x440 [11076.326399] ____fput+0x12/0x20 [11076.327894] task_work_run+0xc6/0x110 [11076.332069] exit_to_usermode_loop+0x1dd/0x1f0 [11076.333563] do_syscall_64+0x3d6/0x440 [11076.337283] entry_SYSCALL_64_after_hwframe+0x49/0xae [11076.343924] RIP: 0033:0x7f151bee3cd8 [11076.348799] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11076.364521] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11076.376083] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [11076.386979] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [11076.392072] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11076.403252] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11076.408571] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11092.052454] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [11092.061119] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11092.092195] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11092.098663] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11092.111695] RIP: 0010:native_safe_halt+0x2e/0x50 [11092.114319] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11092.131349] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11092.138547] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11092.144061] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [11092.153617] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [11092.162509] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [11092.173130] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [11092.180265] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [11092.188193] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11092.199274] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [11092.206678] Call Trace: [11092.214674] [11092.218261] ? show_regs.cold.9+0x22/0x2f [11092.220799] ? watchdog_timer_fn.cold.6+0x62/0x10a [11092.224642] ? __hrtimer_run_queues+0x1c2/0x4d0 [11092.226201] ? watchdog+0x40/0x40 [11092.239376] ? hrtimer_interrupt+0x128/0x2c0 [11092.248202] ? smp_apic_timer_interrupt+0xa6/0x270 [11092.253074] ? apic_timer_interrupt+0xf/0x20 [11092.255270] [11092.256077] ? native_safe_halt+0x2e/0x50 [11092.257559] kvm_wait+0xcd/0xe0 [11092.264220] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11092.270161] do_raw_spin_lock+0x177/0x190 [11092.274978] _raw_spin_lock+0x21/0x30 [11092.277595] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11092.286265] ? do_raw_spin_unlock+0x75/0x190 [11092.289209] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11092.296193] ? do_wait_intr+0xf0/0xf0 [11092.302709] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11092.313777] mdc_reint+0x2d/0x110 [mdc] [11092.318608] mdc_unlink+0x658/0x870 [mdc] [11092.321682] lmv_unlink+0x34d/0xbd0 [lmv] [11092.325637] ll_unlink+0x24d/0x7f0 [lustre] [11092.328107] vfs_unlink+0x202/0x330 [11092.329477] do_unlinkat+0x2fa/0x460 [11092.333692] __x64_sys_unlink+0x37/0x50 [11092.338610] do_syscall_64+0xc1/0x440 [11092.341590] entry_SYSCALL_64_after_hwframe+0x49/0xae [11092.344598] RIP: 0033:0x7f151bdee10b [11092.347528] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [11092.358433] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [11092.362588] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [11092.366656] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [11092.370344] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11092.374088] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11092.377646] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11096.055214] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [11096.057042] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [11096.058424] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11096.058513] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11096.058520] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11096.058523] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11096.058535] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11096.058539] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11096.058554] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [11096.058557] RDX: 00000000000043d7 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [11096.058559] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [11096.058562] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [11096.058565] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11096.058568] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [11096.058571] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11096.058573] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11096.058581] Call Trace: [11096.058587] [11096.058592] ? show_regs.cold.9+0x22/0x2f [11096.058601] ? watchdog_timer_fn.cold.6+0x62/0x10a [11096.058607] ? __hrtimer_run_queues+0x1c2/0x4d0 [11096.058611] ? watchdog+0x40/0x40 [11096.058616] ? hrtimer_interrupt+0x128/0x2c0 [11096.058620] ? smp_apic_timer_interrupt+0xa6/0x270 [11096.058629] ? apic_timer_interrupt+0xf/0x20 [11096.058633] [11096.058636] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11096.058641] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11096.058646] do_raw_spin_lock+0x177/0x190 [11096.058651] _raw_spin_lock+0x21/0x30 [11096.058663] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11096.062824] Modules linked in: [11096.097249] obd_mod_free+0x46/0x140 [ptlrpc] [11096.097558] mdc_commit_open+0x91/0xb0 [mdc] [11096.104526] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11096.104607] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11096.104614] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11096.104620] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11096.105301] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11096.105313] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11096.118278] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11096.118597] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11096.118969] after_reply+0xe2c/0x1780 [ptlrpc] [11096.119211] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11096.119468] ? __next_timer_interrupt+0x160/0x160 [11096.119607] ptlrpcd+0xa82/0xe40 [ptlrpc] [11096.119896] ? do_wait_intr+0xf0/0xf0 [11096.119906] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11096.120107] kthread+0x1d1/0x200 [11096.120114] ? set_kthread_struct+0x70/0x70 [11096.120118] ret_from_fork+0x1f/0x30 [11096.382210] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11096.392482] RAX: 0000000000007513 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11096.404048] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [11096.414405] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [11096.425875] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [11096.436283] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [11096.439834] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [11096.443455] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11096.446416] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11096.454482] Call Trace: [11096.455623] [11096.456943] ? show_regs.cold.9+0x22/0x2f [11096.463713] ? watchdog_timer_fn.cold.6+0x62/0x10a [11096.467929] ? __hrtimer_run_queues+0x1c2/0x4d0 [11096.471947] ? watchdog+0x40/0x40 [11096.473877] ? hrtimer_interrupt+0x128/0x2c0 [11096.478901] ? smp_apic_timer_interrupt+0xa6/0x270 [11096.487521] ? apic_timer_interrupt+0xf/0x20 [11096.489007] [11096.491771] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11096.494282] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11096.497375] do_raw_spin_lock+0x177/0x190 [11096.499069] _raw_spin_lock+0x21/0x30 [11096.501547] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11096.505429] process_one_work+0x2c8/0x700 [11096.509136] worker_thread+0x296/0x6e0 [11096.511319] ? rescuer_thread+0x570/0x570 [11096.515894] kthread+0x1d1/0x200 [11096.517849] ? set_kthread_struct+0x70/0x70 [11096.522674] ret_from_fork+0x1f/0x30 [11096.550454] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9165s! [11096.557015] Showing busy workqueues and worker pools: [11096.559485] workqueue events: flags=0x0 [11096.561610] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11096.563985] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [11096.567521] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11096.569882] pending: kfree_rcu_monitor [11096.571364] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11096.574093] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [11096.577153] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11096.579400] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [11096.582217] workqueue events_long: flags=0x0 [11096.583877] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11096.586641] pending: lu_objects_flush [obdclass] [11096.591327] workqueue events_unbound: flags=0x2 [11096.595037] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11096.597180] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11096.607611] workqueue events_power_efficient: flags=0x80 [11096.612021] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11096.617222] pending: do_cache_clean, neigh_periodic_work, check_lifetime [11096.623011] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11096.629364] pending: neigh_periodic_work [11096.631285] workqueue mm_percpu_wq: flags=0x8 [11096.636026] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11096.644737] pending: vmstat_update, lru_add_drain_per_cpu [11096.649062] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11096.654907] pending: vmstat_update [11096.657283] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11096.663150] pending: lru_add_drain_per_cpu [11096.668144] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [11096.671174] pending: lru_add_drain_per_cpu BAR(48) [11096.673902] workqueue writeback: flags=0x4a [11096.678889] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [11096.683076] pending: wb_workfn [11096.686883] workqueue xprtiod: flags=0xa [11096.689889] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11096.693273] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [11096.703129] workqueue ipv6_addrconf: flags=0x40008 [11096.705848] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11096.711218] pending: addrconf_verify_work [11096.715568] workqueue knbd0-recv: flags=0x1a [11096.716974] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11096.720208] in-flight: 658:recv_work [11096.721566] workqueue ptlrpc_pinger: flags=0x2 [11096.726661] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11096.729688] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11096.737203] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9166s workers=4 idle: 32984 37 8 [11096.747339] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11081s workers=2 idle: 661 [11104.057045] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [11104.060099] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11104.080711] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11104.086509] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11104.090808] RIP: 0010:native_safe_halt+0x2e/0x50 [11104.092342] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11104.101637] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11104.105758] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11104.109127] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [11104.112294] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [11104.115264] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [11104.118682] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [11104.121978] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [11104.125448] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11104.128027] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [11104.131120] Call Trace: [11104.132373] [11104.133661] ? show_regs.cold.9+0x22/0x2f [11104.135980] ? watchdog_timer_fn.cold.6+0x62/0x10a [11104.138440] ? __hrtimer_run_queues+0x1c2/0x4d0 [11104.140394] ? watchdog+0x40/0x40 [11104.142720] ? hrtimer_interrupt+0x128/0x2c0 [11104.144536] ? smp_apic_timer_interrupt+0xa6/0x270 [11104.147212] ? apic_timer_interrupt+0xf/0x20 [11104.149459] [11104.150433] ? native_safe_halt+0x2e/0x50 [11104.152229] kvm_wait+0xcd/0xe0 [11104.153819] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11104.156240] do_raw_spin_lock+0x177/0x190 [11104.158508] _raw_spin_lock+0x21/0x30 [11104.159950] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11104.162615] ? do_raw_spin_unlock+0x75/0x190 [11104.164121] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11104.166559] ? do_wait_intr+0xf0/0xf0 [11104.168095] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11104.170722] mdc_close+0x53b/0xd70 [mdc] [11104.172555] lmv_close+0x18d/0x490 [lmv] [11104.174434] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11104.177239] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11104.180250] ll_md_real_close+0xa5/0x2a0 [lustre] [11104.182984] ll_md_close+0x274/0x800 [lustre] [11104.185445] ll_file_release+0x128/0xc50 [lustre] [11104.187724] __fput+0x10d/0x440 [11104.189124] ____fput+0x12/0x20 [11104.190749] task_work_run+0xc6/0x110 [11104.192666] exit_to_usermode_loop+0x1dd/0x1f0 [11104.194894] do_syscall_64+0x3d6/0x440 [11104.196643] entry_SYSCALL_64_after_hwframe+0x49/0xae [11104.199248] RIP: 0033:0x7f151bee3cd8 [11104.201097] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11104.209362] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11104.213187] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [11104.217584] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [11104.221159] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11104.225110] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11104.228539] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11124.052027] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [11124.056026] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [11124.058547] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11124.058616] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11124.058622] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11124.058625] RIP: 0010:native_safe_halt+0x2e/0x50 [11124.058638] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11124.058642] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11124.058646] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11124.058648] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [11124.058650] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [11124.058652] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [11124.058654] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [11124.058657] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [11124.058660] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11124.058662] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [11124.058670] Call Trace: [11124.058675] [11124.058681] ? show_regs.cold.9+0x22/0x2f [11124.058692] ? watchdog_timer_fn.cold.6+0x62/0x10a [11124.058699] ? __hrtimer_run_queues+0x1c2/0x4d0 [11124.058703] ? watchdog+0x40/0x40 [11124.058708] ? hrtimer_interrupt+0x128/0x2c0 [11124.058712] ? smp_apic_timer_interrupt+0xa6/0x270 [11124.058719] ? apic_timer_interrupt+0xf/0x20 [11124.058724] [11124.058727] ? native_safe_halt+0x2e/0x50 [11124.058730] kvm_wait+0xcd/0xe0 [11124.058737] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11124.058745] do_raw_spin_lock+0x177/0x190 [11124.058750] _raw_spin_lock+0x21/0x30 [11124.058756] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11124.059333] ? do_raw_spin_unlock+0x75/0x190 [11124.059339] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11124.059554] ? do_wait_intr+0xf0/0xf0 [11124.059562] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11124.059758] mdc_reint+0x2d/0x110 [mdc] [11124.059796] mdc_unlink+0x658/0x870 [mdc] [11124.059822] lmv_unlink+0x34d/0xbd0 [lmv] [11124.059864] ll_unlink+0x24d/0x7f0 [lustre] [11124.059969] vfs_unlink+0x202/0x330 [11124.059977] do_unlinkat+0x2fa/0x460 [11124.059982] __x64_sys_unlink+0x37/0x50 [11124.059985] do_syscall_64+0xc1/0x440 [11124.059991] entry_SYSCALL_64_after_hwframe+0x49/0xae [11124.059997] RIP: 0033:0x7f151bdee10b [11124.060004] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [11124.060008] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [11124.060012] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [11124.060015] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [11124.060017] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11124.060019] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11124.060021] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11124.060924] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [11124.068688] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11124.068762] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11124.068770] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11124.068776] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11124.069054] RIP: 0010:native_safe_halt+0x2e/0x50 [11124.069065] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11124.069069] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11124.069073] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11124.069076] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [11124.069078] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [11124.069080] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [11124.069083] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [11124.069086] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [11124.069088] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11124.069090] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11124.069099] Call Trace: [11124.069105] [11124.069110] ? show_regs.cold.9+0x22/0x2f [11124.069119] ? watchdog_timer_fn.cold.6+0x62/0x10a [11124.069126] ? __hrtimer_run_queues+0x1c2/0x4d0 [11124.069131] ? watchdog+0x40/0x40 [11124.069136] ? hrtimer_interrupt+0x128/0x2c0 [11124.069140] ? smp_apic_timer_interrupt+0xa6/0x270 [11124.069284] ? apic_timer_interrupt+0xf/0x20 [11124.069288] [11124.069291] ? native_safe_halt+0x2e/0x50 [11124.069294] kvm_wait+0xcd/0xe0 [11124.069301] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11124.069309] do_raw_spin_lock+0x177/0x190 [11124.069313] _raw_spin_lock+0x21/0x30 [11124.069317] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11124.069520] process_one_work+0x2c8/0x700 [11124.069526] worker_thread+0x296/0x6e0 [11124.069530] ? rescuer_thread+0x570/0x570 [11124.069533] kthread+0x1d1/0x200 [11124.069536] ? set_kthread_struct+0x70/0x70 [11124.069539] ret_from_fork+0x1f/0x30 [11124.530569] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11124.566648] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11124.572575] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11124.575580] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11124.578191] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11124.584956] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11124.588064] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [11124.590725] RDX: 0000000000007f87 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [11124.593418] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [11124.605995] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [11124.612988] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11124.620662] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [11124.629002] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11124.630695] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11124.645913] Call Trace: [11124.648004] [11124.650598] ? show_regs.cold.9+0x22/0x2f [11124.652523] ? watchdog_timer_fn.cold.6+0x62/0x10a [11124.659642] ? __hrtimer_run_queues+0x1c2/0x4d0 [11124.666273] ? watchdog+0x40/0x40 [11124.668604] ? hrtimer_interrupt+0x128/0x2c0 [11124.671381] ? smp_apic_timer_interrupt+0xa6/0x270 [11124.674805] ? apic_timer_interrupt+0xf/0x20 [11124.680218] [11124.682307] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11124.687031] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11124.691079] do_raw_spin_lock+0x177/0x190 [11124.699681] _raw_spin_lock+0x21/0x30 [11124.701971] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11124.709590] obd_mod_free+0x46/0x140 [ptlrpc] [11124.714196] mdc_commit_open+0x91/0xb0 [mdc] [11124.719485] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11124.721850] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11124.733301] after_reply+0xe2c/0x1780 [ptlrpc] [11124.744508] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11124.753841] ? __next_timer_interrupt+0x160/0x160 [11124.757026] ptlrpcd+0xa82/0xe40 [ptlrpc] [11124.761767] ? do_wait_intr+0xf0/0xf0 [11124.763128] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11124.766688] kthread+0x1d1/0x200 [11124.778772] ? set_kthread_struct+0x70/0x70 [11124.782784] ret_from_fork+0x1f/0x30 [11127.269779] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9196s! [11127.275748] Showing busy workqueues and worker pools: [11127.279663] workqueue events: flags=0x0 [11127.281692] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11127.287103] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [11127.295075] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11127.301858] pending: kfree_rcu_monitor [11127.304085] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11127.308900] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [11127.315433] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11127.319711] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [11127.324870] workqueue events_long: flags=0x0 [11127.327017] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11127.331082] pending: lu_objects_flush [obdclass] [11127.333930] workqueue events_unbound: flags=0x2 [11127.336802] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11127.340402] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11127.345990] workqueue events_power_efficient: flags=0x80 [11127.348577] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11127.352049] pending: do_cache_clean, neigh_periodic_work, check_lifetime [11127.355259] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11127.358757] pending: neigh_periodic_work [11127.361298] workqueue mm_percpu_wq: flags=0x8 [11127.363830] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11127.366975] pending: vmstat_update, lru_add_drain_per_cpu [11127.369473] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11127.372883] pending: vmstat_update [11127.374408] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11127.378771] pending: lru_add_drain_per_cpu [11127.381510] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [11127.386934] pending: lru_add_drain_per_cpu BAR(48) [11127.389924] workqueue writeback: flags=0x4a [11127.393835] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [11127.397498] pending: wb_workfn [11127.399879] workqueue xprtiod: flags=0xa [11127.402095] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11127.406046] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [11127.411307] workqueue ipv6_addrconf: flags=0x40008 [11127.416090] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11127.420092] pending: addrconf_verify_work [11127.423074] workqueue knbd0-recv: flags=0x1a [11127.425263] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11127.428474] in-flight: 658:recv_work [11127.430584] workqueue ptlrpc_pinger: flags=0x2 [11127.433053] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11127.436494] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11127.440363] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9196s workers=4 idle: 32984 37 8 [11127.444322] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11112s workers=2 idle: 661 [11132.057155] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [11132.060502] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11132.108023] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11132.116745] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11132.125847] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11132.132975] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11132.144699] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11132.148870] RAX: 000000000000321e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11132.152117] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [11132.156837] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [11132.161952] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [11132.165790] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [11132.168456] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [11132.171904] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11132.174425] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [11132.178156] Call Trace: [11132.181869] [11132.184437] ? show_regs.cold.9+0x22/0x2f [11132.185966] ? watchdog_timer_fn.cold.6+0x62/0x10a [11132.188862] ? __hrtimer_run_queues+0x1c2/0x4d0 [11132.191318] ? watchdog+0x40/0x40 [11132.193661] ? hrtimer_interrupt+0x128/0x2c0 [11132.196814] ? smp_apic_timer_interrupt+0xa6/0x270 [11132.201928] ? apic_timer_interrupt+0xf/0x20 [11132.204696] [11132.205857] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11132.209363] do_raw_spin_lock+0x177/0x190 [11132.211389] _raw_spin_lock+0x21/0x30 [11132.213582] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11132.216724] ? do_raw_spin_unlock+0x75/0x190 [11132.219367] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11132.222667] ? do_wait_intr+0xf0/0xf0 [11132.224660] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11132.227727] mdc_close+0x53b/0xd70 [mdc] [11132.235570] lmv_close+0x18d/0x490 [lmv] [11132.241483] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11132.249162] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11132.257108] ll_md_real_close+0xa5/0x2a0 [lustre] [11132.260678] ll_md_close+0x274/0x800 [lustre] [11132.264798] ll_file_release+0x128/0xc50 [lustre] [11132.267901] __fput+0x10d/0x440 [11132.272366] ____fput+0x12/0x20 [11132.275118] task_work_run+0xc6/0x110 [11132.277600] exit_to_usermode_loop+0x1dd/0x1f0 [11132.285128] do_syscall_64+0x3d6/0x440 [11132.287025] entry_SYSCALL_64_after_hwframe+0x49/0xae [11132.289697] RIP: 0033:0x7f151bee3cd8 [11132.291186] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11132.320956] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11132.333919] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [11132.340875] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [11132.346313] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11132.349665] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11132.353702] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11152.054810] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [11152.055252] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [11152.063602] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) [11152.064443] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [11152.064450] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11152.064517] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11152.064523] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11152.064530] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11152.066152] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11152.066566] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11152.066573] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11152.066577] RAX: 0000000000003f7b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11152.066584] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [11152.066587] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [11152.066589] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [11152.066592] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [11152.066595] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [11152.066598] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11152.066601] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11152.066783] Call Trace: [11152.066794] [11152.066799] ? show_regs.cold.9+0x22/0x2f [11152.066810] ? watchdog_timer_fn.cold.6+0x62/0x10a [11152.066815] ? __hrtimer_run_queues+0x1c2/0x4d0 [11152.066820] ? watchdog+0x40/0x40 [11152.066834] ? hrtimer_interrupt+0x128/0x2c0 [11152.066839] ? smp_apic_timer_interrupt+0xa6/0x270 [11152.066847] ? apic_timer_interrupt+0xf/0x20 [11152.066851] [11152.066854] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11152.066859] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11152.066865] do_raw_spin_lock+0x177/0x190 [11152.066870] _raw_spin_lock+0x21/0x30 [11152.066875] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11152.068233] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11152.068307] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11152.068313] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11152.068414] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11152.068427] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11152.068509] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11152.068515] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [11152.068517] RDX: 00000000000055b9 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [11152.068519] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [11152.068522] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [11152.068524] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11152.068527] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [11152.068530] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11152.068532] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11152.068540] Call Trace: [11152.068545] [11152.068549] ? show_regs.cold.9+0x22/0x2f [11152.068558] ? watchdog_timer_fn.cold.6+0x62/0x10a [11152.068563] ? __hrtimer_run_queues+0x1c2/0x4d0 [11152.068568] ? watchdog+0x40/0x40 [11152.068572] ? hrtimer_interrupt+0x128/0x2c0 [11152.068576] ? smp_apic_timer_interrupt+0xa6/0x270 [11152.068583] ? apic_timer_interrupt+0xf/0x20 [11152.068588] [11152.068591] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11152.068596] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11152.068601] do_raw_spin_lock+0x177/0x190 [11152.068606] _raw_spin_lock+0x21/0x30 [11152.068611] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11152.068875] process_one_work+0x2c8/0x700 [11152.068883] worker_thread+0x296/0x6e0 [11152.068888] ? rescuer_thread+0x570/0x570 [11152.068892] kthread+0x1d1/0x200 [11152.068897] ? set_kthread_struct+0x70/0x70 [11152.068900] ret_from_fork+0x1f/0x30 [11152.069120] obd_mod_free+0x46/0x140 [ptlrpc] [11152.069528] mdc_commit_open+0x91/0xb0 [mdc] [11152.069569] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11152.072384] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11152.075214] after_reply+0xe2c/0x1780 [ptlrpc] [11152.075427] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11152.076311] ? __next_timer_interrupt+0x160/0x160 [11152.077180] ptlrpcd+0xa82/0xe40 [ptlrpc] [11152.077602] ? do_wait_intr+0xf0/0xf0 [11152.077612] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11152.087859] mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11152.087930] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11152.087937] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11152.087940] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11152.087955] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11152.087960] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11152.087965] RAX: 000000000000718a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11152.087967] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [11152.087969] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [11152.087972] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [11152.087974] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [11152.087977] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [11152.087980] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11152.087982] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [11152.087991] Call Trace: [11152.089301] [11152.089312] ? show_regs.cold.9+0x22/0x2f [11152.089340] ? watchdog_timer_fn.cold.6+0x62/0x10a [11152.089346] ? __hrtimer_run_queues+0x1c2/0x4d0 [11152.089351] ? watchdog+0x40/0x40 [11152.089355] ? hrtimer_interrupt+0x128/0x2c0 [11152.089359] ? smp_apic_timer_interrupt+0xa6/0x270 [11152.089369] ? apic_timer_interrupt+0xf/0x20 [11152.089374] [11152.089378] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11152.089385] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11152.089390] do_raw_spin_lock+0x177/0x190 [11152.089396] _raw_spin_lock+0x21/0x30 [11152.089403] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11152.089683] ? do_raw_spin_unlock+0x75/0x190 [11152.089688] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11152.089917] ? do_wait_intr+0xf0/0xf0 [11152.089925] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11152.090158] mdc_reint+0x2d/0x110 [mdc] [11152.090198] mdc_unlink+0x658/0x870 [mdc] [11152.090227] lmv_unlink+0x34d/0xbd0 [lmv] [11152.090263] ll_unlink+0x24d/0x7f0 [lustre] [11152.090395] vfs_unlink+0x202/0x330 [11152.090403] do_unlinkat+0x2fa/0x460 [11152.090409] __x64_sys_unlink+0x37/0x50 [11152.090412] do_syscall_64+0xc1/0x440 [11152.090419] entry_SYSCALL_64_after_hwframe+0x49/0xae [11152.090426] RIP: 0033:0x7f151bdee10b [11152.090435] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [11152.090439] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [11152.090443] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [11152.090445] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [11152.090447] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11152.090449] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11152.090451] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11153.218503] kthread+0x1d1/0x200 [11153.220475] ? set_kthread_struct+0x70/0x70 [11153.229402] ret_from_fork+0x1f/0x30 [11157.476041] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9226s! [11157.488577] Showing busy workqueues and worker pools: [11157.496779] workqueue events: flags=0x0 [11157.501131] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11157.515547] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [11157.533091] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11157.542263] pending: kfree_rcu_monitor [11157.556455] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11157.565806] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [11157.581144] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11157.591902] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [11157.602572] workqueue events_long: flags=0x0 [11157.606485] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11157.615920] pending: lu_objects_flush [obdclass] [11157.619087] workqueue events_unbound: flags=0x2 [11157.628639] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11157.631954] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11157.644323] workqueue events_power_efficient: flags=0x80 [11157.650885] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11157.659815] pending: do_cache_clean, neigh_periodic_work, check_lifetime [11157.676314] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11157.678622] pending: neigh_periodic_work [11157.683751] workqueue mm_percpu_wq: flags=0x8 [11157.685366] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11157.703698] pending: vmstat_update, lru_add_drain_per_cpu [11157.718631] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11157.732974] pending: vmstat_update [11157.742546] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11157.749441] pending: lru_add_drain_per_cpu [11157.762299] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [11157.769178] pending: lru_add_drain_per_cpu BAR(48) [11157.772326] workqueue writeback: flags=0x4a [11157.777985] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [11157.784418] pending: wb_workfn [11157.786338] workqueue xprtiod: flags=0xa [11157.790910] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11157.796156] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [11157.801511] workqueue ipv6_addrconf: flags=0x40008 [11157.806740] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11157.814326] pending: addrconf_verify_work [11157.818493] workqueue knbd0-recv: flags=0x1a [11157.821256] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11157.828754] in-flight: 658:recv_work [11157.836715] workqueue ptlrpc_pinger: flags=0x2 [11157.838765] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11157.851465] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11157.860424] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9227s workers=4 idle: 32984 37 8 [11157.874183] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11142s workers=2 idle: 661 [11160.062865] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [11160.072410] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11160.126531] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11160.144481] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11160.156605] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11160.167385] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11160.184662] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11160.187414] RAX: 0000000000007f8b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11160.189704] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [11160.197675] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [11160.202130] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [11160.205093] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [11160.211497] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [11160.221540] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11160.243262] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [11160.245428] Call Trace: [11160.246238] [11160.246792] ? show_regs.cold.9+0x22/0x2f [11160.252358] ? watchdog_timer_fn.cold.6+0x62/0x10a [11160.258639] ? __hrtimer_run_queues+0x1c2/0x4d0 [11160.261633] ? watchdog+0x40/0x40 [11160.263248] ? hrtimer_interrupt+0x128/0x2c0 [11160.265158] ? smp_apic_timer_interrupt+0xa6/0x270 [11160.267186] ? apic_timer_interrupt+0xf/0x20 [11160.273278] [11160.278654] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11160.282812] do_raw_spin_lock+0x177/0x190 [11160.285667] _raw_spin_lock+0x21/0x30 [11160.289654] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11160.292657] ? do_raw_spin_unlock+0x75/0x190 [11160.293744] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11160.302422] ? do_wait_intr+0xf0/0xf0 [11160.303676] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11160.313608] mdc_close+0x53b/0xd70 [mdc] [11160.315642] lmv_close+0x18d/0x490 [lmv] [11160.319835] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11160.322389] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11160.325099] ll_md_real_close+0xa5/0x2a0 [lustre] [11160.326894] ll_md_close+0x274/0x800 [lustre] [11160.333455] ll_file_release+0x128/0xc50 [lustre] [11160.336517] __fput+0x10d/0x440 [11160.338723] ____fput+0x12/0x20 [11160.339769] task_work_run+0xc6/0x110 [11160.345194] exit_to_usermode_loop+0x1dd/0x1f0 [11160.347190] do_syscall_64+0x3d6/0x440 [11160.352265] entry_SYSCALL_64_after_hwframe+0x49/0xae [11160.354931] RIP: 0033:0x7f151bee3cd8 [11160.357159] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11160.366650] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11160.370028] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [11160.372629] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [11160.378036] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11160.381381] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11160.383767] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11164.842731] rcu: INFO: rcu_sched self-detected stall on CPU [11164.853002] rcu: 1-...!: (5307063 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [11164.876729] (t=9240195 jiffies g=249657 q=5096) [11164.881181] rcu: rcu_sched kthread starved for 9235147 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [11164.886238] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [11164.889622] rcu: RCU grace-period kthread stack dump: [11164.894646] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [11164.909056] Call Trace: [11164.910907] __schedule+0x351/0xcb0 [11164.914586] schedule+0xc0/0x180 [11164.922563] schedule_timeout+0xb4/0x190 [11164.926755] ? do_raw_spin_unlock+0x75/0x190 [11164.930129] ? __next_timer_interrupt+0x160/0x160 [11164.936228] ? prepare_to_swait_event+0xf3/0x140 [11164.941351] rcu_gp_kthread+0x8dd/0x15b0 [11164.944116] ? force_qs_rnp+0x2c0/0x2c0 [11164.952089] kthread+0x1d1/0x200 [11164.955579] ? set_kthread_struct+0x70/0x70 [11164.962635] ret_from_fork+0x1f/0x30 [11164.970201] rcu: Stack dump where RCU GP kthread last ran: [11164.977658] Sending NMI from CPU 1 to CPUs 2: [11164.982510] NMI backtrace for cpu 2 skipped: idling at native_safe_halt+0x2e/0x50 [11164.986704] NMI backtrace for cpu 1 [11164.988536] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11164.993908] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11165.000692] Call Trace: [11165.002199] [11165.003197] ? dump_stack+0xbb/0x10e [11165.005881] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [11165.007869] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [11165.010938] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [11165.014206] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [11165.017180] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [11165.020206] ? rcu_dump_cpu_stacks+0x11b/0x171 [11165.022200] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [11165.025158] ? update_process_times+0x6d/0xc0 [11165.027614] ? tick_sched_handle.isra.18+0x27/0x90 [11165.029972] ? tick_sched_timer+0x53/0xb0 [11165.032950] ? __hrtimer_run_queues+0x1c2/0x4d0 [11165.038174] ? tick_sched_handle.isra.18+0x90/0x90 [11165.043099] ? hrtimer_interrupt+0x128/0x2c0 [11165.048019] ? smp_apic_timer_interrupt+0xa6/0x270 [11165.056127] ? apic_timer_interrupt+0xf/0x20 [11165.060150] [11165.061842] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11165.068621] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11165.071632] ? do_raw_spin_lock+0x177/0x190 [11165.074692] ? _raw_spin_lock+0x21/0x30 [11165.079728] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11165.084821] ? obd_mod_free+0x46/0x140 [ptlrpc] [11165.088511] ? mdc_commit_open+0x91/0xb0 [mdc] [11165.093123] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11165.099057] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11165.104089] ? after_reply+0xe2c/0x1780 [ptlrpc] [11165.109647] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11165.112594] ? __next_timer_interrupt+0x160/0x160 [11165.117805] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [11165.122867] ? do_wait_intr+0xf0/0xf0 [11165.126139] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11165.136951] ? kthread+0x1d1/0x200 [11165.138094] ? set_kthread_struct+0x70/0x70 [11165.145229] ? ret_from_fork+0x1f/0x30 [11184.053486] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [11184.060506] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11184.120127] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11184.141230] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11184.159018] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11184.164047] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11184.183518] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11184.194962] RAX: 0000000000004265 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11184.201292] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [11184.208077] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [11184.217125] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [11184.224002] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [11184.232201] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [11184.239574] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11184.256084] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [11184.264915] Call Trace: [11184.268373] [11184.271187] ? show_regs.cold.9+0x22/0x2f [11184.275384] ? watchdog_timer_fn.cold.6+0x62/0x10a [11184.281318] ? __hrtimer_run_queues+0x1c2/0x4d0 [11184.286245] ? watchdog+0x40/0x40 [11184.287484] ? hrtimer_interrupt+0x128/0x2c0 [11184.295936] ? smp_apic_timer_interrupt+0xa6/0x270 [11184.299871] ? apic_timer_interrupt+0xf/0x20 [11184.306322] [11184.307458] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11184.314852] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11184.318464] do_raw_spin_lock+0x177/0x190 [11184.322217] _raw_spin_lock+0x21/0x30 [11184.326168] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11184.329314] ? do_raw_spin_unlock+0x75/0x190 [11184.338324] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11184.343323] ? do_wait_intr+0xf0/0xf0 [11184.345928] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11184.350839] mdc_reint+0x2d/0x110 [mdc] [11184.353119] mdc_unlink+0x658/0x870 [mdc] [11184.359180] lmv_unlink+0x34d/0xbd0 [lmv] [11184.360703] ll_unlink+0x24d/0x7f0 [lustre] [11184.362420] vfs_unlink+0x202/0x330 [11184.367433] do_unlinkat+0x2fa/0x460 [11184.371536] __x64_sys_unlink+0x37/0x50 [11184.375314] do_syscall_64+0xc1/0x440 [11184.380319] entry_SYSCALL_64_after_hwframe+0x49/0xae [11184.387121] RIP: 0033:0x7f151bdee10b [11184.389106] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [11184.406370] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [11184.410071] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [11184.432145] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [11184.448036] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11184.450330] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11184.471336] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11188.057245] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [11188.060866] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11188.081773] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11188.085636] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11188.089462] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11188.092449] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11188.100110] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11188.102808] RAX: 0000000000005386 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11188.106227] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [11188.109579] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [11188.112900] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [11188.116449] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [11188.120773] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [11188.123169] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11188.125705] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [11188.129139] Call Trace: [11188.130122] [11188.131260] ? show_regs.cold.9+0x22/0x2f [11188.132874] ? watchdog_timer_fn.cold.6+0x62/0x10a [11188.134880] ? __hrtimer_run_queues+0x1c2/0x4d0 [11188.137275] ? watchdog+0x40/0x40 [11188.138766] ? hrtimer_interrupt+0x128/0x2c0 [11188.140667] ? smp_apic_timer_interrupt+0xa6/0x270 [11188.142519] ? apic_timer_interrupt+0xf/0x20 [11188.144613] [11188.146333] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11188.148776] do_raw_spin_lock+0x177/0x190 [11188.150625] _raw_spin_lock+0x21/0x30 [11188.152229] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11188.155056] ? do_raw_spin_unlock+0x75/0x190 [11188.157207] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11188.159457] ? do_wait_intr+0xf0/0xf0 [11188.161180] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11188.163768] mdc_close+0x53b/0xd70 [mdc] [11188.165736] lmv_close+0x18d/0x490 [lmv] [11188.167600] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11188.170384] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11188.173195] ll_md_real_close+0xa5/0x2a0 [lustre] [11188.175365] ll_md_close+0x274/0x800 [lustre] [11188.177591] ll_file_release+0x128/0xc50 [lustre] [11188.179750] __fput+0x10d/0x440 [11188.181224] ____fput+0x12/0x20 [11188.182978] task_work_run+0xc6/0x110 [11188.184391] exit_to_usermode_loop+0x1dd/0x1f0 [11188.186260] do_syscall_64+0x3d6/0x440 [11188.188045] entry_SYSCALL_64_after_hwframe+0x49/0xae [11188.190840] RIP: 0033:0x7f151bee3cd8 [11188.192295] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11188.196251] BUG: workqueue lockup - pool [11188.202757] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11188.202771] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [11188.202774] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [11188.202776] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11188.202779] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11188.202781] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11188.243234] cpus=0-3 flags=0x4 nice=0 stuck for 9257s! [11188.245727] Showing busy workqueues and worker pools: [11188.247953] workqueue events: flags=0x0 [11188.249770] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11188.254688] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [11188.267582] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11188.273057] pending: kfree_rcu_monitor [11188.278613] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11188.289344] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [11188.295235] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11188.299063] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [11188.304419] workqueue events_long: flags=0x0 [11188.308062] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11188.312858] pending: lu_objects_flush [obdclass] [11188.320607] workqueue events_unbound: flags=0x2 [11188.327098] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11188.337067] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11188.346953] workqueue events_power_efficient: flags=0x80 [11188.352445] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11188.359628] pending: do_cache_clean, neigh_periodic_work, check_lifetime [11188.366739] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11188.374422] pending: neigh_periodic_work [11188.375973] workqueue mm_percpu_wq: flags=0x8 [11188.379252] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11188.386427] pending: vmstat_update, lru_add_drain_per_cpu [11188.392124] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11188.398723] pending: vmstat_update [11188.403851] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11188.409790] pending: lru_add_drain_per_cpu [11188.415819] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [11188.420496] pending: lru_add_drain_per_cpu BAR(48) [11188.424525] workqueue writeback: flags=0x4a [11188.426026] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [11188.433581] pending: wb_workfn [11188.439680] workqueue xprtiod: flags=0xa [11188.441815] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11188.445600] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [11188.455251] workqueue ipv6_addrconf: flags=0x40008 [11188.458135] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11188.461962] pending: addrconf_verify_work [11188.464705] workqueue knbd0-recv: flags=0x1a [11188.466145] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11188.469961] in-flight: 658:recv_work [11188.473184] workqueue ptlrpc_pinger: flags=0x2 [11188.484757] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11188.491323] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11188.500745] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9257s workers=4 idle: 32984 37 8 [11188.509332] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11173s workers=2 idle: 661 [11192.056058] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11192.056257] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [11192.061303] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11192.061379] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11192.061386] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11192.061392] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11192.061955] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [11192.061966] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [11192.061970] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11192.061974] RAX: 0000000000007399 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11192.061976] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [11192.061978] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [11192.062091] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [11192.062094] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [11192.062097] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [11192.062100] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11192.062177] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11192.062186] Call Trace: [11192.062191] [11192.062197] ? show_regs.cold.9+0x22/0x2f [11192.062207] ? watchdog_timer_fn.cold.6+0x62/0x10a [11192.062213] ? __hrtimer_run_queues+0x1c2/0x4d0 [11192.062217] ? watchdog+0x40/0x40 [11192.062221] ? hrtimer_interrupt+0x128/0x2c0 [11192.062224] ? smp_apic_timer_interrupt+0xa6/0x270 [11192.062233] ? apic_timer_interrupt+0xf/0x20 [11192.062237] [11192.062239] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [11192.062244] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11192.062249] do_raw_spin_lock+0x177/0x190 [11192.062253] _raw_spin_lock+0x21/0x30 [11192.062257] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11192.062600] process_one_work+0x2c8/0x700 [11192.062607] worker_thread+0x296/0x6e0 [11192.062624] ? rescuer_thread+0x570/0x570 [11192.062628] kthread+0x1d1/0x200 [11192.062632] ? set_kthread_struct+0x70/0x70 [11192.062635] ret_from_fork+0x1f/0x30 [11192.302771] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11192.342063] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11192.348252] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11192.357001] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11192.364902] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11192.385282] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11192.395982] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [11192.401587] RDX: 0000000000004e74 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [11192.409251] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [11192.414210] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [11192.419586] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11192.424336] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [11192.428717] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11192.433362] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11192.436837] Call Trace: [11192.438043] [11192.439095] ? show_regs.cold.9+0x22/0x2f [11192.442303] ? watchdog_timer_fn.cold.6+0x62/0x10a [11192.444552] ? __hrtimer_run_queues+0x1c2/0x4d0 [11192.448422] ? watchdog+0x40/0x40 [11192.453249] ? hrtimer_interrupt+0x128/0x2c0 [11192.455352] ? smp_apic_timer_interrupt+0xa6/0x270 [11192.461099] ? apic_timer_interrupt+0xf/0x20 [11192.464597] [11192.465428] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11192.467444] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11192.476396] do_raw_spin_lock+0x177/0x190 [11192.480456] _raw_spin_lock+0x21/0x30 [11192.483166] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11192.486878] obd_mod_free+0x46/0x140 [ptlrpc] [11192.492400] mdc_commit_open+0x91/0xb0 [mdc] [11192.495383] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11192.501116] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11192.509080] after_reply+0xe2c/0x1780 [ptlrpc] [11192.515327] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11192.521141] ? __next_timer_interrupt+0x160/0x160 [11192.528323] ptlrpcd+0xa82/0xe40 [ptlrpc] [11192.533129] ? do_wait_intr+0xf0/0xf0 [11192.536934] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11192.540117] kthread+0x1d1/0x200 [11192.542363] ? set_kthread_struct+0x70/0x70 [11192.549143] ret_from_fork+0x1f/0x30 [11216.052049] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [11216.054464] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11216.057022] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [11216.080592] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11216.080604] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11216.080607] RIP: 0010:native_safe_halt+0x2e/0x50 [11216.080617] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11216.080628] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11216.080632] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11216.080634] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [11216.080636] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [11216.080640] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [11216.080641] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [11216.080644] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [11216.080646] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11216.080648] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [11216.080654] Call Trace: [11216.080658] [11216.080662] ? show_regs.cold.9+0x22/0x2f [11216.080670] ? watchdog_timer_fn.cold.6+0x62/0x10a [11216.080675] ? __hrtimer_run_queues+0x1c2/0x4d0 [11216.080677] ? watchdog+0x40/0x40 [11216.080680] ? hrtimer_interrupt+0x128/0x2c0 [11216.080683] ? smp_apic_timer_interrupt+0xa6/0x270 [11216.080691] ? apic_timer_interrupt+0xf/0x20 [11216.080695] [11216.080696] ? native_safe_halt+0x2e/0x50 [11216.080699] kvm_wait+0xcd/0xe0 [11216.080705] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11216.080710] do_raw_spin_lock+0x177/0x190 [11216.080713] _raw_spin_lock+0x21/0x30 [11216.080718] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11216.080939] ? do_raw_spin_unlock+0x75/0x190 [11216.080943] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11216.084821] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11216.084881] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11216.084886] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11216.084889] RIP: 0010:native_safe_halt+0x2e/0x50 [11216.084898] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11216.084902] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11216.084905] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11216.084906] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [11216.084908] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [11216.084909] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [11216.084911] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [11216.084916] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [11216.084918] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11216.084919] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [11216.084926] Call Trace: [11216.084930] [11216.084935] ? show_regs.cold.9+0x22/0x2f [11216.084942] ? watchdog_timer_fn.cold.6+0x62/0x10a [11216.084946] ? __hrtimer_run_queues+0x1c2/0x4d0 [11216.084949] ? watchdog+0x40/0x40 [11216.084952] ? hrtimer_interrupt+0x128/0x2c0 [11216.084954] ? smp_apic_timer_interrupt+0xa6/0x270 [11216.084959] ? apic_timer_interrupt+0xf/0x20 [11216.084962] [11216.084965] ? native_safe_halt+0x2e/0x50 [11216.084967] kvm_wait+0xcd/0xe0 [11216.084973] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11216.084979] do_raw_spin_lock+0x177/0x190 [11216.084982] _raw_spin_lock+0x21/0x30 [11216.084985] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11216.085339] ? do_raw_spin_unlock+0x75/0x190 [11216.085345] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11216.092406] ? do_wait_intr+0xf0/0xf0 [11216.092425] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11216.098368] ? do_wait_intr+0xf0/0xf0 [11216.098398] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11216.098805] mdc_close+0x53b/0xd70 [mdc] [11216.098833] lmv_close+0x18d/0x490 [lmv] [11216.098851] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11216.101415] mdc_reint+0x2d/0x110 [mdc] [11216.101467] mdc_unlink+0x658/0x870 [mdc] [11216.101492] lmv_unlink+0x34d/0xbd0 [lmv] [11216.109289] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11216.113069] ll_unlink+0x24d/0x7f0 [lustre] [11216.118955] ll_md_real_close+0xa5/0x2a0 [lustre] [11216.119091] ll_md_close+0x274/0x800 [lustre] [11216.119183] ll_file_release+0x128/0xc50 [lustre] [11216.119289] __fput+0x10d/0x440 [11216.119299] ____fput+0x12/0x20 [11216.119303] task_work_run+0xc6/0x110 [11216.119310] exit_to_usermode_loop+0x1dd/0x1f0 [11216.119316] do_syscall_64+0x3d6/0x440 [11216.119321] entry_SYSCALL_64_after_hwframe+0x49/0xae [11216.119328] RIP: 0033:0x7f151bee3cd8 [11216.119340] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11216.119344] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11216.119348] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [11216.119351] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [11216.119353] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11216.119355] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11216.119357] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11216.290517] vfs_unlink+0x202/0x330 [11216.291922] do_unlinkat+0x2fa/0x460 [11216.293135] __x64_sys_unlink+0x37/0x50 [11216.294898] do_syscall_64+0xc1/0x440 [11216.296175] entry_SYSCALL_64_after_hwframe+0x49/0xae [11216.298143] RIP: 0033:0x7f151bdee10b [11216.299464] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [11216.306444] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [11216.309233] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [11216.311770] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [11216.314202] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11216.316928] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11216.319160] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11218.916762] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9288s! [11218.923028] Showing busy workqueues and worker pools: [11218.925900] workqueue events: flags=0x0 [11218.927951] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11218.931494] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [11218.936307] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11218.939818] pending: kfree_rcu_monitor [11218.941782] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11218.945067] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [11218.950087] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11218.953677] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [11218.958144] workqueue events_long: flags=0x0 [11218.960986] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11218.963179] pending: lu_objects_flush [obdclass] [11218.966020] workqueue events_unbound: flags=0x2 [11218.968143] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11218.971885] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11218.980859] workqueue events_power_efficient: flags=0x80 [11218.986940] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11218.994917] pending: do_cache_clean, neigh_periodic_work, check_lifetime [11219.000736] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11219.004025] pending: neigh_periodic_work [11219.005724] workqueue mm_percpu_wq: flags=0x8 [11219.006833] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11219.009536] pending: vmstat_update, lru_add_drain_per_cpu [11219.013950] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11219.020684] pending: vmstat_update [11219.023883] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11219.029955] pending: lru_add_drain_per_cpu [11219.035124] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [11219.039038] pending: lru_add_drain_per_cpu BAR(48) [11219.045896] workqueue writeback: flags=0x4a [11219.050692] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [11219.057116] pending: wb_workfn [11219.060676] workqueue xprtiod: flags=0xa [11219.063500] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11219.070522] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [11219.077549] workqueue ipv6_addrconf: flags=0x40008 [11219.081893] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11219.088488] pending: addrconf_verify_work [11219.093362] workqueue knbd0-recv: flags=0x1a [11219.097336] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11219.103342] in-flight: 658:recv_work [11219.108121] workqueue ptlrpc_pinger: flags=0x2 [11219.112763] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11219.116684] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11219.123782] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9288s workers=4 idle: 32984 37 8 [11219.128967] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11203s workers=2 idle: 661 [11220.055932] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [11220.057828] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11220.061486] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11220.061565] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11220.061572] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11220.061576] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11220.061590] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11220.061594] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11220.061598] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [11220.061600] RDX: 0000000000004460 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [11220.061602] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [11220.061604] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [11220.061606] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11220.061609] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [11220.061611] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11220.061614] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11220.061623] Call Trace: [11220.061629] [11220.061635] ? show_regs.cold.9+0x22/0x2f [11220.061645] ? watchdog_timer_fn.cold.6+0x62/0x10a [11220.061651] ? __hrtimer_run_queues+0x1c2/0x4d0 [11220.061656] ? watchdog+0x40/0x40 [11220.061660] ? hrtimer_interrupt+0x128/0x2c0 [11220.061664] ? smp_apic_timer_interrupt+0xa6/0x270 [11220.061673] ? apic_timer_interrupt+0xf/0x20 [11220.061678] [11220.061681] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11220.061686] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11220.061690] do_raw_spin_lock+0x177/0x190 [11220.061695] _raw_spin_lock+0x21/0x30 [11220.061701] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11220.065706] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11220.065779] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11220.065787] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11220.065793] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11220.066054] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11220.066065] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11220.066069] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11220.066073] RAX: 0000000000007c84 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11220.066075] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [11220.066078] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [11220.066079] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [11220.066082] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [11220.066086] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [11220.066089] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11220.066091] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11220.066101] Call Trace: [11220.066107] [11220.066112] ? show_regs.cold.9+0x22/0x2f [11220.066121] ? watchdog_timer_fn.cold.6+0x62/0x10a [11220.066128] ? __hrtimer_run_queues+0x1c2/0x4d0 [11220.066132] ? watchdog+0x40/0x40 [11220.066137] ? hrtimer_interrupt+0x128/0x2c0 [11220.066141] ? smp_apic_timer_interrupt+0xa6/0x270 [11220.066150] ? apic_timer_interrupt+0xf/0x20 [11220.066154] [11220.066157] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11220.066162] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11220.066167] do_raw_spin_lock+0x177/0x190 [11220.066171] _raw_spin_lock+0x21/0x30 [11220.066175] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11220.102315] obd_mod_free+0x46/0x140 [ptlrpc] [11220.111025] process_one_work+0x2c8/0x700 [11220.111045] worker_thread+0x296/0x6e0 [11220.111050] ? rescuer_thread+0x570/0x570 [11220.111054] kthread+0x1d1/0x200 [11220.111324] ? set_kthread_struct+0x70/0x70 [11220.111328] ret_from_fork+0x1f/0x30 [11220.531403] mdc_commit_open+0x91/0xb0 [mdc] [11220.534088] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11220.540915] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11220.547029] after_reply+0xe2c/0x1780 [ptlrpc] [11220.556051] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11220.564411] ? __next_timer_interrupt+0x160/0x160 [11220.571085] ptlrpcd+0xa82/0xe40 [ptlrpc] [11220.572814] ? do_wait_intr+0xf0/0xf0 [11220.583797] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11220.585817] kthread+0x1d1/0x200 [11220.591930] ? set_kthread_struct+0x70/0x70 [11220.596137] ret_from_fork+0x1f/0x30 [11244.052150] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [11244.057662] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11244.061463] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [11244.104555] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11244.104570] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11244.104573] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11244.104588] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11244.104593] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11244.104674] RAX: 0000000000007f83 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11244.104677] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [11244.104679] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [11244.104681] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [11244.104684] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [11244.104687] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [11244.104690] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11244.104693] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [11244.104703] Call Trace: [11244.104708] [11244.104713] ? show_regs.cold.9+0x22/0x2f [11244.104723] ? watchdog_timer_fn.cold.6+0x62/0x10a [11244.104730] ? __hrtimer_run_queues+0x1c2/0x4d0 [11244.104734] ? watchdog+0x40/0x40 [11244.104738] ? hrtimer_interrupt+0x128/0x2c0 [11244.104742] ? smp_apic_timer_interrupt+0xa6/0x270 [11244.104751] ? apic_timer_interrupt+0xf/0x20 [11244.104756] [11244.104758] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11244.104764] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11244.104768] do_raw_spin_lock+0x177/0x190 [11244.104773] _raw_spin_lock+0x21/0x30 [11244.104781] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11244.105128] ? do_raw_spin_unlock+0x75/0x190 [11244.105134] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11244.105331] ? do_wait_intr+0xf0/0xf0 [11244.105338] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11244.105527] mdc_reint+0x2d/0x110 [mdc] [11244.105563] mdc_unlink+0x658/0x870 [mdc] [11244.105587] lmv_unlink+0x34d/0xbd0 [lmv] [11244.106156] ll_unlink+0x24d/0x7f0 [lustre] [11244.106267] vfs_unlink+0x202/0x330 [11244.106276] do_unlinkat+0x2fa/0x460 [11244.106280] __x64_sys_unlink+0x37/0x50 [11244.106284] do_syscall_64+0xc1/0x440 [11244.106291] entry_SYSCALL_64_after_hwframe+0x49/0xae [11244.106298] RIP: 0033:0x7f151bdee10b [11244.106667] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [11244.106674] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [11244.106680] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [11244.106682] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [11244.106684] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11244.106686] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11244.106688] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11244.374596] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11244.401510] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11244.406083] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11244.412096] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11244.423317] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11244.444196] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11244.450363] RAX: 0000000000006913 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11244.455331] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [11244.458959] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [11244.468746] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [11244.480629] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [11244.494307] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [11244.507130] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11244.509958] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [11244.515296] Call Trace: [11244.517918] [11244.519108] ? show_regs.cold.9+0x22/0x2f [11244.521871] ? watchdog_timer_fn.cold.6+0x62/0x10a [11244.524643] ? __hrtimer_run_queues+0x1c2/0x4d0 [11244.527187] ? watchdog+0x40/0x40 [11244.528810] ? hrtimer_interrupt+0x128/0x2c0 [11244.531068] ? smp_apic_timer_interrupt+0xa6/0x270 [11244.533862] ? apic_timer_interrupt+0xf/0x20 [11244.536306] [11244.537562] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11244.539677] do_raw_spin_lock+0x177/0x190 [11244.541091] _raw_spin_lock+0x21/0x30 [11244.543181] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11244.546485] ? do_raw_spin_unlock+0x75/0x190 [11244.553165] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11244.559937] ? do_wait_intr+0xf0/0xf0 [11244.564880] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11244.567701] mdc_close+0x53b/0xd70 [mdc] [11244.572322] lmv_close+0x18d/0x490 [lmv] [11244.576125] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11244.582019] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11244.590742] ll_md_real_close+0xa5/0x2a0 [lustre] [11244.592470] ll_md_close+0x274/0x800 [lustre] [11244.596760] ll_file_release+0x128/0xc50 [lustre] [11244.599912] __fput+0x10d/0x440 [11244.604295] ____fput+0x12/0x20 [11244.607325] task_work_run+0xc6/0x110 [11244.611794] exit_to_usermode_loop+0x1dd/0x1f0 [11244.617816] do_syscall_64+0x3d6/0x440 [11244.623727] entry_SYSCALL_64_after_hwframe+0x49/0xae [11244.628199] RIP: 0033:0x7f151bee3cd8 [11244.631292] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11244.647671] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11244.652725] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [11244.656682] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [11244.661180] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11244.665597] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11244.671885] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11248.055110] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [11248.056033] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11248.060137] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11248.060209] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11248.060215] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11248.060218] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11248.060298] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11248.060304] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11248.060308] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [11248.060310] RDX: 0000000000002aa7 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [11248.060313] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [11248.060315] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [11248.060318] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11248.060321] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [11248.060324] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11248.060326] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11248.060335] Call Trace: [11248.060340] [11248.060346] ? show_regs.cold.9+0x22/0x2f [11248.060355] ? watchdog_timer_fn.cold.6+0x62/0x10a [11248.060362] ? __hrtimer_run_queues+0x1c2/0x4d0 [11248.060367] ? watchdog+0x40/0x40 [11248.060371] ? hrtimer_interrupt+0x128/0x2c0 [11248.060376] ? smp_apic_timer_interrupt+0xa6/0x270 [11248.060383] ? apic_timer_interrupt+0xf/0x20 [11248.060388] [11248.060391] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11248.060396] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11248.060400] do_raw_spin_lock+0x177/0x190 [11248.060405] _raw_spin_lock+0x21/0x30 [11248.060411] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11248.066080] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11248.066155] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11248.066163] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11248.066169] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11248.090549] obd_mod_free+0x46/0x140 [ptlrpc] [11248.102478] RIP: 0010:native_safe_halt+0x2e/0x50 [11248.102498] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11248.102502] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11248.102508] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11248.102510] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [11248.102512] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [11248.102514] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [11248.102516] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [11248.102520] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [11248.102522] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11248.102524] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11248.102533] Call Trace: [11248.102538] [11248.102543] ? show_regs.cold.9+0x22/0x2f [11248.102553] ? watchdog_timer_fn.cold.6+0x62/0x10a [11248.102559] ? __hrtimer_run_queues+0x1c2/0x4d0 [11248.102564] ? watchdog+0x40/0x40 [11248.102568] ? hrtimer_interrupt+0x128/0x2c0 [11248.102572] ? smp_apic_timer_interrupt+0xa6/0x270 [11248.102578] ? apic_timer_interrupt+0xf/0x20 [11248.102582] [11248.102585] ? native_safe_halt+0x2e/0x50 [11248.102589] kvm_wait+0xcd/0xe0 [11248.102597] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11248.102608] do_raw_spin_lock+0x177/0x190 [11248.102613] _raw_spin_lock+0x21/0x30 [11248.102618] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11248.103046] process_one_work+0x2c8/0x700 [11248.103053] worker_thread+0x296/0x6e0 [11248.103058] ? rescuer_thread+0x570/0x570 [11248.103061] kthread+0x1d1/0x200 [11248.103076] ? set_kthread_struct+0x70/0x70 [11248.103081] ret_from_fork+0x1f/0x30 [11248.327518] mdc_commit_open+0x91/0xb0 [mdc] [11248.330308] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11248.334650] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11248.336334] after_reply+0xe2c/0x1780 [ptlrpc] [11248.341554] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11248.344487] ? __next_timer_interrupt+0x160/0x160 [11248.346094] ptlrpcd+0xa82/0xe40 [ptlrpc] [11248.348731] ? do_wait_intr+0xf0/0xf0 [11248.350028] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11248.353107] kthread+0x1d1/0x200 [11248.355840] ? set_kthread_struct+0x70/0x70 [11248.359022] ret_from_fork+0x1f/0x30 [11249.634462] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9319s! [11249.642654] Showing busy workqueues and worker pools: [11249.645908] workqueue events: flags=0x0 [11249.647991] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11249.653303] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [11249.659123] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11249.663448] pending: kfree_rcu_monitor [11249.670242] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11249.675183] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [11249.682135] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11249.687440] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [11249.691114] workqueue events_long: flags=0x0 [11249.693242] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11249.696971] pending: lu_objects_flush [obdclass] [11249.700561] workqueue events_unbound: flags=0x2 [11249.703243] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11249.706206] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11249.711534] workqueue events_power_efficient: flags=0x80 [11249.714662] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11249.717789] pending: do_cache_clean, neigh_periodic_work, check_lifetime [11249.721591] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11249.724676] pending: neigh_periodic_work [11249.726827] workqueue mm_percpu_wq: flags=0x8 [11249.729357] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11249.732383] pending: vmstat_update, lru_add_drain_per_cpu [11249.735420] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11249.738590] pending: vmstat_update [11249.743326] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11249.747558] pending: lru_add_drain_per_cpu [11249.750090] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [11249.754076] pending: lru_add_drain_per_cpu BAR(48) [11249.756421] workqueue writeback: flags=0x4a [11249.758288] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [11249.761439] pending: wb_workfn [11249.763328] workqueue xprtiod: flags=0xa [11249.765516] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11249.773837] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [11249.779036] workqueue ipv6_addrconf: flags=0x40008 [11249.781285] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11249.784175] pending: addrconf_verify_work [11249.789070] workqueue knbd0-recv: flags=0x1a [11249.792222] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11249.795035] in-flight: 658:recv_work [11249.796631] workqueue ptlrpc_pinger: flags=0x2 [11249.799012] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11249.801903] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11249.804855] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9319s workers=4 idle: 32984 37 8 [11249.808782] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11234s workers=2 idle: 661 [11272.057050] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [11272.060232] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11272.107314] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11272.116744] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11272.123107] RIP: 0010:native_safe_halt+0x2e/0x50 [11272.127041] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11272.133933] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11272.137178] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11272.142036] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [11272.144927] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [11272.151488] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [11272.158263] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [11272.162946] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [11272.169092] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11272.175220] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [11272.187602] Call Trace: [11272.189500] [11272.193284] ? show_regs.cold.9+0x22/0x2f [11272.198231] ? watchdog_timer_fn.cold.6+0x62/0x10a [11272.204043] ? __hrtimer_run_queues+0x1c2/0x4d0 [11272.212091] ? watchdog+0x40/0x40 [11272.220353] ? hrtimer_interrupt+0x128/0x2c0 [11272.223098] ? smp_apic_timer_interrupt+0xa6/0x270 [11272.225035] ? apic_timer_interrupt+0xf/0x20 [11272.228452] [11272.232206] ? native_safe_halt+0x2e/0x50 [11272.235198] kvm_wait+0xcd/0xe0 [11272.237247] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11272.241986] do_raw_spin_lock+0x177/0x190 [11272.245493] _raw_spin_lock+0x21/0x30 [11272.250090] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11272.257030] ? do_raw_spin_unlock+0x75/0x190 [11272.262130] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11272.266518] ? do_wait_intr+0xf0/0xf0 [11272.267963] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11272.276768] mdc_close+0x53b/0xd70 [mdc] [11272.278336] lmv_close+0x18d/0x490 [lmv] [11272.281304] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11272.287158] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11272.290165] ll_md_real_close+0xa5/0x2a0 [lustre] [11272.297371] ll_md_close+0x274/0x800 [lustre] [11272.303757] ll_file_release+0x128/0xc50 [lustre] [11272.305760] __fput+0x10d/0x440 [11272.308732] ____fput+0x12/0x20 [11272.310191] task_work_run+0xc6/0x110 [11272.313640] exit_to_usermode_loop+0x1dd/0x1f0 [11272.316660] do_syscall_64+0x3d6/0x440 [11272.318963] entry_SYSCALL_64_after_hwframe+0x49/0xae [11272.324199] RIP: 0033:0x7f151bee3cd8 [11272.328243] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11272.338250] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11272.341200] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [11272.349984] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [11272.355716] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11272.357760] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11272.362197] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11276.052378] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [11276.056310] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11276.058851] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11276.058929] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11276.058936] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11276.058939] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11276.058952] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11276.058956] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11276.058960] RAX: 0000000000004f4d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11276.058963] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [11276.058965] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [11276.058967] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [11276.058969] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [11276.058973] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [11276.058975] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11276.058981] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [11276.058988] Call Trace: [11276.058993] [11276.058999] ? show_regs.cold.9+0x22/0x2f [11276.059007] ? watchdog_timer_fn.cold.6+0x62/0x10a [11276.059013] ? __hrtimer_run_queues+0x1c2/0x4d0 [11276.059017] ? watchdog+0x40/0x40 [11276.059020] ? hrtimer_interrupt+0x128/0x2c0 [11276.059028] ? smp_apic_timer_interrupt+0xa6/0x270 [11276.059036] ? apic_timer_interrupt+0xf/0x20 [11276.059040] [11276.059042] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11276.059047] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11276.059051] do_raw_spin_lock+0x177/0x190 [11276.059055] _raw_spin_lock+0x21/0x30 [11276.059060] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11276.059331] ? do_raw_spin_unlock+0x75/0x190 [11276.059337] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11276.059554] ? do_wait_intr+0xf0/0xf0 [11276.059580] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [11276.059585] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11276.059637] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11276.059641] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11276.059644] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11276.059656] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11276.059660] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11276.059664] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [11276.059666] RDX: 0000000000000bf7 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [11276.059669] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [11276.059671] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [11276.059673] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11276.059675] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [11276.059678] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11276.059680] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11276.059685] Call Trace: [11276.059688] [11276.059690] ? show_regs.cold.9+0x22/0x2f [11276.059752] ? watchdog_timer_fn.cold.6+0x62/0x10a [11276.059561] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11276.059757] ? __hrtimer_run_queues+0x1c2/0x4d0 [11276.059760] ? watchdog+0x40/0x40 [11276.059764] ? hrtimer_interrupt+0x128/0x2c0 [11276.059767] ? smp_apic_timer_interrupt+0xa6/0x270 [11276.059772] ? apic_timer_interrupt+0xf/0x20 [11276.059776] [11276.059779] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11276.059784] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11276.059788] do_raw_spin_lock+0x177/0x190 [11276.059792] _raw_spin_lock+0x21/0x30 [11276.059809] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11276.060227] mdc_reint+0x2d/0x110 [mdc] [11276.060068] obd_mod_free+0x46/0x140 [ptlrpc] [11276.060439] mdc_commit_open+0x91/0xb0 [mdc] [11276.060467] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11276.060667] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11276.061386] after_reply+0xe2c/0x1780 [ptlrpc] [11276.062568] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11276.063733] ? __next_timer_interrupt+0x160/0x160 [11276.063777] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11276.063743] ptlrpcd+0xa82/0xe40 [ptlrpc] [11276.063860] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11276.063869] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11276.063876] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11276.064328] ? do_wait_intr+0xf0/0xf0 [11276.064338] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11276.064545] kthread+0x1d1/0x200 [11276.064556] ? set_kthread_struct+0x70/0x70 [11276.064560] ret_from_fork+0x1f/0x30 [11276.101093] mdc_unlink+0x658/0x870 [mdc] [11276.101149] lmv_unlink+0x34d/0xbd0 [lmv] [11276.101176] ll_unlink+0x24d/0x7f0 [lustre] [11276.101287] vfs_unlink+0x202/0x330 [11276.101296] do_unlinkat+0x2fa/0x460 [11276.101302] __x64_sys_unlink+0x37/0x50 [11276.101306] do_syscall_64+0xc1/0x440 [11276.101313] entry_SYSCALL_64_after_hwframe+0x49/0xae [11276.101321] RIP: 0033:0x7f151bdee10b [11276.101328] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [11276.101332] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [11276.101337] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [11276.101339] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [11276.101341] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11276.101343] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11276.101346] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11276.522404] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11276.524122] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11276.540384] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11276.546532] RAX: 0000000000007888 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11276.549515] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [11276.553799] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [11276.558830] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [11276.564415] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [11276.566780] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [11276.569621] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11276.572291] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11276.575324] Call Trace: [11276.576289] [11276.577086] ? show_regs.cold.9+0x22/0x2f [11276.578630] ? watchdog_timer_fn.cold.6+0x62/0x10a [11276.580984] ? __hrtimer_run_queues+0x1c2/0x4d0 [11276.583096] ? watchdog+0x40/0x40 [11276.584583] ? hrtimer_interrupt+0x128/0x2c0 [11276.586474] ? smp_apic_timer_interrupt+0xa6/0x270 [11276.589435] ? apic_timer_interrupt+0xf/0x20 [11276.590919] [11276.591558] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11276.595589] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11276.599045] do_raw_spin_lock+0x177/0x190 [11276.600983] _raw_spin_lock+0x21/0x30 [11276.602251] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11276.604319] process_one_work+0x2c8/0x700 [11276.605430] worker_thread+0x296/0x6e0 [11276.606611] ? rescuer_thread+0x570/0x570 [11276.610442] kthread+0x1d1/0x200 [11276.611568] ? set_kthread_struct+0x70/0x70 [11276.613090] ret_from_fork+0x1f/0x30 [11279.841040] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9349s! [11279.846624] Showing busy workqueues and worker pools: [11279.850789] workqueue events: flags=0x0 [11279.857255] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11279.861053] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [11279.867425] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11279.875301] pending: kfree_rcu_monitor [11279.881215] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11279.887316] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [11279.899566] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11279.907155] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [11279.912420] workqueue events_long: flags=0x0 [11279.914581] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11279.917961] pending: lu_objects_flush [obdclass] [11279.927944] workqueue events_unbound: flags=0x2 [11279.931254] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11279.935895] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11279.947278] workqueue events_power_efficient: flags=0x80 [11279.950896] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11279.959270] pending: do_cache_clean, neigh_periodic_work, check_lifetime [11279.963908] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11279.968227] pending: neigh_periodic_work [11279.975924] workqueue mm_percpu_wq: flags=0x8 [11279.977476] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11279.993432] pending: vmstat_update, lru_add_drain_per_cpu [11279.998135] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11280.007285] pending: vmstat_update [11280.009388] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11280.019159] pending: lru_add_drain_per_cpu [11280.024939] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [11280.030310] pending: lru_add_drain_per_cpu BAR(48) [11280.032295] workqueue writeback: flags=0x4a [11280.034214] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [11280.037338] pending: wb_workfn [11280.039167] workqueue xprtiod: flags=0xa [11280.041171] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11280.047908] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [11280.061288] workqueue ipv6_addrconf: flags=0x40008 [11280.064345] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11280.070154] pending: addrconf_verify_work [11280.073377] workqueue knbd0-recv: flags=0x1a [11280.076660] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11280.081463] in-flight: 658:recv_work [11280.083180] workqueue ptlrpc_pinger: flags=0x2 [11280.085707] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11280.088519] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11280.091615] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9349s workers=4 idle: 32984 37 8 [11280.095865] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11264s workers=2 idle: 661 [11300.058072] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [11300.064299] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11300.106835] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11300.121687] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11300.129644] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11300.132673] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11300.156326] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11300.160031] RAX: 0000000000003d92 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11300.165073] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [11300.172349] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [11300.179681] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [11300.185433] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [11300.189360] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [11300.198896] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11300.202628] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [11300.210355] Call Trace: [11300.214281] [11300.215930] ? show_regs.cold.9+0x22/0x2f [11300.219022] ? watchdog_timer_fn.cold.6+0x62/0x10a [11300.221927] ? __hrtimer_run_queues+0x1c2/0x4d0 [11300.224016] ? watchdog+0x40/0x40 [11300.225506] ? hrtimer_interrupt+0x128/0x2c0 [11300.227590] ? smp_apic_timer_interrupt+0xa6/0x270 [11300.229846] ? apic_timer_interrupt+0xf/0x20 [11300.231535] [11300.232987] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11300.236033] do_raw_spin_lock+0x177/0x190 [11300.239766] _raw_spin_lock+0x21/0x30 [11300.242643] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11300.245399] ? do_raw_spin_unlock+0x75/0x190 [11300.249085] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11300.252294] ? do_wait_intr+0xf0/0xf0 [11300.255373] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11300.258523] mdc_close+0x53b/0xd70 [mdc] [11300.260332] lmv_close+0x18d/0x490 [lmv] [11300.262786] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11300.265759] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11300.268329] ll_md_real_close+0xa5/0x2a0 [lustre] [11300.272151] ll_md_close+0x274/0x800 [lustre] [11300.274822] ll_file_release+0x128/0xc50 [lustre] [11300.278003] __fput+0x10d/0x440 [11300.279824] ____fput+0x12/0x20 [11300.282337] task_work_run+0xc6/0x110 [11300.284260] exit_to_usermode_loop+0x1dd/0x1f0 [11300.287371] do_syscall_64+0x3d6/0x440 [11300.290370] entry_SYSCALL_64_after_hwframe+0x49/0xae [11300.293394] RIP: 0033:0x7f151bee3cd8 [11300.295689] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11300.304128] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11300.308036] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [11300.311687] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [11300.315196] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11300.318159] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11300.320635] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11304.055026] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [11304.058154] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11304.060633] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11304.086856] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11304.086870] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11304.086873] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11304.086886] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11304.086890] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11304.086895] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [11304.086897] RDX: 0000000000006232 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [11304.086899] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [11304.086901] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [11304.086904] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11304.086908] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [11304.086911] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11304.086913] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11304.086921] Call Trace: [11304.086926] [11304.086931] ? show_regs.cold.9+0x22/0x2f [11304.086940] ? watchdog_timer_fn.cold.6+0x62/0x10a [11304.087020] ? __hrtimer_run_queues+0x1c2/0x4d0 [11304.087024] ? watchdog+0x40/0x40 [11304.087028] ? hrtimer_interrupt+0x128/0x2c0 [11304.087032] ? smp_apic_timer_interrupt+0xa6/0x270 [11304.087039] ? apic_timer_interrupt+0xf/0x20 [11304.087043] [11304.087046] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11304.087050] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11304.087054] do_raw_spin_lock+0x177/0x190 [11304.087059] _raw_spin_lock+0x21/0x30 [11304.087065] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11304.087686] obd_mod_free+0x46/0x140 [ptlrpc] [11304.088153] mdc_commit_open+0x91/0xb0 [mdc] [11304.088193] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11304.088636] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11304.089140] after_reply+0xe2c/0x1780 [ptlrpc] [11304.093660] Modules linked in: [11304.102373] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11304.102719] ? __next_timer_interrupt+0x160/0x160 [11304.102796] ptlrpcd+0xa82/0xe40 [ptlrpc] [11304.103369] ? do_wait_intr+0xf0/0xf0 [11304.103381] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11304.103657] kthread+0x1d1/0x200 [11304.103663] ? set_kthread_struct+0x70/0x70 [11304.103667] ret_from_fork+0x1f/0x30 [11304.232762] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11304.273309] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11304.282104] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11304.289167] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11304.291420] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11304.293810] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11304.309208] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11304.313089] RAX: 0000000000004a84 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11304.314917] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [11304.322648] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [11304.328552] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [11304.331507] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [11304.333529] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [11304.336715] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11304.338428] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11304.342490] Call Trace: [11304.343710] [11304.344992] ? show_regs.cold.9+0x22/0x2f [11304.346122] ? watchdog_timer_fn.cold.6+0x62/0x10a [11304.347499] ? __hrtimer_run_queues+0x1c2/0x4d0 [11304.350112] ? watchdog+0x40/0x40 [11304.351354] ? hrtimer_interrupt+0x128/0x2c0 [11304.353747] ? smp_apic_timer_interrupt+0xa6/0x270 [11304.354921] ? apic_timer_interrupt+0xf/0x20 [11304.357151] [11304.357907] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11304.359719] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11304.361949] do_raw_spin_lock+0x177/0x190 [11304.363014] _raw_spin_lock+0x21/0x30 [11304.364166] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11304.367558] process_one_work+0x2c8/0x700 [11304.369115] worker_thread+0x296/0x6e0 [11304.371093] ? rescuer_thread+0x570/0x570 [11304.372124] kthread+0x1d1/0x200 [11304.373641] ? set_kthread_struct+0x70/0x70 [11304.376919] ret_from_fork+0x1f/0x30 [11308.056751] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [11308.060079] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11308.114204] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11308.127065] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11308.131327] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11308.139904] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11308.161320] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [11308.171013] RAX: 0000000000007b92 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11308.186000] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [11308.194494] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [11308.201871] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [11308.206720] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [11308.214801] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [11308.221781] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11308.225766] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [11308.233986] Call Trace: [11308.235031] [11308.239896] ? show_regs.cold.9+0x22/0x2f [11308.244586] ? watchdog_timer_fn.cold.6+0x62/0x10a [11308.250685] ? __hrtimer_run_queues+0x1c2/0x4d0 [11308.253101] ? watchdog+0x40/0x40 [11308.257714] ? hrtimer_interrupt+0x128/0x2c0 [11308.261583] ? smp_apic_timer_interrupt+0xa6/0x270 [11308.266180] ? apic_timer_interrupt+0xf/0x20 [11308.273552] [11308.279297] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11308.289096] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11308.294077] do_raw_spin_lock+0x177/0x190 [11308.299937] _raw_spin_lock+0x21/0x30 [11308.303307] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11308.308052] ? do_raw_spin_unlock+0x75/0x190 [11308.309934] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11308.318032] ? do_wait_intr+0xf0/0xf0 [11308.324179] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11308.331207] mdc_reint+0x2d/0x110 [mdc] [11308.338282] mdc_unlink+0x658/0x870 [mdc] [11308.342342] lmv_unlink+0x34d/0xbd0 [lmv] [11308.345097] ll_unlink+0x24d/0x7f0 [lustre] [11308.351633] vfs_unlink+0x202/0x330 [11308.354783] do_unlinkat+0x2fa/0x460 [11308.357159] __x64_sys_unlink+0x37/0x50 [11308.364090] do_syscall_64+0xc1/0x440 [11308.368179] entry_SYSCALL_64_after_hwframe+0x49/0xae [11308.376389] RIP: 0033:0x7f151bdee10b [11308.379307] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [11308.407859] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [11308.418253] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [11308.427468] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [11308.433695] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11308.442317] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11308.446608] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11310.562098] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9379s! [11310.567092] Showing busy workqueues and worker pools: [11310.569982] workqueue events: flags=0x0 [11310.572405] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11310.576707] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [11310.581432] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11310.585263] pending: kfree_rcu_monitor [11310.586994] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11310.589037] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [11310.594121] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11310.597739] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [11310.602288] workqueue events_long: flags=0x0 [11310.604148] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11310.607027] pending: lu_objects_flush [obdclass] [11310.609639] workqueue events_unbound: flags=0x2 [11310.611401] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11310.614912] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11310.619279] workqueue events_power_efficient: flags=0x80 [11310.621776] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11310.625495] pending: do_cache_clean, neigh_periodic_work, check_lifetime [11310.628584] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11310.631011] pending: neigh_periodic_work [11310.633145] workqueue mm_percpu_wq: flags=0x8 [11310.635959] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11310.639115] pending: vmstat_update, lru_add_drain_per_cpu [11310.641827] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11310.645752] pending: vmstat_update [11310.647764] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11310.650125] pending: lru_add_drain_per_cpu [11310.651614] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [11310.654589] pending: lru_add_drain_per_cpu BAR(48) [11310.657317] workqueue writeback: flags=0x4a [11310.659508] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [11310.662734] pending: wb_workfn [11310.664871] workqueue xprtiod: flags=0xa [11310.666983] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11310.670036] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [11310.674528] workqueue ipv6_addrconf: flags=0x40008 [11310.676921] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11310.680127] pending: addrconf_verify_work [11310.682450] workqueue knbd0-recv: flags=0x1a [11310.684176] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11310.687944] in-flight: 658:recv_work [11310.689807] workqueue ptlrpc_pinger: flags=0x2 [11310.691969] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11310.694768] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11310.697104] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9380s workers=4 idle: 32984 37 8 [11310.700186] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11295s workers=2 idle: 661 [11328.057048] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [11328.061026] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11328.102923] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11328.111384] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11328.129041] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11328.134956] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11328.150995] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11328.155125] RAX: 00000000000011b8 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11328.164952] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [11328.169678] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [11328.174606] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [11328.178405] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [11328.186038] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [11328.192637] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11328.198354] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [11328.203040] Call Trace: [11328.206575] [11328.208067] ? show_regs.cold.9+0x22/0x2f [11328.210938] ? watchdog_timer_fn.cold.6+0x62/0x10a [11328.212542] ? __hrtimer_run_queues+0x1c2/0x4d0 [11328.213866] ? watchdog+0x40/0x40 [11328.215400] ? hrtimer_interrupt+0x128/0x2c0 [11328.216687] ? smp_apic_timer_interrupt+0xa6/0x270 [11328.218675] ? apic_timer_interrupt+0xf/0x20 [11328.223804] [11328.225609] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11328.228432] do_raw_spin_lock+0x177/0x190 [11328.229893] _raw_spin_lock+0x21/0x30 [11328.231402] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11328.235790] ? do_raw_spin_unlock+0x75/0x190 [11328.237662] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11328.239702] ? do_wait_intr+0xf0/0xf0 [11328.241739] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11328.247975] mdc_close+0x53b/0xd70 [mdc] [11328.254941] lmv_close+0x18d/0x490 [lmv] [11328.258057] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11328.262604] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11328.274051] ll_md_real_close+0xa5/0x2a0 [lustre] [11328.277970] ll_md_close+0x274/0x800 [lustre] [11328.281321] ll_file_release+0x128/0xc50 [lustre] [11328.287379] __fput+0x10d/0x440 [11328.288510] ____fput+0x12/0x20 [11328.294699] task_work_run+0xc6/0x110 [11328.302430] exit_to_usermode_loop+0x1dd/0x1f0 [11328.305623] do_syscall_64+0x3d6/0x440 [11328.308237] entry_SYSCALL_64_after_hwframe+0x49/0xae [11328.311475] RIP: 0033:0x7f151bee3cd8 [11328.313903] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11328.326726] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11328.336375] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [11328.342942] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [11328.346644] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11328.353948] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11328.360795] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11332.055017] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [11332.059032] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11332.060508] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11332.114623] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11332.114637] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11332.114641] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11332.114654] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11332.114658] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11332.114663] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [11332.114666] RDX: 0000000000006ac3 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [11332.114668] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [11332.114670] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [11332.114673] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11332.114676] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [11332.114679] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11332.114681] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11332.114690] Call Trace: [11332.114695] [11332.114700] ? show_regs.cold.9+0x22/0x2f [11332.114709] ? watchdog_timer_fn.cold.6+0x62/0x10a [11332.114715] ? __hrtimer_run_queues+0x1c2/0x4d0 [11332.114720] ? watchdog+0x40/0x40 [11332.114724] ? hrtimer_interrupt+0x128/0x2c0 [11332.114729] ? smp_apic_timer_interrupt+0xa6/0x270 [11332.114738] ? apic_timer_interrupt+0xf/0x20 [11332.114743] [11332.114746] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11332.114751] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11332.114756] do_raw_spin_lock+0x177/0x190 [11332.114761] _raw_spin_lock+0x21/0x30 [11332.114769] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11332.115830] obd_mod_free+0x46/0x140 [ptlrpc] [11332.122088] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11332.122187] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11332.122196] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11332.122203] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11332.122508] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11332.122519] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11332.122524] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11332.122528] RAX: 0000000000007c08 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11332.122531] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [11332.122533] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [11332.122535] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [11332.122538] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [11332.122541] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [11332.122544] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11332.122546] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11332.122556] Call Trace: [11332.122561] [11332.122566] ? show_regs.cold.9+0x22/0x2f [11332.122575] ? watchdog_timer_fn.cold.6+0x62/0x10a [11332.122582] ? __hrtimer_run_queues+0x1c2/0x4d0 [11332.122586] ? watchdog+0x40/0x40 [11332.122590] ? hrtimer_interrupt+0x128/0x2c0 [11332.122594] ? smp_apic_timer_interrupt+0xa6/0x270 [11332.122603] ? apic_timer_interrupt+0xf/0x20 [11332.122608] [11332.122611] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11332.122616] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11332.122621] do_raw_spin_lock+0x177/0x190 [11332.122625] _raw_spin_lock+0x21/0x30 [11332.122630] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11332.122846] process_one_work+0x2c8/0x700 [11332.122852] worker_thread+0x296/0x6e0 [11332.122858] ? rescuer_thread+0x570/0x570 [11332.122861] kthread+0x1d1/0x200 [11332.122865] ? set_kthread_struct+0x70/0x70 [11332.122869] ret_from_fork+0x1f/0x30 [11332.440673] mdc_commit_open+0x91/0xb0 [mdc] [11332.444784] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11332.448580] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11332.453410] after_reply+0xe2c/0x1780 [ptlrpc] [11332.455684] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11332.458792] ? __next_timer_interrupt+0x160/0x160 [11332.463705] ptlrpcd+0xa82/0xe40 [ptlrpc] [11332.465761] ? do_wait_intr+0xf0/0xf0 [11332.468625] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11332.473330] kthread+0x1d1/0x200 [11332.477244] ? set_kthread_struct+0x70/0x70 [11332.479467] ret_from_fork+0x1f/0x30 [11336.052050] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [11336.061658] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11336.090607] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11336.095833] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11336.099456] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11336.102383] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11336.111507] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11336.114813] RAX: 00000000000024f5 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11336.118956] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [11336.122337] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [11336.128269] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [11336.132592] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [11336.138092] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [11336.142640] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11336.145703] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [11336.151038] Call Trace: [11336.152432] [11336.154236] ? show_regs.cold.9+0x22/0x2f [11336.156582] ? watchdog_timer_fn.cold.6+0x62/0x10a [11336.158470] ? __hrtimer_run_queues+0x1c2/0x4d0 [11336.160454] ? watchdog+0x40/0x40 [11336.163156] ? hrtimer_interrupt+0x128/0x2c0 [11336.164750] ? smp_apic_timer_interrupt+0xa6/0x270 [11336.167130] ? apic_timer_interrupt+0xf/0x20 [11336.169571] [11336.170363] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11336.172818] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11336.175492] do_raw_spin_lock+0x177/0x190 [11336.177165] _raw_spin_lock+0x21/0x30 [11336.178703] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11336.181454] ? do_raw_spin_unlock+0x75/0x190 [11336.183227] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11336.185658] ? do_wait_intr+0xf0/0xf0 [11336.187236] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11336.191239] mdc_reint+0x2d/0x110 [mdc] [11336.192971] mdc_unlink+0x658/0x870 [mdc] [11336.196687] lmv_unlink+0x34d/0xbd0 [lmv] [11336.198604] ll_unlink+0x24d/0x7f0 [lustre] [11336.201337] vfs_unlink+0x202/0x330 [11336.202812] do_unlinkat+0x2fa/0x460 [11336.207366] __x64_sys_unlink+0x37/0x50 [11336.209352] do_syscall_64+0xc1/0x440 [11336.211944] entry_SYSCALL_64_after_hwframe+0x49/0xae [11336.216718] RIP: 0033:0x7f151bdee10b [11336.218832] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [11336.229152] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [11336.233205] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [11336.240723] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [11336.244586] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11336.248485] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11336.252190] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11340.772705] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9410s! [11340.778855] Showing busy workqueues and worker pools: [11340.787010] workqueue events: flags=0x0 [11340.792013] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11340.803413] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [11340.815811] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11340.821487] pending: kfree_rcu_monitor [11340.822715] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11340.828510] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [11340.844272] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11340.856954] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [11340.864794] workqueue events_long: flags=0x0 [11340.867266] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11340.872365] pending: lu_objects_flush [obdclass] [11340.878645] workqueue events_unbound: flags=0x2 [11340.881348] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11340.890125] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11340.906718] workqueue events_power_efficient: flags=0x80 [11340.917226] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11340.926745] pending: do_cache_clean, neigh_periodic_work, check_lifetime [11340.931826] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11340.939506] pending: neigh_periodic_work [11340.943422] workqueue mm_percpu_wq: flags=0x8 [11340.948809] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11340.958746] pending: vmstat_update, lru_add_drain_per_cpu [11340.965198] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11340.969835] pending: vmstat_update [11340.972257] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11340.976200] pending: lru_add_drain_per_cpu [11340.979306] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [11340.982903] pending: lru_add_drain_per_cpu BAR(48) [11340.985436] workqueue writeback: flags=0x4a [11340.988351] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [11340.990928] pending: wb_workfn [11340.991991] workqueue xprtiod: flags=0xa [11340.993295] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11341.001025] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [11341.008732] workqueue ipv6_addrconf: flags=0x40008 [11341.014221] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11341.017848] pending: addrconf_verify_work [11341.024941] workqueue knbd0-recv: flags=0x1a [11341.030800] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11341.041004] in-flight: 658:recv_work [11341.045977] workqueue ptlrpc_pinger: flags=0x2 [11341.051806] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11341.056948] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11341.066231] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9410s workers=4 idle: 32984 37 8 [11341.078238] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11325s workers=2 idle: 661 [11344.853958] rcu: INFO: rcu_sched self-detected stall on CPU [11344.859980] rcu: 1-...!: (5429737 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [11344.878462] (t=9420206 jiffies g=249657 q=5107) [11344.881157] rcu: rcu_sched kthread starved for 9415158 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [11344.889806] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [11344.904893] rcu: RCU grace-period kthread stack dump: [11344.912787] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [11344.923604] Call Trace: [11344.924801] __schedule+0x351/0xcb0 [11344.930662] schedule+0xc0/0x180 [11344.934809] schedule_timeout+0xb4/0x190 [11344.940426] ? do_raw_spin_unlock+0x75/0x190 [11344.941988] ? __next_timer_interrupt+0x160/0x160 [11344.946737] ? prepare_to_swait_event+0xf3/0x140 [11344.952579] rcu_gp_kthread+0x8dd/0x15b0 [11344.961213] ? force_qs_rnp+0x2c0/0x2c0 [11344.964602] kthread+0x1d1/0x200 [11344.968079] ? set_kthread_struct+0x70/0x70 [11344.972750] ret_from_fork+0x1f/0x30 [11344.976248] rcu: Stack dump where RCU GP kthread last ran: [11344.980625] Sending NMI from CPU 1 to CPUs 2: [11344.993013] NMI backtrace for cpu 2 [11344.993021] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11344.993024] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11344.993026] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11344.993029] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11344.993031] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11344.993033] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 [11344.993035] RAX: 0000000000003595 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11344.993036] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [11344.993038] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [11344.993039] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [11344.993040] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [11344.993041] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [11344.993043] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11344.993044] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11344.993045] Call Trace: [11344.993045] [11344.993046] ? show_regs.cold.9+0x22/0x2f [11344.993047] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [11344.993048] ? nmi_cpu_backtrace_handler+0x15/0x40 [11344.993049] ? nmi_handle+0xbd/0x210 [11344.993050] ? default_do_nmi+0x4b/0x1b0 [11344.993051] ? do_nmi+0x280/0x380 [11344.993052] ? end_repeat_nmi+0x16/0x1f [11344.993053] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11344.993054] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11344.993055] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11344.993056] [11344.993057] do_raw_spin_lock+0x177/0x190 [11344.993058] _raw_spin_lock+0x21/0x30 [11344.993059] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11344.993060] process_one_work+0x2c8/0x700 [11344.993061] worker_thread+0x296/0x6e0 [11344.993062] ? rescuer_thread+0x570/0x570 [11344.993063] kthread+0x1d1/0x200 [11344.993063] ? set_kthread_struct+0x70/0x70 [11344.993064] ret_from_fork+0x1f/0x30 [11344.994750] NMI backtrace for cpu 1 [11345.213062] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11345.233661] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11345.242595] Call Trace: [11345.244059] [11345.245212] ? dump_stack+0xbb/0x10e [11345.247607] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [11345.253363] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [11345.256666] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [11345.259546] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [11345.263510] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [11345.273793] ? rcu_dump_cpu_stacks+0x11b/0x171 [11345.276930] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [11345.283465] ? update_process_times+0x6d/0xc0 [11345.285424] ? tick_sched_handle.isra.18+0x27/0x90 [11345.290875] ? tick_sched_timer+0x53/0xb0 [11345.295141] ? __hrtimer_run_queues+0x1c2/0x4d0 [11345.300049] ? tick_sched_handle.isra.18+0x90/0x90 [11345.309816] ? hrtimer_interrupt+0x128/0x2c0 [11345.313408] ? smp_apic_timer_interrupt+0xa6/0x270 [11345.316604] ? apic_timer_interrupt+0xf/0x20 [11345.321693] [11345.327579] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11345.341395] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11345.344193] ? do_raw_spin_lock+0x177/0x190 [11345.350736] ? _raw_spin_lock+0x21/0x30 [11345.354594] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11345.366387] ? obd_mod_free+0x46/0x140 [ptlrpc] [11345.374183] ? mdc_commit_open+0x91/0xb0 [mdc] [11345.385204] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11345.388488] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11345.390821] ? after_reply+0xe2c/0x1780 [ptlrpc] [11345.399550] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11345.415943] ? __next_timer_interrupt+0x160/0x160 [11345.421995] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [11345.426583] ? do_wait_intr+0xf0/0xf0 [11345.429860] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11345.435035] ? kthread+0x1d1/0x200 [11345.437347] ? set_kthread_struct+0x70/0x70 [11345.442853] ? ret_from_fork+0x1f/0x30 [11356.057087] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [11356.069774] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11356.122290] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11356.128344] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11356.134979] RIP: 0010:native_safe_halt+0x2e/0x50 [11356.141504] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11356.156437] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11356.163399] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11356.172244] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [11356.176136] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [11356.180151] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [11356.183873] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [11356.195081] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [11356.206803] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11356.210601] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [11356.219703] Call Trace: [11356.226371] [11356.227665] ? show_regs.cold.9+0x22/0x2f [11356.229517] ? watchdog_timer_fn.cold.6+0x62/0x10a [11356.232561] ? __hrtimer_run_queues+0x1c2/0x4d0 [11356.235835] ? watchdog+0x40/0x40 [11356.238346] ? hrtimer_interrupt+0x128/0x2c0 [11356.241637] ? smp_apic_timer_interrupt+0xa6/0x270 [11356.252509] ? apic_timer_interrupt+0xf/0x20 [11356.256216] [11356.257007] ? native_safe_halt+0x2e/0x50 [11356.258519] kvm_wait+0xcd/0xe0 [11356.262703] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11356.267759] do_raw_spin_lock+0x177/0x190 [11356.270107] _raw_spin_lock+0x21/0x30 [11356.274424] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11356.279772] ? do_raw_spin_unlock+0x75/0x190 [11356.284003] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11356.286313] ? do_wait_intr+0xf0/0xf0 [11356.287501] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11356.295421] mdc_close+0x53b/0xd70 [mdc] [11356.298290] lmv_close+0x18d/0x490 [lmv] [11356.301649] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11356.303449] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11356.306393] ll_md_real_close+0xa5/0x2a0 [lustre] [11356.308572] ll_md_close+0x274/0x800 [lustre] [11356.310443] ll_file_release+0x128/0xc50 [lustre] [11356.314277] __fput+0x10d/0x440 [11356.315445] ____fput+0x12/0x20 [11356.316632] task_work_run+0xc6/0x110 [11356.319857] exit_to_usermode_loop+0x1dd/0x1f0 [11356.324780] do_syscall_64+0x3d6/0x440 [11356.327716] entry_SYSCALL_64_after_hwframe+0x49/0xae [11356.333395] RIP: 0033:0x7f151bee3cd8 [11356.334827] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11356.353575] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11356.357540] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [11356.363759] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [11356.368745] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11356.371776] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11356.374282] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11368.052676] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [11368.055703] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11368.095846] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11368.104385] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11368.115361] RIP: 0010:native_safe_halt+0x2e/0x50 [11368.118306] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11368.128820] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11368.137750] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11368.144704] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [11368.152397] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [11368.158257] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [11368.164396] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [11368.168508] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [11368.173175] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11368.176241] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [11368.182488] Call Trace: [11368.183749] [11368.185194] ? show_regs.cold.9+0x22/0x2f [11368.189846] ? watchdog_timer_fn.cold.6+0x62/0x10a [11368.192456] ? __hrtimer_run_queues+0x1c2/0x4d0 [11368.194845] ? watchdog+0x40/0x40 [11368.197009] ? hrtimer_interrupt+0x128/0x2c0 [11368.211367] ? smp_apic_timer_interrupt+0xa6/0x270 [11368.213884] ? apic_timer_interrupt+0xf/0x20 [11368.215140] [11368.215759] ? native_safe_halt+0x2e/0x50 [11368.217435] kvm_wait+0xcd/0xe0 [11368.218732] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11368.235666] do_raw_spin_lock+0x177/0x190 [11368.237377] _raw_spin_lock+0x21/0x30 [11368.238874] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11368.241260] ? do_raw_spin_unlock+0x75/0x190 [11368.243059] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11368.245417] ? do_wait_intr+0xf0/0xf0 [11368.247029] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11368.249378] mdc_reint+0x2d/0x110 [mdc] [11368.251069] mdc_unlink+0x658/0x870 [mdc] [11368.252699] lmv_unlink+0x34d/0xbd0 [lmv] [11368.254417] ll_unlink+0x24d/0x7f0 [lustre] [11368.256721] vfs_unlink+0x202/0x330 [11368.258169] do_unlinkat+0x2fa/0x460 [11368.260151] __x64_sys_unlink+0x37/0x50 [11368.261625] do_syscall_64+0xc1/0x440 [11368.263180] entry_SYSCALL_64_after_hwframe+0x49/0xae [11368.265451] RIP: 0033:0x7f151bdee10b [11368.267038] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [11368.274510] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [11368.277439] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [11368.280316] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [11368.283170] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11368.286111] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11368.288808] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11371.488366] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9440s! [11371.498726] Showing busy workqueues and worker pools: [11371.506933] workqueue events: flags=0x0 [11371.512494] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11371.519407] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [11371.525589] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11371.530401] pending: kfree_rcu_monitor [11371.534462] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11371.538513] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [11371.552678] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11371.561210] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [11371.567436] workqueue events_long: flags=0x0 [11371.569499] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11371.580577] pending: lu_objects_flush [obdclass] [11371.585727] workqueue events_unbound: flags=0x2 [11371.591955] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11371.599706] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11371.611709] workqueue events_power_efficient: flags=0x80 [11371.615373] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11371.620399] pending: do_cache_clean, neigh_periodic_work, check_lifetime [11371.625176] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11371.634651] pending: neigh_periodic_work [11371.642380] workqueue mm_percpu_wq: flags=0x8 [11371.654698] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11371.659391] pending: vmstat_update, lru_add_drain_per_cpu [11371.661845] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11371.665506] pending: vmstat_update [11371.670385] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11371.675676] pending: lru_add_drain_per_cpu [11371.680177] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [11371.690092] pending: lru_add_drain_per_cpu BAR(48) [11371.693640] workqueue writeback: flags=0x4a [11371.699024] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [11371.707720] pending: wb_workfn [11371.713701] workqueue xprtiod: flags=0xa [11371.715316] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11371.720283] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [11371.725646] workqueue ipv6_addrconf: flags=0x40008 [11371.728445] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11371.734820] pending: addrconf_verify_work [11371.740140] workqueue knbd0-recv: flags=0x1a [11371.746123] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11371.753259] in-flight: 658:recv_work [11371.759059] workqueue ptlrpc_pinger: flags=0x2 [11371.764267] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11371.769932] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11371.773985] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9441s workers=4 idle: 32984 37 8 [11371.781297] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11356s workers=2 idle: 661 [11372.055029] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [11372.057923] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [11372.058380] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11372.058456] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11372.058461] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11372.058463] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11372.058478] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11372.058482] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11372.058486] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [11372.058489] RDX: 0000000000007214 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [11372.058491] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [11372.058494] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [11372.058496] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11372.058500] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [11372.058502] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11372.058505] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11372.058513] Call Trace: [11372.058518] [11372.058523] ? show_regs.cold.9+0x22/0x2f [11372.058533] ? watchdog_timer_fn.cold.6+0x62/0x10a [11372.058539] ? __hrtimer_run_queues+0x1c2/0x4d0 [11372.058544] ? watchdog+0x40/0x40 [11372.058549] ? hrtimer_interrupt+0x128/0x2c0 [11372.058553] ? smp_apic_timer_interrupt+0xa6/0x270 [11372.059068] ? apic_timer_interrupt+0xf/0x20 [11372.059073] [11372.059076] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11372.059081] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11372.059571] do_raw_spin_lock+0x177/0x190 [11372.059577] _raw_spin_lock+0x21/0x30 [11372.059583] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11372.060367] obd_mod_free+0x46/0x140 [ptlrpc] [11372.063082] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11372.063158] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11372.063166] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11372.063171] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11372.063443] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11372.063454] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11372.063458] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 [11372.063462] RAX: 0000000000005300 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11372.063465] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [11372.063467] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [11372.063470] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [11372.063472] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [11372.063475] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [11372.063478] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11372.063480] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11372.063489] Call Trace: [11372.063495] [11372.063501] ? show_regs.cold.9+0x22/0x2f [11372.063510] ? watchdog_timer_fn.cold.6+0x62/0x10a [11372.063516] ? __hrtimer_run_queues+0x1c2/0x4d0 [11372.063521] ? watchdog+0x40/0x40 [11372.063524] ? hrtimer_interrupt+0x128/0x2c0 [11372.063528] ? smp_apic_timer_interrupt+0xa6/0x270 [11372.063536] ? apic_timer_interrupt+0xf/0x20 [11372.063540] [11372.063543] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11372.063549] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11372.063554] do_raw_spin_lock+0x177/0x190 [11372.063559] _raw_spin_lock+0x21/0x30 [11372.063638] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11372.086237] mdc_commit_open+0x91/0xb0 [mdc] [11372.086296] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11372.091512] process_one_work+0x2c8/0x700 [11372.091534] worker_thread+0x296/0x6e0 [11372.091539] ? rescuer_thread+0x570/0x570 [11372.091543] kthread+0x1d1/0x200 [11372.091548] ? set_kthread_struct+0x70/0x70 [11372.091551] ret_from_fork+0x1f/0x30 [11372.294422] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11372.297346] after_reply+0xe2c/0x1780 [ptlrpc] [11372.299237] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11372.301476] ? __next_timer_interrupt+0x160/0x160 [11372.303080] ptlrpcd+0xa82/0xe40 [ptlrpc] [11372.304554] ? do_wait_intr+0xf0/0xf0 [11372.305941] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11372.307988] kthread+0x1d1/0x200 [11372.309124] ? set_kthread_struct+0x70/0x70 [11372.310661] ret_from_fork+0x1f/0x30 [11384.060021] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [11384.071322] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11384.100604] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11384.105722] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11384.109632] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11384.112209] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11384.120019] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11384.123391] RAX: 0000000000006911 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11384.126394] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [11384.129381] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [11384.132434] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [11384.135536] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [11384.138307] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [11384.141321] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11384.143765] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [11384.146531] Call Trace: [11384.147901] [11384.148835] ? show_regs.cold.9+0x22/0x2f [11384.150459] ? watchdog_timer_fn.cold.6+0x62/0x10a [11384.153019] ? __hrtimer_run_queues+0x1c2/0x4d0 [11384.154899] ? watchdog+0x40/0x40 [11384.155980] ? hrtimer_interrupt+0x128/0x2c0 [11384.157103] ? smp_apic_timer_interrupt+0xa6/0x270 [11384.159036] ? apic_timer_interrupt+0xf/0x20 [11384.161073] [11384.162183] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11384.164409] do_raw_spin_lock+0x177/0x190 [11384.166168] _raw_spin_lock+0x21/0x30 [11384.167491] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11384.169955] ? do_raw_spin_unlock+0x75/0x190 [11384.171960] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11384.174114] ? do_wait_intr+0xf0/0xf0 [11384.175599] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11384.177578] mdc_close+0x53b/0xd70 [mdc] [11384.178954] lmv_close+0x18d/0x490 [lmv] [11384.180287] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11384.182492] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11384.184767] ll_md_real_close+0xa5/0x2a0 [lustre] [11384.186724] ll_md_close+0x274/0x800 [lustre] [11384.188352] ll_file_release+0x128/0xc50 [lustre] [11384.190167] __fput+0x10d/0x440 [11384.191414] ____fput+0x12/0x20 [11384.192806] task_work_run+0xc6/0x110 [11384.194244] exit_to_usermode_loop+0x1dd/0x1f0 [11384.195918] do_syscall_64+0x3d6/0x440 [11384.197430] entry_SYSCALL_64_after_hwframe+0x49/0xae [11384.199254] RIP: 0033:0x7f151bee3cd8 [11384.200592] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11384.207347] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11384.209984] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [11384.212412] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [11384.214827] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11384.217491] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11384.220100] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11396.052948] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [11396.059825] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11396.108325] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11396.124182] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11396.131372] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11396.141347] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11396.168681] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11396.177928] RAX: 000000000000099a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11396.188786] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [11396.196274] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [11396.207398] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [11396.219349] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [11396.230726] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [11396.245228] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11396.264055] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [11396.275453] Call Trace: [11396.282945] [11396.284069] ? show_regs.cold.9+0x22/0x2f [11396.288509] ? watchdog_timer_fn.cold.6+0x62/0x10a [11396.296943] ? __hrtimer_run_queues+0x1c2/0x4d0 [11396.301744] ? watchdog+0x40/0x40 [11396.303887] ? hrtimer_interrupt+0x128/0x2c0 [11396.311152] ? smp_apic_timer_interrupt+0xa6/0x270 [11396.314140] ? apic_timer_interrupt+0xf/0x20 [11396.321055] [11396.323068] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11396.328301] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11396.335413] do_raw_spin_lock+0x177/0x190 [11396.336901] _raw_spin_lock+0x21/0x30 [11396.344256] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11396.354451] ? do_raw_spin_unlock+0x75/0x190 [11396.362949] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11396.367135] ? do_wait_intr+0xf0/0xf0 [11396.369831] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11396.377460] mdc_reint+0x2d/0x110 [mdc] [11396.383695] mdc_unlink+0x658/0x870 [mdc] [11396.387354] lmv_unlink+0x34d/0xbd0 [lmv] [11396.392402] ll_unlink+0x24d/0x7f0 [lustre] [11396.395546] vfs_unlink+0x202/0x330 [11396.397773] do_unlinkat+0x2fa/0x460 [11396.400231] __x64_sys_unlink+0x37/0x50 [11396.402386] do_syscall_64+0xc1/0x440 [11396.404831] entry_SYSCALL_64_after_hwframe+0x49/0xae [11396.408314] RIP: 0033:0x7f151bdee10b [11396.410961] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [11396.420453] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [11396.441047] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [11396.443544] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [11396.461845] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11396.477924] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11396.491802] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11400.062127] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [11400.068899] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [11400.075983] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11400.076072] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11400.076080] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11400.076086] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11400.076372] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11400.076383] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11400.076387] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11400.076392] RAX: 000000000000750c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11400.076395] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [11400.076397] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [11400.076399] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [11400.076402] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [11400.076405] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [11400.076409] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11400.076411] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11400.076421] Call Trace: [11400.076426] [11400.076432] ? show_regs.cold.9+0x22/0x2f [11400.076443] ? watchdog_timer_fn.cold.6+0x62/0x10a [11400.076451] ? __hrtimer_run_queues+0x1c2/0x4d0 [11400.076455] ? watchdog+0x40/0x40 [11400.076460] ? hrtimer_interrupt+0x128/0x2c0 [11400.076464] ? smp_apic_timer_interrupt+0xa6/0x270 [11400.076472] ? apic_timer_interrupt+0xf/0x20 [11400.076477] [11400.076481] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11400.076485] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11400.076491] do_raw_spin_lock+0x177/0x190 [11400.076496] _raw_spin_lock+0x21/0x30 [11400.076502] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11400.076734] process_one_work+0x2c8/0x700 [11400.076742] worker_thread+0x296/0x6e0 [11400.076747] ? rescuer_thread+0x570/0x570 [11400.076750] kthread+0x1d1/0x200 [11400.076755] ? set_kthread_struct+0x70/0x70 [11400.076759] ret_from_fork+0x1f/0x30 [11400.433350] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11400.529861] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11400.539201] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11400.558174] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11400.563778] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11400.607615] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11400.610225] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [11400.631418] RDX: 000000000000116c RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [11400.646748] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [11400.658341] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [11400.676447] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11400.691298] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [11400.714708] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11400.725795] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11400.743230] Call Trace: [11400.756797] [11400.763368] ? show_regs.cold.9+0x22/0x2f [11400.769261] ? watchdog_timer_fn.cold.6+0x62/0x10a [11400.775406] ? __hrtimer_run_queues+0x1c2/0x4d0 [11400.784784] ? watchdog+0x40/0x40 [11400.790085] ? hrtimer_interrupt+0x128/0x2c0 [11400.793679] ? smp_apic_timer_interrupt+0xa6/0x270 [11400.805870] ? apic_timer_interrupt+0xf/0x20 [11400.818111] [11400.820260] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11400.826049] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11400.836084] do_raw_spin_lock+0x177/0x190 [11400.837415] _raw_spin_lock+0x21/0x30 [11400.846626] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11400.857964] obd_mod_free+0x46/0x140 [ptlrpc] [11400.873511] mdc_commit_open+0x91/0xb0 [mdc] [11400.874821] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11400.882578] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11400.891404] after_reply+0xe2c/0x1780 [ptlrpc] [11400.906228] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11400.912144] ? __next_timer_interrupt+0x160/0x160 [11400.926895] ptlrpcd+0xa82/0xe40 [ptlrpc] [11400.939153] ? do_wait_intr+0xf0/0xf0 [11400.944122] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11400.956161] kthread+0x1d1/0x200 [11400.959939] ? set_kthread_struct+0x70/0x70 [11400.966138] ret_from_fork+0x1f/0x30 [11402.210363] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9471s! [11402.225155] Showing busy workqueues and worker pools: [11402.233655] workqueue events: flags=0x0 [11402.237793] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11402.242319] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [11402.250779] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11402.255119] pending: kfree_rcu_monitor [11402.257644] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11402.261693] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [11402.267919] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11402.279322] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [11402.295138] workqueue events_long: flags=0x0 [11402.303235] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11402.313308] pending: lu_objects_flush [obdclass] [11402.323920] workqueue events_unbound: flags=0x2 [11402.331270] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11402.343793] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11402.360262] workqueue events_power_efficient: flags=0x80 [11402.365494] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11402.369953] pending: do_cache_clean, neigh_periodic_work, check_lifetime [11402.374367] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11402.377999] pending: neigh_periodic_work [11402.380706] workqueue mm_percpu_wq: flags=0x8 [11402.383988] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11402.389584] pending: vmstat_update, lru_add_drain_per_cpu [11402.393210] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11402.401262] pending: vmstat_update [11402.406362] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11402.416157] pending: lru_add_drain_per_cpu [11402.426347] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [11402.430678] pending: lru_add_drain_per_cpu BAR(48) [11402.435171] workqueue writeback: flags=0x4a [11402.437983] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [11402.444157] pending: wb_workfn [11402.446134] workqueue xprtiod: flags=0xa [11402.448950] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11402.453352] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [11402.458491] workqueue ipv6_addrconf: flags=0x40008 [11402.461031] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11402.464418] pending: addrconf_verify_work [11402.467370] workqueue knbd0-recv: flags=0x1a [11402.473362] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11402.477908] in-flight: 658:recv_work [11402.480989] workqueue ptlrpc_pinger: flags=0x2 [11402.483446] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11402.490749] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11402.496734] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9471s workers=4 idle: 32984 37 8 [11402.500984] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11387s workers=2 idle: 661 [11412.058981] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [11412.066156] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11412.110070] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11412.120590] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11412.128058] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11412.131323] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11412.144199] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11412.151283] RAX: 0000000000007a11 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11412.161856] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [11412.164401] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [11412.181207] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [11412.190965] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [11412.199630] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [11412.210436] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11412.216819] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [11412.224389] Call Trace: [11412.225891] [11412.230189] ? show_regs.cold.9+0x22/0x2f [11412.236887] ? watchdog_timer_fn.cold.6+0x62/0x10a [11412.242924] ? __hrtimer_run_queues+0x1c2/0x4d0 [11412.246077] ? watchdog+0x40/0x40 [11412.251793] ? hrtimer_interrupt+0x128/0x2c0 [11412.257739] ? smp_apic_timer_interrupt+0xa6/0x270 [11412.260172] ? apic_timer_interrupt+0xf/0x20 [11412.265942] [11412.268210] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11412.272627] do_raw_spin_lock+0x177/0x190 [11412.278530] _raw_spin_lock+0x21/0x30 [11412.282318] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11412.289209] ? do_raw_spin_unlock+0x75/0x190 [11412.294970] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11412.304423] ? do_wait_intr+0xf0/0xf0 [11412.306214] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11412.313192] mdc_close+0x53b/0xd70 [mdc] [11412.322583] lmv_close+0x18d/0x490 [lmv] [11412.333033] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11412.339574] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11412.353013] ll_md_real_close+0xa5/0x2a0 [lustre] [11412.359559] ll_md_close+0x274/0x800 [lustre] [11412.364049] ll_file_release+0x128/0xc50 [lustre] [11412.373594] __fput+0x10d/0x440 [11412.380066] ____fput+0x12/0x20 [11412.382200] task_work_run+0xc6/0x110 [11412.389114] exit_to_usermode_loop+0x1dd/0x1f0 [11412.397674] do_syscall_64+0x3d6/0x440 [11412.403294] entry_SYSCALL_64_after_hwframe+0x49/0xae [11412.407437] RIP: 0033:0x7f151bee3cd8 [11412.410135] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11412.422182] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11412.426275] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [11412.430082] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [11412.432996] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11412.436130] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11412.438974] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11428.052029] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [11428.054657] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11428.055511] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [11428.056250] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [11428.056258] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11428.056347] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11428.056353] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11428.056359] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11428.056650] RIP: 0010:native_safe_halt+0x2e/0x50 [11428.056660] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11428.056664] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11428.056668] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11428.056671] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [11428.056673] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [11428.056675] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [11428.056677] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [11428.056681] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [11428.056684] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11428.056686] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11428.056695] Call Trace: [11428.056700] [11428.056705] ? show_regs.cold.9+0x22/0x2f [11428.056721] ? watchdog_timer_fn.cold.6+0x62/0x10a [11428.056728] ? __hrtimer_run_queues+0x1c2/0x4d0 [11428.056732] ? watchdog+0x40/0x40 [11428.056737] ? hrtimer_interrupt+0x128/0x2c0 [11428.056740] ? smp_apic_timer_interrupt+0xa6/0x270 [11428.056748] ? apic_timer_interrupt+0xf/0x20 [11428.056753] [11428.056756] ? native_safe_halt+0x2e/0x50 [11428.056760] kvm_wait+0xcd/0xe0 [11428.056769] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11428.056788] do_raw_spin_lock+0x177/0x190 [11428.056793] _raw_spin_lock+0x21/0x30 [11428.056797] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11428.057010] process_one_work+0x2c8/0x700 [11428.057017] worker_thread+0x296/0x6e0 [11428.057022] ? rescuer_thread+0x570/0x570 [11428.057027] kthread+0x1d1/0x200 [11428.057032] ? set_kthread_struct+0x70/0x70 [11428.057041] ret_from_fork+0x1f/0x30 [11428.096616] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11428.102352] Modules linked in: [11428.107940] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11428.107951] RIP: 0010:native_safe_halt+0x2e/0x50 [11428.161668] lustre(O) [11428.173869] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11428.173883] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11428.173890] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11428.173893] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [11428.173895] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [11428.173898] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [11428.173900] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [11428.173904] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [11428.173907] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11428.173909] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [11428.173919] Call Trace: [11428.173926] [11428.173932] ? show_regs.cold.9+0x22/0x2f [11428.173947] ? watchdog_timer_fn.cold.6+0x62/0x10a [11428.173954] ? __hrtimer_run_queues+0x1c2/0x4d0 [11428.173959] ? watchdog+0x40/0x40 [11428.192362] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11428.192437] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11428.192445] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11428.192448] RIP: 0010:native_safe_halt+0x2e/0x50 [11428.192461] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11428.192466] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11428.192471] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [11428.192474] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [11428.192476] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [11428.192479] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [11428.198906] ? hrtimer_interrupt+0x128/0x2c0 [11428.198924] ? smp_apic_timer_interrupt+0xa6/0x270 [11428.198932] ? apic_timer_interrupt+0xf/0x20 [11428.198938] [11428.198944] ? native_safe_halt+0x2e/0x50 [11428.198949] kvm_wait+0xcd/0xe0 [11428.198958] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11428.198979] do_raw_spin_lock+0x177/0x190 [11428.198983] _raw_spin_lock+0x21/0x30 [11428.198991] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11428.200678] ? do_raw_spin_unlock+0x75/0x190 [11428.200687] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11428.200879] ? do_wait_intr+0xf0/0xf0 [11428.200886] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11428.201844] mdc_reint+0x2d/0x110 [mdc] [11428.201881] mdc_unlink+0x658/0x870 [mdc] [11428.201907] lmv_unlink+0x34d/0xbd0 [lmv] [11428.201941] ll_unlink+0x24d/0x7f0 [lustre] [11428.202718] vfs_unlink+0x202/0x330 [11428.202727] do_unlinkat+0x2fa/0x460 [11428.202732] __x64_sys_unlink+0x37/0x50 [11428.202736] do_syscall_64+0xc1/0x440 [11428.202743] entry_SYSCALL_64_after_hwframe+0x49/0xae [11428.202751] RIP: 0033:0x7f151bdee10b [11428.202759] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [11428.202763] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [11428.202769] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [11428.202772] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [11428.202774] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11428.202776] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11428.202778] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11428.937891] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11428.942465] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [11428.945188] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11428.955818] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11428.962260] Call Trace: [11428.968259] [11428.969455] ? show_regs.cold.9+0x22/0x2f [11428.976252] ? watchdog_timer_fn.cold.6+0x62/0x10a [11428.981000] ? __hrtimer_run_queues+0x1c2/0x4d0 [11428.983134] ? watchdog+0x40/0x40 [11428.985357] ? hrtimer_interrupt+0x128/0x2c0 [11428.989330] ? smp_apic_timer_interrupt+0xa6/0x270 [11428.993111] ? apic_timer_interrupt+0xf/0x20 [11428.998454] [11428.999396] ? native_safe_halt+0x2e/0x50 [11429.002386] kvm_wait+0xcd/0xe0 [11429.003999] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11429.010573] do_raw_spin_lock+0x177/0x190 [11429.012682] _raw_spin_lock+0x21/0x30 [11429.015011] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11429.021401] obd_mod_free+0x46/0x140 [ptlrpc] [11429.024110] mdc_commit_open+0x91/0xb0 [mdc] [11429.025475] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11429.027568] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11429.030470] after_reply+0xe2c/0x1780 [ptlrpc] [11429.032666] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11429.035160] ? __next_timer_interrupt+0x160/0x160 [11429.037698] ptlrpcd+0xa82/0xe40 [ptlrpc] [11429.040046] ? do_wait_intr+0xf0/0xf0 [11429.042022] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11429.045289] kthread+0x1d1/0x200 [11429.047438] ? set_kthread_struct+0x70/0x70 [11429.050258] ret_from_fork+0x1f/0x30 [11432.929642] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9502s! [11432.934546] Showing busy workqueues and worker pools: [11432.937861] workqueue events: flags=0x0 [11432.940021] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11432.944427] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [11432.950518] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11432.953845] pending: kfree_rcu_monitor [11432.956528] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11432.960207] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [11432.965256] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11432.971191] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [11432.976112] workqueue events_long: flags=0x0 [11432.978493] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11432.982127] pending: lu_objects_flush [obdclass] [11432.984785] workqueue events_unbound: flags=0x2 [11432.987084] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11432.991422] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11432.997324] workqueue events_power_efficient: flags=0x80 [11432.999743] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11433.003676] pending: do_cache_clean, neigh_periodic_work, check_lifetime [11433.007961] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11433.011228] pending: neigh_periodic_work [11433.012985] workqueue mm_percpu_wq: flags=0x8 [11433.014966] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11433.018581] pending: vmstat_update, lru_add_drain_per_cpu [11433.021536] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11433.025915] pending: vmstat_update [11433.027941] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11433.031689] pending: lru_add_drain_per_cpu [11433.034302] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [11433.036804] pending: lru_add_drain_per_cpu BAR(48) [11433.039456] workqueue writeback: flags=0x4a [11433.041610] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [11433.044866] pending: wb_workfn [11433.046826] workqueue xprtiod: flags=0xa [11433.048066] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11433.051125] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [11433.056247] workqueue ipv6_addrconf: flags=0x40008 [11433.058519] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11433.061759] pending: addrconf_verify_work [11433.064134] workqueue knbd0-recv: flags=0x1a [11433.066214] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11433.069505] in-flight: 658:recv_work [11433.071871] workqueue ptlrpc_pinger: flags=0x2 [11433.075156] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11433.078498] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11433.081201] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9502s workers=4 idle: 32984 37 8 [11433.085051] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11417s workers=2 idle: 661 [11440.057024] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [11440.059303] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11440.084912] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11440.094889] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11440.101062] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11440.103579] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11440.133822] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11440.143281] RAX: 0000000000006c8a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11440.153555] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [11440.164315] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [11440.172132] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [11440.182423] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [11440.193670] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [11440.203985] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11440.211788] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [11440.216593] Call Trace: [11440.217715] [11440.221344] ? show_regs.cold.9+0x22/0x2f [11440.224755] ? watchdog_timer_fn.cold.6+0x62/0x10a [11440.232859] ? __hrtimer_run_queues+0x1c2/0x4d0 [11440.238582] ? watchdog+0x40/0x40 [11440.246436] ? hrtimer_interrupt+0x128/0x2c0 [11440.250359] ? smp_apic_timer_interrupt+0xa6/0x270 [11440.260046] ? apic_timer_interrupt+0xf/0x20 [11440.268130] [11440.269500] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11440.284090] do_raw_spin_lock+0x177/0x190 [11440.287953] _raw_spin_lock+0x21/0x30 [11440.295325] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11440.304745] ? do_raw_spin_unlock+0x75/0x190 [11440.309952] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11440.314948] ? do_wait_intr+0xf0/0xf0 [11440.322773] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11440.330523] mdc_close+0x53b/0xd70 [mdc] [11440.334505] lmv_close+0x18d/0x490 [lmv] [11440.337018] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11440.350006] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11440.354875] ll_md_real_close+0xa5/0x2a0 [lustre] [11440.362017] ll_md_close+0x274/0x800 [lustre] [11440.365915] ll_file_release+0x128/0xc50 [lustre] [11440.373465] __fput+0x10d/0x440 [11440.375300] ____fput+0x12/0x20 [11440.377771] task_work_run+0xc6/0x110 [11440.382814] exit_to_usermode_loop+0x1dd/0x1f0 [11440.387522] do_syscall_64+0x3d6/0x440 [11440.393783] entry_SYSCALL_64_after_hwframe+0x49/0xae [11440.400499] RIP: 0033:0x7f151bee3cd8 [11440.402880] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11440.429020] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11440.444735] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [11440.453974] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [11440.462081] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11440.474441] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11440.482639] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11456.055017] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [11456.056033] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [11456.059969] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11456.060060] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11456.060067] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11456.060070] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11456.060085] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11456.060089] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11456.060094] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [11456.060097] RDX: 0000000000000823 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [11456.060099] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [11456.060101] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [11456.060104] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11456.060107] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [11456.060110] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11456.060113] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11456.060121] Call Trace: [11456.060127] [11456.060133] ? show_regs.cold.9+0x22/0x2f [11456.060143] ? watchdog_timer_fn.cold.6+0x62/0x10a [11456.060150] ? __hrtimer_run_queues+0x1c2/0x4d0 [11456.060155] ? watchdog+0x40/0x40 [11456.060159] ? hrtimer_interrupt+0x128/0x2c0 [11456.060163] ? smp_apic_timer_interrupt+0xa6/0x270 [11456.060172] ? apic_timer_interrupt+0xf/0x20 [11456.060176] [11456.060179] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11456.060184] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11456.060188] do_raw_spin_lock+0x177/0x190 [11456.060193] _raw_spin_lock+0x21/0x30 [11456.060199] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11456.060490] obd_mod_free+0x46/0x140 [ptlrpc] [11456.060685] mdc_commit_open+0x91/0xb0 [mdc] [11456.060727] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11456.060928] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11456.061148] after_reply+0xe2c/0x1780 [ptlrpc] [11456.061362] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11456.067883] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11456.068089] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11456.068097] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11456.068104] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11456.068475] RIP: 0010:native_safe_halt+0x2e/0x50 [11456.068496] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11456.068507] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11456.068512] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11456.070596] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [11456.070599] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [11456.070602] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [11456.070604] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [11456.070608] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [11456.070611] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11456.070613] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11456.070621] Call Trace: [11456.070628] [11456.070770] ? show_regs.cold.9+0x22/0x2f [11456.070789] ? watchdog_timer_fn.cold.6+0x62/0x10a [11456.070796] ? __hrtimer_run_queues+0x1c2/0x4d0 [11456.070800] ? watchdog+0x40/0x40 [11456.070804] ? hrtimer_interrupt+0x128/0x2c0 [11456.070808] ? smp_apic_timer_interrupt+0xa6/0x270 [11456.070816] ? apic_timer_interrupt+0xf/0x20 [11456.070821] [11456.070823] ? native_safe_halt+0x2e/0x50 [11456.070829] kvm_wait+0xcd/0xe0 [11456.070836] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11456.070845] do_raw_spin_lock+0x177/0x190 [11456.070850] _raw_spin_lock+0x21/0x30 [11456.070853] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11456.071872] process_one_work+0x2c8/0x700 [11456.071879] worker_thread+0x296/0x6e0 [11456.071883] ? rescuer_thread+0x570/0x570 [11456.071886] kthread+0x1d1/0x200 [11456.071890] ? set_kthread_struct+0x70/0x70 [11456.071893] ret_from_fork+0x1f/0x30 [11456.603497] ? __next_timer_interrupt+0x160/0x160 [11456.606389] ptlrpcd+0xa82/0xe40 [ptlrpc] [11456.617585] ? do_wait_intr+0xf0/0xf0 [11456.622941] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11456.626410] kthread+0x1d1/0x200 [11456.629294] ? set_kthread_struct+0x70/0x70 [11456.634698] ret_from_fork+0x1f/0x30 [11460.052034] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [11460.059060] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11460.106234] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11460.121662] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11460.129201] RIP: 0010:native_safe_halt+0x2e/0x50 [11460.138739] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11460.158871] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11460.167964] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11460.172101] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [11460.176420] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [11460.182595] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [11460.188477] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [11460.192841] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [11460.201117] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11460.205370] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [11460.208557] Call Trace: [11460.210778] [11460.212847] ? show_regs.cold.9+0x22/0x2f [11460.215428] ? watchdog_timer_fn.cold.6+0x62/0x10a [11460.217869] ? __hrtimer_run_queues+0x1c2/0x4d0 [11460.221267] ? watchdog+0x40/0x40 [11460.222905] ? hrtimer_interrupt+0x128/0x2c0 [11460.227609] ? smp_apic_timer_interrupt+0xa6/0x270 [11460.238156] ? apic_timer_interrupt+0xf/0x20 [11460.240763] [11460.245784] ? native_safe_halt+0x2e/0x50 [11460.250333] kvm_wait+0xcd/0xe0 [11460.251462] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11460.270711] do_raw_spin_lock+0x177/0x190 [11460.273719] _raw_spin_lock+0x21/0x30 [11460.276256] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11460.283587] ? do_raw_spin_unlock+0x75/0x190 [11460.286458] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11460.292507] ? do_wait_intr+0xf0/0xf0 [11460.294568] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11460.296705] mdc_reint+0x2d/0x110 [mdc] [11460.308687] mdc_unlink+0x658/0x870 [mdc] [11460.311193] lmv_unlink+0x34d/0xbd0 [lmv] [11460.313686] ll_unlink+0x24d/0x7f0 [lustre] [11460.317791] vfs_unlink+0x202/0x330 [11460.320844] do_unlinkat+0x2fa/0x460 [11460.323548] __x64_sys_unlink+0x37/0x50 [11460.326267] do_syscall_64+0xc1/0x440 [11460.328440] entry_SYSCALL_64_after_hwframe+0x49/0xae [11460.332479] RIP: 0033:0x7f151bdee10b [11460.335791] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [11460.347129] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [11460.354386] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [11460.360994] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [11460.364868] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11460.371840] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11460.378241] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11463.144011] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9532s! [11463.156154] Showing busy workqueues and worker pools: [11463.161292] workqueue events: flags=0x0 [11463.167013] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11463.176345] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [11463.181805] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11463.188864] pending: kfree_rcu_monitor [11463.192225] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11463.195485] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [11463.204491] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11463.207448] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [11463.212548] workqueue events_long: flags=0x0 [11463.214416] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11463.218091] pending: lu_objects_flush [obdclass] [11463.222827] workqueue events_unbound: flags=0x2 [11463.231641] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11463.239264] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11463.247827] workqueue events_power_efficient: flags=0x80 [11463.255791] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11463.262815] pending: do_cache_clean, neigh_periodic_work, check_lifetime [11463.271135] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11463.276278] pending: neigh_periodic_work [11463.280327] workqueue mm_percpu_wq: flags=0x8 [11463.284504] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11463.289300] pending: vmstat_update, lru_add_drain_per_cpu [11463.291762] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11463.298330] pending: vmstat_update [11463.301690] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11463.308891] pending: lru_add_drain_per_cpu [11463.312610] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [11463.316748] pending: lru_add_drain_per_cpu BAR(48) [11463.319728] workqueue writeback: flags=0x4a [11463.323153] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [11463.332174] pending: wb_workfn [11463.337644] workqueue xprtiod: flags=0xa [11463.345613] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11463.355692] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [11463.364260] workqueue ipv6_addrconf: flags=0x40008 [11463.370027] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11463.375772] pending: addrconf_verify_work [11463.380838] workqueue knbd0-recv: flags=0x1a [11463.385503] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11463.390623] in-flight: 658:recv_work [11463.396642] workqueue ptlrpc_pinger: flags=0x2 [11463.407438] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11463.416369] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11463.426278] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9532s workers=4 idle: 32984 37 8 [11463.437911] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11448s workers=2 idle: 661 [11468.057065] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [11468.064857] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11468.128215] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11468.136084] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11468.141449] RIP: 0010:native_safe_halt+0x2e/0x50 [11468.144541] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11468.154619] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11468.158217] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11468.162015] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [11468.165459] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [11468.170105] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [11468.173516] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [11468.177501] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [11468.181143] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11468.185415] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [11468.192367] Call Trace: [11468.198249] [11468.203136] ? show_regs.cold.9+0x22/0x2f [11468.206310] ? watchdog_timer_fn.cold.6+0x62/0x10a [11468.207376] ? __hrtimer_run_queues+0x1c2/0x4d0 [11468.211237] ? watchdog+0x40/0x40 [11468.213951] ? hrtimer_interrupt+0x128/0x2c0 [11468.216225] ? smp_apic_timer_interrupt+0xa6/0x270 [11468.219397] ? apic_timer_interrupt+0xf/0x20 [11468.222210] [11468.223230] ? native_safe_halt+0x2e/0x50 [11468.225403] kvm_wait+0xcd/0xe0 [11468.227459] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11468.230203] do_raw_spin_lock+0x177/0x190 [11468.232237] _raw_spin_lock+0x21/0x30 [11468.234238] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11468.237786] ? do_raw_spin_unlock+0x75/0x190 [11468.240312] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11468.243902] ? do_wait_intr+0xf0/0xf0 [11468.246189] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11468.253366] mdc_close+0x53b/0xd70 [mdc] [11468.262996] lmv_close+0x18d/0x490 [lmv] [11468.266003] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11468.274672] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11468.283234] ll_md_real_close+0xa5/0x2a0 [lustre] [11468.285510] ll_md_close+0x274/0x800 [lustre] [11468.287418] ll_file_release+0x128/0xc50 [lustre] [11468.290499] __fput+0x10d/0x440 [11468.291448] ____fput+0x12/0x20 [11468.295642] task_work_run+0xc6/0x110 [11468.297705] exit_to_usermode_loop+0x1dd/0x1f0 [11468.299374] do_syscall_64+0x3d6/0x440 [11468.303766] entry_SYSCALL_64_after_hwframe+0x49/0xae [11468.310267] RIP: 0033:0x7f151bee3cd8 [11468.312560] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11468.339245] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11468.345050] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [11468.350956] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [11468.355399] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11468.359207] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11468.364111] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11484.056118] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11484.058377] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [11484.062161] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11484.062261] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11484.062268] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11484.062276] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11484.062698] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11484.062709] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11484.062713] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [11484.062717] RAX: 000000000000278a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11484.062720] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [11484.062721] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [11484.062723] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [11484.062725] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [11484.062728] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [11484.062731] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11484.062734] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11484.062741] Call Trace: [11484.062747] [11484.062753] ? show_regs.cold.9+0x22/0x2f [11484.062763] ? watchdog_timer_fn.cold.6+0x62/0x10a [11484.062770] ? __hrtimer_run_queues+0x1c2/0x4d0 [11484.062774] ? watchdog+0x40/0x40 [11484.062789] ? hrtimer_interrupt+0x128/0x2c0 [11484.062793] ? smp_apic_timer_interrupt+0xa6/0x270 [11484.062800] ? apic_timer_interrupt+0xf/0x20 [11484.062805] [11484.062808] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11484.062813] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11484.062817] do_raw_spin_lock+0x177/0x190 [11484.062822] _raw_spin_lock+0x21/0x30 [11484.062827] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11484.063282] process_one_work+0x2c8/0x700 [11484.063298] worker_thread+0x296/0x6e0 [11484.063303] ? rescuer_thread+0x570/0x570 [11484.063307] kthread+0x1d1/0x200 [11484.063317] ? set_kthread_struct+0x70/0x70 [11484.063325] ret_from_fork+0x1f/0x30 [11484.287953] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11484.308306] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11484.323142] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11484.333107] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11484.335201] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11484.362895] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11484.368445] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [11484.377387] RDX: 0000000000007214 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [11484.381390] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [11484.389334] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [11484.395061] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11484.401198] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [11484.405072] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11484.407500] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11484.418393] Call Trace: [11484.419944] [11484.421286] ? show_regs.cold.9+0x22/0x2f [11484.425343] ? watchdog_timer_fn.cold.6+0x62/0x10a [11484.427481] ? __hrtimer_run_queues+0x1c2/0x4d0 [11484.431874] ? watchdog+0x40/0x40 [11484.433054] ? hrtimer_interrupt+0x128/0x2c0 [11484.436294] ? smp_apic_timer_interrupt+0xa6/0x270 [11484.441542] ? apic_timer_interrupt+0xf/0x20 [11484.443910] [11484.445517] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11484.454989] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11484.456670] do_raw_spin_lock+0x177/0x190 [11484.462318] _raw_spin_lock+0x21/0x30 [11484.465428] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11484.472504] obd_mod_free+0x46/0x140 [ptlrpc] [11484.476855] mdc_commit_open+0x91/0xb0 [mdc] [11484.483321] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11484.486282] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11484.493965] after_reply+0xe2c/0x1780 [ptlrpc] [11484.500277] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11484.505455] ? __next_timer_interrupt+0x160/0x160 [11484.511200] ptlrpcd+0xa82/0xe40 [ptlrpc] [11484.513861] ? do_wait_intr+0xf0/0xf0 [11484.515486] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11484.517391] kthread+0x1d1/0x200 [11484.519997] ? set_kthread_struct+0x70/0x70 [11484.522927] ret_from_fork+0x1f/0x30 [11488.052026] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [11488.055068] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11488.093263] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11488.106006] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11488.112326] RIP: 0010:native_safe_halt+0x2e/0x50 [11488.115484] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11488.130356] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [11488.141042] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11488.149506] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [11488.157448] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [11488.161533] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [11488.171202] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [11488.184260] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [11488.193706] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11488.195693] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [11488.200223] Call Trace: [11488.201562] [11488.202801] ? show_regs.cold.9+0x22/0x2f [11488.215753] ? watchdog_timer_fn.cold.6+0x62/0x10a [11488.220176] ? __hrtimer_run_queues+0x1c2/0x4d0 [11488.226310] ? watchdog+0x40/0x40 [11488.231243] ? hrtimer_interrupt+0x128/0x2c0 [11488.237323] ? smp_apic_timer_interrupt+0xa6/0x270 [11488.246606] ? apic_timer_interrupt+0xf/0x20 [11488.251192] [11488.256399] ? native_safe_halt+0x2e/0x50 [11488.260275] kvm_wait+0xcd/0xe0 [11488.264204] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11488.268514] do_raw_spin_lock+0x177/0x190 [11488.272494] _raw_spin_lock+0x21/0x30 [11488.280365] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11488.289404] ? do_raw_spin_unlock+0x75/0x190 [11488.293728] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11488.297067] ? do_wait_intr+0xf0/0xf0 [11488.299324] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11488.302290] mdc_reint+0x2d/0x110 [mdc] [11488.309462] mdc_unlink+0x658/0x870 [mdc] [11488.315967] lmv_unlink+0x34d/0xbd0 [lmv] [11488.317379] ll_unlink+0x24d/0x7f0 [lustre] [11488.321032] vfs_unlink+0x202/0x330 [11488.323362] do_unlinkat+0x2fa/0x460 [11488.327899] __x64_sys_unlink+0x37/0x50 [11488.332901] do_syscall_64+0xc1/0x440 [11488.335429] entry_SYSCALL_64_after_hwframe+0x49/0xae [11488.340498] RIP: 0033:0x7f151bdee10b [11488.341725] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [11488.361770] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [11488.366648] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [11488.373005] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [11488.377912] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11488.384596] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11488.389901] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11493.858317] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9563s! [11493.874562] Showing busy workqueues and worker pools: [11493.886049] workqueue events: flags=0x0 [11493.888323] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11493.891108] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [11493.896488] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11493.907130] pending: kfree_rcu_monitor [11493.913750] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11493.920464] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [11493.932053] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11493.936148] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [11493.940298] workqueue events_long: flags=0x0 [11493.943396] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11493.954173] pending: lu_objects_flush [obdclass] [11493.961867] workqueue events_unbound: flags=0x2 [11493.969079] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11493.974472] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11493.983103] workqueue events_power_efficient: flags=0x80 [11493.985471] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11493.988733] pending: do_cache_clean, neigh_periodic_work, check_lifetime [11493.993327] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11493.997414] pending: neigh_periodic_work [11494.000059] workqueue mm_percpu_wq: flags=0x8 [11494.003022] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11494.006934] pending: vmstat_update, lru_add_drain_per_cpu [11494.010014] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11494.013369] pending: vmstat_update [11494.015173] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11494.018482] pending: lru_add_drain_per_cpu [11494.024051] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [11494.027416] pending: lru_add_drain_per_cpu BAR(48) [11494.032201] workqueue writeback: flags=0x4a [11494.034014] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [11494.043984] pending: wb_workfn [11494.048222] workqueue xprtiod: flags=0xa [11494.056975] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11494.066599] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [11494.079828] workqueue ipv6_addrconf: flags=0x40008 [11494.088449] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11494.094454] pending: addrconf_verify_work [11494.106359] workqueue knbd0-recv: flags=0x1a [11494.111412] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11494.121650] in-flight: 658:recv_work [11494.124725] workqueue ptlrpc_pinger: flags=0x2 [11494.127761] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11494.136863] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11494.145785] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9563s workers=4 idle: 32984 37 8 [11494.151792] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11478s workers=2 idle: 661 [11496.060783] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [11496.074308] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11496.110303] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11496.116744] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11496.129707] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11496.133404] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11496.157510] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11496.164242] RAX: 0000000000005605 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11496.171339] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [11496.181432] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [11496.191649] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [11496.202499] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [11496.220448] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [11496.237970] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11496.248393] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [11496.267696] Call Trace: [11496.275515] [11496.280911] ? show_regs.cold.9+0x22/0x2f [11496.290375] ? watchdog_timer_fn.cold.6+0x62/0x10a [11496.301727] ? __hrtimer_run_queues+0x1c2/0x4d0 [11496.310769] ? watchdog+0x40/0x40 [11496.320972] ? hrtimer_interrupt+0x128/0x2c0 [11496.325237] ? smp_apic_timer_interrupt+0xa6/0x270 [11496.335405] ? apic_timer_interrupt+0xf/0x20 [11496.349259] [11496.350600] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11496.365584] do_raw_spin_lock+0x177/0x190 [11496.376836] _raw_spin_lock+0x21/0x30 [11496.382846] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11496.392199] ? do_raw_spin_unlock+0x75/0x190 [11496.397555] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11496.403149] ? do_wait_intr+0xf0/0xf0 [11496.409098] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11496.413158] mdc_close+0x53b/0xd70 [mdc] [11496.415165] lmv_close+0x18d/0x490 [lmv] [11496.421578] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11496.429923] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11496.438400] ll_md_real_close+0xa5/0x2a0 [lustre] [11496.443655] ll_md_close+0x274/0x800 [lustre] [11496.446470] ll_file_release+0x128/0xc50 [lustre] [11496.450921] __fput+0x10d/0x440 [11496.453249] ____fput+0x12/0x20 [11496.456194] task_work_run+0xc6/0x110 [11496.458450] exit_to_usermode_loop+0x1dd/0x1f0 [11496.461353] do_syscall_64+0x3d6/0x440 [11496.463790] entry_SYSCALL_64_after_hwframe+0x49/0xae [11496.470895] RIP: 0033:0x7f151bee3cd8 [11496.475378] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11496.489982] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11496.502861] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [11496.513465] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [11496.521923] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11496.534970] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11496.543293] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11512.058914] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11512.064876] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11512.065225] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [11512.104118] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11512.108112] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11512.108189] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11512.108197] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11512.108200] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11512.108213] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11512.108218] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11512.108222] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [11512.108224] RDX: 0000000000007277 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [11512.108227] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [11512.108229] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [11512.108231] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11512.108235] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [11512.108237] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11512.108239] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11512.108247] Call Trace: [11512.108252] [11512.108257] ? show_regs.cold.9+0x22/0x2f [11512.108267] ? watchdog_timer_fn.cold.6+0x62/0x10a [11512.108273] ? __hrtimer_run_queues+0x1c2/0x4d0 [11512.108277] ? watchdog+0x40/0x40 [11512.108282] ? hrtimer_interrupt+0x128/0x2c0 [11512.108285] ? smp_apic_timer_interrupt+0xa6/0x270 [11512.108294] ? apic_timer_interrupt+0xf/0x20 [11512.108300] [11512.108303] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11512.108307] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11512.108312] do_raw_spin_lock+0x177/0x190 [11512.108316] _raw_spin_lock+0x21/0x30 [11512.108321] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11512.108832] obd_mod_free+0x46/0x140 [ptlrpc] [11512.109075] mdc_commit_open+0x91/0xb0 [mdc] [11512.109115] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11512.109318] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11512.127594] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11512.127609] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11512.127995] RIP: 0010:native_safe_halt+0x2e/0x50 [11512.128012] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11512.128017] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11512.128021] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11512.128024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [11512.128027] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [11512.128032] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [11512.128035] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [11512.128039] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [11512.128042] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11512.128045] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11512.128052] Call Trace: [11512.128156] [11512.128162] ? show_regs.cold.9+0x22/0x2f [11512.128180] ? watchdog_timer_fn.cold.6+0x62/0x10a [11512.128186] ? __hrtimer_run_queues+0x1c2/0x4d0 [11512.128195] ? watchdog+0x40/0x40 [11512.128200] ? hrtimer_interrupt+0x128/0x2c0 [11512.128204] ? smp_apic_timer_interrupt+0xa6/0x270 [11512.128213] ? apic_timer_interrupt+0xf/0x20 [11512.128218] [11512.128221] ? native_safe_halt+0x2e/0x50 [11512.128225] kvm_wait+0xcd/0xe0 [11512.128234] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11512.128244] do_raw_spin_lock+0x177/0x190 [11512.128249] _raw_spin_lock+0x21/0x30 [11512.128253] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11512.128803] process_one_work+0x2c8/0x700 [11512.128810] worker_thread+0x296/0x6e0 [11512.128815] ? rescuer_thread+0x570/0x570 [11512.128819] kthread+0x1d1/0x200 [11512.128828] ? set_kthread_struct+0x70/0x70 [11512.128835] ret_from_fork+0x1f/0x30 [11512.459678] after_reply+0xe2c/0x1780 [ptlrpc] [11512.462864] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11512.470205] ? __next_timer_interrupt+0x160/0x160 [11512.474194] ptlrpcd+0xa82/0xe40 [ptlrpc] [11512.477124] ? do_wait_intr+0xf0/0xf0 [11512.479157] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11512.482203] kthread+0x1d1/0x200 [11512.484030] ? set_kthread_struct+0x70/0x70 [11512.485670] ret_from_fork+0x1f/0x30 [11520.054569] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [11520.064254] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11520.121125] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11520.133504] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11520.144257] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11520.150051] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11520.165319] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [11520.173830] RAX: 00000000000047a3 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11520.183813] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [11520.191359] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [11520.205177] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [11520.210359] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [11520.220339] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [11520.233736] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11520.245714] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [11520.251768] Call Trace: [11520.253663] [11520.254470] ? show_regs.cold.9+0x22/0x2f [11520.256795] ? watchdog_timer_fn.cold.6+0x62/0x10a [11520.260764] ? __hrtimer_run_queues+0x1c2/0x4d0 [11520.264734] ? watchdog+0x40/0x40 [11520.265867] ? hrtimer_interrupt+0x128/0x2c0 [11520.276263] ? smp_apic_timer_interrupt+0xa6/0x270 [11520.283543] ? apic_timer_interrupt+0xf/0x20 [11520.285529] [11520.289060] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11520.297582] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11520.307033] do_raw_spin_lock+0x177/0x190 [11520.313727] _raw_spin_lock+0x21/0x30 [11520.317996] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11520.326173] ? do_raw_spin_unlock+0x75/0x190 [11520.336773] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11520.347374] ? do_wait_intr+0xf0/0xf0 [11520.353145] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11520.364990] mdc_reint+0x2d/0x110 [mdc] [11520.374025] mdc_unlink+0x658/0x870 [mdc] [11520.377315] lmv_unlink+0x34d/0xbd0 [lmv] [11520.379084] ll_unlink+0x24d/0x7f0 [lustre] [11520.383227] vfs_unlink+0x202/0x330 [11520.386402] do_unlinkat+0x2fa/0x460 [11520.389654] __x64_sys_unlink+0x37/0x50 [11520.391201] do_syscall_64+0xc1/0x440 [11520.392742] entry_SYSCALL_64_after_hwframe+0x49/0xae [11520.397589] RIP: 0033:0x7f151bdee10b [11520.402029] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [11520.412122] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [11520.416465] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [11520.419141] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [11520.421929] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11520.424634] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11520.427267] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11524.057022] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [11524.061408] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11524.117248] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11524.132825] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11524.141706] RIP: 0010:native_safe_halt+0x2e/0x50 [11524.145114] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11524.165797] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11524.173218] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11524.185880] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [11524.196536] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [11524.211522] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [11524.229603] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [11524.232166] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [11524.246363] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11524.250971] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [11524.257977] Call Trace: [11524.261058] [11524.261965] ? show_regs.cold.9+0x22/0x2f [11524.268581] ? watchdog_timer_fn.cold.6+0x62/0x10a [11524.274155] ? __hrtimer_run_queues+0x1c2/0x4d0 [11524.279059] ? watchdog+0x40/0x40 [11524.281711] ? hrtimer_interrupt+0x128/0x2c0 [11524.286814] ? smp_apic_timer_interrupt+0xa6/0x270 [11524.300885] ? apic_timer_interrupt+0xf/0x20 [11524.304500] [11524.305197] ? native_safe_halt+0x2e/0x50 [11524.312864] kvm_wait+0xcd/0xe0 [11524.316348] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11524.332014] do_raw_spin_lock+0x177/0x190 [11524.334993] _raw_spin_lock+0x21/0x30 [11524.342991] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11524.352534] ? do_raw_spin_unlock+0x75/0x190 [11524.356633] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11524.369928] ? do_wait_intr+0xf0/0xf0 [11524.378891] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11524.383624] mdc_close+0x53b/0xd70 [mdc] [11524.385690] lmv_close+0x18d/0x490 [lmv] [11524.387882] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11524.401433] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11524.408508] ll_md_real_close+0xa5/0x2a0 [lustre] [11524.414773] ll_md_close+0x274/0x800 [lustre] [11524.424979] ll_file_release+0x128/0xc50 [lustre] [11524.432832] __fput+0x10d/0x440 [11524.436713] ____fput+0x12/0x20 [11524.441643] task_work_run+0xc6/0x110 [11524.453419] exit_to_usermode_loop+0x1dd/0x1f0 [11524.460739] do_syscall_64+0x3d6/0x440 [11524.467859] entry_SYSCALL_64_after_hwframe+0x49/0xae [11524.469412] RIP: 0033:0x7f151bee3cd8 [11524.470358] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11524.490051] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11524.496782] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [11524.500897] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [11524.508783] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11524.517130] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11524.519520] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11524.577209] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9593s! [11524.581773] Showing busy workqueues and worker pools: [11524.583721] workqueue events: flags=0x0 [11524.585399] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11524.588649] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [11524.594653] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11524.602656] pending: kfree_rcu_monitor [11524.606586] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11524.615225] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [11524.626083] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11524.635502] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [11524.650766] workqueue events_long: flags=0x0 [11524.654495] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11524.662837] pending: lu_objects_flush [obdclass] [11524.670757] workqueue events_unbound: flags=0x2 [11524.677661] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11524.690514] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11524.697494] workqueue events_power_efficient: flags=0x80 [11524.704495] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11524.712981] pending: do_cache_clean, neigh_periodic_work, check_lifetime [11524.721528] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11524.729669] pending: neigh_periodic_work [11524.733686] workqueue mm_percpu_wq: flags=0x8 [11524.735757] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11524.746939] pending: vmstat_update, lru_add_drain_per_cpu [11524.752119] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11524.755521] pending: vmstat_update [11524.757582] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11524.764968] pending: lru_add_drain_per_cpu [11524.769450] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [11524.774947] pending: lru_add_drain_per_cpu BAR(48) [11524.784857] workqueue writeback: flags=0x4a [11524.788979] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [11524.796193] pending: wb_workfn [11524.798041] workqueue xprtiod: flags=0xa [11524.799606] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11524.803868] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [11524.822547] workqueue ipv6_addrconf: flags=0x40008 [11524.826946] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11524.833548] pending: addrconf_verify_work [11524.838654] workqueue knbd0-recv: flags=0x1a [11524.843596] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11524.848675] in-flight: 658:recv_work [11524.850860] workqueue ptlrpc_pinger: flags=0x2 [11524.855463] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11524.856397] rcu: INFO: rcu_sched self-detected stall on CPU [11524.860133] in-flight: [11524.863950] rcu: 1-...!: (5532755 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [11524.860140] 38:ptlrpc_pinger_main [ptlrpc] [11524.865184] (t=9600209 jiffies g=249657 q=5191) [11524.865189] rcu: rcu_sched kthread starved for 9595161 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [11524.865194] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [11524.865195] rcu: RCU grace-period kthread stack dump: [11524.865197] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [11524.865205] Call Trace: [11524.865212] __schedule+0x351/0xcb0 [11524.865225] schedule+0xc0/0x180 [11524.865229] schedule_timeout+0xb4/0x190 [11524.865234] ? do_raw_spin_unlock+0x75/0x190 [11524.865241] ? __next_timer_interrupt+0x160/0x160 [11524.865249] ? prepare_to_swait_event+0xf3/0x140 [11524.865257] rcu_gp_kthread+0x8dd/0x15b0 [11524.865263] ? force_qs_rnp+0x2c0/0x2c0 [11524.865267] kthread+0x1d1/0x200 [11524.865272] ? set_kthread_struct+0x70/0x70 [11524.865275] ret_from_fork+0x1f/0x30 [11524.865283] rcu: Stack dump where RCU GP kthread last ran: [11524.865286] Sending NMI from CPU 1 to CPUs 2: [11524.865338] NMI backtrace for cpu 2 skipped: idling at native_safe_halt+0x2e/0x50 [11524.866305] NMI backtrace for cpu 1 [11524.875837] [11524.878282] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11524.878294] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11524.878296] Call Trace: [11524.878302] [11524.878307] ? dump_stack+0xbb/0x10e [11524.878317] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [11524.878323] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [11524.878328] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [11524.878332] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [11524.878337] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [11524.878340] ? rcu_dump_cpu_stacks+0x11b/0x171 [11524.878347] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [11524.878352] ? update_process_times+0x6d/0xc0 [11524.878361] ? tick_sched_handle.isra.18+0x27/0x90 [11524.878367] ? tick_sched_timer+0x53/0xb0 [11524.878371] ? __hrtimer_run_queues+0x1c2/0x4d0 [11524.878375] ? tick_sched_handle.isra.18+0x90/0x90 [11524.878391] ? hrtimer_interrupt+0x128/0x2c0 [11524.878397] ? smp_apic_timer_interrupt+0xa6/0x270 [11524.878407] ? apic_timer_interrupt+0xf/0x20 [11524.878412] [11524.878416] ? native_safe_halt+0x2e/0x50 [11524.878420] ? kvm_wait+0xcd/0xe0 [11524.878427] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11524.878434] ? do_raw_spin_lock+0x177/0x190 [11524.878439] ? _raw_spin_lock+0x21/0x30 [11524.878445] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11524.882833] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9594s workers=4 idle: 32984 37 8 [11524.882860] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11509s workers=2 idle: 661 [11525.125569] ? obd_mod_free+0x46/0x140 [ptlrpc] [11525.129126] ? mdc_commit_open+0x91/0xb0 [mdc] [11525.133082] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11525.135932] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11525.141884] ? after_reply+0xe2c/0x1780 [ptlrpc] [11525.146986] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11525.151609] ? __next_timer_interrupt+0x160/0x160 [11525.154680] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [11525.157586] ? do_wait_intr+0xf0/0xf0 [11525.160460] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11525.164082] ? kthread+0x1d1/0x200 [11525.165347] ? set_kthread_struct+0x70/0x70 [11525.166937] ? ret_from_fork+0x1f/0x30 [11540.057726] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11540.063413] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11540.111420] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11540.118868] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11540.125887] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11540.134746] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11540.144942] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11540.173273] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11540.184403] RAX: 0000000000007bc1 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11540.193321] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [11540.197467] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [11540.201347] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [11540.205792] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [11540.209869] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [11540.214680] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11540.217808] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11540.221708] Call Trace: [11540.223178] [11540.224030] ? show_regs.cold.9+0x22/0x2f [11540.225515] ? watchdog_timer_fn.cold.6+0x62/0x10a [11540.228717] ? __hrtimer_run_queues+0x1c2/0x4d0 [11540.231319] ? watchdog+0x40/0x40 [11540.233273] ? hrtimer_interrupt+0x128/0x2c0 [11540.235445] ? smp_apic_timer_interrupt+0xa6/0x270 [11540.237845] ? apic_timer_interrupt+0xf/0x20 [11540.240365] [11540.241635] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11540.244699] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11540.247810] do_raw_spin_lock+0x177/0x190 [11540.250512] _raw_spin_lock+0x21/0x30 [11540.252746] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11540.255718] process_one_work+0x2c8/0x700 [11540.257685] worker_thread+0x296/0x6e0 [11540.259460] ? rescuer_thread+0x570/0x570 [11540.261291] kthread+0x1d1/0x200 [11540.262938] ? set_kthread_struct+0x70/0x70 [11540.264810] ret_from_fork+0x1f/0x30 [11552.053166] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [11552.055075] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [11552.058493] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) [11552.060531] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [11552.060541] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11552.060623] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11552.060629] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11552.060635] Modules linked in: [11552.060631] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11552.060639] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11552.060719] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11552.060725] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11552.060727] RIP: 0010:native_safe_halt+0x2e/0x50 [11552.060738] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11552.060743] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [11552.060748] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [11552.060750] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [11552.060753] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [11552.060756] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [11552.060758] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11552.060762] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [11552.060765] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11552.060768] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11552.060777] Call Trace: [11552.060783] [11552.060837] ? show_regs.cold.9+0x22/0x2f [11552.060850] ? watchdog_timer_fn.cold.6+0x62/0x10a [11552.060857] ? __hrtimer_run_queues+0x1c2/0x4d0 [11552.060862] ? watchdog+0x40/0x40 [11552.060866] ? hrtimer_interrupt+0x128/0x2c0 [11552.060871] ? smp_apic_timer_interrupt+0xa6/0x270 [11552.060878] ? apic_timer_interrupt+0xf/0x20 [11552.060883] [11552.060886] ? native_safe_halt+0x2e/0x50 [11552.060891] kvm_wait+0xcd/0xe0 [11552.060899] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11552.061089] do_raw_spin_lock+0x177/0x190 [11552.061096] _raw_spin_lock+0x21/0x30 [11552.061103] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11552.062370] obd_mod_free+0x46/0x140 [ptlrpc] [11552.062574] mdc_commit_open+0x91/0xb0 [mdc] [11552.062615] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11552.063309] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11552.063779] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11552.063842] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11552.063849] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11552.063852] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11552.063863] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11552.063867] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [11552.063871] RAX: 000000000000730b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11552.063873] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [11552.063876] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [11552.063877] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [11552.063880] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [11552.063883] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [11552.063885] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11552.063887] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [11552.063896] Call Trace: [11552.063899] [11552.063902] ? show_regs.cold.9+0x22/0x2f [11552.063911] ? watchdog_timer_fn.cold.6+0x62/0x10a [11552.063918] ? __hrtimer_run_queues+0x1c2/0x4d0 [11552.063922] ? watchdog+0x40/0x40 [11552.063926] ? hrtimer_interrupt+0x128/0x2c0 [11552.063929] ? smp_apic_timer_interrupt+0xa6/0x270 [11552.063936] ? apic_timer_interrupt+0xf/0x20 [11552.063941] [11552.063944] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11552.063949] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11552.063954] do_raw_spin_lock+0x177/0x190 [11552.063959] _raw_spin_lock+0x21/0x30 [11552.063965] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11552.064185] ? do_raw_spin_unlock+0x75/0x190 [11552.064191] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11552.063511] after_reply+0xe2c/0x1780 [ptlrpc] [11552.064527] ? do_wait_intr+0xf0/0xf0 [11552.064535] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11552.064781] mdc_reint+0x2d/0x110 [mdc] [11552.064820] mdc_unlink+0x658/0x870 [mdc] [11552.064848] lmv_unlink+0x34d/0xbd0 [lmv] [11552.064878] ll_unlink+0x24d/0x7f0 [lustre] [11552.064998] vfs_unlink+0x202/0x330 [11552.065007] do_unlinkat+0x2fa/0x460 [11552.065012] __x64_sys_unlink+0x37/0x50 [11552.065015] do_syscall_64+0xc1/0x440 [11552.065022] entry_SYSCALL_64_after_hwframe+0x49/0xae [11552.065028] RIP: 0033:0x7f151bdee10b [11552.065034] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [11552.065037] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [11552.065042] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [11552.065044] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [11552.065047] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11552.065049] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11552.065051] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11552.065320] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11552.065523] ? __next_timer_interrupt+0x160/0x160 [11552.065532] ptlrpcd+0xa82/0xe40 [ptlrpc] [11552.065733] ? do_wait_intr+0xf0/0xf0 [11552.065740] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11552.066874] kthread+0x1d1/0x200 [11552.066881] ? set_kthread_struct+0x70/0x70 [11552.066885] ret_from_fork+0x1f/0x30 [11552.649144] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11552.667225] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [11552.676324] RAX: 0000000000007d09 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11552.682826] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [11552.690869] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [11552.693448] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [11552.701568] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [11552.705435] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [11552.716505] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11552.722094] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [11552.725248] Call Trace: [11552.726308] [11552.727991] ? show_regs.cold.9+0x22/0x2f [11552.730196] ? watchdog_timer_fn.cold.6+0x62/0x10a [11552.732563] ? __hrtimer_run_queues+0x1c2/0x4d0 [11552.735854] ? watchdog+0x40/0x40 [11552.739211] ? hrtimer_interrupt+0x128/0x2c0 [11552.743419] ? smp_apic_timer_interrupt+0xa6/0x270 [11552.747851] ? apic_timer_interrupt+0xf/0x20 [11552.751393] [11552.752377] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11552.756899] do_raw_spin_lock+0x177/0x190 [11552.762258] _raw_spin_lock+0x21/0x30 [11552.764655] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11552.768561] ? do_raw_spin_unlock+0x75/0x190 [11552.774884] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11552.781620] ? do_wait_intr+0xf0/0xf0 [11552.786439] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11552.792302] mdc_close+0x53b/0xd70 [mdc] [11552.795513] lmv_close+0x18d/0x490 [lmv] [11552.798690] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11552.801623] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11552.805351] ll_md_real_close+0xa5/0x2a0 [lustre] [11552.812329] ll_md_close+0x274/0x800 [lustre] [11552.813937] ll_file_release+0x128/0xc50 [lustre] [11552.821502] __fput+0x10d/0x440 [11552.829553] ____fput+0x12/0x20 [11552.831534] task_work_run+0xc6/0x110 [11552.835936] exit_to_usermode_loop+0x1dd/0x1f0 [11552.846000] do_syscall_64+0x3d6/0x440 [11552.850348] entry_SYSCALL_64_after_hwframe+0x49/0xae [11552.853212] RIP: 0033:0x7f151bee3cd8 [11552.855919] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11552.872262] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11552.876143] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [11552.881423] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [11552.886807] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11552.889330] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11552.891925] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11555.307132] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9624s! [11555.312615] Showing busy workqueues and worker pools: [11555.317464] workqueue events: flags=0x0 [11555.322080] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11555.324850] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [11555.331252] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11555.335942] pending: kfree_rcu_monitor [11555.339593] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11555.343336] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [11555.347135] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11555.351286] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [11555.360587] workqueue events_long: flags=0x0 [11555.362604] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11555.365920] pending: lu_objects_flush [obdclass] [11555.369423] workqueue events_unbound: flags=0x2 [11555.371354] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11555.376548] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11555.386671] workqueue events_power_efficient: flags=0x80 [11555.393097] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11555.398654] pending: do_cache_clean, neigh_periodic_work, check_lifetime [11555.403900] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11555.411476] pending: neigh_periodic_work [11555.416217] workqueue mm_percpu_wq: flags=0x8 [11555.418789] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11555.421920] pending: vmstat_update, lru_add_drain_per_cpu [11555.426408] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11555.429231] pending: vmstat_update [11555.430830] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11555.434831] pending: lru_add_drain_per_cpu [11555.437407] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [11555.444915] pending: lru_add_drain_per_cpu BAR(48) [11555.448562] workqueue writeback: flags=0x4a [11555.452043] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [11555.456197] pending: wb_workfn [11555.457887] workqueue xprtiod: flags=0xa [11555.460857] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11555.463745] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [11555.473447] workqueue ipv6_addrconf: flags=0x40008 [11555.476771] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11555.480642] pending: addrconf_verify_work [11555.483865] workqueue knbd0-recv: flags=0x1a [11555.489433] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11555.498036] in-flight: 658:recv_work [11555.499946] workqueue ptlrpc_pinger: flags=0x2 [11555.505356] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11555.507990] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11555.516685] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9624s workers=4 idle: 32984 37 8 [11555.525656] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11540s workers=2 idle: 661 [11568.061044] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11568.078635] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11568.141405] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11568.162431] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11568.186125] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11568.200201] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11568.216852] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11568.241651] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11568.250635] RAX: 0000000000006886 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11568.261360] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [11568.279780] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [11568.300553] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [11568.310236] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [11568.318806] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [11568.329573] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11568.333907] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11568.342806] Call Trace: [11568.352421] [11568.354714] ? show_regs.cold.9+0x22/0x2f [11568.360347] ? watchdog_timer_fn.cold.6+0x62/0x10a [11568.364245] ? __hrtimer_run_queues+0x1c2/0x4d0 [11568.368874] ? watchdog+0x40/0x40 [11568.371713] ? hrtimer_interrupt+0x128/0x2c0 [11568.375470] ? smp_apic_timer_interrupt+0xa6/0x270 [11568.380482] ? apic_timer_interrupt+0xf/0x20 [11568.382199] [11568.382957] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11568.388796] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11568.394946] do_raw_spin_lock+0x177/0x190 [11568.396629] _raw_spin_lock+0x21/0x30 [11568.400641] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11568.405767] process_one_work+0x2c8/0x700 [11568.407859] worker_thread+0x296/0x6e0 [11568.409229] ? rescuer_thread+0x570/0x570 [11568.420303] kthread+0x1d1/0x200 [11568.421718] ? set_kthread_struct+0x70/0x70 [11568.425021] ret_from_fork+0x1f/0x30 [11580.053712] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [11580.056518] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [11580.057294] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [11580.057301] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11580.057387] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11580.057392] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11580.057395] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11580.057409] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11580.057414] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11580.057418] RAX: 0000000000006a16 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11580.057420] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [11580.057422] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [11580.057424] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [11580.057426] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [11580.057429] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [11580.057433] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11580.057435] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [11580.057442] Call Trace: [11580.057447] [11580.057453] ? show_regs.cold.9+0x22/0x2f [11580.057462] ? watchdog_timer_fn.cold.6+0x62/0x10a [11580.057468] ? __hrtimer_run_queues+0x1c2/0x4d0 [11580.057473] ? watchdog+0x40/0x40 [11580.057477] ? hrtimer_interrupt+0x128/0x2c0 [11580.057481] ? smp_apic_timer_interrupt+0xa6/0x270 [11580.057488] ? apic_timer_interrupt+0xf/0x20 [11580.057493] [11580.057496] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11580.057503] do_raw_spin_lock+0x177/0x190 [11580.057508] _raw_spin_lock+0x21/0x30 [11580.057512] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11580.058510] ? do_raw_spin_unlock+0x75/0x190 [11580.058518] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11580.058756] ? do_wait_intr+0xf0/0xf0 [11580.058764] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11580.058980] mdc_close+0x53b/0xd70 [mdc] [11580.059015] lmv_close+0x18d/0x490 [lmv] [11580.059043] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11580.059208] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11580.059322] ll_md_real_close+0xa5/0x2a0 [lustre] [11580.059398] ll_md_close+0x274/0x800 [lustre] [11580.059495] ll_file_release+0x128/0xc50 [lustre] [11580.059576] __fput+0x10d/0x440 [11580.059585] ____fput+0x12/0x20 [11580.059589] task_work_run+0xc6/0x110 [11580.059597] exit_to_usermode_loop+0x1dd/0x1f0 [11580.059606] do_syscall_64+0x3d6/0x440 [11580.059611] entry_SYSCALL_64_after_hwframe+0x49/0xae [11580.059620] RIP: 0033:0x7f151bee3cd8 [11580.059628] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11580.059632] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11580.059636] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [11580.059638] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [11580.059640] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11580.059643] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11580.059645] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11580.070685] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11580.070775] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11580.070784] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11580.070788] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11580.070813] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11580.070817] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11580.070821] RAX: 0000000000001683 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11580.070823] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [11580.070825] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [11580.070827] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [11580.070829] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [11580.070832] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [11580.070834] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11580.070836] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [11580.070844] Call Trace: [11580.070859] [11580.070866] ? show_regs.cold.9+0x22/0x2f [11580.070877] ? watchdog_timer_fn.cold.6+0x62/0x10a [11580.070884] ? __hrtimer_run_queues+0x1c2/0x4d0 [11580.070889] ? watchdog+0x40/0x40 [11580.070894] ? hrtimer_interrupt+0x128/0x2c0 [11580.070898] ? smp_apic_timer_interrupt+0xa6/0x270 [11580.071476] ? apic_timer_interrupt+0xf/0x20 [11580.071482] [11580.071486] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11580.071491] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11580.071495] do_raw_spin_lock+0x177/0x190 [11580.071500] _raw_spin_lock+0x21/0x30 [11580.071506] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11580.071787] ? do_raw_spin_unlock+0x75/0x190 [11580.072337] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11580.072973] ? do_wait_intr+0xf0/0xf0 [11580.072984] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11580.073191] mdc_reint+0x2d/0x110 [mdc] [11580.073374] mdc_unlink+0x658/0x870 [mdc] [11580.073402] lmv_unlink+0x34d/0xbd0 [lmv] [11580.073438] ll_unlink+0x24d/0x7f0 [lustre] [11580.083126] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11580.083218] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11580.083283] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11580.083286] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11580.083301] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11580.083306] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11580.083315] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [11580.083318] RDX: 0000000000001c3b RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [11580.083321] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [11580.083323] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [11580.083326] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11580.083330] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [11580.083333] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11580.083336] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11580.083343] Call Trace: [11580.083349] [11580.083355] ? show_regs.cold.9+0x22/0x2f [11580.083364] ? watchdog_timer_fn.cold.6+0x62/0x10a [11580.083370] ? __hrtimer_run_queues+0x1c2/0x4d0 [11580.083380] ? watchdog+0x40/0x40 [11580.083384] ? hrtimer_interrupt+0x128/0x2c0 [11580.083388] ? smp_apic_timer_interrupt+0xa6/0x270 [11580.083396] ? apic_timer_interrupt+0xf/0x20 [11580.083400] [11580.083403] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11580.083408] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11580.083413] do_raw_spin_lock+0x177/0x190 [11580.083418] _raw_spin_lock+0x21/0x30 [11580.083424] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11580.085972] obd_mod_free+0x46/0x140 [ptlrpc] [11580.095402] vfs_unlink+0x202/0x330 [11580.095428] do_unlinkat+0x2fa/0x460 [11580.095434] __x64_sys_unlink+0x37/0x50 [11580.095438] do_syscall_64+0xc1/0x440 [11580.095446] entry_SYSCALL_64_after_hwframe+0x49/0xae [11580.095455] RIP: 0033:0x7f151bdee10b [11580.095463] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [11580.095467] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [11580.095473] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [11580.095476] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [11580.095478] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11580.095481] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11580.095483] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11580.937240] mdc_commit_open+0x91/0xb0 [mdc] [11580.939336] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11580.943212] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11580.946859] after_reply+0xe2c/0x1780 [ptlrpc] [11580.949445] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11580.953067] ? __next_timer_interrupt+0x160/0x160 [11580.955686] ptlrpcd+0xa82/0xe40 [ptlrpc] [11580.958346] ? do_wait_intr+0xf0/0xf0 [11580.960221] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11580.964057] kthread+0x1d1/0x200 [11580.966001] ? set_kthread_struct+0x70/0x70 [11580.968128] ret_from_fork+0x1f/0x30 [11586.019815] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9655s! [11586.022174] Showing busy workqueues and worker pools: [11586.023930] workqueue events: flags=0x0 [11586.025352] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11586.027485] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [11586.031488] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11586.034852] pending: kfree_rcu_monitor [11586.036397] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11586.039750] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [11586.043822] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11586.047115] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [11586.049772] workqueue events_long: flags=0x0 [11586.050949] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11586.052996] pending: lu_objects_flush [obdclass] [11586.054918] workqueue events_unbound: flags=0x2 [11586.056853] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11586.060072] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11586.064480] workqueue events_power_efficient: flags=0x80 [11586.067161] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11586.069752] pending: do_cache_clean, neigh_periodic_work, check_lifetime [11586.072426] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11586.076068] pending: neigh_periodic_work [11586.078018] workqueue mm_percpu_wq: flags=0x8 [11586.082255] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11586.084373] pending: vmstat_update, lru_add_drain_per_cpu [11586.087061] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11586.089832] pending: vmstat_update [11586.091371] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11586.093999] pending: lru_add_drain_per_cpu [11586.095302] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [11586.097260] pending: lru_add_drain_per_cpu BAR(48) [11586.098821] workqueue writeback: flags=0x4a [11586.099951] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [11586.101751] pending: wb_workfn [11586.103221] workqueue xprtiod: flags=0xa [11586.104326] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11586.106172] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [11586.108296] workqueue ipv6_addrconf: flags=0x40008 [11586.109815] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11586.112173] pending: addrconf_verify_work [11586.114233] workqueue knbd0-recv: flags=0x1a [11586.115632] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11586.117961] in-flight: 658:recv_work [11586.119011] workqueue ptlrpc_pinger: flags=0x2 [11586.120427] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11586.123063] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11586.126103] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9655s workers=4 idle: 32984 37 8 [11586.134045] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11570s workers=2 idle: 661 [11596.056741] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11596.063468] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11596.087467] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11596.093533] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11596.098129] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11596.101377] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11596.104777] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11596.114515] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11596.118622] RAX: 0000000000006b08 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11596.122449] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [11596.127128] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [11596.130970] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [11596.134437] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [11596.138022] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [11596.142512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11596.145452] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11596.149242] Call Trace: [11596.150952] [11596.151987] ? show_regs.cold.9+0x22/0x2f [11596.154206] ? watchdog_timer_fn.cold.6+0x62/0x10a [11596.156776] ? __hrtimer_run_queues+0x1c2/0x4d0 [11596.159277] ? watchdog+0x40/0x40 [11596.161526] ? hrtimer_interrupt+0x128/0x2c0 [11596.163789] ? smp_apic_timer_interrupt+0xa6/0x270 [11596.165586] ? apic_timer_interrupt+0xf/0x20 [11596.168134] [11596.169213] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11596.172021] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11596.175307] do_raw_spin_lock+0x177/0x190 [11596.177811] _raw_spin_lock+0x21/0x30 [11596.179740] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11596.182992] process_one_work+0x2c8/0x700 [11596.185321] worker_thread+0x296/0x6e0 [11596.187257] ? rescuer_thread+0x570/0x570 [11596.189810] kthread+0x1d1/0x200 [11596.191717] ? set_kthread_struct+0x70/0x70 [11596.193969] ret_from_fork+0x1f/0x30 [11608.055101] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [11608.057125] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [11608.058232] Modules linked in: lustre(O) [11608.061983] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11608.062086] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11608.062094] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11608.062097] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11608.062114] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11608.062118] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11608.062123] RAX: 0000000000004308 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11608.062125] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [11608.062128] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [11608.062130] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [11608.062137] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [11608.062141] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [11608.062145] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11608.062147] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [11608.062155] Call Trace: [11608.062161] [11608.062168] ? show_regs.cold.9+0x22/0x2f [11608.062186] ? watchdog_timer_fn.cold.6+0x62/0x10a [11608.062193] ? __hrtimer_run_queues+0x1c2/0x4d0 [11608.062197] ? watchdog+0x40/0x40 [11608.062201] ? hrtimer_interrupt+0x128/0x2c0 [11608.062206] ? smp_apic_timer_interrupt+0xa6/0x270 [11608.062220] ? apic_timer_interrupt+0xf/0x20 [11608.062225] [11608.062228] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11608.062235] do_raw_spin_lock+0x177/0x190 [11608.062240] _raw_spin_lock+0x21/0x30 [11608.062246] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11608.062659] ? do_raw_spin_unlock+0x75/0x190 [11608.062666] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11608.063036] ? do_wait_intr+0xf0/0xf0 [11608.063045] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11608.063257] mdc_close+0x53b/0xd70 [mdc] [11608.063296] lmv_close+0x18d/0x490 [lmv] [11608.063325] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11608.063850] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11608.064046] ll_md_real_close+0xa5/0x2a0 [lustre] [11608.064135] ll_md_close+0x274/0x800 [lustre] [11608.066624] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11608.066737] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11608.066745] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11608.066748] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11608.066761] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11608.066764] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11608.066768] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [11608.066771] RDX: 0000000000004ffb RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [11608.066773] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [11608.066775] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [11608.066777] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11608.066780] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [11608.066782] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11608.066784] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11608.066792] Call Trace: [11608.066797] [11608.066801] ? show_regs.cold.9+0x22/0x2f [11608.066809] ? watchdog_timer_fn.cold.6+0x62/0x10a [11608.066816] ? __hrtimer_run_queues+0x1c2/0x4d0 [11608.066820] ? watchdog+0x40/0x40 [11608.066824] ? hrtimer_interrupt+0x128/0x2c0 [11608.066827] ? smp_apic_timer_interrupt+0xa6/0x270 [11608.066836] ? apic_timer_interrupt+0xf/0x20 [11608.066840] [11608.066843] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11608.066847] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11608.066851] do_raw_spin_lock+0x177/0x190 [11608.066855] _raw_spin_lock+0x21/0x30 [11608.066932] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11608.067193] obd_mod_free+0x46/0x140 [ptlrpc] [11608.067587] mdc_commit_open+0x91/0xb0 [mdc] [11608.067630] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11608.090793] ll_file_release+0x128/0xc50 [lustre] [11608.091003] __fput+0x10d/0x440 [11608.091016] ____fput+0x12/0x20 [11608.091021] task_work_run+0xc6/0x110 [11608.091029] exit_to_usermode_loop+0x1dd/0x1f0 [11608.091036] do_syscall_64+0x3d6/0x440 [11608.091041] entry_SYSCALL_64_after_hwframe+0x49/0xae [11608.091050] RIP: 0033:0x7f151bee3cd8 [11608.091059] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11608.091064] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11608.091070] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [11608.091072] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [11608.091074] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11608.091076] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11608.091079] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11608.531938] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11608.535002] after_reply+0xe2c/0x1780 [ptlrpc] [11608.537999] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11608.540848] ? __next_timer_interrupt+0x160/0x160 [11608.543045] ptlrpcd+0xa82/0xe40 [ptlrpc] [11608.545075] ? do_wait_intr+0xf0/0xf0 [11608.546694] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11608.549458] kthread+0x1d1/0x200 [11608.551065] ? set_kthread_struct+0x70/0x70 [11608.552995] ret_from_fork+0x1f/0x30 [11612.052531] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [11612.064288] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11612.122310] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11612.130529] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11612.140533] RIP: 0010:native_safe_halt+0x2e/0x50 [11612.144114] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11612.158901] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11612.161212] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11612.172963] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [11612.181555] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [11612.186999] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [11612.194660] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [11612.204334] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [11612.218604] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11612.223608] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [11612.225988] Call Trace: [11612.228189] [11612.229192] ? show_regs.cold.9+0x22/0x2f [11612.237342] ? watchdog_timer_fn.cold.6+0x62/0x10a [11612.240390] ? __hrtimer_run_queues+0x1c2/0x4d0 [11612.246155] ? watchdog+0x40/0x40 [11612.248721] ? hrtimer_interrupt+0x128/0x2c0 [11612.251086] ? smp_apic_timer_interrupt+0xa6/0x270 [11612.254110] ? apic_timer_interrupt+0xf/0x20 [11612.256526] [11612.258118] ? native_safe_halt+0x2e/0x50 [11612.269294] kvm_wait+0xcd/0xe0 [11612.275300] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11612.282926] do_raw_spin_lock+0x177/0x190 [11612.285572] _raw_spin_lock+0x21/0x30 [11612.289108] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11612.294341] ? do_raw_spin_unlock+0x75/0x190 [11612.300152] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11612.311069] ? do_wait_intr+0xf0/0xf0 [11612.315040] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11612.328250] mdc_reint+0x2d/0x110 [mdc] [11612.337524] mdc_unlink+0x658/0x870 [mdc] [11612.346077] lmv_unlink+0x34d/0xbd0 [lmv] [11612.350793] ll_unlink+0x24d/0x7f0 [lustre] [11612.353125] vfs_unlink+0x202/0x330 [11612.357029] do_unlinkat+0x2fa/0x460 [11612.360914] __x64_sys_unlink+0x37/0x50 [11612.365117] do_syscall_64+0xc1/0x440 [11612.366499] entry_SYSCALL_64_after_hwframe+0x49/0xae [11612.378202] RIP: 0033:0x7f151bdee10b [11612.382997] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [11612.397596] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [11612.404153] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [11612.407707] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [11612.410793] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11612.413835] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11612.416738] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11616.225050] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9685s! [11616.231315] Showing busy workqueues and worker pools: [11616.234749] workqueue events: flags=0x0 [11616.242253] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11616.251303] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [11616.265286] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11616.271792] pending: kfree_rcu_monitor [11616.275882] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11616.280872] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [11616.289069] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11616.297836] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [11616.302256] workqueue events_long: flags=0x0 [11616.312241] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11616.317080] pending: lu_objects_flush [obdclass] [11616.320345] workqueue events_unbound: flags=0x2 [11616.322983] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11616.327314] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11616.340022] workqueue events_power_efficient: flags=0x80 [11616.344171] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11616.348285] pending: do_cache_clean, neigh_periodic_work, check_lifetime [11616.350777] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11616.358204] pending: neigh_periodic_work [11616.360877] workqueue mm_percpu_wq: flags=0x8 [11616.365064] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11616.369159] pending: vmstat_update, lru_add_drain_per_cpu [11616.373305] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11616.377759] pending: vmstat_update [11616.380317] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11616.383787] pending: lru_add_drain_per_cpu [11616.385097] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [11616.388268] pending: lru_add_drain_per_cpu BAR(48) [11616.391249] workqueue writeback: flags=0x4a [11616.393307] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [11616.400198] pending: wb_workfn [11616.402143] workqueue xprtiod: flags=0xa [11616.403713] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11616.405968] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [11616.412202] workqueue ipv6_addrconf: flags=0x40008 [11616.414739] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11616.423260] pending: addrconf_verify_work [11616.431538] workqueue knbd0-recv: flags=0x1a [11616.439448] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11616.446317] in-flight: 658:recv_work [11616.448824] workqueue ptlrpc_pinger: flags=0x2 [11616.458188] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11616.461713] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11616.468384] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9685s workers=4 idle: 32984 37 8 [11616.472292] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11601s workers=2 idle: 661 [11624.057799] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11624.064825] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11624.100575] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11624.109978] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11624.115391] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11624.122510] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11624.129476] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11624.151610] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11624.158502] RAX: 0000000000007396 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11624.163752] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [11624.167565] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [11624.170914] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [11624.177041] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [11624.182236] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [11624.193722] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11624.195748] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11624.199943] Call Trace: [11624.201389] [11624.202013] ? show_regs.cold.9+0x22/0x2f [11624.204203] ? watchdog_timer_fn.cold.6+0x62/0x10a [11624.212003] ? __hrtimer_run_queues+0x1c2/0x4d0 [11624.217264] ? watchdog+0x40/0x40 [11624.221034] ? hrtimer_interrupt+0x128/0x2c0 [11624.223960] ? smp_apic_timer_interrupt+0xa6/0x270 [11624.231967] ? apic_timer_interrupt+0xf/0x20 [11624.241042] [11624.243943] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11624.251659] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11624.254856] do_raw_spin_lock+0x177/0x190 [11624.259748] _raw_spin_lock+0x21/0x30 [11624.266546] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11624.272352] process_one_work+0x2c8/0x700 [11624.278345] worker_thread+0x296/0x6e0 [11624.281862] ? rescuer_thread+0x570/0x570 [11624.283899] kthread+0x1d1/0x200 [11624.287871] ? set_kthread_struct+0x70/0x70 [11624.290903] ret_from_fork+0x1f/0x30 [11636.055021] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [11636.057061] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [11636.057953] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11636.058031] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11636.058038] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11636.058041] RIP: 0010:native_safe_halt+0x2e/0x50 [11636.058055] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11636.058059] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [11636.058064] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [11636.058066] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [11636.058069] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [11636.058071] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [11636.058074] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11636.058077] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [11636.058080] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11636.058082] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11636.058091] Call Trace: [11636.058097] [11636.058102] ? show_regs.cold.9+0x22/0x2f [11636.058113] ? watchdog_timer_fn.cold.6+0x62/0x10a [11636.058120] ? __hrtimer_run_queues+0x1c2/0x4d0 [11636.058125] ? watchdog+0x40/0x40 [11636.058130] ? hrtimer_interrupt+0x128/0x2c0 [11636.058134] ? smp_apic_timer_interrupt+0xa6/0x270 [11636.058141] ? apic_timer_interrupt+0xf/0x20 [11636.058147] [11636.058149] ? native_safe_halt+0x2e/0x50 [11636.058154] kvm_wait+0xcd/0xe0 [11636.058162] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11636.065208] Modules linked in: [11636.086676] do_raw_spin_lock+0x177/0x190 [11636.086698] _raw_spin_lock+0x21/0x30 [11636.086708] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11636.087288] obd_mod_free+0x46/0x140 [ptlrpc] [11636.098902] lustre(O) [11636.102808] mdc_commit_open+0x91/0xb0 [mdc] [11636.102861] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11636.107518] osp(O) [11636.116575] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11636.116901] after_reply+0xe2c/0x1780 [ptlrpc] [11636.117118] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11636.117370] ? __next_timer_interrupt+0x160/0x160 [11636.117384] ptlrpcd+0xa82/0xe40 [ptlrpc] [11636.117623] ? do_wait_intr+0xf0/0xf0 [11636.117633] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11636.129832] ofd(O) [11636.132793] kthread+0x1d1/0x200 [11636.132811] ? set_kthread_struct+0x70/0x70 [11636.132817] ret_from_fork+0x1f/0x30 [11636.216082] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11636.247443] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11636.257990] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11636.260743] RIP: 0010:native_safe_halt+0x2e/0x50 [11636.262238] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11636.268937] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11636.272571] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11636.275043] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [11636.278112] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [11636.281646] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [11636.284900] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [11636.290895] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [11636.302759] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11636.307541] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [11636.312340] Call Trace: [11636.313563] [11636.314359] ? show_regs.cold.9+0x22/0x2f [11636.316823] ? watchdog_timer_fn.cold.6+0x62/0x10a [11636.323495] ? __hrtimer_run_queues+0x1c2/0x4d0 [11636.328021] ? watchdog+0x40/0x40 [11636.329802] ? hrtimer_interrupt+0x128/0x2c0 [11636.334724] ? smp_apic_timer_interrupt+0xa6/0x270 [11636.338925] ? apic_timer_interrupt+0xf/0x20 [11636.342136] [11636.342889] ? native_safe_halt+0x2e/0x50 [11636.344164] kvm_wait+0xcd/0xe0 [11636.351120] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11636.357197] do_raw_spin_lock+0x177/0x190 [11636.361880] _raw_spin_lock+0x21/0x30 [11636.365815] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11636.369436] ? do_raw_spin_unlock+0x75/0x190 [11636.378902] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11636.382816] ? do_wait_intr+0xf0/0xf0 [11636.386392] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11636.391843] mdc_close+0x53b/0xd70 [mdc] [11636.394498] lmv_close+0x18d/0x490 [lmv] [11636.397127] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11636.405895] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11636.411087] ll_md_real_close+0xa5/0x2a0 [lustre] [11636.415317] ll_md_close+0x274/0x800 [lustre] [11636.422784] ll_file_release+0x128/0xc50 [lustre] [11636.430858] __fput+0x10d/0x440 [11636.433848] ____fput+0x12/0x20 [11636.440500] task_work_run+0xc6/0x110 [11636.441947] exit_to_usermode_loop+0x1dd/0x1f0 [11636.444847] do_syscall_64+0x3d6/0x440 [11636.447956] entry_SYSCALL_64_after_hwframe+0x49/0xae [11636.454026] RIP: 0033:0x7f151bee3cd8 [11636.462526] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11636.478174] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11636.479719] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [11636.489840] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [11636.496508] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11636.505219] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11636.512000] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11644.052021] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [11644.063629] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11644.114587] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11644.126153] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11644.135480] RIP: 0010:native_safe_halt+0x2e/0x50 [11644.137974] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11644.155755] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11644.159258] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11644.162399] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [11644.165985] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [11644.172958] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [11644.176604] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [11644.180560] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [11644.185270] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11644.187824] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [11644.190796] Call Trace: [11644.195185] [11644.195762] ? show_regs.cold.9+0x22/0x2f [11644.198369] ? watchdog_timer_fn.cold.6+0x62/0x10a [11644.200758] ? __hrtimer_run_queues+0x1c2/0x4d0 [11644.203069] ? watchdog+0x40/0x40 [11644.205266] ? hrtimer_interrupt+0x128/0x2c0 [11644.210263] ? smp_apic_timer_interrupt+0xa6/0x270 [11644.211909] ? apic_timer_interrupt+0xf/0x20 [11644.216614] [11644.218457] ? native_safe_halt+0x2e/0x50 [11644.220995] kvm_wait+0xcd/0xe0 [11644.222614] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11644.225577] do_raw_spin_lock+0x177/0x190 [11644.228256] _raw_spin_lock+0x21/0x30 [11644.233718] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11644.237005] ? do_raw_spin_unlock+0x75/0x190 [11644.239085] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11644.241521] ? do_wait_intr+0xf0/0xf0 [11644.243345] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11644.249383] mdc_reint+0x2d/0x110 [mdc] [11644.250972] mdc_unlink+0x658/0x870 [mdc] [11644.253163] lmv_unlink+0x34d/0xbd0 [lmv] [11644.254829] ll_unlink+0x24d/0x7f0 [lustre] [11644.257506] vfs_unlink+0x202/0x330 [11644.261493] do_unlinkat+0x2fa/0x460 [11644.266998] __x64_sys_unlink+0x37/0x50 [11644.269037] do_syscall_64+0xc1/0x440 [11644.274414] entry_SYSCALL_64_after_hwframe+0x49/0xae [11644.276715] RIP: 0033:0x7f151bdee10b [11644.278418] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [11644.291045] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [11644.298480] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [11644.302343] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [11644.306785] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11644.311395] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11644.315840] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11646.944210] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9716s! [11646.949757] Showing busy workqueues and worker pools: [11646.953376] workqueue events: flags=0x0 [11646.958236] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11646.964054] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [11646.982601] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11646.997886] pending: kfree_rcu_monitor [11646.999120] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11647.006504] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [11647.019402] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11647.027974] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [11647.030182] workqueue events_long: flags=0x0 [11647.044577] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11647.047527] pending: lu_objects_flush [obdclass] [11647.051111] workqueue events_unbound: flags=0x2 [11647.056265] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11647.063461] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11647.093472] workqueue events_power_efficient: flags=0x80 [11647.098991] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11647.106132] pending: do_cache_clean, neigh_periodic_work, check_lifetime [11647.116855] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11647.128903] pending: neigh_periodic_work [11647.133446] workqueue mm_percpu_wq: flags=0x8 [11647.142445] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11647.148960] pending: vmstat_update, lru_add_drain_per_cpu [11647.152728] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11647.154763] pending: vmstat_update [11647.156353] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11647.158572] pending: lru_add_drain_per_cpu [11647.166569] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [11647.172056] pending: lru_add_drain_per_cpu BAR(48) [11647.175682] workqueue writeback: flags=0x4a [11647.179710] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [11647.184949] pending: wb_workfn [11647.193099] workqueue xprtiod: flags=0xa [11647.194068] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11647.203003] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [11647.211218] workqueue ipv6_addrconf: flags=0x40008 [11647.216156] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11647.220974] pending: addrconf_verify_work [11647.225248] workqueue knbd0-recv: flags=0x1a [11647.229012] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11647.234042] in-flight: 658:recv_work [11647.237024] workqueue ptlrpc_pinger: flags=0x2 [11647.241393] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11647.252482] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11647.259133] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9716s workers=4 idle: 32984 37 8 [11647.267011] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11632s workers=2 idle: 661 [11652.056134] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11652.062250] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11652.093928] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11652.099246] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11652.103587] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11652.106519] RIP: 0010:native_safe_halt+0x2e/0x50 [11652.108593] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11652.117370] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11652.121174] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11652.123884] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [11652.126835] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [11652.131487] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [11652.134298] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [11652.137061] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [11652.140481] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11652.142870] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11652.145789] Call Trace: [11652.146864] [11652.147830] ? show_regs.cold.9+0x22/0x2f [11652.149462] ? watchdog_timer_fn.cold.6+0x62/0x10a [11652.151571] ? __hrtimer_run_queues+0x1c2/0x4d0 [11652.153466] ? watchdog+0x40/0x40 [11652.154837] ? hrtimer_interrupt+0x128/0x2c0 [11652.156706] ? smp_apic_timer_interrupt+0xa6/0x270 [11652.157978] ? apic_timer_interrupt+0xf/0x20 [11652.159196] [11652.159867] ? native_safe_halt+0x2e/0x50 [11652.161333] kvm_wait+0xcd/0xe0 [11652.162620] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11652.164808] do_raw_spin_lock+0x177/0x190 [11652.166218] _raw_spin_lock+0x21/0x30 [11652.167278] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11652.168971] process_one_work+0x2c8/0x700 [11652.170951] worker_thread+0x296/0x6e0 [11652.173142] ? rescuer_thread+0x570/0x570 [11652.174565] kthread+0x1d1/0x200 [11652.175789] ? set_kthread_struct+0x70/0x70 [11652.177491] ret_from_fork+0x1f/0x30 [11664.058550] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [11664.070724] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [11664.076001] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11664.076102] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11664.076109] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11664.076112] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11664.076127] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11664.076132] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11664.076136] RAX: 000000000000790c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11664.076139] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [11664.076141] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [11664.076144] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [11664.076146] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [11664.076150] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [11664.076153] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11664.076155] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [11664.076165] Call Trace: [11664.076170] [11664.076175] ? show_regs.cold.9+0x22/0x2f [11664.076184] ? watchdog_timer_fn.cold.6+0x62/0x10a [11664.076190] ? __hrtimer_run_queues+0x1c2/0x4d0 [11664.076194] ? watchdog+0x40/0x40 [11664.076198] ? hrtimer_interrupt+0x128/0x2c0 [11664.076201] ? smp_apic_timer_interrupt+0xa6/0x270 [11664.076209] ? apic_timer_interrupt+0xf/0x20 [11664.076214] [11664.076216] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11664.076222] do_raw_spin_lock+0x177/0x190 [11664.076227] _raw_spin_lock+0x21/0x30 [11664.076232] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11664.076504] ? do_raw_spin_unlock+0x75/0x190 [11664.076510] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11664.076703] ? do_wait_intr+0xf0/0xf0 [11664.076710] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11664.076906] mdc_close+0x53b/0xd70 [mdc] [11664.076943] lmv_close+0x18d/0x490 [lmv] [11664.076972] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11664.077142] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11664.077261] ll_md_real_close+0xa5/0x2a0 [lustre] [11664.077346] ll_md_close+0x274/0x800 [lustre] [11664.081510] ll_file_release+0x128/0xc50 [lustre] [11664.081629] __fput+0x10d/0x440 [11664.081639] ____fput+0x12/0x20 [11664.081643] task_work_run+0xc6/0x110 [11664.081651] exit_to_usermode_loop+0x1dd/0x1f0 [11664.081658] do_syscall_64+0x3d6/0x440 [11664.081662] entry_SYSCALL_64_after_hwframe+0x49/0xae [11664.081670] RIP: 0033:0x7f151bee3cd8 [11664.081678] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11664.081682] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11664.081687] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [11664.081690] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [11664.081692] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11664.081694] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11664.081696] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11664.489126] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11664.532755] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11664.543529] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11664.552700] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11664.560700] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11664.575802] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11664.583318] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [11664.587593] RDX: 000000000000778e RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [11664.591835] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [11664.595313] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [11664.597867] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11664.602815] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [11664.610668] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11664.613670] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11664.617310] Call Trace: [11664.618423] [11664.619598] ? show_regs.cold.9+0x22/0x2f [11664.621799] ? watchdog_timer_fn.cold.6+0x62/0x10a [11664.625410] ? __hrtimer_run_queues+0x1c2/0x4d0 [11664.631008] ? watchdog+0x40/0x40 [11664.633648] ? hrtimer_interrupt+0x128/0x2c0 [11664.636164] ? smp_apic_timer_interrupt+0xa6/0x270 [11664.645289] ? apic_timer_interrupt+0xf/0x20 [11664.650578] [11664.652315] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11664.655010] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11664.659088] do_raw_spin_lock+0x177/0x190 [11664.660511] _raw_spin_lock+0x21/0x30 [11664.663546] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11664.666493] obd_mod_free+0x46/0x140 [ptlrpc] [11664.669318] mdc_commit_open+0x91/0xb0 [mdc] [11664.670960] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11664.682829] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11664.692377] after_reply+0xe2c/0x1780 [ptlrpc] [11664.700830] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11664.709716] ? __next_timer_interrupt+0x160/0x160 [11664.711467] ptlrpcd+0xa82/0xe40 [ptlrpc] [11664.721701] ? do_wait_intr+0xf0/0xf0 [11664.727109] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11664.731276] kthread+0x1d1/0x200 [11664.732438] ? set_kthread_struct+0x70/0x70 [11664.737455] ret_from_fork+0x1f/0x30 [11672.052028] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [11672.054997] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11672.074128] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11672.077787] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11672.081060] RIP: 0010:native_safe_halt+0x2e/0x50 [11672.082409] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11672.088152] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11672.090691] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11672.093147] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [11672.095112] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [11672.097121] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [11672.099245] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [11672.101408] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [11672.104229] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11672.106117] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [11672.108099] Call Trace: [11672.108882] [11672.109584] ? show_regs.cold.9+0x22/0x2f [11672.111216] ? watchdog_timer_fn.cold.6+0x62/0x10a [11672.112611] ? __hrtimer_run_queues+0x1c2/0x4d0 [11672.113971] ? watchdog+0x40/0x40 [11672.114935] ? hrtimer_interrupt+0x128/0x2c0 [11672.116408] ? smp_apic_timer_interrupt+0xa6/0x270 [11672.118162] ? apic_timer_interrupt+0xf/0x20 [11672.119500] [11672.120212] ? native_safe_halt+0x2e/0x50 [11672.121397] kvm_wait+0xcd/0xe0 [11672.122394] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11672.124065] do_raw_spin_lock+0x177/0x190 [11672.125199] _raw_spin_lock+0x21/0x30 [11672.126501] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11672.128662] ? do_raw_spin_unlock+0x75/0x190 [11672.130267] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11672.132106] ? do_wait_intr+0xf0/0xf0 [11672.133318] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11672.135106] mdc_reint+0x2d/0x110 [mdc] [11672.136286] mdc_unlink+0x658/0x870 [mdc] [11672.137561] lmv_unlink+0x34d/0xbd0 [lmv] [11672.139047] ll_unlink+0x24d/0x7f0 [lustre] [11672.140525] vfs_unlink+0x202/0x330 [11672.141572] do_unlinkat+0x2fa/0x460 [11672.142800] __x64_sys_unlink+0x37/0x50 [11672.144138] do_syscall_64+0xc1/0x440 [11672.145341] entry_SYSCALL_64_after_hwframe+0x49/0xae [11672.146925] RIP: 0033:0x7f151bdee10b [11672.147999] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [11672.153790] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [11672.156575] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [11672.159252] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [11672.162239] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11672.164406] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11672.166745] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11677.677784] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9747s! [11677.687991] Showing busy workqueues and worker pools: [11677.692126] workqueue events: flags=0x0 [11677.696330] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11677.701250] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [11677.706787] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11677.710492] pending: kfree_rcu_monitor [11677.713094] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11677.715857] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [11677.720892] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11677.723950] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [11677.727122] workqueue events_long: flags=0x0 [11677.735582] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11677.743199] pending: lu_objects_flush [obdclass] [11677.748127] workqueue events_unbound: flags=0x2 [11677.750400] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11677.756441] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11677.767206] workqueue events_power_efficient: flags=0x80 [11677.773508] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11677.783700] pending: do_cache_clean, neigh_periodic_work, check_lifetime [11677.789213] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11677.797694] pending: neigh_periodic_work [11677.803112] workqueue mm_percpu_wq: flags=0x8 [11677.805342] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11677.813122] pending: vmstat_update, lru_add_drain_per_cpu [11677.819169] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11677.826586] pending: vmstat_update [11677.829336] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11677.837344] pending: lru_add_drain_per_cpu [11677.840737] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [11677.848237] pending: lru_add_drain_per_cpu BAR(48) [11677.853988] workqueue writeback: flags=0x4a [11677.855816] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [11677.862255] pending: wb_workfn [11677.864499] workqueue xprtiod: flags=0xa [11677.868889] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11677.871998] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [11677.877262] workqueue ipv6_addrconf: flags=0x40008 [11677.883260] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11677.893621] pending: addrconf_verify_work [11677.902444] workqueue knbd0-recv: flags=0x1a [11677.909917] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11677.916110] in-flight: 658:recv_work [11677.921113] workqueue ptlrpc_pinger: flags=0x2 [11677.924171] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11677.932125] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11677.941909] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9747s workers=4 idle: 32984 37 8 [11677.951824] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11662s workers=2 idle: 661 [11680.056011] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11680.059345] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11680.078596] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11680.083517] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11680.087508] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11680.090361] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11680.092901] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11680.101457] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11680.105270] RAX: 00000000000016f2 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11680.108217] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [11680.111392] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [11680.114409] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [11680.117192] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [11680.120412] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [11680.123649] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11680.126248] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11680.129224] Call Trace: [11680.130183] [11680.131055] ? show_regs.cold.9+0x22/0x2f [11680.132890] ? watchdog_timer_fn.cold.6+0x62/0x10a [11680.135176] ? __hrtimer_run_queues+0x1c2/0x4d0 [11680.137319] ? watchdog+0x40/0x40 [11680.138678] ? hrtimer_interrupt+0x128/0x2c0 [11680.140453] ? smp_apic_timer_interrupt+0xa6/0x270 [11680.142873] ? apic_timer_interrupt+0xf/0x20 [11680.144859] [11680.145804] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11680.148234] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11680.150475] do_raw_spin_lock+0x177/0x190 [11680.152404] _raw_spin_lock+0x21/0x30 [11680.153730] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11680.155612] process_one_work+0x2c8/0x700 [11680.157514] worker_thread+0x296/0x6e0 [11680.158967] ? rescuer_thread+0x570/0x570 [11680.160675] kthread+0x1d1/0x200 [11680.162212] ? set_kthread_struct+0x70/0x70 [11680.164005] ret_from_fork+0x1f/0x30 [11692.061626] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [11692.063054] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [11692.068674] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash [11692.078121] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11692.078197] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11692.078204] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11692.078207] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11692.078221] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11692.078226] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11692.078231] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [11692.078233] RDX: 00000000000024bf RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [11692.078236] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [11692.078238] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [11692.078240] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11692.078243] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [11692.078246] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11692.078249] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11692.078259] Call Trace: [11692.078265] [11692.078271] ? show_regs.cold.9+0x22/0x2f [11692.078281] ? watchdog_timer_fn.cold.6+0x62/0x10a [11692.078288] ? __hrtimer_run_queues+0x1c2/0x4d0 [11692.078293] ? watchdog+0x40/0x40 [11692.078297] ? hrtimer_interrupt+0x128/0x2c0 [11692.078300] ? smp_apic_timer_interrupt+0xa6/0x270 [11692.078309] ? apic_timer_interrupt+0xf/0x20 [11692.078314] [11692.078317] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11692.078321] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11692.078326] do_raw_spin_lock+0x177/0x190 [11692.078331] _raw_spin_lock+0x21/0x30 [11692.078337] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11692.078836] obd_mod_free+0x46/0x140 [ptlrpc] [11692.079030] mdc_commit_open+0x91/0xb0 [mdc] [11692.079066] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11692.079257] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11692.080465] after_reply+0xe2c/0x1780 [ptlrpc] [11692.080772] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11692.080972] ? __next_timer_interrupt+0x160/0x160 [11692.080983] ptlrpcd+0xa82/0xe40 [ptlrpc] [11692.081189] ? do_wait_intr+0xf0/0xf0 [11692.081199] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11692.104027] dm_log dm_mod sha512_ssse3 sha512_generic [11692.104049] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11692.104057] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11692.104059] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11692.104074] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11692.104078] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11692.104083] RAX: 0000000000003b60 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11692.104085] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [11692.104088] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [11692.104090] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [11692.104092] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [11692.104095] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [11692.104098] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11692.104100] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [11692.104108] Call Trace: [11692.104113] [11692.104119] ? show_regs.cold.9+0x22/0x2f [11692.104129] ? watchdog_timer_fn.cold.6+0x62/0x10a [11692.104135] ? __hrtimer_run_queues+0x1c2/0x4d0 [11692.104139] ? watchdog+0x40/0x40 [11692.104144] ? hrtimer_interrupt+0x128/0x2c0 [11692.104148] ? smp_apic_timer_interrupt+0xa6/0x270 [11692.104156] ? apic_timer_interrupt+0xf/0x20 [11692.104160] [11692.104163] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11692.104172] do_raw_spin_lock+0x177/0x190 [11692.104176] _raw_spin_lock+0x21/0x30 [11692.104181] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11692.104446] ? do_raw_spin_unlock+0x75/0x190 [11692.104453] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11692.104976] ? do_wait_intr+0xf0/0xf0 [11692.104985] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11692.105182] mdc_close+0x53b/0xd70 [mdc] [11692.105223] lmv_close+0x18d/0x490 [lmv] [11692.105253] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11692.105412] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11692.105740] ll_md_real_close+0xa5/0x2a0 [lustre] [11692.105833] ll_md_close+0x274/0x800 [lustre] [11692.105929] ll_file_release+0x128/0xc50 [lustre] [11692.106012] __fput+0x10d/0x440 [11692.106022] ____fput+0x12/0x20 [11692.106026] task_work_run+0xc6/0x110 [11692.106033] exit_to_usermode_loop+0x1dd/0x1f0 [11692.106040] do_syscall_64+0x3d6/0x440 [11692.106045] entry_SYSCALL_64_after_hwframe+0x49/0xae [11692.106052] RIP: 0033:0x7f151bee3cd8 [11692.106059] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11692.106064] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11692.106069] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [11692.106071] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [11692.106073] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11692.106075] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11692.106077] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11692.658283] kthread+0x1d1/0x200 [11692.664835] ? set_kthread_struct+0x70/0x70 [11692.669411] ret_from_fork+0x1f/0x30 [11704.053856] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [11704.063615] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11704.119341] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11704.127253] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11704.132334] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11704.137128] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11704.152507] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11704.158329] RAX: 0000000000007e88 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11704.164061] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [11704.167976] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [11704.172259] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [11704.176887] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [11704.180812] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [11704.185849] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11704.188946] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [11704.191517] Call Trace: [11704.194968] [11704.196359] ? show_regs.cold.9+0x22/0x2f [11704.200310] ? watchdog_timer_fn.cold.6+0x62/0x10a [11704.203232] ? __hrtimer_run_queues+0x1c2/0x4d0 [11704.206867] ? watchdog+0x40/0x40 [11704.213340] ? hrtimer_interrupt+0x128/0x2c0 [11704.218173] ? smp_apic_timer_interrupt+0xa6/0x270 [11704.221354] ? apic_timer_interrupt+0xf/0x20 [11704.224394] [11704.230395] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11704.237252] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11704.247144] do_raw_spin_lock+0x177/0x190 [11704.254123] _raw_spin_lock+0x21/0x30 [11704.257991] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11704.263721] ? do_raw_spin_unlock+0x75/0x190 [11704.269157] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11704.273059] ? do_wait_intr+0xf0/0xf0 [11704.279144] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11704.284126] mdc_reint+0x2d/0x110 [mdc] [11704.288284] mdc_unlink+0x658/0x870 [mdc] [11704.293638] lmv_unlink+0x34d/0xbd0 [lmv] [11704.296129] ll_unlink+0x24d/0x7f0 [lustre] [11704.298672] vfs_unlink+0x202/0x330 [11704.303414] do_unlinkat+0x2fa/0x460 [11704.306854] __x64_sys_unlink+0x37/0x50 [11704.317693] do_syscall_64+0xc1/0x440 [11704.321460] entry_SYSCALL_64_after_hwframe+0x49/0xae [11704.328953] RIP: 0033:0x7f151bdee10b [11704.334220] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [11704.347201] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [11704.354264] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [11704.362033] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [11704.368682] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11704.377115] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11704.384946] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11704.859141] rcu: INFO: rcu_sched self-detected stall on CPU [11704.862786] rcu: 1-...!: (5638994 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [11704.869600] (t=9780212 jiffies g=249657 q=5245) [11704.871451] rcu: rcu_sched kthread starved for 9775164 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [11704.877858] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [11704.884408] rcu: RCU grace-period kthread stack dump: [11704.889157] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [11704.896033] Call Trace: [11704.898335] __schedule+0x351/0xcb0 [11704.902287] schedule+0xc0/0x180 [11704.905981] schedule_timeout+0xb4/0x190 [11704.910098] ? do_raw_spin_unlock+0x75/0x190 [11704.911439] ? __next_timer_interrupt+0x160/0x160 [11704.918013] ? prepare_to_swait_event+0xf3/0x140 [11704.925498] rcu_gp_kthread+0x8dd/0x15b0 [11704.930077] ? force_qs_rnp+0x2c0/0x2c0 [11704.933185] kthread+0x1d1/0x200 [11704.934258] ? set_kthread_struct+0x70/0x70 [11704.936928] ret_from_fork+0x1f/0x30 [11704.939000] rcu: Stack dump where RCU GP kthread last ran: [11704.941775] Sending NMI from CPU 1 to CPUs 2: [11704.943470] NMI backtrace for cpu 2 skipped: idling at native_safe_halt+0x2e/0x50 [11704.946734] NMI backtrace for cpu 1 [11704.949473] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11704.957334] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11704.966909] Call Trace: [11704.967916] [11704.968424] ? dump_stack+0xbb/0x10e [11704.970202] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [11704.972257] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [11704.974300] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [11704.976261] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [11704.979961] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [11704.982166] ? rcu_dump_cpu_stacks+0x11b/0x171 [11704.983584] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [11704.985696] ? update_process_times+0x6d/0xc0 [11704.987318] ? tick_sched_handle.isra.18+0x27/0x90 [11704.989479] ? tick_sched_timer+0x53/0xb0 [11704.991121] ? __hrtimer_run_queues+0x1c2/0x4d0 [11704.992804] ? tick_sched_handle.isra.18+0x90/0x90 [11704.994458] ? hrtimer_interrupt+0x128/0x2c0 [11704.995789] ? smp_apic_timer_interrupt+0xa6/0x270 [11704.997022] ? apic_timer_interrupt+0xf/0x20 [11704.998306] [11704.999054] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11705.001072] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11705.002889] ? do_raw_spin_lock+0x177/0x190 [11705.004332] ? _raw_spin_lock+0x21/0x30 [11705.005597] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11705.007591] ? obd_mod_free+0x46/0x140 [ptlrpc] [11705.009281] ? mdc_commit_open+0x91/0xb0 [mdc] [11705.012372] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11705.016075] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11705.018963] ? after_reply+0xe2c/0x1780 [ptlrpc] [11705.021101] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11705.023317] ? __next_timer_interrupt+0x160/0x160 [11705.024800] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [11705.026378] ? do_wait_intr+0xf0/0xf0 [11705.027650] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11705.029736] ? kthread+0x1d1/0x200 [11705.031405] ? set_kthread_struct+0x70/0x70 [11705.033467] ? ret_from_fork+0x1f/0x30 [11708.386312] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9777s! [11708.393172] Showing busy workqueues and worker pools: [11708.396436] workqueue events: flags=0x0 [11708.399248] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11708.403631] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [11708.410275] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11708.414534] pending: kfree_rcu_monitor [11708.417178] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11708.422667] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [11708.428675] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11708.433370] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [11708.438933] workqueue events_long: flags=0x0 [11708.440443] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11708.443941] pending: lu_objects_flush [obdclass] [11708.446038] workqueue events_unbound: flags=0x2 [11708.448169] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11708.451003] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11708.454952] workqueue events_power_efficient: flags=0x80 [11708.457150] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11708.459654] pending: do_cache_clean, neigh_periodic_work, check_lifetime [11708.462304] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11708.465248] pending: neigh_periodic_work [11708.466978] workqueue mm_percpu_wq: flags=0x8 [11708.468699] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11708.471638] pending: vmstat_update, lru_add_drain_per_cpu [11708.474344] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11708.477765] pending: vmstat_update [11708.480015] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11708.483424] pending: lru_add_drain_per_cpu [11708.486052] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [11708.489984] pending: lru_add_drain_per_cpu BAR(48) [11708.492399] workqueue writeback: flags=0x4a [11708.494742] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [11708.498487] pending: wb_workfn [11708.500443] workqueue xprtiod: flags=0xa [11708.504697] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11708.510019] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [11708.514928] workqueue ipv6_addrconf: flags=0x40008 [11708.518167] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11708.522363] pending: addrconf_verify_work [11708.529058] workqueue knbd0-recv: flags=0x1a [11708.533839] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11708.539395] in-flight: 658:recv_work [11708.542177] workqueue ptlrpc_pinger: flags=0x2 [11708.549073] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11708.554883] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11708.565264] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9777s workers=4 idle: 32984 37 8 [11708.575419] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11693s workers=2 idle: 661 [11720.057027] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [11720.060071] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11720.083174] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11720.094467] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11720.098128] RIP: 0010:native_safe_halt+0x2e/0x50 [11720.105788] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11720.115920] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11720.120808] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11720.125915] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [11720.133302] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [11720.136177] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [11720.139186] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [11720.142144] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [11720.150721] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11720.156056] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [11720.161922] Call Trace: [11720.164051] [11720.164987] ? show_regs.cold.9+0x22/0x2f [11720.167180] ? watchdog_timer_fn.cold.6+0x62/0x10a [11720.176412] ? __hrtimer_run_queues+0x1c2/0x4d0 [11720.178483] ? watchdog+0x40/0x40 [11720.180063] ? hrtimer_interrupt+0x128/0x2c0 [11720.182764] ? smp_apic_timer_interrupt+0xa6/0x270 [11720.185272] ? apic_timer_interrupt+0xf/0x20 [11720.186402] [11720.187088] ? native_safe_halt+0x2e/0x50 [11720.191963] kvm_wait+0xcd/0xe0 [11720.195999] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11720.200122] do_raw_spin_lock+0x177/0x190 [11720.202992] _raw_spin_lock+0x21/0x30 [11720.207933] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11720.210983] ? do_raw_spin_unlock+0x75/0x190 [11720.213873] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11720.216030] ? do_wait_intr+0xf0/0xf0 [11720.217843] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11720.221401] mdc_close+0x53b/0xd70 [mdc] [11720.222978] lmv_close+0x18d/0x490 [lmv] [11720.225049] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11720.228441] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11720.230964] ll_md_real_close+0xa5/0x2a0 [lustre] [11720.233494] ll_md_close+0x274/0x800 [lustre] [11720.235708] ll_file_release+0x128/0xc50 [lustre] [11720.237747] __fput+0x10d/0x440 [11720.239118] ____fput+0x12/0x20 [11720.240932] task_work_run+0xc6/0x110 [11720.242512] exit_to_usermode_loop+0x1dd/0x1f0 [11720.244156] do_syscall_64+0x3d6/0x440 [11720.245847] entry_SYSCALL_64_after_hwframe+0x49/0xae [11720.247863] RIP: 0033:0x7f151bee3cd8 [11720.249549] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11720.258462] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11720.260479] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [11720.267140] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [11720.270185] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11720.273214] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11720.276084] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11732.055443] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [11732.058328] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [11732.061779] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11732.061918] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11732.061927] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11732.061930] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11732.061947] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11732.061951] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11732.061956] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [11732.061958] RDX: 00000000000068f8 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [11732.061961] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [11732.061963] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [11732.061965] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11732.061968] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [11732.061971] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11732.061973] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11732.061980] Call Trace: [11732.061986] [11732.061992] ? show_regs.cold.9+0x22/0x2f [11732.062003] ? watchdog_timer_fn.cold.6+0x62/0x10a [11732.062009] ? __hrtimer_run_queues+0x1c2/0x4d0 [11732.062014] ? watchdog+0x40/0x40 [11732.062019] ? hrtimer_interrupt+0x128/0x2c0 [11732.062024] ? smp_apic_timer_interrupt+0xa6/0x270 [11732.062032] ? apic_timer_interrupt+0xf/0x20 [11732.062200] [11732.062204] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11732.062208] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11732.062213] do_raw_spin_lock+0x177/0x190 [11732.062217] _raw_spin_lock+0x21/0x30 [11732.062223] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11732.080736] Modules linked in: lustre(O) [11732.097147] obd_mod_free+0x46/0x140 [ptlrpc] [11732.102841] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) [11732.108004] mdc_commit_open+0x91/0xb0 [mdc] [11732.108132] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11732.111291] fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11732.111344] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11732.111351] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11732.111408] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11732.111895] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11732.111907] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11732.111912] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11732.111916] RAX: 0000000000007c1f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11732.111919] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [11732.111921] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [11732.111923] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [11732.111926] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [11732.111930] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [11732.111933] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11732.111935] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11732.111943] Call Trace: [11732.111953] [11732.111959] ? show_regs.cold.9+0x22/0x2f [11732.111968] ? watchdog_timer_fn.cold.6+0x62/0x10a [11732.111974] ? __hrtimer_run_queues+0x1c2/0x4d0 [11732.111978] ? watchdog+0x40/0x40 [11732.111983] ? hrtimer_interrupt+0x128/0x2c0 [11732.111987] ? smp_apic_timer_interrupt+0xa6/0x270 [11732.111995] ? apic_timer_interrupt+0xf/0x20 [11732.111999] [11732.112002] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11732.112007] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11732.112015] do_raw_spin_lock+0x177/0x190 [11732.112020] _raw_spin_lock+0x21/0x30 [11732.112025] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11732.121307] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11732.121855] after_reply+0xe2c/0x1780 [ptlrpc] [11732.122116] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11732.122337] ? __next_timer_interrupt+0x160/0x160 [11732.122349] ptlrpcd+0xa82/0xe40 [ptlrpc] [11732.122839] ? do_wait_intr+0xf0/0xf0 [11732.122849] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11732.123065] kthread+0x1d1/0x200 [11732.123070] ? set_kthread_struct+0x70/0x70 [11732.123074] ret_from_fork+0x1f/0x30 [11732.434145] process_one_work+0x2c8/0x700 [11732.435252] worker_thread+0x296/0x6e0 [11732.437002] ? rescuer_thread+0x570/0x570 [11732.438869] kthread+0x1d1/0x200 [11732.440827] ? set_kthread_struct+0x70/0x70 [11732.442847] ret_from_fork+0x1f/0x30 [11736.052088] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [11736.054992] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11736.094149] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11736.098663] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11736.107561] RIP: 0010:native_safe_halt+0x2e/0x50 [11736.110063] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11736.136434] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11736.140438] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11736.145989] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [11736.149408] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [11736.152604] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [11736.155671] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [11736.159084] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [11736.164672] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11736.166443] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [11736.169732] Call Trace: [11736.171784] [11736.172623] ? show_regs.cold.9+0x22/0x2f [11736.174965] ? watchdog_timer_fn.cold.6+0x62/0x10a [11736.179012] ? __hrtimer_run_queues+0x1c2/0x4d0 [11736.181389] ? watchdog+0x40/0x40 [11736.182499] ? hrtimer_interrupt+0x128/0x2c0 [11736.184193] ? smp_apic_timer_interrupt+0xa6/0x270 [11736.186587] ? apic_timer_interrupt+0xf/0x20 [11736.188298] [11736.190695] ? native_safe_halt+0x2e/0x50 [11736.192500] kvm_wait+0xcd/0xe0 [11736.194137] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11736.196712] do_raw_spin_lock+0x177/0x190 [11736.200109] _raw_spin_lock+0x21/0x30 [11736.203681] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11736.214449] ? do_raw_spin_unlock+0x75/0x190 [11736.217539] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11736.222464] ? do_wait_intr+0xf0/0xf0 [11736.226478] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11736.229592] mdc_reint+0x2d/0x110 [mdc] [11736.231669] mdc_unlink+0x658/0x870 [mdc] [11736.233390] lmv_unlink+0x34d/0xbd0 [lmv] [11736.235556] ll_unlink+0x24d/0x7f0 [lustre] [11736.237341] vfs_unlink+0x202/0x330 [11736.238361] do_unlinkat+0x2fa/0x460 [11736.239254] __x64_sys_unlink+0x37/0x50 [11736.240153] do_syscall_64+0xc1/0x440 [11736.241669] entry_SYSCALL_64_after_hwframe+0x49/0xae [11736.243735] RIP: 0033:0x7f151bdee10b [11736.245280] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [11736.252654] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [11736.255728] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [11736.259693] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [11736.262857] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11736.265776] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11736.269991] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11738.593574] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9807s! [11738.598764] Showing busy workqueues and worker pools: [11738.602564] workqueue events: flags=0x0 [11738.603987] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11738.612983] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [11738.624762] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11738.628949] pending: kfree_rcu_monitor [11738.630887] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11738.633748] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [11738.638178] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11738.645093] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [11738.650331] workqueue events_long: flags=0x0 [11738.652986] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11738.660557] pending: lu_objects_flush [obdclass] [11738.664583] workqueue events_unbound: flags=0x2 [11738.668554] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11738.673454] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11738.680415] workqueue events_power_efficient: flags=0x80 [11738.682725] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11738.687367] pending: do_cache_clean, neigh_periodic_work, check_lifetime [11738.693538] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11738.699041] pending: neigh_periodic_work [11738.702284] workqueue mm_percpu_wq: flags=0x8 [11738.704033] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11738.707321] pending: vmstat_update, lru_add_drain_per_cpu [11738.710076] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11738.714104] pending: vmstat_update [11738.715758] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11738.719358] pending: lru_add_drain_per_cpu [11738.721129] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [11738.726119] pending: lru_add_drain_per_cpu BAR(48) [11738.731130] workqueue writeback: flags=0x4a [11738.733217] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [11738.737069] pending: wb_workfn [11738.738588] workqueue xprtiod: flags=0xa [11738.741575] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11738.743803] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [11738.747391] workqueue ipv6_addrconf: flags=0x40008 [11738.749575] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11738.751650] pending: addrconf_verify_work [11738.754908] workqueue knbd0-recv: flags=0x1a [11738.756079] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11738.758528] in-flight: 658:recv_work [11738.762334] workqueue ptlrpc_pinger: flags=0x2 [11738.765079] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11738.768334] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11738.770449] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9808s workers=4 idle: 32984 37 8 [11738.783095] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11723s workers=2 idle: 661 [11748.057109] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [11748.060418] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11748.085629] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11748.091579] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11748.096535] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11748.099646] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11748.109779] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11748.113879] RAX: 00000000000072a3 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11748.118367] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [11748.123022] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [11748.127226] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [11748.131325] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [11748.135676] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [11748.140339] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11748.144280] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [11748.149520] Call Trace: [11748.150899] [11748.151848] ? show_regs.cold.9+0x22/0x2f [11748.154523] ? watchdog_timer_fn.cold.6+0x62/0x10a [11748.157373] ? __hrtimer_run_queues+0x1c2/0x4d0 [11748.159906] ? watchdog+0x40/0x40 [11748.162383] ? hrtimer_interrupt+0x128/0x2c0 [11748.165528] ? smp_apic_timer_interrupt+0xa6/0x270 [11748.168793] ? apic_timer_interrupt+0xf/0x20 [11748.171428] [11748.173617] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11748.176662] do_raw_spin_lock+0x177/0x190 [11748.179230] _raw_spin_lock+0x21/0x30 [11748.181351] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11748.185596] ? do_raw_spin_unlock+0x75/0x190 [11748.188716] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11748.191783] ? do_wait_intr+0xf0/0xf0 [11748.194375] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11748.197645] mdc_close+0x53b/0xd70 [mdc] [11748.200157] lmv_close+0x18d/0x490 [lmv] [11748.202809] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11748.205559] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11748.208428] ll_md_real_close+0xa5/0x2a0 [lustre] [11748.211405] ll_md_close+0x274/0x800 [lustre] [11748.213909] ll_file_release+0x128/0xc50 [lustre] [11748.216712] __fput+0x10d/0x440 [11748.218680] ____fput+0x12/0x20 [11748.220509] task_work_run+0xc6/0x110 [11748.222414] exit_to_usermode_loop+0x1dd/0x1f0 [11748.224545] do_syscall_64+0x3d6/0x440 [11748.226423] entry_SYSCALL_64_after_hwframe+0x49/0xae [11748.228734] RIP: 0033:0x7f151bee3cd8 [11748.230492] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11748.239447] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11748.243242] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [11748.246036] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [11748.248575] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11748.251708] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11748.255265] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11760.057587] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11760.059608] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [11760.066816] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11760.066907] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11760.066915] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11760.066921] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11760.067249] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11760.067261] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11760.067265] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11760.067269] RAX: 0000000000007586 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11760.067272] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [11760.067274] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [11760.067276] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [11760.067278] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [11760.067282] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [11760.067284] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11760.067287] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11760.067295] Call Trace: [11760.067301] [11760.067307] ? show_regs.cold.9+0x22/0x2f [11760.067316] ? watchdog_timer_fn.cold.6+0x62/0x10a [11760.067322] ? __hrtimer_run_queues+0x1c2/0x4d0 [11760.067326] ? watchdog+0x40/0x40 [11760.067331] ? hrtimer_interrupt+0x128/0x2c0 [11760.067335] ? smp_apic_timer_interrupt+0xa6/0x270 [11760.067343] ? apic_timer_interrupt+0xf/0x20 [11760.067348] [11760.067351] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11760.067356] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11760.067361] do_raw_spin_lock+0x177/0x190 [11760.067366] _raw_spin_lock+0x21/0x30 [11760.067370] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11760.067595] process_one_work+0x2c8/0x700 [11760.067602] worker_thread+0x296/0x6e0 [11760.067607] ? rescuer_thread+0x570/0x570 [11760.067610] kthread+0x1d1/0x200 [11760.067614] ? set_kthread_struct+0x70/0x70 [11760.067617] ret_from_fork+0x1f/0x30 [11760.452491] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11760.517642] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11760.532300] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11760.541282] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11760.544553] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11760.560443] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11760.575565] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [11760.590584] RDX: 0000000000003652 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [11760.603412] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [11760.618541] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [11760.623157] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11760.628810] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [11760.637656] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11760.646384] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11760.654716] Call Trace: [11760.655888] [11760.657705] ? show_regs.cold.9+0x22/0x2f [11760.665536] ? watchdog_timer_fn.cold.6+0x62/0x10a [11760.670560] ? __hrtimer_run_queues+0x1c2/0x4d0 [11760.672577] ? watchdog+0x40/0x40 [11760.675568] ? hrtimer_interrupt+0x128/0x2c0 [11760.678146] ? smp_apic_timer_interrupt+0xa6/0x270 [11760.680527] ? apic_timer_interrupt+0xf/0x20 [11760.681829] [11760.682673] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11760.685794] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11760.690396] do_raw_spin_lock+0x177/0x190 [11760.692932] _raw_spin_lock+0x21/0x30 [11760.695323] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11760.700388] obd_mod_free+0x46/0x140 [ptlrpc] [11760.704334] mdc_commit_open+0x91/0xb0 [mdc] [11760.710593] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11760.712674] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11760.714416] after_reply+0xe2c/0x1780 [ptlrpc] [11760.717527] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11760.722122] ? __next_timer_interrupt+0x160/0x160 [11760.729522] ptlrpcd+0xa82/0xe40 [ptlrpc] [11760.735338] ? do_wait_intr+0xf0/0xf0 [11760.738692] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11760.747283] kthread+0x1d1/0x200 [11760.751617] ? set_kthread_struct+0x70/0x70 [11760.762343] ret_from_fork+0x1f/0x30 [11764.056252] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [11764.063752] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11764.131812] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11764.138045] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11764.142341] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11764.146267] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11764.154320] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11764.165984] RAX: 0000000000003a86 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11764.175372] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [11764.187746] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [11764.198196] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [11764.205912] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [11764.212867] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [11764.224754] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11764.226756] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [11764.234964] Call Trace: [11764.238788] [11764.242025] ? show_regs.cold.9+0x22/0x2f [11764.247802] ? watchdog_timer_fn.cold.6+0x62/0x10a [11764.257445] ? __hrtimer_run_queues+0x1c2/0x4d0 [11764.266759] ? watchdog+0x40/0x40 [11764.267914] ? hrtimer_interrupt+0x128/0x2c0 [11764.278306] ? smp_apic_timer_interrupt+0xa6/0x270 [11764.284911] ? apic_timer_interrupt+0xf/0x20 [11764.288665] [11764.289667] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11764.293735] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11764.296290] do_raw_spin_lock+0x177/0x190 [11764.300272] _raw_spin_lock+0x21/0x30 [11764.302217] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11764.307920] ? do_raw_spin_unlock+0x75/0x190 [11764.310126] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11764.316248] ? do_wait_intr+0xf0/0xf0 [11764.320043] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11764.321700] mdc_reint+0x2d/0x110 [mdc] [11764.327805] mdc_unlink+0x658/0x870 [mdc] [11764.329838] lmv_unlink+0x34d/0xbd0 [lmv] [11764.333426] ll_unlink+0x24d/0x7f0 [lustre] [11764.338456] vfs_unlink+0x202/0x330 [11764.341459] do_unlinkat+0x2fa/0x460 [11764.343826] __x64_sys_unlink+0x37/0x50 [11764.347813] do_syscall_64+0xc1/0x440 [11764.350660] entry_SYSCALL_64_after_hwframe+0x49/0xae [11764.352205] RIP: 0033:0x7f151bdee10b [11764.358284] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [11764.386419] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [11764.394610] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [11764.402450] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [11764.409343] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11764.416383] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11764.423222] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11768.801822] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9838s! [11768.817683] Showing busy workqueues and worker pools: [11768.825241] workqueue events: flags=0x0 [11768.832140] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11768.844231] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [11768.853322] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11768.859595] pending: kfree_rcu_monitor [11768.861568] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11768.870205] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [11768.877796] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11768.888949] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [11768.897441] workqueue events_long: flags=0x0 [11768.901555] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11768.908539] pending: lu_objects_flush [obdclass] [11768.914667] workqueue events_unbound: flags=0x2 [11768.919456] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11768.923821] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11768.936184] workqueue events_power_efficient: flags=0x80 [11768.938968] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11768.944615] pending: do_cache_clean, neigh_periodic_work, check_lifetime [11768.948073] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11768.951273] pending: neigh_periodic_work [11768.953188] workqueue mm_percpu_wq: flags=0x8 [11768.955210] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11768.958541] pending: vmstat_update, lru_add_drain_per_cpu [11768.961116] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11768.977893] pending: vmstat_update [11768.979396] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11768.984838] pending: lru_add_drain_per_cpu [11768.989854] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [11768.996462] pending: lru_add_drain_per_cpu BAR(48) [11769.004874] workqueue writeback: flags=0x4a [11769.008106] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [11769.017354] pending: wb_workfn [11769.026672] workqueue xprtiod: flags=0xa [11769.033128] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11769.041344] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [11769.056303] workqueue ipv6_addrconf: flags=0x40008 [11769.075442] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11769.080444] pending: addrconf_verify_work [11769.085354] workqueue knbd0-recv: flags=0x1a [11769.096810] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11769.101644] in-flight: 658:recv_work [11769.112861] workqueue ptlrpc_pinger: flags=0x2 [11769.114183] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11769.123657] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11769.129651] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9838s workers=4 idle: 32984 37 8 [11769.132630] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11753s workers=2 idle: 661 [11776.057245] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [11776.062182] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11776.104171] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11776.109750] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11776.115039] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11776.117509] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11776.125279] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11776.128201] RAX: 0000000000005a93 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11776.130791] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [11776.133535] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [11776.136228] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [11776.138962] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [11776.142083] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [11776.145655] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11776.148249] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [11776.151720] Call Trace: [11776.152847] [11776.153753] ? show_regs.cold.9+0x22/0x2f [11776.155770] ? watchdog_timer_fn.cold.6+0x62/0x10a [11776.158325] ? __hrtimer_run_queues+0x1c2/0x4d0 [11776.160079] ? watchdog+0x40/0x40 [11776.161159] ? hrtimer_interrupt+0x128/0x2c0 [11776.163261] ? smp_apic_timer_interrupt+0xa6/0x270 [11776.165251] ? apic_timer_interrupt+0xf/0x20 [11776.167614] [11776.169249] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11776.171760] do_raw_spin_lock+0x177/0x190 [11776.174221] _raw_spin_lock+0x21/0x30 [11776.175998] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11776.179021] ? do_raw_spin_unlock+0x75/0x190 [11776.181055] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11776.184096] ? do_wait_intr+0xf0/0xf0 [11776.186119] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11776.188773] mdc_close+0x53b/0xd70 [mdc] [11776.190753] lmv_close+0x18d/0x490 [lmv] [11776.192682] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11776.195572] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11776.198317] ll_md_real_close+0xa5/0x2a0 [lustre] [11776.200640] ll_md_close+0x274/0x800 [lustre] [11776.205521] ll_file_release+0x128/0xc50 [lustre] [11776.209129] __fput+0x10d/0x440 [11776.211332] ____fput+0x12/0x20 [11776.213575] task_work_run+0xc6/0x110 [11776.216502] exit_to_usermode_loop+0x1dd/0x1f0 [11776.218670] do_syscall_64+0x3d6/0x440 [11776.219851] entry_SYSCALL_64_after_hwframe+0x49/0xae [11776.221475] RIP: 0033:0x7f151bee3cd8 [11776.222681] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11776.228126] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11776.230206] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [11776.236697] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [11776.239861] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11776.243280] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11776.246043] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11788.055932] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [11788.066723] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11788.068699] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11788.068788] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11788.068795] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11788.068799] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11788.068814] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11788.068818] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11788.068822] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [11788.068825] RDX: 000000000000614c RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [11788.068836] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [11788.068839] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [11788.068841] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11788.068844] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [11788.068847] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11788.068849] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11788.068857] Call Trace: [11788.069351] [11788.069359] ? show_regs.cold.9+0x22/0x2f [11788.069369] ? watchdog_timer_fn.cold.6+0x62/0x10a [11788.069375] ? __hrtimer_run_queues+0x1c2/0x4d0 [11788.069379] ? watchdog+0x40/0x40 [11788.069383] ? hrtimer_interrupt+0x128/0x2c0 [11788.069387] ? smp_apic_timer_interrupt+0xa6/0x270 [11788.069396] ? apic_timer_interrupt+0xf/0x20 [11788.069400] [11788.069403] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11788.069408] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11788.069413] do_raw_spin_lock+0x177/0x190 [11788.069418] _raw_spin_lock+0x21/0x30 [11788.069425] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11788.069753] obd_mod_free+0x46/0x140 [ptlrpc] [11788.070093] mdc_commit_open+0x91/0xb0 [mdc] [11788.070131] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11788.073380] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11788.073633] after_reply+0xe2c/0x1780 [ptlrpc] [11788.075198] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11788.076643] ? __next_timer_interrupt+0x160/0x160 [11788.076655] ptlrpcd+0xa82/0xe40 [ptlrpc] [11788.076893] ? do_wait_intr+0xf0/0xf0 [11788.076904] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11788.078536] kthread+0x1d1/0x200 [11788.078544] ? set_kthread_struct+0x70/0x70 [11788.078548] ret_from_fork+0x1f/0x30 [11788.614151] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11788.694017] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11788.711378] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11788.724254] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11788.735056] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11788.744916] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11788.772106] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11788.783303] RAX: 0000000000007e88 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11788.797092] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [11788.803177] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [11788.812981] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [11788.847881] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [11788.865041] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [11788.874348] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11788.889378] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11788.899180] Call Trace: [11788.905366] [11788.906171] ? show_regs.cold.9+0x22/0x2f [11788.919930] ? watchdog_timer_fn.cold.6+0x62/0x10a [11788.938198] ? __hrtimer_run_queues+0x1c2/0x4d0 [11788.945137] ? watchdog+0x40/0x40 [11788.949983] ? hrtimer_interrupt+0x128/0x2c0 [11788.960216] ? smp_apic_timer_interrupt+0xa6/0x270 [11788.964953] ? apic_timer_interrupt+0xf/0x20 [11788.970571] [11788.975980] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11788.983059] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11788.989313] do_raw_spin_lock+0x177/0x190 [11788.996353] _raw_spin_lock+0x21/0x30 [11788.998921] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11789.009816] process_one_work+0x2c8/0x700 [11789.013716] worker_thread+0x296/0x6e0 [11789.015065] ? rescuer_thread+0x570/0x570 [11789.026595] kthread+0x1d1/0x200 [11789.027856] ? set_kthread_struct+0x70/0x70 [11789.033379] ret_from_fork+0x1f/0x30 [11796.056885] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [11796.058857] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11796.097119] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11796.111056] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11796.119447] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11796.123342] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11796.137159] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11796.143577] RAX: 0000000000003d89 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11796.149381] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [11796.153905] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [11796.157059] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [11796.167305] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [11796.172061] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [11796.180831] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11796.182964] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [11796.192347] Call Trace: [11796.193301] [11796.194627] ? show_regs.cold.9+0x22/0x2f [11796.196057] ? watchdog_timer_fn.cold.6+0x62/0x10a [11796.199400] ? __hrtimer_run_queues+0x1c2/0x4d0 [11796.204062] ? watchdog+0x40/0x40 [11796.205363] ? hrtimer_interrupt+0x128/0x2c0 [11796.210769] ? smp_apic_timer_interrupt+0xa6/0x270 [11796.215138] ? apic_timer_interrupt+0xf/0x20 [11796.219522] [11796.220350] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11796.223855] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11796.227688] do_raw_spin_lock+0x177/0x190 [11796.232088] _raw_spin_lock+0x21/0x30 [11796.238124] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11796.241227] ? do_raw_spin_unlock+0x75/0x190 [11796.243355] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11796.249544] ? do_wait_intr+0xf0/0xf0 [11796.253197] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11796.256476] mdc_reint+0x2d/0x110 [mdc] [11796.259116] mdc_unlink+0x658/0x870 [mdc] [11796.260646] lmv_unlink+0x34d/0xbd0 [lmv] [11796.263090] ll_unlink+0x24d/0x7f0 [lustre] [11796.267076] vfs_unlink+0x202/0x330 [11796.267996] do_unlinkat+0x2fa/0x460 [11796.270341] __x64_sys_unlink+0x37/0x50 [11796.271673] do_syscall_64+0xc1/0x440 [11796.272960] entry_SYSCALL_64_after_hwframe+0x49/0xae [11796.276333] RIP: 0033:0x7f151bdee10b [11796.278390] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [11796.295109] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [11796.299431] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [11796.303699] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [11796.306037] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11796.310783] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11796.314086] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11799.524645] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9868s! [11799.534439] Showing busy workqueues and worker pools: [11799.537292] workqueue events: flags=0x0 [11799.539430] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11799.544288] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [11799.555074] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11799.559179] pending: kfree_rcu_monitor [11799.561748] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11799.567534] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [11799.573275] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11799.579343] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [11799.584449] workqueue events_long: flags=0x0 [11799.591225] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11799.598121] pending: lu_objects_flush [obdclass] [11799.603231] workqueue events_unbound: flags=0x2 [11799.606162] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11799.609545] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11799.616030] workqueue events_power_efficient: flags=0x80 [11799.618828] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11799.622235] pending: do_cache_clean, neigh_periodic_work, check_lifetime [11799.626041] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11799.629464] pending: neigh_periodic_work [11799.631541] workqueue mm_percpu_wq: flags=0x8 [11799.636451] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11799.645455] pending: vmstat_update, lru_add_drain_per_cpu [11799.649239] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11799.653361] pending: vmstat_update [11799.655074] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11799.659035] pending: lru_add_drain_per_cpu [11799.661280] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [11799.671276] pending: lru_add_drain_per_cpu BAR(48) [11799.674272] workqueue writeback: flags=0x4a [11799.676378] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [11799.680406] pending: wb_workfn [11799.682808] workqueue xprtiod: flags=0xa [11799.684826] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11799.688037] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [11799.692815] workqueue ipv6_addrconf: flags=0x40008 [11799.695239] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11799.698366] pending: addrconf_verify_work [11799.700512] workqueue knbd0-recv: flags=0x1a [11799.702988] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11799.706058] in-flight: 658:recv_work [11799.707913] workqueue ptlrpc_pinger: flags=0x2 [11799.711446] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11799.715096] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11799.718011] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9869s workers=4 idle: 32984 37 8 [11799.725326] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11784s workers=2 idle: 661 [11804.057264] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [11804.063160] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11804.088260] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11804.102996] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11804.111646] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11804.116572] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11804.130523] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11804.134997] RAX: 000000000000237b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11804.139166] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [11804.143613] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [11804.148332] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [11804.153374] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [11804.157835] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [11804.164645] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11804.167978] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [11804.172966] Call Trace: [11804.174522] [11804.175752] ? show_regs.cold.9+0x22/0x2f [11804.178999] ? watchdog_timer_fn.cold.6+0x62/0x10a [11804.181459] ? __hrtimer_run_queues+0x1c2/0x4d0 [11804.183876] ? watchdog+0x40/0x40 [11804.185721] ? hrtimer_interrupt+0x128/0x2c0 [11804.187867] ? smp_apic_timer_interrupt+0xa6/0x270 [11804.189976] ? apic_timer_interrupt+0xf/0x20 [11804.191884] [11804.193088] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11804.195302] do_raw_spin_lock+0x177/0x190 [11804.197086] _raw_spin_lock+0x21/0x30 [11804.199277] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11804.202124] ? do_raw_spin_unlock+0x75/0x190 [11804.204254] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11804.206862] ? do_wait_intr+0xf0/0xf0 [11804.208467] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11804.211286] mdc_close+0x53b/0xd70 [mdc] [11804.213033] lmv_close+0x18d/0x490 [lmv] [11804.215480] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11804.218436] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11804.222810] ll_md_real_close+0xa5/0x2a0 [lustre] [11804.226023] ll_md_close+0x274/0x800 [lustre] [11804.229081] ll_file_release+0x128/0xc50 [lustre] [11804.231218] __fput+0x10d/0x440 [11804.233140] ____fput+0x12/0x20 [11804.234874] task_work_run+0xc6/0x110 [11804.237263] exit_to_usermode_loop+0x1dd/0x1f0 [11804.239828] do_syscall_64+0x3d6/0x440 [11804.243373] entry_SYSCALL_64_after_hwframe+0x49/0xae [11804.252885] RIP: 0033:0x7f151bee3cd8 [11804.256873] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11804.278342] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11804.286485] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [11804.294273] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [11804.301713] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11804.306281] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11804.315470] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11816.056059] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11816.058705] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11816.059394] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [11816.089120] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11816.089133] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11816.089139] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11816.089393] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11816.089404] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11816.089408] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11816.089413] RAX: 0000000000007189 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11816.089415] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [11816.089417] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [11816.089419] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [11816.089994] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [11816.089997] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [11816.090001] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11816.090003] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11816.090010] Call Trace: [11816.090015] [11816.090019] ? show_regs.cold.9+0x22/0x2f [11816.090029] ? watchdog_timer_fn.cold.6+0x62/0x10a [11816.090035] ? __hrtimer_run_queues+0x1c2/0x4d0 [11816.090039] ? watchdog+0x40/0x40 [11816.090091] ? hrtimer_interrupt+0x128/0x2c0 [11816.090095] ? smp_apic_timer_interrupt+0xa6/0x270 [11816.090102] ? apic_timer_interrupt+0xf/0x20 [11816.090107] [11816.090109] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11816.090114] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11816.090119] do_raw_spin_lock+0x177/0x190 [11816.090123] _raw_spin_lock+0x21/0x30 [11816.090128] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11816.090357] process_one_work+0x2c8/0x700 [11816.090365] worker_thread+0x296/0x6e0 [11816.090369] ? rescuer_thread+0x570/0x570 [11816.090372] kthread+0x1d1/0x200 [11816.090376] ? set_kthread_struct+0x70/0x70 [11816.090379] ret_from_fork+0x1f/0x30 [11816.250837] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11816.314349] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11816.325630] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11816.338980] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11816.344203] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11816.375148] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11816.387408] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [11816.397369] RDX: 0000000000003dde RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [11816.404311] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [11816.410135] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [11816.414383] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11816.417564] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [11816.421643] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11816.425037] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11816.428924] Call Trace: [11816.430075] [11816.431338] ? show_regs.cold.9+0x22/0x2f [11816.433364] ? watchdog_timer_fn.cold.6+0x62/0x10a [11816.435956] ? __hrtimer_run_queues+0x1c2/0x4d0 [11816.438061] ? watchdog+0x40/0x40 [11816.440640] ? hrtimer_interrupt+0x128/0x2c0 [11816.443137] ? smp_apic_timer_interrupt+0xa6/0x270 [11816.444443] ? apic_timer_interrupt+0xf/0x20 [11816.446086] [11816.448647] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11816.451140] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11816.452649] do_raw_spin_lock+0x177/0x190 [11816.453676] _raw_spin_lock+0x21/0x30 [11816.463355] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11816.465346] obd_mod_free+0x46/0x140 [ptlrpc] [11816.476931] mdc_commit_open+0x91/0xb0 [mdc] [11816.478100] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11816.479626] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11816.493155] after_reply+0xe2c/0x1780 [ptlrpc] [11816.500605] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11816.509535] ? __next_timer_interrupt+0x160/0x160 [11816.515146] ptlrpcd+0xa82/0xe40 [ptlrpc] [11816.520208] ? do_wait_intr+0xf0/0xf0 [11816.524168] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11816.531352] kthread+0x1d1/0x200 [11816.533168] ? set_kthread_struct+0x70/0x70 [11816.535205] ret_from_fork+0x1f/0x30 [11824.053397] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [11824.056087] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11824.111307] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11824.115017] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11824.122828] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11824.130257] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11824.145373] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [11824.158066] RAX: 000000000000477e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11824.167033] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [11824.183051] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [11824.185612] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [11824.198243] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [11824.203175] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [11824.208044] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11824.211251] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [11824.215789] Call Trace: [11824.220149] [11824.223280] ? show_regs.cold.9+0x22/0x2f [11824.226430] ? watchdog_timer_fn.cold.6+0x62/0x10a [11824.229116] ? __hrtimer_run_queues+0x1c2/0x4d0 [11824.235727] ? watchdog+0x40/0x40 [11824.237671] ? hrtimer_interrupt+0x128/0x2c0 [11824.240123] ? smp_apic_timer_interrupt+0xa6/0x270 [11824.245301] ? apic_timer_interrupt+0xf/0x20 [11824.248155] [11824.250604] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11824.252146] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11824.253521] do_raw_spin_lock+0x177/0x190 [11824.255839] _raw_spin_lock+0x21/0x30 [11824.257690] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11824.260202] ? do_raw_spin_unlock+0x75/0x190 [11824.262163] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11824.264643] ? do_wait_intr+0xf0/0xf0 [11824.266060] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11824.268238] mdc_reint+0x2d/0x110 [mdc] [11824.270140] mdc_unlink+0x658/0x870 [mdc] [11824.271953] lmv_unlink+0x34d/0xbd0 [lmv] [11824.273842] ll_unlink+0x24d/0x7f0 [lustre] [11824.275796] vfs_unlink+0x202/0x330 [11824.277129] do_unlinkat+0x2fa/0x460 [11824.284811] __x64_sys_unlink+0x37/0x50 [11824.288218] do_syscall_64+0xc1/0x440 [11824.291122] entry_SYSCALL_64_after_hwframe+0x49/0xae [11824.297288] RIP: 0033:0x7f151bdee10b [11824.300095] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [11824.316018] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [11824.326392] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [11824.334624] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [11824.338904] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11824.343864] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11824.350729] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11829.730203] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9899s! [11829.736619] Showing busy workqueues and worker pools: [11829.739695] workqueue events: flags=0x0 [11829.746827] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11829.753662] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [11829.761465] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11829.767410] pending: kfree_rcu_monitor [11829.771040] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11829.775051] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [11829.781377] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11829.784742] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [11829.792830] workqueue events_long: flags=0x0 [11829.796068] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11829.798055] pending: lu_objects_flush [obdclass] [11829.802469] workqueue events_unbound: flags=0x2 [11829.808286] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11829.814857] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11829.831716] workqueue events_power_efficient: flags=0x80 [11829.836533] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11829.838664] pending: do_cache_clean, neigh_periodic_work, check_lifetime [11829.843741] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11829.850243] pending: neigh_periodic_work [11829.852800] workqueue mm_percpu_wq: flags=0x8 [11829.858244] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11829.860161] pending: vmstat_update, lru_add_drain_per_cpu [11829.862008] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11829.866260] pending: vmstat_update [11829.867554] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11829.873088] pending: lru_add_drain_per_cpu [11829.874417] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [11829.881481] pending: lru_add_drain_per_cpu BAR(48) [11829.886629] workqueue writeback: flags=0x4a [11829.888616] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [11829.891514] pending: wb_workfn [11829.894174] workqueue xprtiod: flags=0xa [11829.896000] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11829.898660] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [11829.901680] workqueue ipv6_addrconf: flags=0x40008 [11829.904085] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11829.906996] pending: addrconf_verify_work [11829.908764] workqueue knbd0-recv: flags=0x1a [11829.910813] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11829.913731] in-flight: 658:recv_work [11829.916114] workqueue ptlrpc_pinger: flags=0x2 [11829.918013] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11829.921498] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11829.924072] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9899s workers=4 idle: 32984 37 8 [11829.927555] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11814s workers=2 idle: 661 [11832.059304] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [11832.066083] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11832.089636] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11832.094489] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11832.097967] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11832.100800] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11832.109028] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11832.113056] RAX: 0000000000001df8 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11832.118262] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [11832.121757] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [11832.125660] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [11832.137259] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [11832.150150] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [11832.161760] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11832.164786] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [11832.169083] Call Trace: [11832.170075] [11832.170761] ? show_regs.cold.9+0x22/0x2f [11832.171933] ? watchdog_timer_fn.cold.6+0x62/0x10a [11832.176946] ? __hrtimer_run_queues+0x1c2/0x4d0 [11832.179810] ? watchdog+0x40/0x40 [11832.183900] ? hrtimer_interrupt+0x128/0x2c0 [11832.186091] ? smp_apic_timer_interrupt+0xa6/0x270 [11832.188757] ? apic_timer_interrupt+0xf/0x20 [11832.190570] [11832.191761] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11832.194005] do_raw_spin_lock+0x177/0x190 [11832.196435] _raw_spin_lock+0x21/0x30 [11832.198062] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11832.200860] ? do_raw_spin_unlock+0x75/0x190 [11832.202549] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11832.204822] ? do_wait_intr+0xf0/0xf0 [11832.206258] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11832.209102] mdc_close+0x53b/0xd70 [mdc] [11832.210861] lmv_close+0x18d/0x490 [lmv] [11832.212803] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11832.221877] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11832.228949] ll_md_real_close+0xa5/0x2a0 [lustre] [11832.240853] ll_md_close+0x274/0x800 [lustre] [11832.244012] ll_file_release+0x128/0xc50 [lustre] [11832.251062] __fput+0x10d/0x440 [11832.253248] ____fput+0x12/0x20 [11832.257006] task_work_run+0xc6/0x110 [11832.259829] exit_to_usermode_loop+0x1dd/0x1f0 [11832.263881] do_syscall_64+0x3d6/0x440 [11832.266014] entry_SYSCALL_64_after_hwframe+0x49/0xae [11832.269071] RIP: 0033:0x7f151bee3cd8 [11832.272257] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11832.293582] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11832.299266] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [11832.304250] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [11832.312073] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11832.318085] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11832.330071] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11844.055034] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [11844.056948] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11844.058373] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11844.058448] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11844.058455] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11844.058458] RIP: 0010:native_safe_halt+0x2e/0x50 [11844.058471] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11844.058474] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11844.058479] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [11844.058481] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [11844.058483] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [11844.058485] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [11844.058487] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11844.058490] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [11844.058493] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11844.058495] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11844.058503] Call Trace: [11844.058508] [11844.058514] ? show_regs.cold.9+0x22/0x2f [11844.058524] ? watchdog_timer_fn.cold.6+0x62/0x10a [11844.058531] ? __hrtimer_run_queues+0x1c2/0x4d0 [11844.058535] ? watchdog+0x40/0x40 [11844.058540] ? hrtimer_interrupt+0x128/0x2c0 [11844.058544] ? smp_apic_timer_interrupt+0xa6/0x270 [11844.058552] ? apic_timer_interrupt+0xf/0x20 [11844.058557] [11844.058560] ? native_safe_halt+0x2e/0x50 [11844.058565] kvm_wait+0xcd/0xe0 [11844.058573] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11844.058579] do_raw_spin_lock+0x177/0x190 [11844.058583] _raw_spin_lock+0x21/0x30 [11844.058589] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11844.061912] Modules linked in: [11844.080423] obd_mod_free+0x46/0x140 [ptlrpc] [11844.080826] mdc_commit_open+0x91/0xb0 [mdc] [11844.080863] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11844.081081] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11844.081436] after_reply+0xe2c/0x1780 [ptlrpc] [11844.081663] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11844.082219] ? __next_timer_interrupt+0x160/0x160 [11844.082229] ptlrpcd+0xa82/0xe40 [ptlrpc] [11844.097266] lustre(O) [11844.099823] ? do_wait_intr+0xf0/0xf0 [11844.103642] osp(O) [11844.112941] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11844.123176] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11844.123256] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11844.127458] kthread+0x1d1/0x200 [11844.127490] ? set_kthread_struct+0x70/0x70 [11844.127495] ret_from_fork+0x1f/0x30 [11844.318293] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11844.322168] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11844.332056] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11844.338622] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11844.353992] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11844.357780] RAX: 0000000000007a9a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11844.361066] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [11844.365304] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [11844.371857] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [11844.374252] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [11844.376903] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [11844.380024] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11844.382253] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11844.385017] Call Trace: [11844.386556] [11844.387132] ? show_regs.cold.9+0x22/0x2f [11844.390719] ? watchdog_timer_fn.cold.6+0x62/0x10a [11844.395389] ? __hrtimer_run_queues+0x1c2/0x4d0 [11844.397054] ? watchdog+0x40/0x40 [11844.400394] ? hrtimer_interrupt+0x128/0x2c0 [11844.403423] ? smp_apic_timer_interrupt+0xa6/0x270 [11844.405423] ? apic_timer_interrupt+0xf/0x20 [11844.414060] [11844.414778] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11844.419604] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11844.421487] do_raw_spin_lock+0x177/0x190 [11844.424011] _raw_spin_lock+0x21/0x30 [11844.425769] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11844.430909] process_one_work+0x2c8/0x700 [11844.431832] worker_thread+0x296/0x6e0 [11844.432764] ? rescuer_thread+0x570/0x570 [11844.436633] kthread+0x1d1/0x200 [11844.437798] ? set_kthread_struct+0x70/0x70 [11844.440306] ret_from_fork+0x1f/0x30 [11856.052025] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [11856.054547] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11856.069099] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11856.072595] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11856.075404] RIP: 0010:native_safe_halt+0x2e/0x50 [11856.077278] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11856.082996] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11856.085360] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11856.087820] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [11856.090714] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [11856.092822] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [11856.095647] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [11856.098277] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [11856.100622] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11856.102820] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [11856.104818] Call Trace: [11856.105764] [11856.106538] ? show_regs.cold.9+0x22/0x2f [11856.108099] ? watchdog_timer_fn.cold.6+0x62/0x10a [11856.109598] ? __hrtimer_run_queues+0x1c2/0x4d0 [11856.110942] ? watchdog+0x40/0x40 [11856.112019] ? hrtimer_interrupt+0x128/0x2c0 [11856.113607] ? smp_apic_timer_interrupt+0xa6/0x270 [11856.115340] ? apic_timer_interrupt+0xf/0x20 [11856.116750] [11856.117402] ? native_safe_halt+0x2e/0x50 [11856.118755] kvm_wait+0xcd/0xe0 [11856.119654] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11856.121057] do_raw_spin_lock+0x177/0x190 [11856.122161] _raw_spin_lock+0x21/0x30 [11856.123198] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11856.124835] ? do_raw_spin_unlock+0x75/0x190 [11856.126106] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11856.128007] ? do_wait_intr+0xf0/0xf0 [11856.129450] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11856.131352] mdc_reint+0x2d/0x110 [mdc] [11856.132567] mdc_unlink+0x658/0x870 [mdc] [11856.134225] lmv_unlink+0x34d/0xbd0 [lmv] [11856.135773] ll_unlink+0x24d/0x7f0 [lustre] [11856.137266] vfs_unlink+0x202/0x330 [11856.138362] do_unlinkat+0x2fa/0x460 [11856.139484] __x64_sys_unlink+0x37/0x50 [11856.140783] do_syscall_64+0xc1/0x440 [11856.141816] entry_SYSCALL_64_after_hwframe+0x49/0xae [11856.143464] RIP: 0033:0x7f151bdee10b [11856.144617] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [11856.151707] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [11856.154444] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [11856.156925] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [11856.159514] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11856.161908] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11856.164739] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11859.937419] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9929s! [11859.941749] Showing busy workqueues and worker pools: [11859.944772] workqueue events: flags=0x0 [11859.947742] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11859.953010] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [11859.956803] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11859.961753] pending: kfree_rcu_monitor [11859.967898] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11859.970735] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [11859.982036] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11859.987564] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [11859.994320] workqueue events_long: flags=0x0 [11859.997274] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11860.003491] pending: lu_objects_flush [obdclass] [11860.010093] workqueue events_unbound: flags=0x2 [11860.014399] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11860.017542] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11860.029200] workqueue events_power_efficient: flags=0x80 [11860.035183] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11860.043272] pending: do_cache_clean, neigh_periodic_work, check_lifetime [11860.049270] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11860.053976] pending: neigh_periodic_work [11860.057378] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [11860.057780] workqueue mm_percpu_wq: flags=0x8 [11860.057785] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11860.057798] pending: vmstat_update, lru_add_drain_per_cpu [11860.057815] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11860.057822] pending: vmstat_update [11860.057828] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11860.057834] pending: lru_add_drain_per_cpu [11860.057840] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [11860.057846] pending: lru_add_drain_per_cpu BAR(48) [11860.057857] workqueue writeback: flags=0x4a [11860.057860] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [11860.057866] pending: wb_workfn [11860.057884] workqueue xprtiod: flags=0xa [11860.057886] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11860.057892] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [11860.057911] workqueue ipv6_addrconf: flags=0x40008 [11860.057914] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11860.057920] pending: addrconf_verify_work [11860.057931] workqueue knbd0-recv: flags=0x1a [11860.057933] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11860.057939] in-flight: 658:recv_work [11860.057948] workqueue ptlrpc_pinger: flags=0x2 [11860.057950] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11860.057954] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11860.058284] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9929s workers=4 idle: 32984 37 8 [11860.058298] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11844s workers=2 idle: 661 [11860.196414] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11860.236366] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11860.240829] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11860.244570] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11860.247514] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11860.257713] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11860.260969] RAX: 000000000000799c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11860.270277] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [11860.277074] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [11860.280183] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [11860.283087] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [11860.287832] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [11860.293085] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11860.296119] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [11860.300486] Call Trace: [11860.301750] [11860.302854] ? show_regs.cold.9+0x22/0x2f [11860.305068] ? watchdog_timer_fn.cold.6+0x62/0x10a [11860.307949] ? __hrtimer_run_queues+0x1c2/0x4d0 [11860.310630] ? watchdog+0x40/0x40 [11860.312909] ? hrtimer_interrupt+0x128/0x2c0 [11860.315501] ? smp_apic_timer_interrupt+0xa6/0x270 [11860.318154] ? apic_timer_interrupt+0xf/0x20 [11860.320385] [11860.321091] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11860.323985] do_raw_spin_lock+0x177/0x190 [11860.325763] _raw_spin_lock+0x21/0x30 [11860.327633] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11860.330655] ? do_raw_spin_unlock+0x75/0x190 [11860.333186] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11860.336100] ? do_wait_intr+0xf0/0xf0 [11860.337589] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11860.340177] mdc_close+0x53b/0xd70 [mdc] [11860.341845] lmv_close+0x18d/0x490 [lmv] [11860.342971] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11860.344672] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11860.346769] ll_md_real_close+0xa5/0x2a0 [lustre] [11860.348442] ll_md_close+0x274/0x800 [lustre] [11860.350047] ll_file_release+0x128/0xc50 [lustre] [11860.351778] __fput+0x10d/0x440 [11860.352773] ____fput+0x12/0x20 [11860.353874] task_work_run+0xc6/0x110 [11860.354970] exit_to_usermode_loop+0x1dd/0x1f0 [11860.356347] do_syscall_64+0x3d6/0x440 [11860.357529] entry_SYSCALL_64_after_hwframe+0x49/0xae [11860.359160] RIP: 0033:0x7f151bee3cd8 [11860.360489] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11860.365587] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11860.367496] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [11860.370101] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [11860.372275] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11860.374676] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11860.376860] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11872.055690] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [11872.056034] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11872.062033] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11872.062196] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11872.062205] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11872.062208] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11872.062223] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11872.062228] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11872.062233] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [11872.062235] RDX: 0000000000003a39 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [11872.062238] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [11872.062240] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [11872.062242] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11872.062246] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [11872.062248] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11872.062251] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11872.062259] Call Trace: [11872.062265] [11872.062270] ? show_regs.cold.9+0x22/0x2f [11872.062289] ? watchdog_timer_fn.cold.6+0x62/0x10a [11872.062296] ? __hrtimer_run_queues+0x1c2/0x4d0 [11872.062300] ? watchdog+0x40/0x40 [11872.062305] ? hrtimer_interrupt+0x128/0x2c0 [11872.062309] ? smp_apic_timer_interrupt+0xa6/0x270 [11872.062317] ? apic_timer_interrupt+0xf/0x20 [11872.062321] [11872.062324] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11872.062329] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11872.062334] do_raw_spin_lock+0x177/0x190 [11872.062339] _raw_spin_lock+0x21/0x30 [11872.062346] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11872.062634] obd_mod_free+0x46/0x140 [ptlrpc] [11872.067490] Modules linked in: lustre(O) [11872.095206] mdc_commit_open+0x91/0xb0 [mdc] [11872.104235] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) [11872.109655] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11872.112458] mgc(O) [11872.122701] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11872.123231] after_reply+0xe2c/0x1780 [ptlrpc] [11872.123473] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11872.124477] ? __next_timer_interrupt+0x160/0x160 [11872.124489] ptlrpcd+0xa82/0xe40 [ptlrpc] [11872.124725] ? do_wait_intr+0xf0/0xf0 [11872.124735] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11872.124978] kthread+0x1d1/0x200 [11872.124983] ? set_kthread_struct+0x70/0x70 [11872.124987] ret_from_fork+0x1f/0x30 [11872.253466] mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11872.274195] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11872.282989] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11872.289534] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11872.298145] RIP: 0010:native_safe_halt+0x2e/0x50 [11872.300219] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11872.317282] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11872.323638] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11872.331569] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [11872.340218] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [11872.347824] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [11872.356796] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [11872.370550] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [11872.381489] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11872.387619] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11872.398302] Call Trace: [11872.399393] [11872.401335] ? show_regs.cold.9+0x22/0x2f [11872.405564] ? watchdog_timer_fn.cold.6+0x62/0x10a [11872.407002] ? __hrtimer_run_queues+0x1c2/0x4d0 [11872.415121] ? watchdog+0x40/0x40 [11872.416295] ? hrtimer_interrupt+0x128/0x2c0 [11872.428220] ? smp_apic_timer_interrupt+0xa6/0x270 [11872.432522] ? apic_timer_interrupt+0xf/0x20 [11872.439800] [11872.442737] ? native_safe_halt+0x2e/0x50 [11872.448726] kvm_wait+0xcd/0xe0 [11872.452229] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11872.459987] do_raw_spin_lock+0x177/0x190 [11872.462376] _raw_spin_lock+0x21/0x30 [11872.468142] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11872.473227] process_one_work+0x2c8/0x700 [11872.481530] worker_thread+0x296/0x6e0 [11872.488036] ? rescuer_thread+0x570/0x570 [11872.496022] kthread+0x1d1/0x200 [11872.499308] ? set_kthread_struct+0x70/0x70 [11872.505910] ret_from_fork+0x1f/0x30 [11884.862011] rcu: INFO: rcu_sched self-detected stall on CPU [11884.868524] rcu: 1-...!: (5740988 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [11884.876493] (t=9960215 jiffies g=249657 q=5318) [11884.878541] rcu: rcu_sched kthread starved for 9955167 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [11884.882384] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [11884.885848] rcu: RCU grace-period kthread stack dump: [11884.887969] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [11884.891819] Call Trace: [11884.892622] __schedule+0x351/0xcb0 [11884.893952] schedule+0xc0/0x180 [11884.894998] schedule_timeout+0xb4/0x190 [11884.896481] ? do_raw_spin_unlock+0x75/0x190 [11884.898516] ? __next_timer_interrupt+0x160/0x160 [11884.900041] ? prepare_to_swait_event+0xf3/0x140 [11884.901797] rcu_gp_kthread+0x8dd/0x15b0 [11884.903476] ? force_qs_rnp+0x2c0/0x2c0 [11884.904852] kthread+0x1d1/0x200 [11884.906130] ? set_kthread_struct+0x70/0x70 [11884.907876] ret_from_fork+0x1f/0x30 [11884.909370] rcu: Stack dump where RCU GP kthread last ran: [11884.911997] Sending NMI from CPU 1 to CPUs 3: [11884.915062] NMI backtrace for cpu 3 [11884.915072] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11884.915074] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11884.915076] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11884.915078] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11884.915080] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 [11884.915083] RAX: 000000000000412e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11884.915084] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [11884.915086] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [11884.915087] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [11884.915089] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [11884.915094] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [11884.915095] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11884.915097] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [11884.915098] Call Trace: [11884.915099] [11884.915100] ? show_regs.cold.9+0x22/0x2f [11884.915101] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [11884.915102] ? nmi_cpu_backtrace_handler+0x15/0x40 [11884.915103] ? nmi_handle+0xbd/0x210 [11884.915105] ? default_do_nmi+0x4b/0x1b0 [11884.915106] ? do_nmi+0x280/0x380 [11884.915107] ? end_repeat_nmi+0x16/0x1f [11884.915108] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11884.915110] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11884.915113] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11884.915115] [11884.915116] do_raw_spin_lock+0x177/0x190 [11884.915117] _raw_spin_lock+0x21/0x30 [11884.915118] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11884.915119] ? do_raw_spin_unlock+0x75/0x190 [11884.915120] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11884.915121] ? do_wait_intr+0xf0/0xf0 [11884.915123] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11884.915124] mdc_close+0x53b/0xd70 [mdc] [11884.915125] lmv_close+0x18d/0x490 [lmv] [11884.915126] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11884.915127] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11884.915128] ll_md_real_close+0xa5/0x2a0 [lustre] [11884.915128] ll_md_close+0x274/0x800 [lustre] [11884.915129] ll_file_release+0x128/0xc50 [lustre] [11884.915130] __fput+0x10d/0x440 [11884.915131] ____fput+0x12/0x20 [11884.915132] task_work_run+0xc6/0x110 [11884.915132] exit_to_usermode_loop+0x1dd/0x1f0 [11884.915133] do_syscall_64+0x3d6/0x440 [11884.915134] entry_SYSCALL_64_after_hwframe+0x49/0xae [11884.915135] RIP: 0033:0x7f151bee3cd8 [11884.915136] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11884.915141] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11884.915144] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [11884.915145] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [11884.915147] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11884.915148] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11884.915149] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11884.918617] NMI backtrace for cpu 1 [11885.034321] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11885.038690] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11885.041966] Call Trace: [11885.042869] [11885.043636] ? dump_stack+0xbb/0x10e [11885.045196] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [11885.047377] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [11885.049040] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [11885.051227] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [11885.053017] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [11885.055015] ? rcu_dump_cpu_stacks+0x11b/0x171 [11885.056488] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [11885.058736] ? update_process_times+0x6d/0xc0 [11885.060384] ? tick_sched_handle.isra.18+0x27/0x90 [11885.061965] ? tick_sched_timer+0x53/0xb0 [11885.063203] ? __hrtimer_run_queues+0x1c2/0x4d0 [11885.064564] ? tick_sched_handle.isra.18+0x90/0x90 [11885.066127] ? hrtimer_interrupt+0x128/0x2c0 [11885.067985] ? smp_apic_timer_interrupt+0xa6/0x270 [11885.069488] ? apic_timer_interrupt+0xf/0x20 [11885.070643] [11885.071389] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11885.073161] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11885.074932] ? do_raw_spin_lock+0x177/0x190 [11885.076168] ? _raw_spin_lock+0x21/0x30 [11885.077281] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11885.079795] ? obd_mod_free+0x46/0x140 [ptlrpc] [11885.081672] ? mdc_commit_open+0x91/0xb0 [mdc] [11885.083558] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11885.086038] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11885.088260] ? after_reply+0xe2c/0x1780 [ptlrpc] [11885.090711] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11885.093052] ? __next_timer_interrupt+0x160/0x160 [11885.095020] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [11885.097251] ? do_wait_intr+0xf0/0xf0 [11885.098927] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11885.101271] ? kthread+0x1d1/0x200 [11885.102803] ? set_kthread_struct+0x70/0x70 [11885.104479] ? ret_from_fork+0x1f/0x30 [11888.052531] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [11888.057134] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11888.082374] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11888.087437] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11888.093247] RIP: 0010:native_safe_halt+0x2e/0x50 [11888.094881] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11888.111381] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11888.116785] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11888.119667] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [11888.123059] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [11888.129885] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [11888.134339] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [11888.139191] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [11888.144209] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11888.156598] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [11888.164702] Call Trace: [11888.169183] [11888.173747] ? show_regs.cold.9+0x22/0x2f [11888.178603] ? watchdog_timer_fn.cold.6+0x62/0x10a [11888.184146] ? __hrtimer_run_queues+0x1c2/0x4d0 [11888.192606] ? watchdog+0x40/0x40 [11888.198686] ? hrtimer_interrupt+0x128/0x2c0 [11888.202699] ? smp_apic_timer_interrupt+0xa6/0x270 [11888.207038] ? apic_timer_interrupt+0xf/0x20 [11888.209223] [11888.214872] ? native_safe_halt+0x2e/0x50 [11888.222102] kvm_wait+0xcd/0xe0 [11888.228846] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11888.239281] do_raw_spin_lock+0x177/0x190 [11888.242628] _raw_spin_lock+0x21/0x30 [11888.247053] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11888.251388] ? do_raw_spin_unlock+0x75/0x190 [11888.253366] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11888.256543] ? do_wait_intr+0xf0/0xf0 [11888.258603] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11888.265661] mdc_reint+0x2d/0x110 [mdc] [11888.269368] mdc_unlink+0x658/0x870 [mdc] [11888.273198] lmv_unlink+0x34d/0xbd0 [lmv] [11888.280450] ll_unlink+0x24d/0x7f0 [lustre] [11888.285539] vfs_unlink+0x202/0x330 [11888.289832] do_unlinkat+0x2fa/0x460 [11888.291200] __x64_sys_unlink+0x37/0x50 [11888.296840] do_syscall_64+0xc1/0x440 [11888.303778] entry_SYSCALL_64_after_hwframe+0x49/0xae [11888.306885] RIP: 0033:0x7f151bdee10b [11888.310155] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [11888.340977] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [11888.350018] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [11888.356192] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [11888.361926] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11888.372070] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11888.379917] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11890.147677] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9959s! [11890.155245] Showing busy workqueues and worker pools: [11890.166926] workqueue events: flags=0x0 [11890.174898] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11890.185003] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [11890.202554] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11890.218018] pending: kfree_rcu_monitor [11890.225442] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11890.237317] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [11890.246059] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11890.257028] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [11890.266142] workqueue events_long: flags=0x0 [11890.271040] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11890.280095] pending: lu_objects_flush [obdclass] [11890.287197] workqueue events_unbound: flags=0x2 [11890.289575] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11890.294254] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11890.300580] workqueue events_power_efficient: flags=0x80 [11890.303611] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11890.310559] pending: do_cache_clean, neigh_periodic_work, check_lifetime [11890.316468] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11890.320267] pending: neigh_periodic_work [11890.323250] workqueue mm_percpu_wq: flags=0x8 [11890.328472] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11890.339803] pending: vmstat_update, lru_add_drain_per_cpu [11890.348380] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11890.360990] pending: vmstat_update [11890.367141] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11890.381181] pending: lru_add_drain_per_cpu [11890.386577] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [11890.392329] pending: lru_add_drain_per_cpu BAR(48) [11890.396506] workqueue writeback: flags=0x4a [11890.399873] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [11890.403548] pending: wb_workfn [11890.405495] workqueue xprtiod: flags=0xa [11890.408278] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11890.411422] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [11890.416287] workqueue ipv6_addrconf: flags=0x40008 [11890.418886] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11890.422653] pending: addrconf_verify_work [11890.426182] workqueue knbd0-recv: flags=0x1a [11890.428384] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11890.432782] in-flight: 658:recv_work [11890.435353] workqueue ptlrpc_pinger: flags=0x2 [11890.437934] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11890.441523] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11890.445237] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9959s workers=4 idle: 32984 37 8 [11890.449889] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11875s workers=2 idle: 661 [11900.056030] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11900.060375] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11900.098234] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11900.107535] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11900.117345] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11900.122552] RIP: 0010:native_safe_halt+0x2e/0x50 [11900.125998] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11900.146554] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11900.158948] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11900.164196] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [11900.173584] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [11900.178332] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [11900.183195] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [11900.188024] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [11900.195082] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11900.199475] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11900.204239] Call Trace: [11900.207296] [11900.208186] ? show_regs.cold.9+0x22/0x2f [11900.211117] ? watchdog_timer_fn.cold.6+0x62/0x10a [11900.220513] ? __hrtimer_run_queues+0x1c2/0x4d0 [11900.227429] ? watchdog+0x40/0x40 [11900.231068] ? hrtimer_interrupt+0x128/0x2c0 [11900.232656] ? smp_apic_timer_interrupt+0xa6/0x270 [11900.238925] ? apic_timer_interrupt+0xf/0x20 [11900.241998] [11900.244197] ? native_safe_halt+0x2e/0x50 [11900.246524] kvm_wait+0xcd/0xe0 [11900.248464] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11900.251340] do_raw_spin_lock+0x177/0x190 [11900.253393] _raw_spin_lock+0x21/0x30 [11900.255755] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11900.257856] process_one_work+0x2c8/0x700 [11900.260399] worker_thread+0x296/0x6e0 [11900.263027] ? rescuer_thread+0x570/0x570 [11900.265473] kthread+0x1d1/0x200 [11900.267885] ? set_kthread_struct+0x70/0x70 [11900.270563] ret_from_fork+0x1f/0x30 [11912.055703] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [11912.060248] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11912.060875] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [11912.117959] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11912.117975] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11912.117979] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11912.117993] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11912.118248] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11912.118254] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [11912.118257] RDX: 00000000000050ae RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [11912.118259] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [11912.118261] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [11912.118263] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11912.118266] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [11912.118273] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11912.118275] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11912.118283] Call Trace: [11912.118288] [11912.118294] ? show_regs.cold.9+0x22/0x2f [11912.118365] ? watchdog_timer_fn.cold.6+0x62/0x10a [11912.118371] ? __hrtimer_run_queues+0x1c2/0x4d0 [11912.118376] ? watchdog+0x40/0x40 [11912.118379] ? hrtimer_interrupt+0x128/0x2c0 [11912.118384] ? smp_apic_timer_interrupt+0xa6/0x270 [11912.118393] ? apic_timer_interrupt+0xf/0x20 [11912.118398] [11912.118401] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11912.118406] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11912.118410] do_raw_spin_lock+0x177/0x190 [11912.118415] _raw_spin_lock+0x21/0x30 [11912.118422] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11912.118915] obd_mod_free+0x46/0x140 [ptlrpc] [11912.119111] mdc_commit_open+0x91/0xb0 [mdc] [11912.119148] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11912.119755] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11912.120108] after_reply+0xe2c/0x1780 [ptlrpc] [11912.120725] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11912.121183] ? __next_timer_interrupt+0x160/0x160 [11912.121194] ptlrpcd+0xa82/0xe40 [ptlrpc] [11912.121443] ? do_wait_intr+0xf0/0xf0 [11912.121453] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11912.121938] kthread+0x1d1/0x200 [11912.121945] ? set_kthread_struct+0x70/0x70 [11912.121949] ret_from_fork+0x1f/0x30 [11912.425153] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11912.447699] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11912.453955] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11912.458857] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11912.461137] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11912.470986] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11912.475860] RAX: 0000000000001fbe RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11912.480943] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [11912.486118] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [11912.489816] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [11912.493024] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [11912.496181] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [11912.499749] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11912.502228] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [11912.505924] Call Trace: [11912.507055] [11912.508054] ? show_regs.cold.9+0x22/0x2f [11912.509971] ? watchdog_timer_fn.cold.6+0x62/0x10a [11912.514429] ? __hrtimer_run_queues+0x1c2/0x4d0 [11912.521215] ? watchdog+0x40/0x40 [11912.525357] ? hrtimer_interrupt+0x128/0x2c0 [11912.529119] ? smp_apic_timer_interrupt+0xa6/0x270 [11912.532930] ? apic_timer_interrupt+0xf/0x20 [11912.543144] [11912.544848] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11912.557147] do_raw_spin_lock+0x177/0x190 [11912.560223] _raw_spin_lock+0x21/0x30 [11912.563806] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11912.567120] ? do_raw_spin_unlock+0x75/0x190 [11912.569820] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11912.575182] ? do_wait_intr+0xf0/0xf0 [11912.576958] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11912.588652] mdc_close+0x53b/0xd70 [mdc] [11912.593452] lmv_close+0x18d/0x490 [lmv] [11912.599958] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11912.614608] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11912.629754] ll_md_real_close+0xa5/0x2a0 [lustre] [11912.638097] ll_md_close+0x274/0x800 [lustre] [11912.646493] ll_file_release+0x128/0xc50 [lustre] [11912.652548] __fput+0x10d/0x440 [11912.666144] ____fput+0x12/0x20 [11912.687102] task_work_run+0xc6/0x110 [11912.702580] exit_to_usermode_loop+0x1dd/0x1f0 [11912.706429] do_syscall_64+0x3d6/0x440 [11912.717172] entry_SYSCALL_64_after_hwframe+0x49/0xae [11912.721409] RIP: 0033:0x7f151bee3cd8 [11912.729752] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11912.748967] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11912.755960] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [11912.770154] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [11912.777417] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11912.781182] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11912.785839] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11916.052142] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [11916.056371] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11916.078526] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11916.083779] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11916.089300] RIP: 0010:native_safe_halt+0x2e/0x50 [11916.091210] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11916.099357] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11916.104094] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11916.109924] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [11916.113213] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [11916.116981] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [11916.122324] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [11916.125397] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [11916.129142] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11916.133642] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [11916.137923] Call Trace: [11916.138895] [11916.140039] ? show_regs.cold.9+0x22/0x2f [11916.142088] ? watchdog_timer_fn.cold.6+0x62/0x10a [11916.145804] ? __hrtimer_run_queues+0x1c2/0x4d0 [11916.148328] ? watchdog+0x40/0x40 [11916.150782] ? hrtimer_interrupt+0x128/0x2c0 [11916.155025] ? smp_apic_timer_interrupt+0xa6/0x270 [11916.157172] ? apic_timer_interrupt+0xf/0x20 [11916.164162] [11916.165667] ? native_safe_halt+0x2e/0x50 [11916.166747] kvm_wait+0xcd/0xe0 [11916.170962] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11916.175629] do_raw_spin_lock+0x177/0x190 [11916.180106] _raw_spin_lock+0x21/0x30 [11916.184109] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11916.191562] ? do_raw_spin_unlock+0x75/0x190 [11916.193393] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11916.198798] ? do_wait_intr+0xf0/0xf0 [11916.203846] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11916.210362] mdc_reint+0x2d/0x110 [mdc] [11916.214041] mdc_unlink+0x658/0x870 [mdc] [11916.219054] lmv_unlink+0x34d/0xbd0 [lmv] [11916.221802] ll_unlink+0x24d/0x7f0 [lustre] [11916.227860] vfs_unlink+0x202/0x330 [11916.231563] do_unlinkat+0x2fa/0x460 [11916.235669] __x64_sys_unlink+0x37/0x50 [11916.239128] do_syscall_64+0xc1/0x440 [11916.243569] entry_SYSCALL_64_after_hwframe+0x49/0xae [11916.248844] RIP: 0033:0x7f151bdee10b [11916.250358] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [11916.269897] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [11916.275888] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [11916.282384] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [11916.292757] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11916.298695] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11916.307137] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11920.868767] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9990s! [11920.875116] Showing busy workqueues and worker pools: [11920.894339] workqueue events: flags=0x0 [11920.899244] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11920.905609] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [11920.910748] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11920.922236] pending: kfree_rcu_monitor [11920.924313] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11920.928744] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [11920.945119] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11920.951768] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [11920.972272] workqueue events_long: flags=0x0 [11920.975686] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11920.981952] pending: lu_objects_flush [obdclass] [11920.985832] workqueue events_unbound: flags=0x2 [11920.987836] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11920.997776] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11921.008273] workqueue events_power_efficient: flags=0x80 [11921.036827] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11921.046296] pending: do_cache_clean, neigh_periodic_work, check_lifetime [11921.056286] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11921.067371] pending: neigh_periodic_work [11921.076531] workqueue mm_percpu_wq: flags=0x8 [11921.082606] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11921.091396] pending: vmstat_update, lru_add_drain_per_cpu [11921.112878] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11921.129769] pending: vmstat_update [11921.134434] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11921.145488] pending: lru_add_drain_per_cpu [11921.154530] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [11921.159861] pending: lru_add_drain_per_cpu BAR(48) [11921.168053] workqueue writeback: flags=0x4a [11921.175072] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [11921.186359] pending: wb_workfn [11921.194211] workqueue xprtiod: flags=0xa [11921.195491] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11921.203271] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [11921.219019] workqueue ipv6_addrconf: flags=0x40008 [11921.226586] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11921.234027] pending: addrconf_verify_work [11921.236979] workqueue knbd0-recv: flags=0x1a [11921.243680] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11921.249601] in-flight: 658:recv_work [11921.259315] workqueue ptlrpc_pinger: flags=0x2 [11921.260810] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11921.262453] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11921.286994] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9990s workers=4 idle: 32984 37 8 [11921.296232] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11906s workers=2 idle: 661 [11928.056570] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11928.061113] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11928.118015] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11928.123897] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11928.129125] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11928.132311] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11928.135872] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11928.145467] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11928.149409] RAX: 0000000000003412 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11928.152692] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [11928.156948] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [11928.161007] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [11928.164329] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [11928.167497] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [11928.174024] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11928.180581] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11928.188584] Call Trace: [11928.191313] [11928.192718] ? show_regs.cold.9+0x22/0x2f [11928.195368] ? watchdog_timer_fn.cold.6+0x62/0x10a [11928.200912] ? __hrtimer_run_queues+0x1c2/0x4d0 [11928.206650] ? watchdog+0x40/0x40 [11928.209575] ? hrtimer_interrupt+0x128/0x2c0 [11928.216335] ? smp_apic_timer_interrupt+0xa6/0x270 [11928.219990] ? apic_timer_interrupt+0xf/0x20 [11928.223782] [11928.225025] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11928.230721] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11928.233583] do_raw_spin_lock+0x177/0x190 [11928.235544] _raw_spin_lock+0x21/0x30 [11928.240088] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11928.246850] process_one_work+0x2c8/0x700 [11928.248826] worker_thread+0x296/0x6e0 [11928.251701] ? rescuer_thread+0x570/0x570 [11928.253038] kthread+0x1d1/0x200 [11928.255314] ? set_kthread_struct+0x70/0x70 [11928.257620] ret_from_fork+0x1f/0x30 [11940.057369] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [11940.057410] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [11940.066461] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11940.066547] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11940.066556] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11940.066559] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11940.066575] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11940.066579] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11940.066583] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [11940.066586] RDX: 00000000000031ec RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [11940.066589] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [11940.066592] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [11940.066595] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11940.066598] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [11940.066601] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11940.066604] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11940.066613] Call Trace: [11940.066619] [11940.066625] ? show_regs.cold.9+0x22/0x2f [11940.066634] ? watchdog_timer_fn.cold.6+0x62/0x10a [11940.066641] ? __hrtimer_run_queues+0x1c2/0x4d0 [11940.066646] ? watchdog+0x40/0x40 [11940.066650] ? hrtimer_interrupt+0x128/0x2c0 [11940.066655] ? smp_apic_timer_interrupt+0xa6/0x270 [11940.066664] ? apic_timer_interrupt+0xf/0x20 [11940.066668] [11940.066671] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11940.066676] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11940.066681] do_raw_spin_lock+0x177/0x190 [11940.066686] _raw_spin_lock+0x21/0x30 [11940.066692] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11940.066970] obd_mod_free+0x46/0x140 [ptlrpc] [11940.067158] mdc_commit_open+0x91/0xb0 [mdc] [11940.067195] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11940.067401] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11940.067604] after_reply+0xe2c/0x1780 [ptlrpc] [11940.067816] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11940.068029] ? __next_timer_interrupt+0x160/0x160 [11940.068038] ptlrpcd+0xa82/0xe40 [ptlrpc] [11940.068260] ? do_wait_intr+0xf0/0xf0 [11940.068269] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11940.079058] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11940.079137] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11940.079145] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11940.079148] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11940.079162] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11940.079167] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11940.079171] RAX: 0000000000001e83 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11940.079174] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [11940.079176] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [11940.079179] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [11940.079181] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [11940.079185] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [11940.079188] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11940.079190] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [11940.079200] Call Trace: [11940.079206] [11940.079212] ? show_regs.cold.9+0x22/0x2f [11940.079222] ? watchdog_timer_fn.cold.6+0x62/0x10a [11940.079229] ? __hrtimer_run_queues+0x1c2/0x4d0 [11940.079233] ? watchdog+0x40/0x40 [11940.079237] ? hrtimer_interrupt+0x128/0x2c0 [11940.079241] ? smp_apic_timer_interrupt+0xa6/0x270 [11940.079248] ? apic_timer_interrupt+0xf/0x20 [11940.079252] [11940.079255] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11940.079262] do_raw_spin_lock+0x177/0x190 [11940.079267] _raw_spin_lock+0x21/0x30 [11940.079272] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11940.134345] kthread+0x1d1/0x200 [11940.151723] ? do_raw_spin_unlock+0x75/0x190 [11940.151743] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11940.159801] ? set_kthread_struct+0x70/0x70 [11940.160129] ret_from_fork+0x1f/0x30 [11940.650798] ? do_wait_intr+0xf0/0xf0 [11940.653996] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11940.661687] mdc_close+0x53b/0xd70 [mdc] [11940.665053] lmv_close+0x18d/0x490 [lmv] [11940.667962] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11940.671777] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11940.678882] ll_md_real_close+0xa5/0x2a0 [lustre] [11940.682987] ll_md_close+0x274/0x800 [lustre] [11940.685662] ll_file_release+0x128/0xc50 [lustre] [11940.688695] __fput+0x10d/0x440 [11940.690625] ____fput+0x12/0x20 [11940.691996] task_work_run+0xc6/0x110 [11940.694492] exit_to_usermode_loop+0x1dd/0x1f0 [11940.696729] do_syscall_64+0x3d6/0x440 [11940.699007] entry_SYSCALL_64_after_hwframe+0x49/0xae [11940.701199] RIP: 0033:0x7f151bee3cd8 [11940.703626] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11940.728091] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11940.735916] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [11940.747787] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [11940.753557] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11940.760822] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11940.769584] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11948.052016] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [11948.056196] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11948.100050] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11948.109002] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11948.116476] RIP: 0010:native_safe_halt+0x2e/0x50 [11948.121601] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11948.136363] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11948.143401] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11948.149786] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [11948.156922] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [11948.162427] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [11948.169892] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [11948.175841] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [11948.183988] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11948.189848] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [11948.196794] Call Trace: [11948.199440] [11948.203821] ? show_regs.cold.9+0x22/0x2f [11948.205384] ? watchdog_timer_fn.cold.6+0x62/0x10a [11948.209725] ? __hrtimer_run_queues+0x1c2/0x4d0 [11948.213415] ? watchdog+0x40/0x40 [11948.218023] ? hrtimer_interrupt+0x128/0x2c0 [11948.219593] ? smp_apic_timer_interrupt+0xa6/0x270 [11948.225190] ? apic_timer_interrupt+0xf/0x20 [11948.226900] [11948.231129] ? native_safe_halt+0x2e/0x50 [11948.232553] kvm_wait+0xcd/0xe0 [11948.233827] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11948.239644] do_raw_spin_lock+0x177/0x190 [11948.244652] _raw_spin_lock+0x21/0x30 [11948.246228] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11948.252231] ? do_raw_spin_unlock+0x75/0x190 [11948.253809] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11948.259990] ? do_wait_intr+0xf0/0xf0 [11948.261700] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11948.267396] mdc_reint+0x2d/0x110 [mdc] [11948.270163] mdc_unlink+0x658/0x870 [mdc] [11948.272189] lmv_unlink+0x34d/0xbd0 [lmv] [11948.274316] ll_unlink+0x24d/0x7f0 [lustre] [11948.276193] vfs_unlink+0x202/0x330 [11948.278059] do_unlinkat+0x2fa/0x460 [11948.279648] __x64_sys_unlink+0x37/0x50 [11948.281821] do_syscall_64+0xc1/0x440 [11948.283436] entry_SYSCALL_64_after_hwframe+0x49/0xae [11948.285842] RIP: 0033:0x7f151bdee10b [11948.287463] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [11948.296623] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [11948.300012] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [11948.303402] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [11948.306980] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11948.310627] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11948.314064] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11951.591173] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10020s! [11951.599102] Showing busy workqueues and worker pools: [11951.604020] workqueue events: flags=0x0 [11951.605899] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11951.612521] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [11951.627024] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11951.642958] pending: kfree_rcu_monitor [11951.645897] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11951.654941] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [11951.665657] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11951.669014] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [11951.673554] workqueue events_long: flags=0x0 [11951.678500] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11951.686490] pending: lu_objects_flush [obdclass] [11951.690427] workqueue events_unbound: flags=0x2 [11951.693965] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11951.702350] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11951.711911] workqueue events_power_efficient: flags=0x80 [11951.715968] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11951.719688] pending: do_cache_clean, neigh_periodic_work, check_lifetime [11951.726234] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11951.734497] pending: neigh_periodic_work [11951.736062] workqueue mm_percpu_wq: flags=0x8 [11951.739017] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11951.752877] pending: vmstat_update, lru_add_drain_per_cpu [11951.757636] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11951.765701] pending: vmstat_update [11951.769371] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11951.778567] pending: lru_add_drain_per_cpu [11951.782298] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [11951.787591] pending: lru_add_drain_per_cpu BAR(48) [11951.790076] workqueue writeback: flags=0x4a [11951.791350] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [11951.798504] pending: wb_workfn [11951.800464] workqueue xprtiod: flags=0xa [11951.806040] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11951.812993] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [11951.819735] workqueue ipv6_addrconf: flags=0x40008 [11951.823396] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11951.828352] pending: addrconf_verify_work [11951.831595] workqueue knbd0-recv: flags=0x1a [11951.833758] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11951.846287] in-flight: 658:recv_work [11951.850537] workqueue ptlrpc_pinger: flags=0x2 [11951.852518] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11951.857035] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11951.864592] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10021s workers=4 idle: 32984 37 8 [11951.876400] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11936s workers=2 idle: 661 [11956.056033] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [11956.060559] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11956.105884] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11956.110932] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11956.116520] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11956.121224] RIP: 0010:native_safe_halt+0x2e/0x50 [11956.124120] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11956.135671] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11956.140794] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11956.144920] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [11956.149107] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [11956.153219] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [11956.156186] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [11956.163234] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [11956.171026] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11956.175768] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11956.179317] Call Trace: [11956.180674] [11956.181670] ? show_regs.cold.9+0x22/0x2f [11956.183748] ? watchdog_timer_fn.cold.6+0x62/0x10a [11956.186603] ? __hrtimer_run_queues+0x1c2/0x4d0 [11956.188923] ? watchdog+0x40/0x40 [11956.190787] ? hrtimer_interrupt+0x128/0x2c0 [11956.192651] ? smp_apic_timer_interrupt+0xa6/0x270 [11956.196239] ? apic_timer_interrupt+0xf/0x20 [11956.198546] [11956.199684] ? native_safe_halt+0x2e/0x50 [11956.201944] kvm_wait+0xcd/0xe0 [11956.203333] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11956.205657] do_raw_spin_lock+0x177/0x190 [11956.207983] _raw_spin_lock+0x21/0x30 [11956.209545] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11956.213529] process_one_work+0x2c8/0x700 [11956.215726] worker_thread+0x296/0x6e0 [11956.217433] ? rescuer_thread+0x570/0x570 [11956.219493] kthread+0x1d1/0x200 [11956.221107] ? set_kthread_struct+0x70/0x70 [11956.222978] ret_from_fork+0x1f/0x30 [11968.056863] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [11968.057262] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [11968.066195] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) [11968.071801] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11968.072111] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11968.072119] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11968.072121] RIP: 0010:native_safe_halt+0x2e/0x50 [11968.072142] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11968.072146] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11968.072151] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11968.072153] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [11968.072155] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [11968.072157] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [11968.072159] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [11968.072163] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [11968.072166] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11968.072174] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [11968.072182] Call Trace: [11968.072187] [11968.072194] ? show_regs.cold.9+0x22/0x2f [11968.072203] ? watchdog_timer_fn.cold.6+0x62/0x10a [11968.072209] ? __hrtimer_run_queues+0x1c2/0x4d0 [11968.072214] ? watchdog+0x40/0x40 [11968.072217] ? hrtimer_interrupt+0x128/0x2c0 [11968.072222] ? smp_apic_timer_interrupt+0xa6/0x270 [11968.072227] ? apic_timer_interrupt+0xf/0x20 [11968.072232] [11968.072234] ? native_safe_halt+0x2e/0x50 [11968.072237] kvm_wait+0xcd/0xe0 [11968.072245] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11968.072254] do_raw_spin_lock+0x177/0x190 [11968.072259] _raw_spin_lock+0x21/0x30 [11968.072262] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11968.072533] ? do_raw_spin_unlock+0x75/0x190 [11968.072540] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11968.085433] lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix [11968.126384] ? do_wait_intr+0xf0/0xf0 [11968.126415] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11968.126957] mdc_close+0x53b/0xd70 [mdc] [11968.127126] lmv_close+0x18d/0x490 [lmv] [11968.127156] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11968.127384] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11968.127765] ll_md_real_close+0xa5/0x2a0 [lustre] [11968.127900] ll_md_close+0x274/0x800 [lustre] [11968.128009] ll_file_release+0x128/0xc50 [lustre] [11968.128104] __fput+0x10d/0x440 [11968.128114] ____fput+0x12/0x20 [11968.128119] task_work_run+0xc6/0x110 [11968.128127] exit_to_usermode_loop+0x1dd/0x1f0 [11968.128135] do_syscall_64+0x3d6/0x440 [11968.128140] entry_SYSCALL_64_after_hwframe+0x49/0xae [11968.128148] RIP: 0033:0x7f151bee3cd8 [11968.128155] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11968.128159] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11968.128164] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [11968.128167] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [11968.128169] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11968.128172] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11968.128185] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11968.381670] ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11968.387268] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11968.393893] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11968.398059] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11968.401694] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11968.412114] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11968.416338] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [11968.420354] RDX: 0000000000000e02 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [11968.423799] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [11968.427804] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [11968.431843] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11968.435808] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [11968.441099] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11968.445470] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11968.449967] Call Trace: [11968.452910] [11968.454125] ? show_regs.cold.9+0x22/0x2f [11968.456069] ? watchdog_timer_fn.cold.6+0x62/0x10a [11968.460032] ? __hrtimer_run_queues+0x1c2/0x4d0 [11968.463197] ? watchdog+0x40/0x40 [11968.465885] ? hrtimer_interrupt+0x128/0x2c0 [11968.468871] ? smp_apic_timer_interrupt+0xa6/0x270 [11968.473306] ? apic_timer_interrupt+0xf/0x20 [11968.476371] [11968.477380] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11968.481377] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11968.483710] do_raw_spin_lock+0x177/0x190 [11968.486854] _raw_spin_lock+0x21/0x30 [11968.488296] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11968.490940] obd_mod_free+0x46/0x140 [ptlrpc] [11968.493454] mdc_commit_open+0x91/0xb0 [mdc] [11968.495710] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11968.499299] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11968.502542] after_reply+0xe2c/0x1780 [ptlrpc] [11968.508475] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11968.512463] ? __next_timer_interrupt+0x160/0x160 [11968.520538] ptlrpcd+0xa82/0xe40 [ptlrpc] [11968.525318] ? do_wait_intr+0xf0/0xf0 [11968.530096] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11968.533505] kthread+0x1d1/0x200 [11968.536334] ? set_kthread_struct+0x70/0x70 [11968.538700] ret_from_fork+0x1f/0x30 [11976.055137] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [11976.065203] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11976.096829] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11976.101567] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11976.104758] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11976.107329] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11976.114047] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11976.116818] RAX: 0000000000007f8e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11976.119305] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [11976.122686] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [11976.125740] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [11976.129149] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [11976.132376] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [11976.135482] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11976.137691] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [11976.140681] Call Trace: [11976.141633] [11976.142627] ? show_regs.cold.9+0x22/0x2f [11976.144394] ? watchdog_timer_fn.cold.6+0x62/0x10a [11976.145929] ? __hrtimer_run_queues+0x1c2/0x4d0 [11976.147613] ? watchdog+0x40/0x40 [11976.148719] ? hrtimer_interrupt+0x128/0x2c0 [11976.150115] ? smp_apic_timer_interrupt+0xa6/0x270 [11976.151835] ? apic_timer_interrupt+0xf/0x20 [11976.153417] [11976.154222] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11976.156344] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11976.158361] do_raw_spin_lock+0x177/0x190 [11976.159943] _raw_spin_lock+0x21/0x30 [11976.161479] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11976.163671] ? do_raw_spin_unlock+0x75/0x190 [11976.165446] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11976.170848] ? do_wait_intr+0xf0/0xf0 [11976.172212] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11976.177283] mdc_reint+0x2d/0x110 [mdc] [11976.178848] mdc_unlink+0x658/0x870 [mdc] [11976.184123] lmv_unlink+0x34d/0xbd0 [lmv] [11976.185569] ll_unlink+0x24d/0x7f0 [lustre] [11976.190560] vfs_unlink+0x202/0x330 [11976.191844] do_unlinkat+0x2fa/0x460 [11976.196502] __x64_sys_unlink+0x37/0x50 [11976.197684] do_syscall_64+0xc1/0x440 [11976.200197] entry_SYSCALL_64_after_hwframe+0x49/0xae [11976.207945] RIP: 0033:0x7f151bdee10b [11976.210167] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [11976.224932] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [11976.231562] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [11976.237339] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [11976.240465] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11976.241967] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11976.256278] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11982.305230] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10051s! [11982.310150] Showing busy workqueues and worker pools: [11982.314156] workqueue events: flags=0x0 [11982.315340] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11982.322596] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [11982.332973] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11982.341381] pending: kfree_rcu_monitor [11982.345115] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11982.348279] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [11982.354084] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11982.360093] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [11982.367991] workqueue events_long: flags=0x0 [11982.371025] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11982.377250] pending: lu_objects_flush [obdclass] [11982.381010] workqueue events_unbound: flags=0x2 [11982.383527] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [11982.390051] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [11982.405764] workqueue events_power_efficient: flags=0x80 [11982.409203] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11982.415656] pending: do_cache_clean, neigh_periodic_work, check_lifetime [11982.419228] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11982.425076] pending: neigh_periodic_work [11982.426883] workqueue mm_percpu_wq: flags=0x8 [11982.428125] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11982.429796] pending: vmstat_update, lru_add_drain_per_cpu [11982.435028] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11982.441990] pending: vmstat_update [11982.444939] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11982.449613] pending: lru_add_drain_per_cpu [11982.455327] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [11982.462333] pending: lru_add_drain_per_cpu BAR(48) [11982.472395] workqueue writeback: flags=0x4a [11982.477552] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [11982.483991] pending: wb_workfn [11982.485583] workqueue xprtiod: flags=0xa [11982.487675] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [11982.490258] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [11982.495776] workqueue ipv6_addrconf: flags=0x40008 [11982.499072] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11982.504555] pending: addrconf_verify_work [11982.509747] workqueue knbd0-recv: flags=0x1a [11982.512390] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11982.516077] in-flight: 658:recv_work [11982.517651] workqueue ptlrpc_pinger: flags=0x2 [11982.519393] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11982.522101] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [11982.524975] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10051s workers=4 idle: 32984 37 8 [11982.531070] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11967s workers=2 idle: 661 [11984.056026] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [11984.058965] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11984.095815] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11984.104054] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11984.115477] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11984.119520] RIP: 0010:native_safe_halt+0x2e/0x50 [11984.126029] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11984.142337] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11984.150082] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11984.157046] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [11984.164691] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [11984.169914] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [11984.176588] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [11984.189227] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [11984.198772] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11984.204736] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11984.214257] Call Trace: [11984.224452] [11984.225290] ? show_regs.cold.9+0x22/0x2f [11984.230647] ? watchdog_timer_fn.cold.6+0x62/0x10a [11984.236247] ? __hrtimer_run_queues+0x1c2/0x4d0 [11984.241323] ? watchdog+0x40/0x40 [11984.245744] ? hrtimer_interrupt+0x128/0x2c0 [11984.251568] ? smp_apic_timer_interrupt+0xa6/0x270 [11984.256220] ? apic_timer_interrupt+0xf/0x20 [11984.262383] [11984.263253] ? native_safe_halt+0x2e/0x50 [11984.269189] kvm_wait+0xcd/0xe0 [11984.270427] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11984.276587] do_raw_spin_lock+0x177/0x190 [11984.283545] _raw_spin_lock+0x21/0x30 [11984.287333] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11984.293260] process_one_work+0x2c8/0x700 [11984.297834] worker_thread+0x296/0x6e0 [11984.309502] ? rescuer_thread+0x570/0x570 [11984.310965] kthread+0x1d1/0x200 [11984.312100] ? set_kthread_struct+0x70/0x70 [11984.320671] ret_from_fork+0x1f/0x30 [11996.055021] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [11996.057266] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [11996.064485] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11996.064740] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11996.064749] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11996.064754] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11996.064771] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11996.064775] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11996.064780] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [11996.064783] RDX: 000000000000428f RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [11996.064785] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [11996.064788] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [11996.064790] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11996.064794] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [11996.064797] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11996.064800] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [11996.064807] Call Trace: [11996.064820] [11996.064827] ? show_regs.cold.9+0x22/0x2f [11996.064837] ? watchdog_timer_fn.cold.6+0x62/0x10a [11996.064844] ? __hrtimer_run_queues+0x1c2/0x4d0 [11996.064848] ? watchdog+0x40/0x40 [11996.064852] ? hrtimer_interrupt+0x128/0x2c0 [11996.064856] ? smp_apic_timer_interrupt+0xa6/0x270 [11996.064866] ? apic_timer_interrupt+0xf/0x20 [11996.064871] [11996.064873] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11996.064877] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11996.064881] do_raw_spin_lock+0x177/0x190 [11996.064886] _raw_spin_lock+0x21/0x30 [11996.064892] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11996.065249] obd_mod_free+0x46/0x140 [ptlrpc] [11996.073897] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11996.073977] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11996.073984] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11996.073987] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11996.074273] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11996.074278] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11996.074282] RAX: 0000000000003613 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [11996.074284] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [11996.074287] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [11996.074289] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [11996.074291] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [11996.074294] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [11996.074418] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11996.074421] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [11996.074430] Call Trace: [11996.074435] [11996.074442] ? show_regs.cold.9+0x22/0x2f [11996.074453] ? watchdog_timer_fn.cold.6+0x62/0x10a [11996.074459] ? __hrtimer_run_queues+0x1c2/0x4d0 [11996.074463] ? watchdog+0x40/0x40 [11996.074467] ? hrtimer_interrupt+0x128/0x2c0 [11996.074470] ? smp_apic_timer_interrupt+0xa6/0x270 [11996.074477] ? apic_timer_interrupt+0xf/0x20 [11996.074482] [11996.074485] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11996.074492] do_raw_spin_lock+0x177/0x190 [11996.074497] _raw_spin_lock+0x21/0x30 [11996.074501] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11996.119627] mdc_commit_open+0x91/0xb0 [mdc] [11996.119824] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11996.120504] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11996.120963] after_reply+0xe2c/0x1780 [ptlrpc] [11996.121503] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11996.129122] ? do_raw_spin_unlock+0x75/0x190 [11996.129139] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11996.129368] ? do_wait_intr+0xf0/0xf0 [11996.129374] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11996.129527] mdc_close+0x53b/0xd70 [mdc] [11996.129567] lmv_close+0x18d/0x490 [lmv] [11996.129594] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11996.135482] ? __next_timer_interrupt+0x160/0x160 [11996.135525] ptlrpcd+0xa82/0xe40 [ptlrpc] [11996.136014] ? do_wait_intr+0xf0/0xf0 [11996.136025] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11996.140135] ll_close_inode_openhandle+0x493/0xea0 [lustre] [11996.140639] ll_md_real_close+0xa5/0x2a0 [lustre] [11996.140741] ll_md_close+0x274/0x800 [lustre] [11996.140840] ll_file_release+0x128/0xc50 [lustre] [11996.140927] __fput+0x10d/0x440 [11996.140940] ____fput+0x12/0x20 [11996.140946] task_work_run+0xc6/0x110 [11996.140956] exit_to_usermode_loop+0x1dd/0x1f0 [11996.140965] do_syscall_64+0x3d6/0x440 [11996.140970] entry_SYSCALL_64_after_hwframe+0x49/0xae [11996.140981] RIP: 0033:0x7f151bee3cd8 [11996.140991] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [11996.140995] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [11996.141002] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [11996.141005] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [11996.141008] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [11996.141010] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [11996.141013] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [11996.435285] kthread+0x1d1/0x200 [11996.438915] ? set_kthread_struct+0x70/0x70 [11996.441371] ret_from_fork+0x1f/0x30 [12008.054885] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [12008.061908] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12008.119224] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12008.129172] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12008.135133] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12008.138198] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12008.148202] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12008.152130] RAX: 0000000000002433 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12008.155815] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [12008.159401] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [12008.162313] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [12008.164435] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [12008.167090] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [12008.170818] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12008.173084] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [12008.176303] Call Trace: [12008.177786] [12008.178708] ? show_regs.cold.9+0x22/0x2f [12008.180414] ? watchdog_timer_fn.cold.6+0x62/0x10a [12008.183275] ? __hrtimer_run_queues+0x1c2/0x4d0 [12008.184910] ? watchdog+0x40/0x40 [12008.186282] ? hrtimer_interrupt+0x128/0x2c0 [12008.189369] ? smp_apic_timer_interrupt+0xa6/0x270 [12008.192168] ? apic_timer_interrupt+0xf/0x20 [12008.194159] [12008.195105] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12008.197727] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12008.200910] do_raw_spin_lock+0x177/0x190 [12008.202804] _raw_spin_lock+0x21/0x30 [12008.204432] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12008.207378] ? do_raw_spin_unlock+0x75/0x190 [12008.209846] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12008.212567] ? do_wait_intr+0xf0/0xf0 [12008.214285] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12008.216951] mdc_reint+0x2d/0x110 [mdc] [12008.218919] mdc_unlink+0x658/0x870 [mdc] [12008.220904] lmv_unlink+0x34d/0xbd0 [lmv] [12008.222828] ll_unlink+0x24d/0x7f0 [lustre] [12008.224843] vfs_unlink+0x202/0x330 [12008.226418] do_unlinkat+0x2fa/0x460 [12008.228324] __x64_sys_unlink+0x37/0x50 [12008.230100] do_syscall_64+0xc1/0x440 [12008.231920] entry_SYSCALL_64_after_hwframe+0x49/0xae [12008.235185] RIP: 0033:0x7f151bdee10b [12008.236819] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [12008.246711] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [12008.250161] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [12008.254300] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [12008.258126] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12008.261984] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12008.265731] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12012.061408] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [12012.068342] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12012.123764] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12012.131929] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12012.138753] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12012.142794] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12012.146492] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12012.158871] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12012.163808] RAX: 0000000000005618 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12012.171867] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [12012.185219] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [12012.192345] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [12012.203372] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [12012.215508] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [12012.225224] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12012.228023] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12012.236082] Call Trace: [12012.239032] [12012.239573] ? show_regs.cold.9+0x22/0x2f [12012.244064] ? watchdog_timer_fn.cold.6+0x62/0x10a [12012.248511] ? __hrtimer_run_queues+0x1c2/0x4d0 [12012.252194] ? watchdog+0x40/0x40 [12012.256982] ? hrtimer_interrupt+0x128/0x2c0 [12012.258573] ? smp_apic_timer_interrupt+0xa6/0x270 [12012.268397] ? apic_timer_interrupt+0xf/0x20 [12012.271212] [12012.272733] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12012.280777] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12012.286000] do_raw_spin_lock+0x177/0x190 [12012.288126] _raw_spin_lock+0x21/0x30 [12012.289368] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12012.293089] process_one_work+0x2c8/0x700 [12012.295705] worker_thread+0x296/0x6e0 [12012.298373] ? rescuer_thread+0x570/0x570 [12012.305625] kthread+0x1d1/0x200 [12012.309322] ? set_kthread_struct+0x70/0x70 [12012.317534] ret_from_fork+0x1f/0x30 [12013.031513] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10082s! [12013.044292] Showing busy workqueues and worker pools: [12013.048688] workqueue events: flags=0x0 [12013.055810] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12013.062666] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [12013.070812] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12013.078136] pending: kfree_rcu_monitor [12013.082669] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12013.087829] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [12013.098415] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12013.104266] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [12013.114391] workqueue events_long: flags=0x0 [12013.118469] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12013.126176] pending: lu_objects_flush [obdclass] [12013.128582] workqueue events_unbound: flags=0x2 [12013.134347] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12013.141472] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12013.151603] workqueue events_power_efficient: flags=0x80 [12013.155262] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12013.160727] pending: do_cache_clean, neigh_periodic_work, check_lifetime [12013.165015] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12013.171032] pending: neigh_periodic_work [12013.175421] workqueue mm_percpu_wq: flags=0x8 [12013.179193] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12013.183748] pending: vmstat_update, lru_add_drain_per_cpu [12013.189464] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12013.197847] pending: vmstat_update [12013.203613] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12013.211381] pending: lru_add_drain_per_cpu [12013.215415] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [12013.218347] pending: lru_add_drain_per_cpu BAR(48) [12013.221036] workqueue writeback: flags=0x4a [12013.223134] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [12013.228061] pending: wb_workfn [12013.229763] workqueue xprtiod: flags=0xa [12013.231844] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12013.235093] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [12013.239322] workqueue ipv6_addrconf: flags=0x40008 [12013.241823] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12013.245035] pending: addrconf_verify_work [12013.247104] workqueue knbd0-recv: flags=0x1a [12013.249346] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12013.253465] in-flight: 658:recv_work [12013.255694] workqueue ptlrpc_pinger: flags=0x2 [12013.258234] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12013.262227] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12013.265748] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10082s workers=4 idle: 32984 37 8 [12013.269160] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11998s workers=2 idle: 661 [12024.056633] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [12024.059487] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [12024.070004] Modules linked in: [12024.079133] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12024.079211] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12024.079218] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12024.079221] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12024.079995] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12024.080006] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12024.080011] RAX: 000000000000220a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12024.080014] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [12024.080016] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [12024.080018] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [12024.080021] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [12024.080024] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [12024.080027] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12024.080029] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [12024.080038] Call Trace: [12024.080043] [12024.080048] ? show_regs.cold.9+0x22/0x2f [12024.080059] ? watchdog_timer_fn.cold.6+0x62/0x10a [12024.080066] ? __hrtimer_run_queues+0x1c2/0x4d0 [12024.080070] ? watchdog+0x40/0x40 [12024.080074] ? hrtimer_interrupt+0x128/0x2c0 [12024.080078] ? smp_apic_timer_interrupt+0xa6/0x270 [12024.080086] ? apic_timer_interrupt+0xf/0x20 [12024.080091] [12024.080094] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12024.080102] do_raw_spin_lock+0x177/0x190 [12024.080106] _raw_spin_lock+0x21/0x30 [12024.080112] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12024.080550] ? do_raw_spin_unlock+0x75/0x190 [12024.080557] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12024.080836] ? do_wait_intr+0xf0/0xf0 [12024.080844] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12024.081048] mdc_close+0x53b/0xd70 [mdc] [12024.081088] lmv_close+0x18d/0x490 [lmv] [12024.081117] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12024.081280] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12024.081393] ll_md_real_close+0xa5/0x2a0 [lustre] [12024.082559] ll_md_close+0x274/0x800 [lustre] [12024.082703] ll_file_release+0x128/0xc50 [lustre] [12024.082786] __fput+0x10d/0x440 [12024.082794] ____fput+0x12/0x20 [12024.082799] task_work_run+0xc6/0x110 [12024.082806] exit_to_usermode_loop+0x1dd/0x1f0 [12024.082813] do_syscall_64+0x3d6/0x440 [12024.082817] entry_SYSCALL_64_after_hwframe+0x49/0xae [12024.082825] RIP: 0033:0x7f151bee3cd8 [12024.082832] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12024.082836] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12024.082840] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [12024.082843] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [12024.082845] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12024.082848] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12024.082850] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12024.440046] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12024.476164] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12024.485299] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12024.490835] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12024.498202] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12024.508024] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12024.516293] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [12024.522679] RDX: 00000000000046c1 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [12024.532860] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [12024.542857] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [12024.553757] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12024.561764] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [12024.577512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12024.580380] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12024.586758] Call Trace: [12024.588097] [12024.589149] ? show_regs.cold.9+0x22/0x2f [12024.594110] ? watchdog_timer_fn.cold.6+0x62/0x10a [12024.596727] ? __hrtimer_run_queues+0x1c2/0x4d0 [12024.602160] ? watchdog+0x40/0x40 [12024.609350] ? hrtimer_interrupt+0x128/0x2c0 [12024.616949] ? smp_apic_timer_interrupt+0xa6/0x270 [12024.626168] ? apic_timer_interrupt+0xf/0x20 [12024.634870] [12024.635653] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12024.638426] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12024.645642] do_raw_spin_lock+0x177/0x190 [12024.650136] _raw_spin_lock+0x21/0x30 [12024.653885] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12024.662226] obd_mod_free+0x46/0x140 [ptlrpc] [12024.672145] mdc_commit_open+0x91/0xb0 [mdc] [12024.677940] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12024.687759] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12024.692674] after_reply+0xe2c/0x1780 [ptlrpc] [12024.695075] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12024.704111] ? __next_timer_interrupt+0x160/0x160 [12024.709330] ptlrpcd+0xa82/0xe40 [ptlrpc] [12024.720082] ? do_wait_intr+0xf0/0xf0 [12024.723747] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12024.726314] kthread+0x1d1/0x200 [12024.728008] ? set_kthread_struct+0x70/0x70 [12024.733107] ret_from_fork+0x1f/0x30 [12040.052125] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [12040.057522] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [12040.062118] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12040.062856] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12040.062863] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12040.062866] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12040.062892] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12040.062896] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12040.062901] RAX: 000000000000492c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12040.062904] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [12040.062906] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [12040.062908] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [12040.062911] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [12040.062915] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [12040.062917] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12040.062920] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [12040.062929] Call Trace: [12040.062944] [12040.062951] ? show_regs.cold.9+0x22/0x2f [12040.062961] ? watchdog_timer_fn.cold.6+0x62/0x10a [12040.062968] ? __hrtimer_run_queues+0x1c2/0x4d0 [12040.062973] ? watchdog+0x40/0x40 [12040.062976] ? hrtimer_interrupt+0x128/0x2c0 [12040.062980] ? smp_apic_timer_interrupt+0xa6/0x270 [12040.062990] ? apic_timer_interrupt+0xf/0x20 [12040.062995] [12040.062998] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12040.063004] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12040.063008] do_raw_spin_lock+0x177/0x190 [12040.063014] _raw_spin_lock+0x21/0x30 [12040.063021] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12040.071991] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12040.072084] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12040.072095] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12040.072130] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12040.072690] RIP: 0010:native_safe_halt+0x2e/0x50 [12040.072701] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12040.072709] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12040.072714] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12040.072716] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [12040.072719] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [12040.072721] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [12040.072724] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [12040.072728] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [12040.072735] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12040.072738] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12040.072745] Call Trace: [12040.072751] [12040.072756] ? show_regs.cold.9+0x22/0x2f [12040.073068] ? watchdog_timer_fn.cold.6+0x62/0x10a [12040.073080] ? __hrtimer_run_queues+0x1c2/0x4d0 [12040.073085] ? watchdog+0x40/0x40 [12040.073089] ? hrtimer_interrupt+0x128/0x2c0 [12040.073096] ? smp_apic_timer_interrupt+0xa6/0x270 [12040.073103] ? apic_timer_interrupt+0xf/0x20 [12040.073107] [12040.073113] ? native_safe_halt+0x2e/0x50 [12040.073247] kvm_wait+0xcd/0xe0 [12040.073257] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12040.073368] do_raw_spin_lock+0x177/0x190 [12040.073375] _raw_spin_lock+0x21/0x30 [12040.073463] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12040.075084] process_one_work+0x2c8/0x700 [12040.075096] worker_thread+0x296/0x6e0 [12040.075101] ? rescuer_thread+0x570/0x570 [12040.075104] kthread+0x1d1/0x200 [12040.075108] ? set_kthread_struct+0x70/0x70 [12040.075112] ret_from_fork+0x1f/0x30 [12040.477729] ? do_raw_spin_unlock+0x75/0x190 [12040.482094] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12040.489052] ? do_wait_intr+0xf0/0xf0 [12040.492133] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12040.498988] mdc_reint+0x2d/0x110 [mdc] [12040.500983] mdc_unlink+0x658/0x870 [mdc] [12040.504314] lmv_unlink+0x34d/0xbd0 [lmv] [12040.506052] ll_unlink+0x24d/0x7f0 [lustre] [12040.510027] vfs_unlink+0x202/0x330 [12040.515494] do_unlinkat+0x2fa/0x460 [12040.516835] __x64_sys_unlink+0x37/0x50 [12040.519622] do_syscall_64+0xc1/0x440 [12040.521115] entry_SYSCALL_64_after_hwframe+0x49/0xae [12040.524571] RIP: 0033:0x7f151bdee10b [12040.526058] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [12040.534974] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [12040.540911] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [12040.544450] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [12040.549475] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12040.551797] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12040.558211] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12043.744301] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10113s! [12043.750043] Showing busy workqueues and worker pools: [12043.752622] workqueue events: flags=0x0 [12043.754877] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12043.758192] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [12043.765098] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12043.768102] pending: kfree_rcu_monitor [12043.769827] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12043.772339] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [12043.776895] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12043.779711] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [12043.783164] workqueue events_long: flags=0x0 [12043.785592] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12043.789031] pending: lu_objects_flush [obdclass] [12043.791324] workqueue events_unbound: flags=0x2 [12043.794389] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12043.797803] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12043.802526] workqueue events_power_efficient: flags=0x80 [12043.804950] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12043.808201] pending: do_cache_clean, neigh_periodic_work, check_lifetime [12043.811172] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12043.813852] pending: neigh_periodic_work [12043.815602] workqueue mm_percpu_wq: flags=0x8 [12043.817712] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12043.820663] pending: vmstat_update, lru_add_drain_per_cpu [12043.823343] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12043.826107] pending: vmstat_update [12043.827534] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12043.830755] pending: lru_add_drain_per_cpu [12043.832592] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [12043.835813] pending: lru_add_drain_per_cpu BAR(48) [12043.837843] workqueue writeback: flags=0x4a [12043.839746] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [12043.842727] pending: wb_workfn [12043.844029] workqueue xprtiod: flags=0xa [12043.846030] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12043.848157] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [12043.851575] workqueue ipv6_addrconf: flags=0x40008 [12043.854066] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12043.857092] pending: addrconf_verify_work [12043.859229] workqueue knbd0-recv: flags=0x1a [12043.860585] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12043.863176] in-flight: 658:recv_work [12043.864893] workqueue ptlrpc_pinger: flags=0x2 [12043.867107] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12043.869561] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12043.872371] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10113s workers=4 idle: 32984 37 8 [12043.875473] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12028s workers=2 idle: 661 [12052.055156] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [12052.057025] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [12052.060480] Modules linked in: [12052.063108] Modules linked in: [12052.063114] lustre(O) [12052.063821] lustre(O) [12052.066196] osp(O) ofd(O) lod(O) mdt(O) [12052.066722] osp(O) [12052.067847] mdd(O) [12052.069403] ofd(O) [12052.070956] mgs(O) [12052.071621] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12052.071669] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12052.071673] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12052.071676] RIP: 0010:native_safe_halt+0x2e/0x50 [12052.071686] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12052.071689] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12052.071693] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12052.071694] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [12052.071696] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [12052.071697] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [12052.071699] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [12052.071702] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [12052.071704] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12052.071705] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [12052.071712] Call Trace: [12052.071716] [12052.071721] ? show_regs.cold.9+0x22/0x2f [12052.071727] ? watchdog_timer_fn.cold.6+0x62/0x10a [12052.071732] ? __hrtimer_run_queues+0x1c2/0x4d0 [12052.071735] ? watchdog+0x40/0x40 [12052.071737] ? hrtimer_interrupt+0x128/0x2c0 [12052.071740] ? smp_apic_timer_interrupt+0xa6/0x270 [12052.071745] ? apic_timer_interrupt+0xf/0x20 [12052.072863] lquota(O) lfsck(O) [12052.073590] [12052.108211] obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) [12052.132874] ? native_safe_halt+0x2e/0x50 [12052.140482] ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12052.140532] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12052.140540] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12052.140543] RIP: 0010:native_safe_halt+0x2e/0x50 [12052.140557] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12052.140561] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12052.140566] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [12052.140568] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [12052.140570] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [12052.140572] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [12052.140574] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12052.140577] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [12052.140580] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12052.140582] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12052.140591] Call Trace: [12052.140596] [12052.140602] ? show_regs.cold.9+0x22/0x2f [12052.140612] ? watchdog_timer_fn.cold.6+0x62/0x10a [12052.140618] ? __hrtimer_run_queues+0x1c2/0x4d0 [12052.140623] ? watchdog+0x40/0x40 [12052.140627] ? hrtimer_interrupt+0x128/0x2c0 [12052.140631] ? smp_apic_timer_interrupt+0xa6/0x270 [12052.140638] ? apic_timer_interrupt+0xf/0x20 [12052.140643] [12052.140646] ? native_safe_halt+0x2e/0x50 [12052.140649] kvm_wait+0xcd/0xe0 [12052.140657] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12052.140664] do_raw_spin_lock+0x177/0x190 [12052.140668] _raw_spin_lock+0x21/0x30 [12052.140674] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12052.145335] kvm_wait+0xcd/0xe0 [12052.165641] obd_mod_free+0x46/0x140 [ptlrpc] [12052.170310] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12052.170339] do_raw_spin_lock+0x177/0x190 [12052.170345] _raw_spin_lock+0x21/0x30 [12052.170352] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12052.170702] ? do_raw_spin_unlock+0x75/0x190 [12052.170709] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12052.171112] ? do_wait_intr+0xf0/0xf0 [12052.171121] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12052.174042] mdc_commit_open+0x91/0xb0 [mdc] [12052.178260] mdc_close+0x53b/0xd70 [mdc] [12052.185566] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12052.188721] lmv_close+0x18d/0x490 [lmv] [12052.198193] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12052.202349] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12052.202601] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12052.202716] ll_md_real_close+0xa5/0x2a0 [lustre] [12052.202800] ll_md_close+0x274/0x800 [lustre] [12052.205516] after_reply+0xe2c/0x1780 [ptlrpc] [12052.208913] ll_file_release+0x128/0xc50 [lustre] [12052.210249] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12052.211608] __fput+0x10d/0x440 [12052.213138] ? __next_timer_interrupt+0x160/0x160 [12052.213156] ptlrpcd+0xa82/0xe40 [ptlrpc] [12052.221100] ____fput+0x12/0x20 [12052.224681] ? do_wait_intr+0xf0/0xf0 [12052.224704] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12052.231947] task_work_run+0xc6/0x110 [12052.233278] kthread+0x1d1/0x200 [12052.243400] exit_to_usermode_loop+0x1dd/0x1f0 [12052.243423] do_syscall_64+0x3d6/0x440 [12052.243429] entry_SYSCALL_64_after_hwframe+0x49/0xae [12052.243437] RIP: 0033:0x7f151bee3cd8 [12052.243475] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12052.243480] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12052.243485] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [12052.243488] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [12052.243490] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12052.243492] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12052.243495] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12052.574210] ? set_kthread_struct+0x70/0x70 [12052.583192] ret_from_fork+0x1f/0x30 [12064.865028] rcu: INFO: rcu_sched self-detected stall on CPU [12064.867695] rcu: 1-...!: (5852298 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [12064.875576] (t=10140218 jiffies g=249657 q=5394) [12064.879645] rcu: rcu_sched kthread starved for 10135170 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [12064.888946] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [12064.896779] rcu: RCU grace-period kthread stack dump: [12064.901630] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [12064.909779] Call Trace: [12064.911623] __schedule+0x351/0xcb0 [12064.914787] schedule+0xc0/0x180 [12064.917649] schedule_timeout+0xb4/0x190 [12064.920707] ? do_raw_spin_unlock+0x75/0x190 [12064.924759] ? __next_timer_interrupt+0x160/0x160 [12064.928677] ? prepare_to_swait_event+0xf3/0x140 [12064.931076] rcu_gp_kthread+0x8dd/0x15b0 [12064.934271] ? force_qs_rnp+0x2c0/0x2c0 [12064.936895] kthread+0x1d1/0x200 [12064.941562] ? set_kthread_struct+0x70/0x70 [12064.943487] ret_from_fork+0x1f/0x30 [12064.945581] rcu: Stack dump where RCU GP kthread last ran: [12064.948678] Sending NMI from CPU 1 to CPUs 0: [12064.951224] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [12064.954264] NMI backtrace for cpu 1 [12064.956724] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12064.966645] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12064.972691] Call Trace: [12064.975574] [12064.977539] ? dump_stack+0xbb/0x10e [12064.979924] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [12064.984526] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [12064.987435] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [12064.989412] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [12064.994697] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [12064.998476] ? rcu_dump_cpu_stacks+0x11b/0x171 [12065.001672] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [12065.005088] ? update_process_times+0x6d/0xc0 [12065.012103] ? tick_sched_handle.isra.18+0x27/0x90 [12065.017591] ? tick_sched_timer+0x53/0xb0 [12065.019255] ? __hrtimer_run_queues+0x1c2/0x4d0 [12065.022487] ? tick_sched_handle.isra.18+0x90/0x90 [12065.025847] ? hrtimer_interrupt+0x128/0x2c0 [12065.028676] ? smp_apic_timer_interrupt+0xa6/0x270 [12065.031219] ? apic_timer_interrupt+0xf/0x20 [12065.033455] [12065.034937] ? native_safe_halt+0x2e/0x50 [12065.037580] ? kvm_wait+0xcd/0xe0 [12065.042542] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12065.046358] ? do_raw_spin_lock+0x177/0x190 [12065.049802] ? _raw_spin_lock+0x21/0x30 [12065.052873] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12065.057566] ? obd_mod_free+0x46/0x140 [ptlrpc] [12065.061488] ? mdc_commit_open+0x91/0xb0 [mdc] [12065.064239] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12065.069517] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12065.076877] ? after_reply+0xe2c/0x1780 [ptlrpc] [12065.082910] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12065.087728] ? __next_timer_interrupt+0x160/0x160 [12065.090990] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [12065.095415] ? do_wait_intr+0xf0/0xf0 [12065.098475] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12065.101827] ? kthread+0x1d1/0x200 [12065.103136] ? set_kthread_struct+0x70/0x70 [12065.106850] ? ret_from_fork+0x1f/0x30 [12068.056569] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [12068.066784] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12068.113211] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12068.122718] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12068.130658] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12068.141418] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12068.146724] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12068.158825] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12068.162975] RAX: 0000000000007f8a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12068.166593] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [12068.173834] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [12068.182913] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [12068.191182] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [12068.195880] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [12068.199852] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12068.204245] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12068.209505] Call Trace: [12068.211618] [12068.212425] ? show_regs.cold.9+0x22/0x2f [12068.214179] ? watchdog_timer_fn.cold.6+0x62/0x10a [12068.216829] ? __hrtimer_run_queues+0x1c2/0x4d0 [12068.219809] ? watchdog+0x40/0x40 [12068.221248] ? hrtimer_interrupt+0x128/0x2c0 [12068.223263] ? smp_apic_timer_interrupt+0xa6/0x270 [12068.225503] ? apic_timer_interrupt+0xf/0x20 [12068.227487] [12068.228854] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12068.231248] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12068.236744] do_raw_spin_lock+0x177/0x190 [12068.237711] _raw_spin_lock+0x21/0x30 [12068.238641] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12068.244906] process_one_work+0x2c8/0x700 [12068.255021] worker_thread+0x296/0x6e0 [12068.256258] ? rescuer_thread+0x570/0x570 [12068.261239] kthread+0x1d1/0x200 [12068.264365] ? set_kthread_struct+0x70/0x70 [12068.267445] ret_from_fork+0x1f/0x30 [12073.952208] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10143s! [12073.954847] Showing busy workqueues and worker pools: [12073.957425] workqueue events: flags=0x0 [12073.959155] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12073.962299] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [12073.967030] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12073.970612] pending: kfree_rcu_monitor [12073.972899] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12073.976291] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [12073.980891] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12073.983905] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [12073.987860] workqueue events_long: flags=0x0 [12073.990375] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12073.994215] pending: lu_objects_flush [obdclass] [12073.996888] workqueue events_unbound: flags=0x2 [12073.999280] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12074.002904] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12074.008342] workqueue events_power_efficient: flags=0x80 [12074.010641] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12074.014604] pending: do_cache_clean, neigh_periodic_work, check_lifetime [12074.018455] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12074.022373] pending: neigh_periodic_work [12074.024607] workqueue mm_percpu_wq: flags=0x8 [12074.032772] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12074.037229] pending: vmstat_update, lru_add_drain_per_cpu [12074.044560] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12074.056281] pending: vmstat_update [12074.061718] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12074.070675] pending: lru_add_drain_per_cpu [12074.073226] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [12074.077416] pending: lru_add_drain_per_cpu BAR(48) [12074.082734] workqueue writeback: flags=0x4a [12074.084647] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [12074.091061] pending: wb_workfn [12074.097871] workqueue xprtiod: flags=0xa [12074.100341] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12074.108211] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [12074.116933] workqueue ipv6_addrconf: flags=0x40008 [12074.122434] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12074.127481] pending: addrconf_verify_work [12074.132455] workqueue knbd0-recv: flags=0x1a [12074.139097] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12074.145908] in-flight: 658:recv_work [12074.148022] workqueue ptlrpc_pinger: flags=0x2 [12074.150883] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12074.155285] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12074.162580] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10143s workers=4 idle: 32984 37 8 [12074.170685] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12058s workers=2 idle: 661 [12080.057251] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [12080.064757] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12080.089956] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12080.094721] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12080.100761] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12080.104573] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12080.112826] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12080.116093] RAX: 0000000000003489 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12080.119210] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [12080.122400] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [12080.125700] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [12080.127911] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [12080.135337] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [12080.139750] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12080.146400] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [12080.152329] Call Trace: [12080.154539] [12080.155922] ? show_regs.cold.9+0x22/0x2f [12080.157501] ? watchdog_timer_fn.cold.6+0x62/0x10a [12080.161449] ? __hrtimer_run_queues+0x1c2/0x4d0 [12080.163354] ? watchdog+0x40/0x40 [12080.165115] ? hrtimer_interrupt+0x128/0x2c0 [12080.170854] ? smp_apic_timer_interrupt+0xa6/0x270 [12080.173298] ? apic_timer_interrupt+0xf/0x20 [12080.177319] [12080.178846] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12080.183086] do_raw_spin_lock+0x177/0x190 [12080.186699] _raw_spin_lock+0x21/0x30 [12080.187972] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12080.194432] ? do_raw_spin_unlock+0x75/0x190 [12080.195683] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12080.202118] ? do_wait_intr+0xf0/0xf0 [12080.204846] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12080.209403] mdc_close+0x53b/0xd70 [mdc] [12080.211464] lmv_close+0x18d/0x490 [lmv] [12080.213404] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12080.216449] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12080.223250] ll_md_real_close+0xa5/0x2a0 [lustre] [12080.225206] ll_md_close+0x274/0x800 [lustre] [12080.227356] ll_file_release+0x128/0xc50 [lustre] [12080.231240] __fput+0x10d/0x440 [12080.232590] ____fput+0x12/0x20 [12080.234616] task_work_run+0xc6/0x110 [12080.241056] exit_to_usermode_loop+0x1dd/0x1f0 [12080.243796] do_syscall_64+0x3d6/0x440 [12080.246200] entry_SYSCALL_64_after_hwframe+0x49/0xae [12080.250180] RIP: 0033:0x7f151bee3cd8 [12080.251697] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12080.292350] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12080.307949] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [12080.314633] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [12080.320155] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12080.327413] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12080.331268] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12092.055028] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [12092.058778] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12092.078552] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12092.086059] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12092.089975] RIP: 0010:native_safe_halt+0x2e/0x50 [12092.091704] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12092.098223] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12092.100450] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [12092.103718] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [12092.106950] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [12092.109370] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [12092.111547] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12092.114244] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [12092.117054] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12092.118731] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12092.121230] Call Trace: [12092.122176] [12092.123172] ? show_regs.cold.9+0x22/0x2f [12092.124411] ? watchdog_timer_fn.cold.6+0x62/0x10a [12092.126246] ? __hrtimer_run_queues+0x1c2/0x4d0 [12092.127961] ? watchdog+0x40/0x40 [12092.128991] ? hrtimer_interrupt+0x128/0x2c0 [12092.130804] ? smp_apic_timer_interrupt+0xa6/0x270 [12092.132317] ? apic_timer_interrupt+0xf/0x20 [12092.133733] [12092.134449] ? native_safe_halt+0x2e/0x50 [12092.135947] kvm_wait+0xcd/0xe0 [12092.137251] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12092.139205] do_raw_spin_lock+0x177/0x190 [12092.140476] _raw_spin_lock+0x21/0x30 [12092.142026] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12092.144159] obd_mod_free+0x46/0x140 [ptlrpc] [12092.146224] mdc_commit_open+0x91/0xb0 [mdc] [12092.147693] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12092.149651] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12092.151738] after_reply+0xe2c/0x1780 [ptlrpc] [12092.153972] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12092.156101] ? __next_timer_interrupt+0x160/0x160 [12092.157585] ptlrpcd+0xa82/0xe40 [ptlrpc] [12092.159361] ? do_wait_intr+0xf0/0xf0 [12092.160490] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12092.162390] kthread+0x1d1/0x200 [12092.163508] ? set_kthread_struct+0x70/0x70 [12092.164959] ret_from_fork+0x1f/0x30 [12096.056015] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [12096.059613] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12096.087608] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12096.096019] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12096.102980] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12096.107524] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12096.112373] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12096.125864] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12096.129761] RAX: 00000000000053d4 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12096.133942] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [12096.138027] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [12096.141780] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [12096.145626] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [12096.149568] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [12096.153702] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12096.156707] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12096.160407] Call Trace: [12096.161574] [12096.162500] ? show_regs.cold.9+0x22/0x2f [12096.165540] ? watchdog_timer_fn.cold.6+0x62/0x10a [12096.168665] ? __hrtimer_run_queues+0x1c2/0x4d0 [12096.173552] ? watchdog+0x40/0x40 [12096.175431] ? hrtimer_interrupt+0x128/0x2c0 [12096.177498] ? smp_apic_timer_interrupt+0xa6/0x270 [12096.180083] ? apic_timer_interrupt+0xf/0x20 [12096.182418] [12096.183264] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12096.185692] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12096.188410] do_raw_spin_lock+0x177/0x190 [12096.190651] _raw_spin_lock+0x21/0x30 [12096.193432] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12096.197666] process_one_work+0x2c8/0x700 [12096.199997] worker_thread+0x296/0x6e0 [12096.202351] ? rescuer_thread+0x570/0x570 [12096.204408] kthread+0x1d1/0x200 [12096.206162] ? set_kthread_struct+0x70/0x70 [12096.208381] ret_from_fork+0x1f/0x30 [12100.052023] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [12100.054415] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12100.072406] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12100.076447] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12100.080115] RIP: 0010:native_safe_halt+0x2e/0x50 [12100.081830] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12100.088922] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12100.091967] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12100.094773] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [12100.097320] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [12100.099746] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [12100.102529] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [12100.105040] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [12100.108009] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12100.110567] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [12100.113903] Call Trace: [12100.114833] [12100.115672] ? show_regs.cold.9+0x22/0x2f [12100.117503] ? watchdog_timer_fn.cold.6+0x62/0x10a [12100.119827] ? __hrtimer_run_queues+0x1c2/0x4d0 [12100.121726] ? watchdog+0x40/0x40 [12100.122999] ? hrtimer_interrupt+0x128/0x2c0 [12100.125157] ? smp_apic_timer_interrupt+0xa6/0x270 [12100.127322] ? apic_timer_interrupt+0xf/0x20 [12100.129458] [12100.130358] ? native_safe_halt+0x2e/0x50 [12100.132524] kvm_wait+0xcd/0xe0 [12100.133933] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12100.136549] do_raw_spin_lock+0x177/0x190 [12100.138331] _raw_spin_lock+0x21/0x30 [12100.140437] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12100.142990] ? do_raw_spin_unlock+0x75/0x190 [12100.145049] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12100.147541] ? do_wait_intr+0xf0/0xf0 [12100.149074] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12100.151960] mdc_reint+0x2d/0x110 [mdc] [12100.154031] mdc_unlink+0x658/0x870 [mdc] [12100.156459] lmv_unlink+0x34d/0xbd0 [lmv] [12100.158545] ll_unlink+0x24d/0x7f0 [lustre] [12100.160755] vfs_unlink+0x202/0x330 [12100.162648] do_unlinkat+0x2fa/0x460 [12100.164248] __x64_sys_unlink+0x37/0x50 [12100.166121] do_syscall_64+0xc1/0x440 [12100.168085] entry_SYSCALL_64_after_hwframe+0x49/0xae [12100.170417] RIP: 0033:0x7f151bdee10b [12100.172589] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [12100.181655] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [12100.185373] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [12100.190640] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [12100.196055] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12100.200095] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12100.206535] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12104.672223] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10174s! [12104.675128] Showing busy workqueues and worker pools: [12104.678138] workqueue events: flags=0x0 [12104.684286] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12104.687588] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [12104.697607] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12104.701023] pending: kfree_rcu_monitor [12104.702477] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12104.705357] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [12104.710503] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12104.713940] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [12104.717247] workqueue events_long: flags=0x0 [12104.722367] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12104.726290] pending: lu_objects_flush [obdclass] [12104.730968] workqueue events_unbound: flags=0x2 [12104.732886] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12104.735466] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12104.748592] workqueue events_power_efficient: flags=0x80 [12104.754309] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12104.759614] pending: do_cache_clean, neigh_periodic_work, check_lifetime [12104.764115] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12104.766460] pending: neigh_periodic_work [12104.768986] workqueue mm_percpu_wq: flags=0x8 [12104.770935] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12104.775224] pending: vmstat_update, lru_add_drain_per_cpu [12104.780175] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12104.787480] pending: vmstat_update [12104.788963] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12104.792920] pending: lru_add_drain_per_cpu [12104.794601] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [12104.805541] pending: lru_add_drain_per_cpu BAR(48) [12104.809156] workqueue writeback: flags=0x4a [12104.812415] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [12104.815521] pending: wb_workfn [12104.821058] workqueue xprtiod: flags=0xa [12104.822467] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12104.825714] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [12104.831474] workqueue ipv6_addrconf: flags=0x40008 [12104.836486] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12104.839848] pending: addrconf_verify_work [12104.841980] workqueue knbd0-recv: flags=0x1a [12104.844802] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12104.847269] in-flight: 658:recv_work [12104.849432] workqueue ptlrpc_pinger: flags=0x2 [12104.851894] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12104.855083] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12104.858384] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10174s workers=4 idle: 32984 37 8 [12104.863843] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12089s workers=2 idle: 661 [12108.057023] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [12108.068765] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12108.098923] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12108.112231] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12108.125550] RIP: 0010:native_safe_halt+0x2e/0x50 [12108.128208] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12108.143970] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12108.149791] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12108.167532] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [12108.184139] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [12108.196194] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [12108.213095] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [12108.225024] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [12108.227686] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12108.246660] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [12108.258647] Call Trace: [12108.259432] [12108.265808] ? show_regs.cold.9+0x22/0x2f [12108.274450] ? watchdog_timer_fn.cold.6+0x62/0x10a [12108.283975] ? __hrtimer_run_queues+0x1c2/0x4d0 [12108.289030] ? watchdog+0x40/0x40 [12108.295718] ? hrtimer_interrupt+0x128/0x2c0 [12108.302479] ? smp_apic_timer_interrupt+0xa6/0x270 [12108.308184] ? apic_timer_interrupt+0xf/0x20 [12108.309641] [12108.310448] ? native_safe_halt+0x2e/0x50 [12108.323277] kvm_wait+0xcd/0xe0 [12108.331019] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12108.337008] do_raw_spin_lock+0x177/0x190 [12108.338293] _raw_spin_lock+0x21/0x30 [12108.353032] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12108.358009] ? do_raw_spin_unlock+0x75/0x190 [12108.364441] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12108.369510] ? do_wait_intr+0xf0/0xf0 [12108.370668] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12108.378700] mdc_close+0x53b/0xd70 [mdc] [12108.381993] lmv_close+0x18d/0x490 [lmv] [12108.388674] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12108.394833] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12108.401466] ll_md_real_close+0xa5/0x2a0 [lustre] [12108.406606] ll_md_close+0x274/0x800 [lustre] [12108.409185] ll_file_release+0x128/0xc50 [lustre] [12108.414035] __fput+0x10d/0x440 [12108.416205] ____fput+0x12/0x20 [12108.417870] task_work_run+0xc6/0x110 [12108.419899] exit_to_usermode_loop+0x1dd/0x1f0 [12108.422060] do_syscall_64+0x3d6/0x440 [12108.424029] entry_SYSCALL_64_after_hwframe+0x49/0xae [12108.426766] RIP: 0033:0x7f151bee3cd8 [12108.428322] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12108.435559] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12108.438395] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [12108.440942] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [12108.443397] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12108.449363] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12108.456006] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12120.056746] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [12120.061552] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12120.103402] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12120.113337] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12120.123798] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12120.127347] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12120.146340] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12120.151813] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [12120.159089] RDX: 00000000000079da RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [12120.162462] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [12120.166725] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [12120.169933] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12120.175504] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [12120.180578] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12120.184540] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12120.192444] Call Trace: [12120.193854] [12120.195043] ? show_regs.cold.9+0x22/0x2f [12120.201011] ? watchdog_timer_fn.cold.6+0x62/0x10a [12120.204380] ? __hrtimer_run_queues+0x1c2/0x4d0 [12120.210723] ? watchdog+0x40/0x40 [12120.215819] ? hrtimer_interrupt+0x128/0x2c0 [12120.221397] ? smp_apic_timer_interrupt+0xa6/0x270 [12120.228576] ? apic_timer_interrupt+0xf/0x20 [12120.236797] [12120.237459] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12120.243019] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12120.247347] do_raw_spin_lock+0x177/0x190 [12120.250085] _raw_spin_lock+0x21/0x30 [12120.255076] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12120.259145] obd_mod_free+0x46/0x140 [ptlrpc] [12120.264422] mdc_commit_open+0x91/0xb0 [mdc] [12120.267277] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12120.280541] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12120.286072] after_reply+0xe2c/0x1780 [ptlrpc] [12120.293454] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12120.296443] ? __next_timer_interrupt+0x160/0x160 [12120.300370] ptlrpcd+0xa82/0xe40 [ptlrpc] [12120.307208] ? do_wait_intr+0xf0/0xf0 [12120.310444] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12120.316980] kthread+0x1d1/0x200 [12120.318167] ? set_kthread_struct+0x70/0x70 [12120.321225] ret_from_fork+0x1f/0x30 [12124.057243] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12124.060542] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12124.085766] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12124.092275] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12124.096946] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12124.100629] RIP: 0010:native_safe_halt+0x2e/0x50 [12124.102908] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12124.112926] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12124.117699] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12124.121621] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [12124.125939] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [12124.129867] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [12124.132806] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [12124.137011] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [12124.140750] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12124.143770] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12124.147771] Call Trace: [12124.148775] [12124.149793] ? show_regs.cold.9+0x22/0x2f [12124.152194] ? watchdog_timer_fn.cold.6+0x62/0x10a [12124.154710] ? __hrtimer_run_queues+0x1c2/0x4d0 [12124.157006] ? watchdog+0x40/0x40 [12124.158664] ? hrtimer_interrupt+0x128/0x2c0 [12124.160787] ? smp_apic_timer_interrupt+0xa6/0x270 [12124.163417] ? apic_timer_interrupt+0xf/0x20 [12124.166237] [12124.167273] ? native_safe_halt+0x2e/0x50 [12124.169356] kvm_wait+0xcd/0xe0 [12124.171390] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12124.175916] do_raw_spin_lock+0x177/0x190 [12124.177999] _raw_spin_lock+0x21/0x30 [12124.180423] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12124.183365] process_one_work+0x2c8/0x700 [12124.188734] worker_thread+0x296/0x6e0 [12124.189947] ? rescuer_thread+0x570/0x570 [12124.191146] kthread+0x1d1/0x200 [12124.193039] ? set_kthread_struct+0x70/0x70 [12124.196786] ret_from_fork+0x1f/0x30 [12132.052042] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [12132.055426] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12132.080597] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12132.086448] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12132.090682] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12132.092819] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12132.101691] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12132.105462] RAX: 0000000000002d8d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12132.110826] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [12132.120452] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [12132.126850] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [12132.134246] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [12132.138380] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [12132.142514] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12132.145559] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [12132.149029] Call Trace: [12132.150727] [12132.151882] ? show_regs.cold.9+0x22/0x2f [12132.153894] ? watchdog_timer_fn.cold.6+0x62/0x10a [12132.156070] ? __hrtimer_run_queues+0x1c2/0x4d0 [12132.158414] ? watchdog+0x40/0x40 [12132.159955] ? hrtimer_interrupt+0x128/0x2c0 [12132.162231] ? smp_apic_timer_interrupt+0xa6/0x270 [12132.164596] ? apic_timer_interrupt+0xf/0x20 [12132.166625] [12132.167581] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12132.170787] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12132.172384] do_raw_spin_lock+0x177/0x190 [12132.174604] _raw_spin_lock+0x21/0x30 [12132.176473] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12132.179292] ? do_raw_spin_unlock+0x75/0x190 [12132.180935] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12132.184020] ? do_wait_intr+0xf0/0xf0 [12132.186117] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12132.188940] mdc_reint+0x2d/0x110 [mdc] [12132.191230] mdc_unlink+0x658/0x870 [mdc] [12132.193296] lmv_unlink+0x34d/0xbd0 [lmv] [12132.195030] ll_unlink+0x24d/0x7f0 [lustre] [12132.197069] vfs_unlink+0x202/0x330 [12132.198859] do_unlinkat+0x2fa/0x460 [12132.200699] __x64_sys_unlink+0x37/0x50 [12132.202484] do_syscall_64+0xc1/0x440 [12132.204228] entry_SYSCALL_64_after_hwframe+0x49/0xae [12132.206121] RIP: 0033:0x7f151bdee10b [12132.207459] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [12132.213999] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [12132.216684] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [12132.218960] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [12132.221724] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12132.223986] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12132.226665] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12134.880821] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10204s! [12134.884888] Showing busy workqueues and worker pools: [12134.890785] workqueue events: flags=0x0 [12134.892165] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12134.898093] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [12134.910555] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12134.922650] pending: kfree_rcu_monitor [12134.926821] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12134.935384] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [12134.943242] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12134.946831] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [12134.953669] workqueue events_long: flags=0x0 [12134.960711] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12134.968455] pending: lu_objects_flush [obdclass] [12134.973717] workqueue events_unbound: flags=0x2 [12134.976992] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12134.992070] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12135.006705] workqueue events_power_efficient: flags=0x80 [12135.010754] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12135.016697] pending: do_cache_clean, neigh_periodic_work, check_lifetime [12135.033824] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12135.040251] pending: neigh_periodic_work [12135.049020] workqueue mm_percpu_wq: flags=0x8 [12135.054796] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12135.066865] pending: vmstat_update, lru_add_drain_per_cpu [12135.074807] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12135.087538] pending: vmstat_update [12135.089809] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12135.094654] pending: lru_add_drain_per_cpu [12135.103538] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [12135.110936] pending: lru_add_drain_per_cpu BAR(48) [12135.119769] workqueue writeback: flags=0x4a [12135.124317] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [12135.128496] pending: wb_workfn [12135.136848] workqueue xprtiod: flags=0xa [12135.139106] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12135.150766] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [12135.161552] workqueue ipv6_addrconf: flags=0x40008 [12135.164409] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12135.170978] pending: addrconf_verify_work [12135.174307] workqueue knbd0-recv: flags=0x1a [12135.175615] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12135.179971] in-flight: 658:recv_work [12135.182029] workqueue ptlrpc_pinger: flags=0x2 [12135.193518] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12135.197847] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12135.205734] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10204s workers=4 idle: 32984 37 8 [12135.209618] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12119s workers=2 idle: 661 [12136.061226] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [12136.078520] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12136.120175] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12136.128222] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12136.135573] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12136.141831] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12136.157738] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12136.161720] RAX: 0000000000000211 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12136.169381] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [12136.173752] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [12136.177422] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [12136.181698] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [12136.184792] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [12136.195750] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12136.203846] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [12136.208543] Call Trace: [12136.212439] [12136.213235] ? show_regs.cold.9+0x22/0x2f [12136.217889] ? watchdog_timer_fn.cold.6+0x62/0x10a [12136.221091] ? __hrtimer_run_queues+0x1c2/0x4d0 [12136.227093] ? watchdog+0x40/0x40 [12136.229389] ? hrtimer_interrupt+0x128/0x2c0 [12136.232613] ? smp_apic_timer_interrupt+0xa6/0x270 [12136.238238] ? apic_timer_interrupt+0xf/0x20 [12136.244029] [12136.244637] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12136.250795] do_raw_spin_lock+0x177/0x190 [12136.251955] _raw_spin_lock+0x21/0x30 [12136.257160] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12136.262856] ? do_raw_spin_unlock+0x75/0x190 [12136.266699] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12136.273350] ? do_wait_intr+0xf0/0xf0 [12136.279251] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12136.281906] mdc_close+0x53b/0xd70 [mdc] [12136.285672] lmv_close+0x18d/0x490 [lmv] [12136.287723] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12136.291141] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12136.300144] ll_md_real_close+0xa5/0x2a0 [lustre] [12136.303205] ll_md_close+0x274/0x800 [lustre] [12136.313799] ll_file_release+0x128/0xc50 [lustre] [12136.320141] __fput+0x10d/0x440 [12136.322007] ____fput+0x12/0x20 [12136.328001] task_work_run+0xc6/0x110 [12136.330744] exit_to_usermode_loop+0x1dd/0x1f0 [12136.335611] do_syscall_64+0x3d6/0x440 [12136.337100] entry_SYSCALL_64_after_hwframe+0x49/0xae [12136.343138] RIP: 0033:0x7f151bee3cd8 [12136.344577] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12136.355503] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12136.367087] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [12136.374696] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [12136.388064] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12136.399844] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12136.411282] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12148.061211] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [12148.065267] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12148.126437] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12148.133633] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12148.137627] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12148.142775] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12148.157721] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12148.162712] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [12148.170562] RDX: 0000000000000193 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [12148.179585] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [12148.186387] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [12148.190025] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12148.196662] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [12148.204862] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12148.207130] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12148.212999] Call Trace: [12148.215125] [12148.216671] ? show_regs.cold.9+0x22/0x2f [12148.219072] ? watchdog_timer_fn.cold.6+0x62/0x10a [12148.221517] ? __hrtimer_run_queues+0x1c2/0x4d0 [12148.223818] ? watchdog+0x40/0x40 [12148.224964] ? hrtimer_interrupt+0x128/0x2c0 [12148.226927] ? smp_apic_timer_interrupt+0xa6/0x270 [12148.233028] ? apic_timer_interrupt+0xf/0x20 [12148.235940] [12148.237260] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12148.239184] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12148.242109] do_raw_spin_lock+0x177/0x190 [12148.244095] _raw_spin_lock+0x21/0x30 [12148.248842] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12148.253566] obd_mod_free+0x46/0x140 [ptlrpc] [12148.255256] mdc_commit_open+0x91/0xb0 [mdc] [12148.264035] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12148.269753] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12148.277779] after_reply+0xe2c/0x1780 [ptlrpc] [12148.281911] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12148.284685] ? __next_timer_interrupt+0x160/0x160 [12148.295150] ptlrpcd+0xa82/0xe40 [ptlrpc] [12148.300972] ? do_wait_intr+0xf0/0xf0 [12148.304733] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12148.307558] kthread+0x1d1/0x200 [12148.309031] ? set_kthread_struct+0x70/0x70 [12148.311065] ret_from_fork+0x1f/0x30 [12152.056367] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12152.061033] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12152.133341] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12152.142462] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12152.151895] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12152.156808] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12152.159815] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12152.172466] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12152.177530] RAX: 00000000000059c1 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12152.181709] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [12152.185345] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [12152.191869] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [12152.199857] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [12152.207652] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [12152.213243] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12152.221594] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12152.230582] Call Trace: [12152.232138] [12152.232966] ? show_regs.cold.9+0x22/0x2f [12152.238757] ? watchdog_timer_fn.cold.6+0x62/0x10a [12152.248484] ? __hrtimer_run_queues+0x1c2/0x4d0 [12152.254827] ? watchdog+0x40/0x40 [12152.271668] ? hrtimer_interrupt+0x128/0x2c0 [12152.277587] ? smp_apic_timer_interrupt+0xa6/0x270 [12152.282052] ? apic_timer_interrupt+0xf/0x20 [12152.286003] [12152.290895] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12152.296239] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12152.299669] do_raw_spin_lock+0x177/0x190 [12152.304899] _raw_spin_lock+0x21/0x30 [12152.313595] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12152.317717] process_one_work+0x2c8/0x700 [12152.322811] worker_thread+0x296/0x6e0 [12152.330776] ? rescuer_thread+0x570/0x570 [12152.333805] kthread+0x1d1/0x200 [12152.337809] ? set_kthread_struct+0x70/0x70 [12152.343145] ret_from_fork+0x1f/0x30 [12160.052033] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [12160.056777] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12160.084607] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12160.091302] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12160.097972] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12160.103584] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12160.113241] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12160.117945] RAX: 00000000000048d0 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12160.121680] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [12160.125400] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [12160.129405] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [12160.133048] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [12160.137148] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [12160.141885] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12160.145872] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [12160.149784] Call Trace: [12160.150951] [12160.151997] ? show_regs.cold.9+0x22/0x2f [12160.154123] ? watchdog_timer_fn.cold.6+0x62/0x10a [12160.156598] ? __hrtimer_run_queues+0x1c2/0x4d0 [12160.159403] ? watchdog+0x40/0x40 [12160.161018] ? hrtimer_interrupt+0x128/0x2c0 [12160.163387] ? smp_apic_timer_interrupt+0xa6/0x270 [12160.165928] ? apic_timer_interrupt+0xf/0x20 [12160.168292] [12160.169853] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12160.172726] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12160.175492] do_raw_spin_lock+0x177/0x190 [12160.177830] _raw_spin_lock+0x21/0x30 [12160.182819] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12160.192311] ? do_raw_spin_unlock+0x75/0x190 [12160.196666] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12160.202189] ? do_wait_intr+0xf0/0xf0 [12160.205905] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12160.209574] mdc_reint+0x2d/0x110 [mdc] [12160.212035] mdc_unlink+0x658/0x870 [mdc] [12160.219021] lmv_unlink+0x34d/0xbd0 [lmv] [12160.222797] ll_unlink+0x24d/0x7f0 [lustre] [12160.228624] vfs_unlink+0x202/0x330 [12160.234462] do_unlinkat+0x2fa/0x460 [12160.235841] __x64_sys_unlink+0x37/0x50 [12160.241377] do_syscall_64+0xc1/0x440 [12160.242741] entry_SYSCALL_64_after_hwframe+0x49/0xae [12160.244954] RIP: 0033:0x7f151bdee10b [12160.246655] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [12160.257632] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [12160.261158] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [12160.265254] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [12160.270362] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12160.272523] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12160.279810] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12164.058493] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [12164.069634] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12164.124596] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12164.132291] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12164.137746] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12164.140967] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12164.172749] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12164.181494] RAX: 0000000000007e8d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12164.193273] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [12164.203475] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [12164.205545] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [12164.214106] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [12164.223707] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [12164.230026] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12164.238849] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [12164.245011] Call Trace: [12164.252662] [12164.254630] ? show_regs.cold.9+0x22/0x2f [12164.258436] ? watchdog_timer_fn.cold.6+0x62/0x10a [12164.260244] ? __hrtimer_run_queues+0x1c2/0x4d0 [12164.268996] ? watchdog+0x40/0x40 [12164.272387] ? hrtimer_interrupt+0x128/0x2c0 [12164.275089] ? smp_apic_timer_interrupt+0xa6/0x270 [12164.281132] ? apic_timer_interrupt+0xf/0x20 [12164.285925] [12164.288671] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12164.292080] do_raw_spin_lock+0x177/0x190 [12164.294707] _raw_spin_lock+0x21/0x30 [12164.297202] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12164.305372] ? do_raw_spin_unlock+0x75/0x190 [12164.307741] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12164.317033] ? do_wait_intr+0xf0/0xf0 [12164.323880] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12164.329244] mdc_close+0x53b/0xd70 [mdc] [12164.335631] lmv_close+0x18d/0x490 [lmv] [12164.342427] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12164.349917] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12164.356667] ll_md_real_close+0xa5/0x2a0 [lustre] [12164.362703] ll_md_close+0x274/0x800 [lustre] [12164.366713] ll_file_release+0x128/0xc50 [lustre] [12164.374118] __fput+0x10d/0x440 [12164.376678] ____fput+0x12/0x20 [12164.379865] task_work_run+0xc6/0x110 [12164.381082] exit_to_usermode_loop+0x1dd/0x1f0 [12164.390081] do_syscall_64+0x3d6/0x440 [12164.393857] entry_SYSCALL_64_after_hwframe+0x49/0xae [12164.399716] RIP: 0033:0x7f151bee3cd8 [12164.402767] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12164.423158] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12164.427938] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [12164.436370] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [12164.441785] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12164.445287] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12164.448457] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12165.603046] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10235s! [12165.606983] Showing busy workqueues and worker pools: [12165.609608] workqueue events: flags=0x0 [12165.611887] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12165.615549] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [12165.621092] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12165.624500] pending: kfree_rcu_monitor [12165.629293] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12165.633625] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [12165.639341] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12165.642559] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [12165.647092] workqueue events_long: flags=0x0 [12165.649069] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12165.652023] pending: lu_objects_flush [obdclass] [12165.654549] workqueue events_unbound: flags=0x2 [12165.656995] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12165.659713] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12165.664448] workqueue events_power_efficient: flags=0x80 [12165.667581] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12165.670760] pending: do_cache_clean, neigh_periodic_work, check_lifetime [12165.674478] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12165.677596] pending: neigh_periodic_work [12165.679593] workqueue mm_percpu_wq: flags=0x8 [12165.681753] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12165.684944] pending: vmstat_update, lru_add_drain_per_cpu [12165.687602] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12165.690703] pending: vmstat_update [12165.692591] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12165.695674] pending: lru_add_drain_per_cpu [12165.697732] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [12165.700911] pending: lru_add_drain_per_cpu BAR(48) [12165.703452] workqueue writeback: flags=0x4a [12165.705473] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [12165.708530] pending: wb_workfn [12165.710085] workqueue xprtiod: flags=0xa [12165.711934] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12165.715113] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [12165.719138] workqueue ipv6_addrconf: flags=0x40008 [12165.721615] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12165.724704] pending: addrconf_verify_work [12165.726997] workqueue knbd0-recv: flags=0x1a [12165.729272] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12165.733017] in-flight: 658:recv_work [12165.735022] workqueue ptlrpc_pinger: flags=0x2 [12165.737265] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12165.740092] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12165.743178] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10235s workers=4 idle: 32984 37 8 [12165.748677] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12150s workers=2 idle: 661 [12176.056380] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [12176.061557] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12176.097024] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12176.103368] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12176.109271] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12176.114519] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12176.129577] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12176.134629] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [12176.140721] RDX: 0000000000000016 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [12176.146196] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [12176.149618] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [12176.153879] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12176.158367] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [12176.163316] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12176.166263] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12176.170452] Call Trace: [12176.172586] [12176.173636] ? show_regs.cold.9+0x22/0x2f [12176.177833] ? watchdog_timer_fn.cold.6+0x62/0x10a [12176.180374] ? __hrtimer_run_queues+0x1c2/0x4d0 [12176.183229] ? watchdog+0x40/0x40 [12176.185217] ? hrtimer_interrupt+0x128/0x2c0 [12176.187783] ? smp_apic_timer_interrupt+0xa6/0x270 [12176.190221] ? apic_timer_interrupt+0xf/0x20 [12176.192555] [12176.193927] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12176.196586] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12176.199561] do_raw_spin_lock+0x177/0x190 [12176.201629] _raw_spin_lock+0x21/0x30 [12176.203447] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12176.207800] obd_mod_free+0x46/0x140 [ptlrpc] [12176.210653] mdc_commit_open+0x91/0xb0 [mdc] [12176.213322] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12176.216289] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12176.219842] after_reply+0xe2c/0x1780 [ptlrpc] [12176.223053] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12176.226223] ? __next_timer_interrupt+0x160/0x160 [12176.229123] ptlrpcd+0xa82/0xe40 [ptlrpc] [12176.232192] ? do_wait_intr+0xf0/0xf0 [12176.234475] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12176.239509] kthread+0x1d1/0x200 [12176.241310] ? set_kthread_struct+0x70/0x70 [12176.243521] ret_from_fork+0x1f/0x30 [12180.056036] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12180.058685] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12180.086081] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12180.095112] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12180.098863] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12180.101550] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12180.104146] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12180.113397] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [12180.118106] RAX: 0000000000001d87 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12180.120906] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [12180.126641] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [12180.132880] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [12180.136162] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [12180.142257] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [12180.146889] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12180.152409] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12180.155344] Call Trace: [12180.156304] [12180.160876] ? show_regs.cold.9+0x22/0x2f [12180.162534] ? watchdog_timer_fn.cold.6+0x62/0x10a [12180.166628] ? __hrtimer_run_queues+0x1c2/0x4d0 [12180.169187] ? watchdog+0x40/0x40 [12180.171317] ? hrtimer_interrupt+0x128/0x2c0 [12180.172975] ? smp_apic_timer_interrupt+0xa6/0x270 [12180.178089] ? apic_timer_interrupt+0xf/0x20 [12180.179843] [12180.183867] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12180.187577] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12180.193410] do_raw_spin_lock+0x177/0x190 [12180.197190] _raw_spin_lock+0x21/0x30 [12180.200542] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12180.204395] process_one_work+0x2c8/0x700 [12180.208463] worker_thread+0x296/0x6e0 [12180.213012] ? rescuer_thread+0x570/0x570 [12180.217769] kthread+0x1d1/0x200 [12180.221093] ? set_kthread_struct+0x70/0x70 [12180.224751] ret_from_fork+0x1f/0x30 [12192.056106] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [12192.057019] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [12192.061402] Modules linked in: [12192.064205] Modules linked in: lustre(O) [12192.065272] lustre(O) [12192.065273] osp(O) [12192.066629] osp(O) [12192.067634] ofd(O) lod(O) mdt(O) mdd(O) [12192.068559] ofd(O) [12192.069473] mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12192.069526] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12192.069531] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12192.069534] RIP: 0010:native_safe_halt+0x2e/0x50 [12192.069544] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12192.069548] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12192.069552] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12192.069555] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [12192.069556] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [12192.069559] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [12192.069561] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [12192.069564] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [12192.069566] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12192.069568] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [12192.069575] Call Trace: [12192.069580] [12192.069584] ? show_regs.cold.9+0x22/0x2f [12192.069594] ? watchdog_timer_fn.cold.6+0x62/0x10a [12192.069600] ? __hrtimer_run_queues+0x1c2/0x4d0 [12192.069604] ? watchdog+0x40/0x40 [12192.069608] ? hrtimer_interrupt+0x128/0x2c0 [12192.069612] ? smp_apic_timer_interrupt+0xa6/0x270 [12192.069620] ? apic_timer_interrupt+0xf/0x20 [12192.069624] [12192.069627] ? native_safe_halt+0x2e/0x50 [12192.069631] kvm_wait+0xcd/0xe0 [12192.069637] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12192.069646] do_raw_spin_lock+0x177/0x190 [12192.069650] _raw_spin_lock+0x21/0x30 [12192.073538] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) [12192.074489] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12192.097881] osc(O) [12192.103010] ? do_raw_spin_unlock+0x75/0x190 [12192.112870] lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12192.112940] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12192.115179] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12192.144468] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12192.144480] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12192.144496] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12192.145764] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12192.145902] RAX: 0000000000001ad9 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12192.145905] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [12192.145915] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [12192.145918] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [12192.145921] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [12192.145924] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [12192.145927] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12192.145930] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [12192.145938] Call Trace: [12192.145945] [12192.145951] ? show_regs.cold.9+0x22/0x2f [12192.145964] ? watchdog_timer_fn.cold.6+0x62/0x10a [12192.145971] ? __hrtimer_run_queues+0x1c2/0x4d0 [12192.145975] ? watchdog+0x40/0x40 [12192.145980] ? hrtimer_interrupt+0x128/0x2c0 [12192.145984] ? smp_apic_timer_interrupt+0xa6/0x270 [12192.145993] ? apic_timer_interrupt+0xf/0x20 [12192.145998] [12192.146002] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12192.146008] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12192.146768] do_raw_spin_lock+0x177/0x190 [12192.146882] _raw_spin_lock+0x21/0x30 [12192.146890] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12192.148363] ? do_raw_spin_unlock+0x75/0x190 [12192.148371] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12192.149093] ? do_wait_intr+0xf0/0xf0 [12192.149103] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12192.149385] mdc_reint+0x2d/0x110 [mdc] [12192.149422] mdc_unlink+0x658/0x870 [mdc] [12192.149450] lmv_unlink+0x34d/0xbd0 [lmv] [12192.149485] ll_unlink+0x24d/0x7f0 [lustre] [12192.149626] vfs_unlink+0x202/0x330 [12192.149634] do_unlinkat+0x2fa/0x460 [12192.149638] __x64_sys_unlink+0x37/0x50 [12192.149641] do_syscall_64+0xc1/0x440 [12192.149648] entry_SYSCALL_64_after_hwframe+0x49/0xae [12192.149656] RIP: 0033:0x7f151bdee10b [12192.149664] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [12192.149667] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [12192.149672] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [12192.149674] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [12192.149677] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12192.149679] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12192.149681] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12192.442412] ? do_wait_intr+0xf0/0xf0 [12192.444716] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12192.446597] mdc_close+0x53b/0xd70 [mdc] [12192.447874] lmv_close+0x18d/0x490 [lmv] [12192.456458] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12192.459595] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12192.462276] ll_md_real_close+0xa5/0x2a0 [lustre] [12192.465017] ll_md_close+0x274/0x800 [lustre] [12192.466443] ll_file_release+0x128/0xc50 [lustre] [12192.477055] __fput+0x10d/0x440 [12192.478199] ____fput+0x12/0x20 [12192.480239] task_work_run+0xc6/0x110 [12192.481684] exit_to_usermode_loop+0x1dd/0x1f0 [12192.485147] do_syscall_64+0x3d6/0x440 [12192.486750] entry_SYSCALL_64_after_hwframe+0x49/0xae [12192.488404] RIP: 0033:0x7f151bee3cd8 [12192.489442] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12192.509365] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12192.513237] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [12192.519918] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [12192.528126] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12192.532185] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12192.535284] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12195.813414] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10265s! [12195.824577] Showing busy workqueues and worker pools: [12195.830265] workqueue events: flags=0x0 [12195.834212] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12195.841095] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [12195.852398] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12195.856230] pending: kfree_rcu_monitor [12195.859390] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12195.865582] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [12195.876373] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12195.879104] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [12195.883724] workqueue events_long: flags=0x0 [12195.885916] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12195.888375] pending: lu_objects_flush [obdclass] [12195.892136] workqueue events_unbound: flags=0x2 [12195.893845] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12195.896238] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12195.900923] workqueue events_power_efficient: flags=0x80 [12195.908270] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12195.913587] pending: do_cache_clean, neigh_periodic_work, check_lifetime [12195.917242] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12195.923348] pending: neigh_periodic_work [12195.926656] workqueue mm_percpu_wq: flags=0x8 [12195.928859] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12195.932471] pending: vmstat_update, lru_add_drain_per_cpu [12195.935163] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12195.939442] pending: vmstat_update [12195.942574] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12195.946622] pending: lru_add_drain_per_cpu [12195.952737] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [12195.958058] pending: lru_add_drain_per_cpu BAR(48) [12195.962530] workqueue writeback: flags=0x4a [12195.967461] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [12195.971090] pending: wb_workfn [12195.973021] workqueue xprtiod: flags=0xa [12195.975133] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12195.978090] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [12195.982663] workqueue ipv6_addrconf: flags=0x40008 [12195.985155] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12195.988713] pending: addrconf_verify_work [12195.991211] workqueue knbd0-recv: flags=0x1a [12195.993264] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12195.996369] in-flight: 658:recv_work [12195.998104] workqueue ptlrpc_pinger: flags=0x2 [12196.000353] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12196.002720] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12196.005267] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10265s workers=4 idle: 32984 37 8 [12196.012161] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12180s workers=2 idle: 661 [12204.056397] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [12204.061410] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12204.114516] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12204.122205] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12204.130681] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12204.132410] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12204.148205] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12204.158109] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [12204.176184] RDX: 000000000000654f RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [12204.180454] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [12204.184442] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [12204.188266] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12204.193506] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [12204.198123] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12204.202027] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12204.214177] Call Trace: [12204.215079] [12204.215773] ? show_regs.cold.9+0x22/0x2f [12204.224847] ? watchdog_timer_fn.cold.6+0x62/0x10a [12204.226534] ? __hrtimer_run_queues+0x1c2/0x4d0 [12204.231856] ? watchdog+0x40/0x40 [12204.233090] ? hrtimer_interrupt+0x128/0x2c0 [12204.240351] ? smp_apic_timer_interrupt+0xa6/0x270 [12204.242712] ? apic_timer_interrupt+0xf/0x20 [12204.245860] [12204.246850] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12204.251191] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12204.255148] do_raw_spin_lock+0x177/0x190 [12204.258466] _raw_spin_lock+0x21/0x30 [12204.262098] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12204.269938] obd_mod_free+0x46/0x140 [ptlrpc] [12204.274074] mdc_commit_open+0x91/0xb0 [mdc] [12204.279199] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12204.284652] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12204.289367] after_reply+0xe2c/0x1780 [ptlrpc] [12204.292883] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12204.303272] ? __next_timer_interrupt+0x160/0x160 [12204.313307] ptlrpcd+0xa82/0xe40 [ptlrpc] [12204.316923] ? do_wait_intr+0xf0/0xf0 [12204.322736] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12204.324729] kthread+0x1d1/0x200 [12204.338633] ? set_kthread_struct+0x70/0x70 [12204.344250] ret_from_fork+0x1f/0x30 [12208.061647] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12208.069549] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12208.108157] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12208.119942] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12208.126591] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12208.129915] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12208.134937] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12208.155425] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12208.157557] RAX: 0000000000007f84 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12208.164336] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [12208.172123] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [12208.175238] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [12208.181237] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [12208.187856] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [12208.191529] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12208.199115] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12208.203597] Call Trace: [12208.204778] [12208.205504] ? show_regs.cold.9+0x22/0x2f [12208.207468] ? watchdog_timer_fn.cold.6+0x62/0x10a [12208.211278] ? __hrtimer_run_queues+0x1c2/0x4d0 [12208.213582] ? watchdog+0x40/0x40 [12208.216233] ? hrtimer_interrupt+0x128/0x2c0 [12208.219454] ? smp_apic_timer_interrupt+0xa6/0x270 [12208.222291] ? apic_timer_interrupt+0xf/0x20 [12208.225364] [12208.226623] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12208.230262] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12208.238701] do_raw_spin_lock+0x177/0x190 [12208.241942] _raw_spin_lock+0x21/0x30 [12208.244807] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12208.253210] process_one_work+0x2c8/0x700 [12208.256377] worker_thread+0x296/0x6e0 [12208.260428] ? rescuer_thread+0x570/0x570 [12208.263351] kthread+0x1d1/0x200 [12208.266218] ? set_kthread_struct+0x70/0x70 [12208.269506] ret_from_fork+0x1f/0x30 [12220.057613] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [12220.060917] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [12220.067236] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12220.067305] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12220.067312] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12220.067430] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12220.067445] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12220.067448] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12220.067453] RAX: 000000000000551b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12220.067456] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [12220.067458] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [12220.067460] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [12220.067463] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [12220.067466] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [12220.067469] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12220.067471] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [12220.067477] Call Trace: [12220.067482] [12220.067486] ? show_regs.cold.9+0x22/0x2f [12220.067494] ? watchdog_timer_fn.cold.6+0x62/0x10a [12220.067498] ? __hrtimer_run_queues+0x1c2/0x4d0 [12220.067657] ? watchdog+0x40/0x40 [12220.067661] ? hrtimer_interrupt+0x128/0x2c0 [12220.067666] ? smp_apic_timer_interrupt+0xa6/0x270 [12220.067674] ? apic_timer_interrupt+0xf/0x20 [12220.067678] [12220.067682] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12220.067687] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12220.067692] do_raw_spin_lock+0x177/0x190 [12220.067697] _raw_spin_lock+0x21/0x30 [12220.067703] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12220.067934] ? do_raw_spin_unlock+0x75/0x190 [12220.067938] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12220.068085] ? do_wait_intr+0xf0/0xf0 [12220.068090] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12220.068215] mdc_reint+0x2d/0x110 [mdc] [12220.068243] mdc_unlink+0x658/0x870 [mdc] [12220.068260] lmv_unlink+0x34d/0xbd0 [lmv] [12220.068278] ll_unlink+0x24d/0x7f0 [lustre] [12220.068419] vfs_unlink+0x202/0x330 [12220.068426] do_unlinkat+0x2fa/0x460 [12220.068429] __x64_sys_unlink+0x37/0x50 [12220.068431] do_syscall_64+0xc1/0x440 [12220.068436] entry_SYSCALL_64_after_hwframe+0x49/0xae [12220.068441] RIP: 0033:0x7f151bdee10b [12220.068446] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [12220.068449] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [12220.068452] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [12220.068453] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [12220.068455] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12220.068456] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12220.068458] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12220.174176] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12220.187766] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12220.191013] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12220.193768] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12220.195276] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12220.200833] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12220.203243] RAX: 0000000000007619 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12220.205478] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [12220.207821] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [12220.209777] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [12220.211819] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [12220.214147] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [12220.216419] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12220.218486] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [12220.220493] Call Trace: [12220.221383] [12220.222071] ? show_regs.cold.9+0x22/0x2f [12220.223286] ? watchdog_timer_fn.cold.6+0x62/0x10a [12220.224952] ? __hrtimer_run_queues+0x1c2/0x4d0 [12220.226496] ? watchdog+0x40/0x40 [12220.227626] ? hrtimer_interrupt+0x128/0x2c0 [12220.229217] ? smp_apic_timer_interrupt+0xa6/0x270 [12220.230307] ? apic_timer_interrupt+0xf/0x20 [12220.231472] [12220.232268] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12220.233857] do_raw_spin_lock+0x177/0x190 [12220.235067] _raw_spin_lock+0x21/0x30 [12220.236167] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12220.238082] ? do_raw_spin_unlock+0x75/0x190 [12220.239470] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12220.241297] ? do_wait_intr+0xf0/0xf0 [12220.242694] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12220.244263] mdc_close+0x53b/0xd70 [mdc] [12220.245650] lmv_close+0x18d/0x490 [lmv] [12220.246835] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12220.248671] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12220.250468] ll_md_real_close+0xa5/0x2a0 [lustre] [12220.252060] ll_md_close+0x274/0x800 [lustre] [12220.253690] ll_file_release+0x128/0xc50 [lustre] [12220.255187] __fput+0x10d/0x440 [12220.256347] ____fput+0x12/0x20 [12220.257185] task_work_run+0xc6/0x110 [12220.258149] exit_to_usermode_loop+0x1dd/0x1f0 [12220.259575] do_syscall_64+0x3d6/0x440 [12220.260594] entry_SYSCALL_64_after_hwframe+0x49/0xae [12220.262494] RIP: 0033:0x7f151bee3cd8 [12220.263677] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12220.270144] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12220.272106] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [12220.276138] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [12220.279749] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12220.283890] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12220.287251] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12226.018601] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10295s! [12226.027079] Showing busy workqueues and worker pools: [12226.032015] workqueue events: flags=0x0 [12226.034804] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12226.040314] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [12226.061852] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12226.071847] pending: kfree_rcu_monitor [12226.073380] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12226.091331] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [12226.107827] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12226.112258] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [12226.126681] workqueue events_long: flags=0x0 [12226.131426] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12226.133113] pending: lu_objects_flush [obdclass] [12226.146388] workqueue events_unbound: flags=0x2 [12226.156669] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12226.166136] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12226.196203] workqueue events_power_efficient: flags=0x80 [12226.209262] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12226.218025] pending: do_cache_clean, neigh_periodic_work, check_lifetime [12226.226154] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12226.234463] pending: neigh_periodic_work [12226.237532] workqueue mm_percpu_wq: flags=0x8 [12226.243153] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12226.253403] pending: vmstat_update, lru_add_drain_per_cpu [12226.262834] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12226.267913] pending: vmstat_update [12226.270785] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12226.278959] pending: lru_add_drain_per_cpu [12226.296305] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [12226.306788] pending: lru_add_drain_per_cpu BAR(48) [12226.321881] workqueue writeback: flags=0x4a [12226.329896] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [12226.335222] pending: wb_workfn [12226.344687] workqueue xprtiod: flags=0xa [12226.346432] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12226.358887] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [12226.369062] workqueue ipv6_addrconf: flags=0x40008 [12226.381082] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12226.393194] pending: addrconf_verify_work [12226.399990] workqueue knbd0-recv: flags=0x1a [12226.403016] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12226.415134] in-flight: 658:recv_work [12226.419377] workqueue ptlrpc_pinger: flags=0x2 [12226.421067] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12226.430002] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12226.433057] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10295s workers=4 idle: 32984 37 8 [12226.436766] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12211s workers=2 idle: 661 [12232.060665] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [12232.071706] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12232.106221] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12232.111218] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12232.114570] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12232.118944] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12232.127204] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12232.131353] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [12232.139985] RDX: 00000000000047af RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [12232.145080] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [12232.149976] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [12232.152805] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12232.155204] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [12232.160022] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12232.164113] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12232.172194] Call Trace: [12232.173018] [12232.174169] ? show_regs.cold.9+0x22/0x2f [12232.175657] ? watchdog_timer_fn.cold.6+0x62/0x10a [12232.177274] ? __hrtimer_run_queues+0x1c2/0x4d0 [12232.179169] ? watchdog+0x40/0x40 [12232.180322] ? hrtimer_interrupt+0x128/0x2c0 [12232.182016] ? smp_apic_timer_interrupt+0xa6/0x270 [12232.189204] ? apic_timer_interrupt+0xf/0x20 [12232.190730] [12232.191278] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12232.193951] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12232.195826] do_raw_spin_lock+0x177/0x190 [12232.198916] _raw_spin_lock+0x21/0x30 [12232.200586] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12232.202644] obd_mod_free+0x46/0x140 [ptlrpc] [12232.204645] mdc_commit_open+0x91/0xb0 [mdc] [12232.206318] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12232.209804] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12232.212165] after_reply+0xe2c/0x1780 [ptlrpc] [12232.214143] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12232.216275] ? __next_timer_interrupt+0x160/0x160 [12232.218654] ptlrpcd+0xa82/0xe40 [ptlrpc] [12232.221314] ? do_wait_intr+0xf0/0xf0 [12232.224718] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12232.228763] kthread+0x1d1/0x200 [12232.230835] ? set_kthread_struct+0x70/0x70 [12232.235995] ret_from_fork+0x1f/0x30 [12236.064603] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12236.076028] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12236.151468] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12236.186013] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12236.219898] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12236.237262] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12236.258121] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12236.305949] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12236.331286] RAX: 0000000000003f0c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12236.350576] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [12236.374749] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [12236.387448] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [12236.400681] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [12236.410316] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [12236.428629] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12236.436169] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12236.447460] Call Trace: [12236.448412] [12236.453412] ? show_regs.cold.9+0x22/0x2f [12236.454963] ? watchdog_timer_fn.cold.6+0x62/0x10a [12236.468224] ? __hrtimer_run_queues+0x1c2/0x4d0 [12236.480087] ? watchdog+0x40/0x40 [12236.485140] ? hrtimer_interrupt+0x128/0x2c0 [12236.495006] ? smp_apic_timer_interrupt+0xa6/0x270 [12236.508824] ? apic_timer_interrupt+0xf/0x20 [12236.510421] [12236.511169] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12236.529272] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12236.540730] do_raw_spin_lock+0x177/0x190 [12236.563693] _raw_spin_lock+0x21/0x30 [12236.565083] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12236.578554] process_one_work+0x2c8/0x700 [12236.587533] worker_thread+0x296/0x6e0 [12236.594694] ? rescuer_thread+0x570/0x570 [12236.600897] kthread+0x1d1/0x200 [12236.619176] ? set_kthread_struct+0x70/0x70 [12236.620873] ret_from_fork+0x1f/0x30 [12244.874409] rcu: INFO: rcu_sched self-detected stall on CPU [12244.877719] rcu: 1-...!: (5957472 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [12244.889699] (t=10320227 jiffies g=249657 q=5445) [12244.895990] rcu: rcu_sched kthread starved for 10315179 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [12244.916883] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [12244.929736] rcu: RCU grace-period kthread stack dump: [12244.936901] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [12244.952805] Call Trace: [12244.954674] __schedule+0x351/0xcb0 [12244.958758] schedule+0xc0/0x180 [12244.962704] schedule_timeout+0xb4/0x190 [12244.968124] ? do_raw_spin_unlock+0x75/0x190 [12244.972732] ? __next_timer_interrupt+0x160/0x160 [12244.976995] ? prepare_to_swait_event+0xf3/0x140 [12244.983642] rcu_gp_kthread+0x8dd/0x15b0 [12244.986412] ? force_qs_rnp+0x2c0/0x2c0 [12244.992282] kthread+0x1d1/0x200 [12244.996667] ? set_kthread_struct+0x70/0x70 [12245.001521] ret_from_fork+0x1f/0x30 [12245.005669] rcu: Stack dump where RCU GP kthread last ran: [12245.011512] Sending NMI from CPU 1 to CPUs 2: [12245.017877] NMI backtrace for cpu 2 [12245.017884] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12245.017886] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12245.017891] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12245.017894] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12245.017897] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12245.017899] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 [12245.017901] RAX: 00000000000026c3 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12245.017903] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [12245.017905] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [12245.017907] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [12245.017911] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [12245.017912] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [12245.017913] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12245.017915] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12245.017916] Call Trace: [12245.017917] [12245.017918] ? show_regs.cold.9+0x22/0x2f [12245.017919] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [12245.017920] ? nmi_cpu_backtrace_handler+0x15/0x40 [12245.017921] ? nmi_handle+0xbd/0x210 [12245.017922] ? default_do_nmi+0x4b/0x1b0 [12245.017923] ? do_nmi+0x280/0x380 [12245.017924] ? end_repeat_nmi+0x16/0x1f [12245.017925] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12245.017930] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12245.017931] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12245.017932] [12245.017933] do_raw_spin_lock+0x177/0x190 [12245.017934] _raw_spin_lock+0x21/0x30 [12245.017936] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12245.017937] process_one_work+0x2c8/0x700 [12245.017938] worker_thread+0x296/0x6e0 [12245.017939] ? rescuer_thread+0x570/0x570 [12245.017940] kthread+0x1d1/0x200 [12245.017941] ? set_kthread_struct+0x70/0x70 [12245.017943] ret_from_fork+0x1f/0x30 [12245.026260] NMI backtrace for cpu 1 [12245.107010] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12245.110676] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12245.113235] Call Trace: [12245.114008] [12245.114701] ? dump_stack+0xbb/0x10e [12245.115764] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [12245.117246] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [12245.118755] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [12245.120342] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [12245.121842] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [12245.123445] ? rcu_dump_cpu_stacks+0x11b/0x171 [12245.124764] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [12245.126328] ? update_process_times+0x6d/0xc0 [12245.127646] ? tick_sched_handle.isra.18+0x27/0x90 [12245.129036] ? tick_sched_timer+0x53/0xb0 [12245.130231] ? __hrtimer_run_queues+0x1c2/0x4d0 [12245.131614] ? tick_sched_handle.isra.18+0x90/0x90 [12245.133093] ? hrtimer_interrupt+0x128/0x2c0 [12245.134387] ? smp_apic_timer_interrupt+0xa6/0x270 [12245.135816] ? apic_timer_interrupt+0xf/0x20 [12245.137414] [12245.138045] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12245.139765] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12245.141409] ? do_raw_spin_lock+0x177/0x190 [12245.142679] ? _raw_spin_lock+0x21/0x30 [12245.143883] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12245.145423] ? obd_mod_free+0x46/0x140 [ptlrpc] [12245.147243] ? mdc_commit_open+0x91/0xb0 [mdc] [12245.148659] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12245.150644] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12245.152662] ? after_reply+0xe2c/0x1780 [ptlrpc] [12245.154402] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12245.156246] ? __next_timer_interrupt+0x160/0x160 [12245.157705] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [12245.159569] ? do_wait_intr+0xf0/0xf0 [12245.160740] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12245.162634] ? kthread+0x1d1/0x200 [12245.163739] ? set_kthread_struct+0x70/0x70 [12245.165001] ? ret_from_fork+0x1f/0x30 [12248.062052] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [12248.072518] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12248.115022] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12248.122403] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12248.129120] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12248.133416] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12248.147214] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12248.153911] RAX: 000000000000659f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12248.162422] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [12248.167693] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [12248.173450] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [12248.177255] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [12248.182968] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [12248.188002] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12248.191809] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [12248.195761] Call Trace: [12248.197125] [12248.200033] ? show_regs.cold.9+0x22/0x2f [12248.202636] ? watchdog_timer_fn.cold.6+0x62/0x10a [12248.205965] ? __hrtimer_run_queues+0x1c2/0x4d0 [12248.208742] ? watchdog+0x40/0x40 [12248.210939] ? hrtimer_interrupt+0x128/0x2c0 [12248.212395] ? smp_apic_timer_interrupt+0xa6/0x270 [12248.213888] ? apic_timer_interrupt+0xf/0x20 [12248.215387] [12248.216174] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12248.218023] do_raw_spin_lock+0x177/0x190 [12248.219354] _raw_spin_lock+0x21/0x30 [12248.220597] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12248.222471] ? do_raw_spin_unlock+0x75/0x190 [12248.224051] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12248.226020] ? do_wait_intr+0xf0/0xf0 [12248.227197] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12248.229205] mdc_close+0x53b/0xd70 [mdc] [12248.230648] lmv_close+0x18d/0x490 [lmv] [12248.231894] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12248.233687] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12248.235444] ll_md_real_close+0xa5/0x2a0 [lustre] [12248.236827] ll_md_close+0x274/0x800 [lustre] [12248.238155] ll_file_release+0x128/0xc50 [lustre] [12248.239748] __fput+0x10d/0x440 [12248.240693] ____fput+0x12/0x20 [12248.241615] task_work_run+0xc6/0x110 [12248.242647] exit_to_usermode_loop+0x1dd/0x1f0 [12248.243945] do_syscall_64+0x3d6/0x440 [12248.244893] entry_SYSCALL_64_after_hwframe+0x49/0xae [12248.246330] RIP: 0033:0x7f151bee3cd8 [12248.247303] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12248.252776] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12248.258111] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [12248.260979] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [12248.266811] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12248.273839] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12248.277995] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12252.058266] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [12252.067384] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12252.104509] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12252.111846] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12252.126729] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12252.131053] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12252.142376] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12252.150686] RAX: 0000000000005961 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12252.158548] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [12252.162549] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [12252.170132] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [12252.173823] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [12252.179227] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [12252.188790] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12252.192658] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [12252.201671] Call Trace: [12252.203549] [12252.205630] ? show_regs.cold.9+0x22/0x2f [12252.207819] ? watchdog_timer_fn.cold.6+0x62/0x10a [12252.214419] ? __hrtimer_run_queues+0x1c2/0x4d0 [12252.216187] ? watchdog+0x40/0x40 [12252.218446] ? hrtimer_interrupt+0x128/0x2c0 [12252.220894] ? smp_apic_timer_interrupt+0xa6/0x270 [12252.222960] ? apic_timer_interrupt+0xf/0x20 [12252.224729] [12252.225862] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12252.233642] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12252.236142] do_raw_spin_lock+0x177/0x190 [12252.238715] _raw_spin_lock+0x21/0x30 [12252.240514] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12252.243157] ? do_raw_spin_unlock+0x75/0x190 [12252.245248] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12252.247675] ? do_wait_intr+0xf0/0xf0 [12252.248981] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12252.250981] mdc_reint+0x2d/0x110 [mdc] [12252.252579] mdc_unlink+0x658/0x870 [mdc] [12252.257899] lmv_unlink+0x34d/0xbd0 [lmv] [12252.260035] ll_unlink+0x24d/0x7f0 [lustre] [12252.263035] vfs_unlink+0x202/0x330 [12252.264965] do_unlinkat+0x2fa/0x460 [12252.271300] __x64_sys_unlink+0x37/0x50 [12252.273001] do_syscall_64+0xc1/0x440 [12252.275019] entry_SYSCALL_64_after_hwframe+0x49/0xae [12252.282510] RIP: 0033:0x7f151bdee10b [12252.284091] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [12252.292100] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [12252.295823] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [12252.304206] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [12252.308591] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12252.312731] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12252.315786] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12256.739171] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10326s! [12256.761489] Showing busy workqueues and worker pools: [12256.770382] workqueue events: flags=0x0 [12256.776448] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12256.779036] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [12256.801344] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12256.806991] pending: kfree_rcu_monitor [12256.818961] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12256.831612] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [12256.842205] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12256.849815] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [12256.858030] workqueue events_long: flags=0x0 [12256.860250] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12256.868567] pending: lu_objects_flush [obdclass] [12256.883824] workqueue events_unbound: flags=0x2 [12256.896765] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12256.905055] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12256.924935] workqueue events_power_efficient: flags=0x80 [12256.937744] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12256.949625] pending: do_cache_clean, neigh_periodic_work, check_lifetime [12256.959447] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12256.963351] pending: neigh_periodic_work [12256.970470] workqueue mm_percpu_wq: flags=0x8 [12256.975690] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12256.979446] pending: vmstat_update, lru_add_drain_per_cpu [12256.983383] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12256.988804] pending: vmstat_update [12256.990891] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12256.994563] pending: lru_add_drain_per_cpu [12256.996691] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [12256.999975] pending: lru_add_drain_per_cpu BAR(48) [12257.004549] workqueue writeback: flags=0x4a [12257.006221] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [12257.011173] pending: wb_workfn [12257.013381] workqueue xprtiod: flags=0xa [12257.015585] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12257.018402] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [12257.022692] workqueue ipv6_addrconf: flags=0x40008 [12257.024877] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12257.028542] pending: addrconf_verify_work [12257.030904] workqueue knbd0-recv: flags=0x1a [12257.033292] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12257.036569] in-flight: 658:recv_work [12257.038832] workqueue ptlrpc_pinger: flags=0x2 [12257.041095] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12257.044052] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12257.047836] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10326s workers=4 idle: 32984 37 8 [12257.051443] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12241s workers=2 idle: 661 [12272.055024] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [12272.056048] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [12272.059485] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12272.059559] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12272.070723] Modules linked in: [12272.095244] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12272.095258] RIP: 0010:native_safe_halt+0x2e/0x50 [12272.095275] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12272.095279] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12272.095285] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [12272.095287] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [12272.095289] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [12272.095291] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [12272.095293] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12272.095297] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [12272.095299] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12272.095302] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12272.095310] Call Trace: [12272.095315] [12272.095321] ? show_regs.cold.9+0x22/0x2f [12272.095330] ? watchdog_timer_fn.cold.6+0x62/0x10a [12272.095336] ? __hrtimer_run_queues+0x1c2/0x4d0 [12272.095341] ? watchdog+0x40/0x40 [12272.095345] ? hrtimer_interrupt+0x128/0x2c0 [12272.095349] ? smp_apic_timer_interrupt+0xa6/0x270 [12272.095624] ? apic_timer_interrupt+0xf/0x20 [12272.095630] [12272.095633] ? native_safe_halt+0x2e/0x50 [12272.095637] kvm_wait+0xcd/0xe0 [12272.095656] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12272.095664] do_raw_spin_lock+0x177/0x190 [12272.095668] _raw_spin_lock+0x21/0x30 [12272.095674] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12272.095969] obd_mod_free+0x46/0x140 [ptlrpc] [12272.096324] mdc_commit_open+0x91/0xb0 [mdc] [12272.096427] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12272.096625] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12272.096833] after_reply+0xe2c/0x1780 [ptlrpc] [12272.108181] lustre(O) [12272.109470] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12272.119854] osp(O) [12272.121819] ? __next_timer_interrupt+0x160/0x160 [12272.121838] ptlrpcd+0xa82/0xe40 [ptlrpc] [12272.122305] ? do_wait_intr+0xf0/0xf0 [12272.122315] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12272.150720] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12272.150805] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12272.150813] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12272.150820] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12272.159290] kthread+0x1d1/0x200 [12272.159310] ? set_kthread_struct+0x70/0x70 [12272.159315] ret_from_fork+0x1f/0x30 [12272.289015] [12272.289577] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12272.296743] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12272.315486] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12272.325594] RAX: 0000000000004787 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12272.329611] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [12272.335790] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [12272.341913] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [12272.347296] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [12272.351262] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [12272.355979] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12272.360238] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12272.363877] Call Trace: [12272.365104] [12272.366546] ? show_regs.cold.9+0x22/0x2f [12272.368613] ? watchdog_timer_fn.cold.6+0x62/0x10a [12272.372994] ? __hrtimer_run_queues+0x1c2/0x4d0 [12272.374999] ? watchdog+0x40/0x40 [12272.376700] ? hrtimer_interrupt+0x128/0x2c0 [12272.378907] ? smp_apic_timer_interrupt+0xa6/0x270 [12272.384701] ? apic_timer_interrupt+0xf/0x20 [12272.388581] [12272.389505] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12272.392751] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12272.397036] do_raw_spin_lock+0x177/0x190 [12272.399828] _raw_spin_lock+0x21/0x30 [12272.402231] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12272.405334] process_one_work+0x2c8/0x700 [12272.407323] worker_thread+0x296/0x6e0 [12272.409218] ? rescuer_thread+0x570/0x570 [12272.411607] kthread+0x1d1/0x200 [12272.417626] ? set_kthread_struct+0x70/0x70 [12272.420071] ret_from_fork+0x1f/0x30 [12276.057051] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [12276.065870] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12276.100655] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12276.125568] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12276.140644] RIP: 0010:native_safe_halt+0x2e/0x50 [12276.158585] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12276.194218] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12276.206579] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12276.221861] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [12276.227467] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [12276.235238] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [12276.244310] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [12276.254691] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [12276.266581] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12276.276237] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [12276.293658] Call Trace: [12276.297293] [12276.299897] ? show_regs.cold.9+0x22/0x2f [12276.304442] ? watchdog_timer_fn.cold.6+0x62/0x10a [12276.314269] ? __hrtimer_run_queues+0x1c2/0x4d0 [12276.320693] ? watchdog+0x40/0x40 [12276.331606] ? hrtimer_interrupt+0x128/0x2c0 [12276.338499] ? smp_apic_timer_interrupt+0xa6/0x270 [12276.342438] ? apic_timer_interrupt+0xf/0x20 [12276.354616] [12276.358425] ? native_safe_halt+0x2e/0x50 [12276.367722] kvm_wait+0xcd/0xe0 [12276.368915] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12276.386856] do_raw_spin_lock+0x177/0x190 [12276.389984] _raw_spin_lock+0x21/0x30 [12276.393741] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12276.407023] ? do_raw_spin_unlock+0x75/0x190 [12276.416384] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12276.426621] ? do_wait_intr+0xf0/0xf0 [12276.436138] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12276.444278] mdc_close+0x53b/0xd70 [mdc] [12276.450419] lmv_close+0x18d/0x490 [lmv] [12276.455396] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12276.461320] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12276.473762] ll_md_real_close+0xa5/0x2a0 [lustre] [12276.480423] ll_md_close+0x274/0x800 [lustre] [12276.485402] ll_file_release+0x128/0xc50 [lustre] [12276.498468] __fput+0x10d/0x440 [12276.501480] ____fput+0x12/0x20 [12276.511237] task_work_run+0xc6/0x110 [12276.517331] exit_to_usermode_loop+0x1dd/0x1f0 [12276.522928] do_syscall_64+0x3d6/0x440 [12276.530099] entry_SYSCALL_64_after_hwframe+0x49/0xae [12276.540619] RIP: 0033:0x7f151bee3cd8 [12276.549834] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12276.571926] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12276.580576] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [12276.593842] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [12276.599475] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12276.613409] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12276.624323] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12284.052037] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [12284.061581] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12284.106125] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12284.111722] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12284.121324] RIP: 0010:native_safe_halt+0x2e/0x50 [12284.122434] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12284.133098] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12284.136924] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12284.140514] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [12284.144970] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [12284.150854] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [12284.153605] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [12284.160259] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [12284.167292] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12284.171944] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [12284.179689] Call Trace: [12284.182593] [12284.184516] ? show_regs.cold.9+0x22/0x2f [12284.186488] ? watchdog_timer_fn.cold.6+0x62/0x10a [12284.188734] ? __hrtimer_run_queues+0x1c2/0x4d0 [12284.195701] ? watchdog+0x40/0x40 [12284.198872] ? hrtimer_interrupt+0x128/0x2c0 [12284.203824] ? smp_apic_timer_interrupt+0xa6/0x270 [12284.206420] ? apic_timer_interrupt+0xf/0x20 [12284.209722] [12284.211398] ? native_safe_halt+0x2e/0x50 [12284.214409] kvm_wait+0xcd/0xe0 [12284.216411] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12284.220027] do_raw_spin_lock+0x177/0x190 [12284.225182] _raw_spin_lock+0x21/0x30 [12284.229593] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12284.233412] ? do_raw_spin_unlock+0x75/0x190 [12284.234510] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12284.235988] ? do_wait_intr+0xf0/0xf0 [12284.244643] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12284.249383] mdc_reint+0x2d/0x110 [mdc] [12284.251720] mdc_unlink+0x658/0x870 [mdc] [12284.254859] lmv_unlink+0x34d/0xbd0 [lmv] [12284.257724] ll_unlink+0x24d/0x7f0 [lustre] [12284.260177] vfs_unlink+0x202/0x330 [12284.262025] do_unlinkat+0x2fa/0x460 [12284.263746] __x64_sys_unlink+0x37/0x50 [12284.265552] do_syscall_64+0xc1/0x440 [12284.266997] entry_SYSCALL_64_after_hwframe+0x49/0xae [12284.268648] RIP: 0033:0x7f151bdee10b [12284.269697] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [12284.290909] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [12284.296679] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [12284.301814] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [12284.314719] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12284.322162] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12284.324683] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12287.457130] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10356s! [12287.464171] Showing busy workqueues and worker pools: [12287.470104] workqueue events: flags=0x0 [12287.473825] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12287.574740] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [12287.580236] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12287.586831] pending: kfree_rcu_monitor [12287.589301] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12287.596271] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [12287.601187] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12287.603412] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [12287.608590] workqueue events_long: flags=0x0 [12287.610328] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12287.614589] pending: lu_objects_flush [obdclass] [12287.618844] workqueue events_unbound: flags=0x2 [12287.621049] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12287.623237] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12287.629360] workqueue events_power_efficient: flags=0x80 [12287.633561] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12287.638306] pending: do_cache_clean, neigh_periodic_work, check_lifetime [12287.642171] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12287.645205] pending: neigh_periodic_work [12287.653489] workqueue mm_percpu_wq: flags=0x8 [12287.654844] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12287.673108] pending: vmstat_update, lru_add_drain_per_cpu [12287.677505] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12287.681224] pending: vmstat_update [12287.683752] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12287.686787] pending: lru_add_drain_per_cpu [12287.695444] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [12287.705904] pending: lru_add_drain_per_cpu BAR(48) [12287.707371] workqueue writeback: flags=0x4a [12287.708622] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [12287.713716] pending: wb_workfn [12287.719434] workqueue xprtiod: flags=0xa [12287.722400] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12287.729773] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [12287.740661] workqueue ipv6_addrconf: flags=0x40008 [12287.747855] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12287.753672] pending: addrconf_verify_work [12287.759386] workqueue knbd0-recv: flags=0x1a [12287.762695] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12287.768016] in-flight: 658:recv_work [12287.769635] workqueue ptlrpc_pinger: flags=0x2 [12287.771780] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12287.774618] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12287.777440] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10357s workers=4 idle: 32984 37 8 [12287.780423] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12272s workers=2 idle: 661 [12300.059745] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [12300.061521] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [12300.073685] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12300.073779] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12300.073787] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12300.073794] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12300.074078] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12300.074089] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12300.074093] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12300.074098] RAX: 000000000000532f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12300.074100] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [12300.074103] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [12300.074105] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [12300.074108] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [12300.074111] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [12300.074114] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12300.074116] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12300.074125] Call Trace: [12300.074131] [12300.074137] ? show_regs.cold.9+0x22/0x2f [12300.074148] ? watchdog_timer_fn.cold.6+0x62/0x10a [12300.074155] ? __hrtimer_run_queues+0x1c2/0x4d0 [12300.074159] ? watchdog+0x40/0x40 [12300.074162] ? hrtimer_interrupt+0x128/0x2c0 [12300.074166] ? smp_apic_timer_interrupt+0xa6/0x270 [12300.074173] ? apic_timer_interrupt+0xf/0x20 [12300.074178] [12300.074180] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12300.074185] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12300.074190] do_raw_spin_lock+0x177/0x190 [12300.074195] _raw_spin_lock+0x21/0x30 [12300.074200] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12300.074402] process_one_work+0x2c8/0x700 [12300.074410] worker_thread+0x296/0x6e0 [12300.074415] ? rescuer_thread+0x570/0x570 [12300.074419] kthread+0x1d1/0x200 [12300.074423] ? set_kthread_struct+0x70/0x70 [12300.074426] ret_from_fork+0x1f/0x30 [12300.392890] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12300.448549] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12300.457346] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12300.463144] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12300.466436] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12300.477057] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12300.481020] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [12300.484396] RDX: 0000000000007b9d RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [12300.488558] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [12300.491964] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [12300.503420] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12300.510115] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [12300.518337] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12300.527928] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12300.535566] Call Trace: [12300.536527] [12300.537897] ? show_regs.cold.9+0x22/0x2f [12300.540549] ? watchdog_timer_fn.cold.6+0x62/0x10a [12300.542673] ? __hrtimer_run_queues+0x1c2/0x4d0 [12300.546096] ? watchdog+0x40/0x40 [12300.547916] ? hrtimer_interrupt+0x128/0x2c0 [12300.557058] ? smp_apic_timer_interrupt+0xa6/0x270 [12300.569199] ? apic_timer_interrupt+0xf/0x20 [12300.572034] [12300.576941] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12300.583841] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12300.586819] do_raw_spin_lock+0x177/0x190 [12300.592500] _raw_spin_lock+0x21/0x30 [12300.594385] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12300.601918] obd_mod_free+0x46/0x140 [ptlrpc] [12300.606673] mdc_commit_open+0x91/0xb0 [mdc] [12300.615648] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12300.620660] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12300.627656] after_reply+0xe2c/0x1780 [ptlrpc] [12300.633450] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12300.640822] ? __next_timer_interrupt+0x160/0x160 [12300.645807] ptlrpcd+0xa82/0xe40 [ptlrpc] [12300.649921] ? do_wait_intr+0xf0/0xf0 [12300.656367] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12300.659217] kthread+0x1d1/0x200 [12300.664230] ? set_kthread_struct+0x70/0x70 [12300.668452] ret_from_fork+0x1f/0x30 [12304.057010] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [12304.061262] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12304.086391] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12304.092360] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12304.097066] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12304.100397] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12304.109021] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12304.112543] RAX: 0000000000006de9 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12304.115621] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [12304.118656] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [12304.122336] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [12304.125677] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [12304.129110] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [12304.143251] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12304.155558] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [12304.164461] Call Trace: [12304.167915] [12304.174090] ? show_regs.cold.9+0x22/0x2f [12304.175803] ? watchdog_timer_fn.cold.6+0x62/0x10a [12304.186990] ? __hrtimer_run_queues+0x1c2/0x4d0 [12304.189551] ? watchdog+0x40/0x40 [12304.193682] ? hrtimer_interrupt+0x128/0x2c0 [12304.195553] ? smp_apic_timer_interrupt+0xa6/0x270 [12304.200590] ? apic_timer_interrupt+0xf/0x20 [12304.203846] [12304.204454] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12304.205997] do_raw_spin_lock+0x177/0x190 [12304.217633] _raw_spin_lock+0x21/0x30 [12304.220231] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12304.223046] ? do_raw_spin_unlock+0x75/0x190 [12304.225083] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12304.228414] ? do_wait_intr+0xf0/0xf0 [12304.230033] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12304.234623] mdc_close+0x53b/0xd70 [mdc] [12304.236048] lmv_close+0x18d/0x490 [lmv] [12304.237240] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12304.239528] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12304.244141] ll_md_real_close+0xa5/0x2a0 [lustre] [12304.246904] ll_md_close+0x274/0x800 [lustre] [12304.249451] ll_file_release+0x128/0xc50 [lustre] [12304.252424] __fput+0x10d/0x440 [12304.253544] ____fput+0x12/0x20 [12304.257416] task_work_run+0xc6/0x110 [12304.261869] exit_to_usermode_loop+0x1dd/0x1f0 [12304.265461] do_syscall_64+0x3d6/0x440 [12304.267420] entry_SYSCALL_64_after_hwframe+0x49/0xae [12304.271323] RIP: 0033:0x7f151bee3cd8 [12304.276459] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12304.298532] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12304.305312] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [12304.314447] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [12304.321805] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12304.329113] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12304.338071] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12312.052026] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [12312.055165] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12312.072099] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12312.076782] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12312.080543] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12312.082856] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12312.089446] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12312.092593] RAX: 000000000000649a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12312.096073] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [12312.099049] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [12312.103112] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [12312.107931] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [12312.111191] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [12312.115155] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12312.117532] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [12312.121025] Call Trace: [12312.122063] [12312.123113] ? show_regs.cold.9+0x22/0x2f [12312.124931] ? watchdog_timer_fn.cold.6+0x62/0x10a [12312.127184] ? __hrtimer_run_queues+0x1c2/0x4d0 [12312.129398] ? watchdog+0x40/0x40 [12312.131166] ? hrtimer_interrupt+0x128/0x2c0 [12312.133857] ? smp_apic_timer_interrupt+0xa6/0x270 [12312.135780] ? apic_timer_interrupt+0xf/0x20 [12312.137316] [12312.138168] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12312.149262] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12312.151919] do_raw_spin_lock+0x177/0x190 [12312.155318] _raw_spin_lock+0x21/0x30 [12312.156559] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12312.160151] ? do_raw_spin_unlock+0x75/0x190 [12312.161680] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12312.164338] ? do_wait_intr+0xf0/0xf0 [12312.166285] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12312.169096] mdc_reint+0x2d/0x110 [mdc] [12312.170573] mdc_unlink+0x658/0x870 [mdc] [12312.172407] lmv_unlink+0x34d/0xbd0 [lmv] [12312.174444] ll_unlink+0x24d/0x7f0 [lustre] [12312.178993] vfs_unlink+0x202/0x330 [12312.180329] do_unlinkat+0x2fa/0x460 [12312.184925] __x64_sys_unlink+0x37/0x50 [12312.186367] do_syscall_64+0xc1/0x440 [12312.189212] entry_SYSCALL_64_after_hwframe+0x49/0xae [12312.194963] RIP: 0033:0x7f151bdee10b [12312.199557] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [12312.228533] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [12312.239360] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [12312.252233] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [12312.257306] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12312.267317] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12312.273717] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12318.177025] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10387s! [12318.180066] Showing busy workqueues and worker pools: [12318.182311] workqueue events: flags=0x0 [12318.184224] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12318.186968] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [12318.190874] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12318.193095] pending: kfree_rcu_monitor [12318.194315] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12318.197125] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [12318.201438] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12318.204956] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [12318.209075] workqueue events_long: flags=0x0 [12318.210433] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12318.213150] pending: lu_objects_flush [obdclass] [12318.215720] workqueue events_unbound: flags=0x2 [12318.217621] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12318.220802] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12318.225427] workqueue events_power_efficient: flags=0x80 [12318.227624] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12318.229715] pending: do_cache_clean, neigh_periodic_work, check_lifetime [12318.231953] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12318.234888] pending: neigh_periodic_work [12318.236194] workqueue mm_percpu_wq: flags=0x8 [12318.238072] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12318.240538] pending: vmstat_update, lru_add_drain_per_cpu [12318.242829] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12318.245758] pending: vmstat_update [12318.247136] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12318.250039] pending: lru_add_drain_per_cpu [12318.252258] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [12318.256225] pending: lru_add_drain_per_cpu BAR(48) [12318.258490] workqueue writeback: flags=0x4a [12318.260018] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [12318.262714] pending: wb_workfn [12318.264392] workqueue xprtiod: flags=0xa [12318.266389] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12318.269262] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [12318.272216] workqueue ipv6_addrconf: flags=0x40008 [12318.274122] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12318.276969] pending: addrconf_verify_work [12318.278847] workqueue knbd0-recv: flags=0x1a [12318.280228] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12318.283120] in-flight: 658:recv_work [12318.285622] workqueue ptlrpc_pinger: flags=0x2 [12318.288216] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12318.290949] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12318.293946] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10387s workers=4 idle: 32984 37 8 [12318.296841] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12303s workers=2 idle: 661 [12328.056503] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [12328.059828] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12328.066719] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12328.066824] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12328.066833] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12328.066897] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12328.066917] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12328.066922] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12328.066927] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [12328.066929] RDX: 000000000000554d RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [12328.066932] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [12328.066934] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [12328.066937] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12328.066941] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [12328.066944] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12328.066946] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12328.066956] Call Trace: [12328.066962] [12328.067041] ? show_regs.cold.9+0x22/0x2f [12328.067056] ? watchdog_timer_fn.cold.6+0x62/0x10a [12328.067062] ? __hrtimer_run_queues+0x1c2/0x4d0 [12328.067066] ? watchdog+0x40/0x40 [12328.067071] ? hrtimer_interrupt+0x128/0x2c0 [12328.067074] ? smp_apic_timer_interrupt+0xa6/0x270 [12328.067083] ? apic_timer_interrupt+0xf/0x20 [12328.067088] [12328.067091] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12328.067095] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12328.067100] do_raw_spin_lock+0x177/0x190 [12328.067110] _raw_spin_lock+0x21/0x30 [12328.067117] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12328.073236] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12328.073315] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12328.073324] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12328.073331] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12328.073619] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12328.073631] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12328.073635] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12328.073640] RAX: 0000000000002a89 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12328.073642] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [12328.073644] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [12328.073646] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [12328.073649] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [12328.073652] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [12328.073654] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12328.073657] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12328.073666] Call Trace: [12328.073671] [12328.073678] ? show_regs.cold.9+0x22/0x2f [12328.073688] ? watchdog_timer_fn.cold.6+0x62/0x10a [12328.073695] ? __hrtimer_run_queues+0x1c2/0x4d0 [12328.073699] ? watchdog+0x40/0x40 [12328.073703] ? hrtimer_interrupt+0x128/0x2c0 [12328.073707] ? smp_apic_timer_interrupt+0xa6/0x270 [12328.073715] ? apic_timer_interrupt+0xf/0x20 [12328.073719] [12328.073722] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12328.073726] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12328.073731] do_raw_spin_lock+0x177/0x190 [12328.073736] _raw_spin_lock+0x21/0x30 [12328.073741] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12328.073951] process_one_work+0x2c8/0x700 [12328.073958] worker_thread+0x296/0x6e0 [12328.073962] ? rescuer_thread+0x570/0x570 [12328.073967] kthread+0x1d1/0x200 [12328.073971] ? set_kthread_struct+0x70/0x70 [12328.073975] ret_from_fork+0x1f/0x30 [12328.436493] obd_mod_free+0x46/0x140 [ptlrpc] [12328.441336] mdc_commit_open+0x91/0xb0 [mdc] [12328.444286] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12328.451511] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12328.456317] after_reply+0xe2c/0x1780 [ptlrpc] [12328.459766] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12328.463381] ? __next_timer_interrupt+0x160/0x160 [12328.465781] ptlrpcd+0xa82/0xe40 [ptlrpc] [12328.469142] ? do_wait_intr+0xf0/0xf0 [12328.470637] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12328.473545] kthread+0x1d1/0x200 [12328.475275] ? set_kthread_struct+0x70/0x70 [12328.478164] ret_from_fork+0x1f/0x30 [12332.057046] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [12332.059911] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12332.075288] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12332.078862] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12332.081322] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12332.083622] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12332.090893] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 [12332.093814] RAX: 0000000000005900 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12332.096878] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [12332.099778] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [12332.102807] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [12332.105279] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [12332.108104] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [12332.110966] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12332.113606] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [12332.116327] Call Trace: [12332.117375] [12332.118177] ? show_regs.cold.9+0x22/0x2f [12332.119355] ? watchdog_timer_fn.cold.6+0x62/0x10a [12332.121050] ? __hrtimer_run_queues+0x1c2/0x4d0 [12332.122389] ? watchdog+0x40/0x40 [12332.124646] ? hrtimer_interrupt+0x128/0x2c0 [12332.126103] ? smp_apic_timer_interrupt+0xa6/0x270 [12332.127977] ? apic_timer_interrupt+0xf/0x20 [12332.129643] [12332.130262] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12332.132081] do_raw_spin_lock+0x177/0x190 [12332.133745] _raw_spin_lock+0x21/0x30 [12332.135245] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12332.137890] ? do_raw_spin_unlock+0x75/0x190 [12332.139116] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12332.141199] ? do_wait_intr+0xf0/0xf0 [12332.142655] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12332.144920] mdc_close+0x53b/0xd70 [mdc] [12332.146351] lmv_close+0x18d/0x490 [lmv] [12332.147703] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12332.149747] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12332.151596] ll_md_real_close+0xa5/0x2a0 [lustre] [12332.153042] ll_md_close+0x274/0x800 [lustre] [12332.154458] ll_file_release+0x128/0xc50 [lustre] [12332.156264] __fput+0x10d/0x440 [12332.157262] ____fput+0x12/0x20 [12332.158255] task_work_run+0xc6/0x110 [12332.159715] exit_to_usermode_loop+0x1dd/0x1f0 [12332.161679] do_syscall_64+0x3d6/0x440 [12332.163152] entry_SYSCALL_64_after_hwframe+0x49/0xae [12332.164881] RIP: 0033:0x7f151bee3cd8 [12332.166063] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12332.172104] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12332.174812] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [12332.177246] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [12332.180330] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12332.182954] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12332.186574] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12344.062106] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [12344.078858] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12344.114897] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12344.121540] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12344.125753] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12344.130122] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12344.145791] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 [12344.150848] RAX: 0000000000007100 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12344.158982] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [12344.165995] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [12344.175215] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [12344.182323] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [12344.188508] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [12344.196645] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12344.202965] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [12344.208964] Call Trace: [12344.210072] [12344.210964] ? show_regs.cold.9+0x22/0x2f [12344.215151] ? watchdog_timer_fn.cold.6+0x62/0x10a [12344.220949] ? __hrtimer_run_queues+0x1c2/0x4d0 [12344.224625] ? watchdog+0x40/0x40 [12344.229647] ? hrtimer_interrupt+0x128/0x2c0 [12344.231445] ? smp_apic_timer_interrupt+0xa6/0x270 [12344.236626] ? apic_timer_interrupt+0xf/0x20 [12344.248543] [12344.249625] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12344.257162] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12344.258984] do_raw_spin_lock+0x177/0x190 [12344.260409] _raw_spin_lock+0x21/0x30 [12344.268534] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12344.276271] ? do_raw_spin_unlock+0x75/0x190 [12344.278196] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12344.287895] ? do_wait_intr+0xf0/0xf0 [12344.292735] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12344.302701] mdc_reint+0x2d/0x110 [mdc] [12344.306408] mdc_unlink+0x658/0x870 [mdc] [12344.312522] lmv_unlink+0x34d/0xbd0 [lmv] [12344.313986] ll_unlink+0x24d/0x7f0 [lustre] [12344.321011] vfs_unlink+0x202/0x330 [12344.323422] do_unlinkat+0x2fa/0x460 [12344.325453] __x64_sys_unlink+0x37/0x50 [12344.334629] do_syscall_64+0xc1/0x440 [12344.337873] entry_SYSCALL_64_after_hwframe+0x49/0xae [12344.341754] RIP: 0033:0x7f151bdee10b [12344.344813] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [12344.364819] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [12344.368400] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [12344.378553] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [12344.391968] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12344.399944] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12344.409248] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12348.384116] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10417s! [12348.388029] Showing busy workqueues and worker pools: [12348.394144] workqueue events: flags=0x0 [12348.400917] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12348.407838] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [12348.418455] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12348.422122] pending: kfree_rcu_monitor [12348.425546] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12348.430097] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [12348.438772] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12348.445496] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [12348.449995] workqueue events_long: flags=0x0 [12348.451987] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12348.455669] pending: lu_objects_flush [obdclass] [12348.461238] workqueue events_unbound: flags=0x2 [12348.465818] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12348.470365] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12348.478982] workqueue events_power_efficient: flags=0x80 [12348.482734] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12348.487937] pending: do_cache_clean, neigh_periodic_work, check_lifetime [12348.496381] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12348.504941] pending: neigh_periodic_work [12348.508532] workqueue mm_percpu_wq: flags=0x8 [12348.510152] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12348.517019] pending: vmstat_update, lru_add_drain_per_cpu [12348.519804] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12348.523754] pending: vmstat_update [12348.537404] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12348.545462] pending: lru_add_drain_per_cpu [12348.547605] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [12348.560178] pending: lru_add_drain_per_cpu BAR(48) [12348.569757] workqueue writeback: flags=0x4a [12348.575817] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [12348.585563] pending: wb_workfn [12348.595051] workqueue xprtiod: flags=0xa [12348.602338] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12348.604602] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [12348.615840] workqueue ipv6_addrconf: flags=0x40008 [12348.617994] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12348.624431] pending: addrconf_verify_work [12348.630380] workqueue knbd0-recv: flags=0x1a [12348.635951] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12348.642328] in-flight: 658:recv_work [12348.647179] workqueue ptlrpc_pinger: flags=0x2 [12348.650014] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12348.653255] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12348.662874] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10418s workers=4 idle: 32984 37 8 [12348.667779] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12333s workers=2 idle: 661 [12356.055242] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [12356.056086] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12356.061720] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) [12356.072250] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12356.072348] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12356.072360] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12356.072367] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12356.072736] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12356.072747] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12356.072751] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12356.072756] RAX: 0000000000001287 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12356.072758] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [12356.072760] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [12356.072763] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [12356.072765] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [12356.072768] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [12356.072771] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12356.072773] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12356.072781] Call Trace: [12356.072787] [12356.072794] ? show_regs.cold.9+0x22/0x2f [12356.072803] ? watchdog_timer_fn.cold.6+0x62/0x10a [12356.072810] ? __hrtimer_run_queues+0x1c2/0x4d0 [12356.072814] ? watchdog+0x40/0x40 [12356.072819] ? hrtimer_interrupt+0x128/0x2c0 [12356.072824] ? smp_apic_timer_interrupt+0xa6/0x270 [12356.072832] ? apic_timer_interrupt+0xf/0x20 [12356.072837] [12356.072840] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12356.072844] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12356.072848] do_raw_spin_lock+0x177/0x190 [12356.072852] _raw_spin_lock+0x21/0x30 [12356.072856] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12356.073426] process_one_work+0x2c8/0x700 [12356.073432] worker_thread+0x296/0x6e0 [12356.073437] ? rescuer_thread+0x570/0x570 [12356.073440] kthread+0x1d1/0x200 [12356.073444] ? set_kthread_struct+0x70/0x70 [12356.073448] ret_from_fork+0x1f/0x30 [12356.271960] libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12356.287100] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12356.292747] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12356.297328] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12356.300846] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12356.319977] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12356.328111] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [12356.332979] RDX: 0000000000007f6e RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [12356.336219] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [12356.339830] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [12356.343065] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12356.346027] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [12356.351172] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12356.353655] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12356.357372] Call Trace: [12356.358362] [12356.359287] ? show_regs.cold.9+0x22/0x2f [12356.361133] ? watchdog_timer_fn.cold.6+0x62/0x10a [12356.364135] ? __hrtimer_run_queues+0x1c2/0x4d0 [12356.367775] ? watchdog+0x40/0x40 [12356.369177] ? hrtimer_interrupt+0x128/0x2c0 [12356.371663] ? smp_apic_timer_interrupt+0xa6/0x270 [12356.373768] ? apic_timer_interrupt+0xf/0x20 [12356.375691] [12356.376640] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12356.378746] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12356.380902] do_raw_spin_lock+0x177/0x190 [12356.383152] _raw_spin_lock+0x21/0x30 [12356.387245] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12356.389048] obd_mod_free+0x46/0x140 [ptlrpc] [12356.390979] mdc_commit_open+0x91/0xb0 [mdc] [12356.394112] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12356.397077] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12356.400009] after_reply+0xe2c/0x1780 [ptlrpc] [12356.403250] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12356.406121] ? __next_timer_interrupt+0x160/0x160 [12356.408806] ptlrpcd+0xa82/0xe40 [ptlrpc] [12356.412302] ? do_wait_intr+0xf0/0xf0 [12356.414304] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12356.418064] kthread+0x1d1/0x200 [12356.419746] ? set_kthread_struct+0x70/0x70 [12356.424665] ret_from_fork+0x1f/0x30 [12360.057023] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [12360.059702] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12360.087032] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12360.095344] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12360.099395] RIP: 0010:native_safe_halt+0x2e/0x50 [12360.102670] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12360.114513] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12360.118421] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12360.128023] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [12360.131903] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [12360.136597] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [12360.140367] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [12360.145951] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [12360.152490] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12360.156214] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [12360.159983] Call Trace: [12360.161596] [12360.162572] ? show_regs.cold.9+0x22/0x2f [12360.164992] ? watchdog_timer_fn.cold.6+0x62/0x10a [12360.168893] ? __hrtimer_run_queues+0x1c2/0x4d0 [12360.172449] ? watchdog+0x40/0x40 [12360.175483] ? hrtimer_interrupt+0x128/0x2c0 [12360.177939] ? smp_apic_timer_interrupt+0xa6/0x270 [12360.180842] ? apic_timer_interrupt+0xf/0x20 [12360.185354] [12360.187466] ? native_safe_halt+0x2e/0x50 [12360.189810] kvm_wait+0xcd/0xe0 [12360.193892] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12360.200080] do_raw_spin_lock+0x177/0x190 [12360.202846] _raw_spin_lock+0x21/0x30 [12360.205636] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12360.212918] ? do_raw_spin_unlock+0x75/0x190 [12360.215747] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12360.222876] ? do_wait_intr+0xf0/0xf0 [12360.226143] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12360.229207] mdc_close+0x53b/0xd70 [mdc] [12360.233073] lmv_close+0x18d/0x490 [lmv] [12360.238968] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12360.241427] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12360.243928] ll_md_real_close+0xa5/0x2a0 [lustre] [12360.246254] ll_md_close+0x274/0x800 [lustre] [12360.248087] ll_file_release+0x128/0xc50 [lustre] [12360.250196] __fput+0x10d/0x440 [12360.251135] ____fput+0x12/0x20 [12360.252294] task_work_run+0xc6/0x110 [12360.253804] exit_to_usermode_loop+0x1dd/0x1f0 [12360.255665] do_syscall_64+0x3d6/0x440 [12360.257409] entry_SYSCALL_64_after_hwframe+0x49/0xae [12360.259545] RIP: 0033:0x7f151bee3cd8 [12360.261318] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12360.268333] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12360.271275] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [12360.274231] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [12360.277388] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12360.280228] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12360.282691] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12376.052022] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [12376.054873] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12376.072888] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12376.077839] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12376.082063] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12376.084035] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12376.092932] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12376.095599] RAX: 0000000000005597 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12376.099393] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [12376.103333] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [12376.106266] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [12376.110914] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [12376.113741] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [12376.117878] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12376.120750] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [12376.123725] Call Trace: [12376.124940] [12376.125668] ? show_regs.cold.9+0x22/0x2f [12376.127402] ? watchdog_timer_fn.cold.6+0x62/0x10a [12376.129042] ? __hrtimer_run_queues+0x1c2/0x4d0 [12376.130863] ? watchdog+0x40/0x40 [12376.132252] ? hrtimer_interrupt+0x128/0x2c0 [12376.133987] ? smp_apic_timer_interrupt+0xa6/0x270 [12376.136481] ? apic_timer_interrupt+0xf/0x20 [12376.138464] [12376.139210] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12376.141087] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12376.142761] do_raw_spin_lock+0x177/0x190 [12376.144655] _raw_spin_lock+0x21/0x30 [12376.146249] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12376.148717] ? do_raw_spin_unlock+0x75/0x190 [12376.150048] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12376.151903] ? do_wait_intr+0xf0/0xf0 [12376.153449] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12376.156101] mdc_reint+0x2d/0x110 [mdc] [12376.157779] mdc_unlink+0x658/0x870 [mdc] [12376.159763] lmv_unlink+0x34d/0xbd0 [lmv] [12376.161600] ll_unlink+0x24d/0x7f0 [lustre] [12376.162864] vfs_unlink+0x202/0x330 [12376.163834] do_unlinkat+0x2fa/0x460 [12376.164814] __x64_sys_unlink+0x37/0x50 [12376.165790] do_syscall_64+0xc1/0x440 [12376.167261] entry_SYSCALL_64_after_hwframe+0x49/0xae [12376.169665] RIP: 0033:0x7f151bdee10b [12376.170767] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [12376.178624] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [12376.182063] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [12376.186133] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [12376.189818] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12376.193586] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12376.197666] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12379.107223] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10448s! [12379.113170] Showing busy workqueues and worker pools: [12379.115338] workqueue events: flags=0x0 [12379.121823] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12379.127033] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [12379.135177] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12379.140034] pending: kfree_rcu_monitor [12379.143902] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12379.150721] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [12379.158725] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12379.162379] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [12379.167920] workqueue events_long: flags=0x0 [12379.171499] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12379.175819] pending: lu_objects_flush [obdclass] [12379.178624] workqueue events_unbound: flags=0x2 [12379.180683] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12379.191231] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12379.198229] workqueue events_power_efficient: flags=0x80 [12379.200155] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12379.203674] pending: do_cache_clean, neigh_periodic_work, check_lifetime [12379.207041] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12379.210310] pending: neigh_periodic_work [12379.213575] workqueue mm_percpu_wq: flags=0x8 [12379.217309] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12379.222294] pending: vmstat_update, lru_add_drain_per_cpu [12379.227345] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12379.236755] pending: vmstat_update [12379.240276] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12379.243995] pending: lru_add_drain_per_cpu [12379.245910] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [12379.249177] pending: lru_add_drain_per_cpu BAR(48) [12379.252192] workqueue writeback: flags=0x4a [12379.253864] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [12379.256516] pending: wb_workfn [12379.257569] workqueue xprtiod: flags=0xa [12379.260934] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12379.267493] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [12379.275029] workqueue ipv6_addrconf: flags=0x40008 [12379.280891] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12379.284261] pending: addrconf_verify_work [12379.285775] workqueue knbd0-recv: flags=0x1a [12379.288050] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12379.290797] in-flight: 658:recv_work [12379.292273] workqueue ptlrpc_pinger: flags=0x2 [12379.294452] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12379.299409] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12379.306064] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10448s workers=4 idle: 32984 37 8 [12379.310347] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12364s workers=2 idle: 661 [12384.055030] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [12384.056172] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12384.060754] Modules linked in: [12384.062914] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12384.062987] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12384.062994] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12384.063001] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12384.063368] RIP: 0010:native_safe_halt+0x2e/0x50 [12384.063376] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12384.063379] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12384.063383] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12384.063385] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [12384.063387] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [12384.063388] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [12384.063390] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [12384.063392] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [12384.063394] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12384.063396] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12384.063403] Call Trace: [12384.063407] [12384.063411] ? show_regs.cold.9+0x22/0x2f [12384.063418] ? watchdog_timer_fn.cold.6+0x62/0x10a [12384.063423] ? __hrtimer_run_queues+0x1c2/0x4d0 [12384.063427] ? watchdog+0x40/0x40 [12384.063432] ? hrtimer_interrupt+0x128/0x2c0 [12384.063442] ? smp_apic_timer_interrupt+0xa6/0x270 [12384.063449] ? apic_timer_interrupt+0xf/0x20 [12384.063453] [12384.063456] ? native_safe_halt+0x2e/0x50 [12384.063464] kvm_wait+0xcd/0xe0 [12384.063472] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12384.066456] lustre(O) [12384.081648] do_raw_spin_lock+0x177/0x190 [12384.091393] osp(O) [12384.094835] _raw_spin_lock+0x21/0x30 [12384.100440] ofd(O) [12384.102461] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12384.110241] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12384.110311] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12384.110318] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12384.110322] RIP: 0010:native_safe_halt+0x2e/0x50 [12384.110341] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12384.110346] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12384.110350] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [12384.110353] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [12384.110356] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [12384.110358] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [12384.110361] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12384.110364] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [12384.110368] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12384.110370] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12384.110382] Call Trace: [12384.110387] [12384.110393] ? show_regs.cold.9+0x22/0x2f [12384.110404] ? watchdog_timer_fn.cold.6+0x62/0x10a [12384.110410] ? __hrtimer_run_queues+0x1c2/0x4d0 [12384.110413] ? watchdog+0x40/0x40 [12384.110417] ? hrtimer_interrupt+0x128/0x2c0 [12384.110421] ? smp_apic_timer_interrupt+0xa6/0x270 [12384.110433] ? apic_timer_interrupt+0xf/0x20 [12384.110438] [12384.110444] ? native_safe_halt+0x2e/0x50 [12384.110448] kvm_wait+0xcd/0xe0 [12384.110456] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12384.110465] do_raw_spin_lock+0x177/0x190 [12384.110469] _raw_spin_lock+0x21/0x30 [12384.110474] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12384.113123] process_one_work+0x2c8/0x700 [12384.113141] worker_thread+0x296/0x6e0 [12384.113146] ? rescuer_thread+0x570/0x570 [12384.113149] kthread+0x1d1/0x200 [12384.113153] ? set_kthread_struct+0x70/0x70 [12384.113157] ret_from_fork+0x1f/0x30 [12384.343426] obd_mod_free+0x46/0x140 [ptlrpc] [12384.348793] mdc_commit_open+0x91/0xb0 [mdc] [12384.351492] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12384.357076] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12384.363036] after_reply+0xe2c/0x1780 [ptlrpc] [12384.367396] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12384.371580] ? __next_timer_interrupt+0x160/0x160 [12384.375372] ptlrpcd+0xa82/0xe40 [ptlrpc] [12384.378437] ? do_wait_intr+0xf0/0xf0 [12384.379460] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12384.385370] kthread+0x1d1/0x200 [12384.387930] ? set_kthread_struct+0x70/0x70 [12384.392682] ret_from_fork+0x1f/0x30 [12388.061114] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [12388.068684] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12388.115584] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12388.125197] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12388.129690] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12388.132475] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12388.151390] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12388.153836] RAX: 0000000000005635 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12388.157282] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [12388.163077] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [12388.166836] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [12388.171117] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [12388.176183] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [12388.180020] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12388.183439] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [12388.186977] Call Trace: [12388.190981] [12388.193133] ? show_regs.cold.9+0x22/0x2f [12388.199059] ? watchdog_timer_fn.cold.6+0x62/0x10a [12388.205433] ? __hrtimer_run_queues+0x1c2/0x4d0 [12388.207075] ? watchdog+0x40/0x40 [12388.209734] ? hrtimer_interrupt+0x128/0x2c0 [12388.214111] ? smp_apic_timer_interrupt+0xa6/0x270 [12388.219593] ? apic_timer_interrupt+0xf/0x20 [12388.225047] [12388.225786] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12388.230673] do_raw_spin_lock+0x177/0x190 [12388.234472] _raw_spin_lock+0x21/0x30 [12388.239743] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12388.241613] ? do_raw_spin_unlock+0x75/0x190 [12388.246349] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12388.250052] ? do_wait_intr+0xf0/0xf0 [12388.252039] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12388.254266] mdc_close+0x53b/0xd70 [mdc] [12388.258536] lmv_close+0x18d/0x490 [lmv] [12388.263391] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12388.268249] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12388.274359] ll_md_real_close+0xa5/0x2a0 [lustre] [12388.279635] ll_md_close+0x274/0x800 [lustre] [12388.285212] ll_file_release+0x128/0xc50 [lustre] [12388.288413] __fput+0x10d/0x440 [12388.291839] ____fput+0x12/0x20 [12388.296362] task_work_run+0xc6/0x110 [12388.299846] exit_to_usermode_loop+0x1dd/0x1f0 [12388.303214] do_syscall_64+0x3d6/0x440 [12388.314039] entry_SYSCALL_64_after_hwframe+0x49/0xae [12388.321658] RIP: 0033:0x7f151bee3cd8 [12388.326527] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12388.356745] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12388.363114] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [12388.371626] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [12388.378147] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12388.393945] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12388.401787] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12404.052046] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [12404.055286] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12404.119558] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12404.137969] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12404.151052] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12404.170686] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12404.197679] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12404.202676] RAX: 0000000000004f88 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12404.208590] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [12404.220376] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [12404.229359] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [12404.239467] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [12404.250135] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [12404.263083] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12404.272703] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [12404.284818] Call Trace: [12404.287347] [12404.288397] ? show_regs.cold.9+0x22/0x2f [12404.292996] ? watchdog_timer_fn.cold.6+0x62/0x10a [12404.295905] ? __hrtimer_run_queues+0x1c2/0x4d0 [12404.298397] ? watchdog+0x40/0x40 [12404.300372] ? hrtimer_interrupt+0x128/0x2c0 [12404.303902] ? smp_apic_timer_interrupt+0xa6/0x270 [12404.306094] ? apic_timer_interrupt+0xf/0x20 [12404.309336] [12404.315114] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12404.318779] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12404.331189] do_raw_spin_lock+0x177/0x190 [12404.334102] _raw_spin_lock+0x21/0x30 [12404.336218] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12404.340147] ? do_raw_spin_unlock+0x75/0x190 [12404.344885] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12404.350174] ? do_wait_intr+0xf0/0xf0 [12404.352929] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12404.363381] mdc_reint+0x2d/0x110 [mdc] [12404.365509] mdc_unlink+0x658/0x870 [mdc] [12404.373373] lmv_unlink+0x34d/0xbd0 [lmv] [12404.380233] ll_unlink+0x24d/0x7f0 [lustre] [12404.389145] vfs_unlink+0x202/0x330 [12404.391620] do_unlinkat+0x2fa/0x460 [12404.395427] __x64_sys_unlink+0x37/0x50 [12404.398449] do_syscall_64+0xc1/0x440 [12404.400700] entry_SYSCALL_64_after_hwframe+0x49/0xae [12404.404890] RIP: 0033:0x7f151bdee10b [12404.409399] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [12404.423062] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [12404.427389] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [12404.431343] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [12404.437065] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12404.443091] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12404.448373] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12409.825055] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10479s! [12409.829426] Showing busy workqueues and worker pools: [12409.831699] workqueue events: flags=0x0 [12409.833137] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12409.835555] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [12409.840310] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12409.843541] pending: kfree_rcu_monitor [12409.845118] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12409.848603] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [12409.854792] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12409.858047] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [12409.862116] workqueue events_long: flags=0x0 [12409.864706] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12409.867989] pending: lu_objects_flush [obdclass] [12409.870509] workqueue events_unbound: flags=0x2 [12409.872916] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12409.876215] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12409.881339] workqueue events_power_efficient: flags=0x80 [12409.883786] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12409.886960] pending: do_cache_clean, neigh_periodic_work, check_lifetime [12409.890183] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12409.893506] pending: neigh_periodic_work [12409.895555] workqueue mm_percpu_wq: flags=0x8 [12409.897563] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12409.900529] pending: vmstat_update, lru_add_drain_per_cpu [12409.902777] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12409.905289] pending: vmstat_update [12409.906781] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12409.909852] pending: lru_add_drain_per_cpu [12409.911348] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [12409.914200] pending: lru_add_drain_per_cpu BAR(48) [12409.916882] workqueue writeback: flags=0x4a [12409.918939] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [12409.923263] pending: wb_workfn [12409.925618] workqueue xprtiod: flags=0xa [12409.927112] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12409.930101] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [12409.934165] workqueue ipv6_addrconf: flags=0x40008 [12409.936433] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12409.939545] pending: addrconf_verify_work [12409.941628] workqueue knbd0-recv: flags=0x1a [12409.943104] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12409.946006] in-flight: 658:recv_work [12409.947872] workqueue ptlrpc_pinger: flags=0x2 [12409.950035] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12409.952225] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12409.955362] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10479s workers=4 idle: 32984 37 8 [12409.958319] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12394s workers=2 idle: 661 [12412.057529] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12412.060095] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [12412.066499] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12412.066581] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12412.066592] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12412.066598] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12412.067373] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12412.067386] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12412.067390] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12412.067394] RAX: 0000000000007b22 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12412.067396] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [12412.067398] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [12412.067400] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [12412.067403] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [12412.067406] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [12412.067409] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12412.067411] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12412.067419] Call Trace: [12412.067424] [12412.067429] ? show_regs.cold.9+0x22/0x2f [12412.067445] ? watchdog_timer_fn.cold.6+0x62/0x10a [12412.067452] ? __hrtimer_run_queues+0x1c2/0x4d0 [12412.067456] ? watchdog+0x40/0x40 [12412.067460] ? hrtimer_interrupt+0x128/0x2c0 [12412.067464] ? smp_apic_timer_interrupt+0xa6/0x270 [12412.067477] ? apic_timer_interrupt+0xf/0x20 [12412.067482] [12412.067485] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12412.067490] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12412.067495] do_raw_spin_lock+0x177/0x190 [12412.067500] _raw_spin_lock+0x21/0x30 [12412.067504] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12412.067830] process_one_work+0x2c8/0x700 [12412.067837] worker_thread+0x296/0x6e0 [12412.067842] ? rescuer_thread+0x570/0x570 [12412.078432] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12412.078509] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12412.078517] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12412.078520] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12412.078534] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12412.078539] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12412.078543] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [12412.078545] RDX: 0000000000000bfa RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [12412.078548] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [12412.078550] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [12412.078552] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12412.078555] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [12412.079356] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12412.079361] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12412.079369] Call Trace: [12412.079377] [12412.079383] ? show_regs.cold.9+0x22/0x2f [12412.079395] ? watchdog_timer_fn.cold.6+0x62/0x10a [12412.079402] ? __hrtimer_run_queues+0x1c2/0x4d0 [12412.079406] ? watchdog+0x40/0x40 [12412.079410] ? hrtimer_interrupt+0x128/0x2c0 [12412.079415] ? smp_apic_timer_interrupt+0xa6/0x270 [12412.079423] ? apic_timer_interrupt+0xf/0x20 [12412.079427] [12412.079431] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12412.079437] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12412.079441] do_raw_spin_lock+0x177/0x190 [12412.079446] _raw_spin_lock+0x21/0x30 [12412.079453] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12412.080394] obd_mod_free+0x46/0x140 [ptlrpc] [12412.127801] kthread+0x1d1/0x200 [12412.127819] ? set_kthread_struct+0x70/0x70 [12412.127824] ret_from_fork+0x1f/0x30 [12412.514328] mdc_commit_open+0x91/0xb0 [mdc] [12412.521885] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12412.530077] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12412.533200] after_reply+0xe2c/0x1780 [ptlrpc] [12412.541243] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12412.551972] ? __next_timer_interrupt+0x160/0x160 [12412.561135] ptlrpcd+0xa82/0xe40 [ptlrpc] [12412.562768] ? do_wait_intr+0xf0/0xf0 [12412.570659] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12412.574558] kthread+0x1d1/0x200 [12412.577002] ? set_kthread_struct+0x70/0x70 [12412.580551] ret_from_fork+0x1f/0x30 [12416.057053] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [12416.061172] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12416.086530] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12416.092905] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12416.097728] RIP: 0010:native_safe_halt+0x2e/0x50 [12416.101307] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12416.113818] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12416.118419] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12416.123732] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [12416.128524] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [12416.133133] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [12416.137950] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [12416.142647] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [12416.147528] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12416.154744] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [12416.161890] Call Trace: [12416.163344] [12416.164550] ? show_regs.cold.9+0x22/0x2f [12416.166689] ? watchdog_timer_fn.cold.6+0x62/0x10a [12416.168380] ? __hrtimer_run_queues+0x1c2/0x4d0 [12416.170860] ? watchdog+0x40/0x40 [12416.172256] ? hrtimer_interrupt+0x128/0x2c0 [12416.173835] ? smp_apic_timer_interrupt+0xa6/0x270 [12416.176511] ? apic_timer_interrupt+0xf/0x20 [12416.179049] [12416.180172] ? native_safe_halt+0x2e/0x50 [12416.183315] kvm_wait+0xcd/0xe0 [12416.186357] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12416.191128] do_raw_spin_lock+0x177/0x190 [12416.194970] _raw_spin_lock+0x21/0x30 [12416.196226] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12416.199880] ? do_raw_spin_unlock+0x75/0x190 [12416.201336] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12416.203659] ? do_wait_intr+0xf0/0xf0 [12416.205087] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12416.207500] mdc_close+0x53b/0xd70 [mdc] [12416.208872] lmv_close+0x18d/0x490 [lmv] [12416.210303] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12416.212453] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12416.214692] ll_md_real_close+0xa5/0x2a0 [lustre] [12416.216279] ll_md_close+0x274/0x800 [lustre] [12416.217920] ll_file_release+0x128/0xc50 [lustre] [12416.219611] __fput+0x10d/0x440 [12416.220775] ____fput+0x12/0x20 [12416.222221] task_work_run+0xc6/0x110 [12416.223363] exit_to_usermode_loop+0x1dd/0x1f0 [12416.225068] do_syscall_64+0x3d6/0x440 [12416.226407] entry_SYSCALL_64_after_hwframe+0x49/0xae [12416.227900] RIP: 0033:0x7f151bee3cd8 [12416.229299] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12416.235225] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12416.237547] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [12416.241582] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [12416.249495] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12416.260650] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12416.266573] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12424.877749] rcu: INFO: rcu_sched self-detected stall on CPU [12424.880959] rcu: 1-...!: (6059367 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [12424.888212] (t=10500230 jiffies g=249657 q=5508) [12424.890943] rcu: rcu_sched kthread starved for 10495182 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [12424.898488] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [12424.902253] rcu: RCU grace-period kthread stack dump: [12424.904662] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [12424.911371] Call Trace: [12424.912938] __schedule+0x351/0xcb0 [12424.914629] schedule+0xc0/0x180 [12424.916578] schedule_timeout+0xb4/0x190 [12424.919724] ? do_raw_spin_unlock+0x75/0x190 [12424.922840] ? __next_timer_interrupt+0x160/0x160 [12424.927084] ? prepare_to_swait_event+0xf3/0x140 [12424.933672] rcu_gp_kthread+0x8dd/0x15b0 [12424.936681] ? force_qs_rnp+0x2c0/0x2c0 [12424.939096] kthread+0x1d1/0x200 [12424.940862] ? set_kthread_struct+0x70/0x70 [12424.943883] ret_from_fork+0x1f/0x30 [12424.946523] rcu: Stack dump where RCU GP kthread last ran: [12424.949643] Sending NMI from CPU 1 to CPUs 0: [12424.956898] NMI backtrace for cpu 0 [12424.956904] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12424.956907] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12424.956908] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12424.956910] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12424.956916] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 [12424.956918] RAX: 0000000000001c90 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12424.956920] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [12424.956924] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [12424.956925] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [12424.956927] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [12424.956928] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [12424.956930] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12424.956931] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [12424.956932] Call Trace: [12424.956933] [12424.957001] ? show_regs.cold.9+0x22/0x2f [12424.957003] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [12424.957004] ? nmi_cpu_backtrace_handler+0x15/0x40 [12424.957005] ? nmi_handle+0xbd/0x210 [12424.957006] ? default_do_nmi+0x4b/0x1b0 [12424.957007] ? do_nmi+0x280/0x380 [12424.957008] ? end_repeat_nmi+0x16/0x1f [12424.957009] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12424.957011] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12424.957012] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12424.957013] [12424.957014] do_raw_spin_lock+0x177/0x190 [12424.957015] _raw_spin_lock+0x21/0x30 [12424.957016] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12424.957017] ? do_raw_spin_unlock+0x75/0x190 [12424.957018] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12424.957019] ? do_wait_intr+0xf0/0xf0 [12424.957020] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12424.957068] mdc_reint+0x2d/0x110 [mdc] [12424.957070] mdc_unlink+0x658/0x870 [mdc] [12424.957071] lmv_unlink+0x34d/0xbd0 [lmv] [12424.957072] ll_unlink+0x24d/0x7f0 [lustre] [12424.957073] vfs_unlink+0x202/0x330 [12424.957074] do_unlinkat+0x2fa/0x460 [12424.957078] __x64_sys_unlink+0x37/0x50 [12424.957079] do_syscall_64+0xc1/0x440 [12424.957081] entry_SYSCALL_64_after_hwframe+0x49/0xae [12424.957082] RIP: 0033:0x7f151bdee10b [12424.957084] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [12424.957086] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [12424.957089] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [12424.957090] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [12424.957091] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12424.957093] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12424.957094] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12424.957643] NMI backtrace for cpu 1 [12425.138495] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12425.143261] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12425.147579] Call Trace: [12425.148881] [12425.149755] ? dump_stack+0xbb/0x10e [12425.151613] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [12425.154722] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [12425.156716] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [12425.158993] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [12425.161169] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [12425.163382] ? rcu_dump_cpu_stacks+0x11b/0x171 [12425.165992] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [12425.168645] ? update_process_times+0x6d/0xc0 [12425.170491] ? tick_sched_handle.isra.18+0x27/0x90 [12425.173569] ? tick_sched_timer+0x53/0xb0 [12425.175057] ? __hrtimer_run_queues+0x1c2/0x4d0 [12425.177695] ? tick_sched_handle.isra.18+0x90/0x90 [12425.179597] ? hrtimer_interrupt+0x128/0x2c0 [12425.181203] ? smp_apic_timer_interrupt+0xa6/0x270 [12425.183176] ? apic_timer_interrupt+0xf/0x20 [12425.185401] [12425.186586] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12425.188949] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12425.191090] ? do_raw_spin_lock+0x177/0x190 [12425.192360] ? _raw_spin_lock+0x21/0x30 [12425.193607] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12425.195464] ? obd_mod_free+0x46/0x140 [ptlrpc] [12425.197304] ? mdc_commit_open+0x91/0xb0 [mdc] [12425.198761] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12425.200764] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12425.203241] ? after_reply+0xe2c/0x1780 [ptlrpc] [12425.205402] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12425.207311] ? __next_timer_interrupt+0x160/0x160 [12425.208911] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [12425.210370] ? do_wait_intr+0xf0/0xf0 [12425.211476] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12425.213619] ? kthread+0x1d1/0x200 [12425.218086] ? set_kthread_struct+0x70/0x70 [12425.219172] ? ret_from_fork+0x1f/0x30 [12440.032260] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10509s! [12440.035374] Showing busy workqueues and worker pools: [12440.038823] workqueue events: flags=0x0 [12440.042746] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12440.047265] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [12440.055171] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12440.058366] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12440.064203] pending: kfree_rcu_monitor [12440.064243] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12440.064256] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [12440.064430] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12440.064439] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [12440.064733] workqueue events_long: flags=0x0 [12440.064736] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12440.064743] pending: lu_objects_flush [obdclass] [12440.064894] workqueue events_unbound: flags=0x2 [12440.064897] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12440.064904] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12440.064925] workqueue events_power_efficient: flags=0x80 [12440.064927] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12440.064936] pending: do_cache_clean, neigh_periodic_work, check_lifetime [12440.064953] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12440.064959] pending: neigh_periodic_work [12440.064967] workqueue mm_percpu_wq: flags=0x8 [12440.064969] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12440.064975] pending: vmstat_update, lru_add_drain_per_cpu [12440.064985] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12440.064990] pending: vmstat_update [12440.064994] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12440.065005] pending: lru_add_drain_per_cpu [12440.065028] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [12440.065037] pending: lru_add_drain_per_cpu BAR(48) [12440.065050] workqueue writeback: flags=0x4a [12440.065052] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [12440.065060] pending: wb_workfn [12440.065071] workqueue xprtiod: flags=0xa [12440.065073] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12440.065079] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [12440.065096] workqueue ipv6_addrconf: flags=0x40008 [12440.065099] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12440.065106] pending: addrconf_verify_work [12440.065116] workqueue knbd0-recv: flags=0x1a [12440.065119] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12440.065126] in-flight: 658:recv_work [12440.065136] workqueue ptlrpc_pinger: flags=0x2 [12440.065138] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12440.065143] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12440.065357] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10509s workers=4 idle: 32984 37 8 [12440.065370] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12424s workers=2 idle: 661 [12440.278971] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12440.308455] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12440.313108] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12440.316756] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12440.319951] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12440.322250] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12440.329311] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12440.332121] RAX: 0000000000007eab RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12440.342682] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [12440.350756] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [12440.355605] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [12440.360273] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [12440.365354] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [12440.372536] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12440.378326] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12440.387385] Call Trace: [12440.388173] [12440.388862] ? show_regs.cold.9+0x22/0x2f [12440.406570] ? watchdog_timer_fn.cold.6+0x62/0x10a [12440.414129] ? __hrtimer_run_queues+0x1c2/0x4d0 [12440.419647] ? watchdog+0x40/0x40 [12440.421344] ? hrtimer_interrupt+0x128/0x2c0 [12440.426493] ? smp_apic_timer_interrupt+0xa6/0x270 [12440.431939] ? apic_timer_interrupt+0xf/0x20 [12440.433529] [12440.434217] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12440.440663] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12440.444647] do_raw_spin_lock+0x177/0x190 [12440.448205] _raw_spin_lock+0x21/0x30 [12440.451263] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12440.456698] process_one_work+0x2c8/0x700 [12440.459230] worker_thread+0x296/0x6e0 [12440.466848] ? rescuer_thread+0x570/0x570 [12440.469752] kthread+0x1d1/0x200 [12440.471734] ? set_kthread_struct+0x70/0x70 [12440.478498] ret_from_fork+0x1f/0x30 [12444.057024] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [12444.062593] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12444.086839] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12444.093625] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12444.107648] RIP: 0010:native_safe_halt+0x2e/0x50 [12444.109892] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12444.129179] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12444.137647] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12444.145276] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [12444.153222] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [12444.160281] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [12444.167553] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [12444.183011] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [12444.194029] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12444.202584] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [12444.209824] Call Trace: [12444.213696] [12444.214647] ? show_regs.cold.9+0x22/0x2f [12444.221024] ? watchdog_timer_fn.cold.6+0x62/0x10a [12444.225314] ? __hrtimer_run_queues+0x1c2/0x4d0 [12444.229840] ? watchdog+0x40/0x40 [12444.231784] ? hrtimer_interrupt+0x128/0x2c0 [12444.234042] ? smp_apic_timer_interrupt+0xa6/0x270 [12444.236283] ? apic_timer_interrupt+0xf/0x20 [12444.242061] [12444.242829] ? native_safe_halt+0x2e/0x50 [12444.248314] kvm_wait+0xcd/0xe0 [12444.250153] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12444.257834] do_raw_spin_lock+0x177/0x190 [12444.260259] _raw_spin_lock+0x21/0x30 [12444.267411] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12444.274365] ? do_raw_spin_unlock+0x75/0x190 [12444.276968] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12444.285438] ? do_wait_intr+0xf0/0xf0 [12444.290885] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12444.294904] mdc_close+0x53b/0xd70 [mdc] [12444.300773] lmv_close+0x18d/0x490 [lmv] [12444.304798] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12444.310872] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12444.318400] ll_md_real_close+0xa5/0x2a0 [lustre] [12444.321334] ll_md_close+0x274/0x800 [lustre] [12444.323808] ll_file_release+0x128/0xc50 [lustre] [12444.326167] __fput+0x10d/0x440 [12444.327634] ____fput+0x12/0x20 [12444.329069] task_work_run+0xc6/0x110 [12444.330689] exit_to_usermode_loop+0x1dd/0x1f0 [12444.333064] do_syscall_64+0x3d6/0x440 [12444.334682] entry_SYSCALL_64_after_hwframe+0x49/0xae [12444.337906] RIP: 0033:0x7f151bee3cd8 [12444.339583] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12444.348100] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12444.351402] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [12444.354418] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [12444.357759] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12444.361580] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12444.364823] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12452.055043] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [12452.058378] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12452.109672] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12452.118581] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12452.122917] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12452.132595] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12452.150538] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12452.162448] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [12452.169090] RDX: 0000000000002e45 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [12452.176881] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [12452.180594] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [12452.187482] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12452.194276] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [12452.203284] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12452.208474] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12452.216667] Call Trace: [12452.219561] [12452.220490] ? show_regs.cold.9+0x22/0x2f [12452.223037] ? watchdog_timer_fn.cold.6+0x62/0x10a [12452.225693] ? __hrtimer_run_queues+0x1c2/0x4d0 [12452.227813] ? watchdog+0x40/0x40 [12452.229791] ? hrtimer_interrupt+0x128/0x2c0 [12452.232435] ? smp_apic_timer_interrupt+0xa6/0x270 [12452.241269] ? apic_timer_interrupt+0xf/0x20 [12452.244252] [12452.246189] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12452.250117] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12452.255435] do_raw_spin_lock+0x177/0x190 [12452.262156] _raw_spin_lock+0x21/0x30 [12452.266463] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12452.271141] obd_mod_free+0x46/0x140 [ptlrpc] [12452.273414] mdc_commit_open+0x91/0xb0 [mdc] [12452.275555] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12452.278230] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12452.284793] after_reply+0xe2c/0x1780 [ptlrpc] [12452.288326] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12452.293228] ? __next_timer_interrupt+0x160/0x160 [12452.296590] ptlrpcd+0xa82/0xe40 [ptlrpc] [12452.300754] ? do_wait_intr+0xf0/0xf0 [12452.304138] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12452.309186] kthread+0x1d1/0x200 [12452.310894] ? set_kthread_struct+0x70/0x70 [12452.318681] ret_from_fork+0x1f/0x30 [12468.052012] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [12468.056724] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12468.057676] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12468.061771] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12468.061859] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12468.061869] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12468.061876] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12468.062375] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12468.062393] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12468.062397] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12468.062402] RAX: 000000000000298c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12468.062404] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [12468.062407] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [12468.062409] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [12468.062411] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [12468.062415] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [12468.062418] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12468.062424] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12468.062604] Call Trace: [12468.062609] [12468.062614] ? show_regs.cold.9+0x22/0x2f [12468.062623] ? watchdog_timer_fn.cold.6+0x62/0x10a [12468.062630] ? __hrtimer_run_queues+0x1c2/0x4d0 [12468.062635] ? watchdog+0x40/0x40 [12468.062639] ? hrtimer_interrupt+0x128/0x2c0 [12468.062643] ? smp_apic_timer_interrupt+0xa6/0x270 [12468.062653] ? apic_timer_interrupt+0xf/0x20 [12468.062658] [12468.062661] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12468.062671] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12468.062677] do_raw_spin_lock+0x177/0x190 [12468.062693] _raw_spin_lock+0x21/0x30 [12468.062700] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12468.062960] process_one_work+0x2c8/0x700 [12468.062968] worker_thread+0x296/0x6e0 [12468.062973] ? rescuer_thread+0x570/0x570 [12468.062981] kthread+0x1d1/0x200 [12468.062987] ? set_kthread_struct+0x70/0x70 [12468.062990] ret_from_fork+0x1f/0x30 [12468.205321] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12468.210077] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12468.212906] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12468.214969] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12468.222030] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12468.224743] RAX: 0000000000001f8e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12468.227303] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [12468.229872] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [12468.232575] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [12468.235270] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [12468.238269] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [12468.241320] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12468.243486] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [12468.246309] Call Trace: [12468.246973] [12468.247655] ? show_regs.cold.9+0x22/0x2f [12468.249242] ? watchdog_timer_fn.cold.6+0x62/0x10a [12468.250970] ? __hrtimer_run_queues+0x1c2/0x4d0 [12468.252669] ? watchdog+0x40/0x40 [12468.253902] ? hrtimer_interrupt+0x128/0x2c0 [12468.255863] ? smp_apic_timer_interrupt+0xa6/0x270 [12468.257620] ? apic_timer_interrupt+0xf/0x20 [12468.262391] [12468.263898] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12468.266523] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12468.268989] do_raw_spin_lock+0x177/0x190 [12468.271596] _raw_spin_lock+0x21/0x30 [12468.275595] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12468.281276] ? do_raw_spin_unlock+0x75/0x190 [12468.286830] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12468.294458] ? do_wait_intr+0xf0/0xf0 [12468.297481] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12468.302910] mdc_reint+0x2d/0x110 [mdc] [12468.305373] mdc_unlink+0x658/0x870 [mdc] [12468.307023] lmv_unlink+0x34d/0xbd0 [lmv] [12468.310378] ll_unlink+0x24d/0x7f0 [lustre] [12468.312048] vfs_unlink+0x202/0x330 [12468.313373] do_unlinkat+0x2fa/0x460 [12468.314801] __x64_sys_unlink+0x37/0x50 [12468.316321] do_syscall_64+0xc1/0x440 [12468.317601] entry_SYSCALL_64_after_hwframe+0x49/0xae [12468.319326] RIP: 0033:0x7f151bdee10b [12468.321381] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [12468.330294] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [12468.333847] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [12468.336487] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [12468.339577] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12468.342665] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12468.347974] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12470.243035] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10539s! [12470.251132] Showing busy workqueues and worker pools: [12470.256758] workqueue events: flags=0x0 [12470.258777] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12470.261754] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [12470.267891] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12470.271367] pending: kfree_rcu_monitor [12470.275331] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12470.278345] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [12470.283337] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12470.286183] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [12470.289351] workqueue events_long: flags=0x0 [12470.292103] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12470.299478] pending: lu_objects_flush [obdclass] [12470.301638] workqueue events_unbound: flags=0x2 [12470.305653] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12470.311884] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12470.319930] workqueue events_power_efficient: flags=0x80 [12470.324797] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12470.327688] pending: do_cache_clean, neigh_periodic_work, check_lifetime [12470.332134] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12470.337762] pending: neigh_periodic_work [12470.341978] workqueue mm_percpu_wq: flags=0x8 [12470.345016] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12470.351754] pending: vmstat_update, lru_add_drain_per_cpu [12470.353888] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12470.358445] pending: vmstat_update [12470.363968] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12470.366813] pending: lru_add_drain_per_cpu [12470.368993] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [12470.372237] pending: lru_add_drain_per_cpu BAR(48) [12470.375921] workqueue writeback: flags=0x4a [12470.380217] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [12470.383915] pending: wb_workfn [12470.388245] workqueue xprtiod: flags=0xa [12470.391381] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12470.396267] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [12470.409037] workqueue ipv6_addrconf: flags=0x40008 [12470.411690] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12470.417901] pending: addrconf_verify_work [12470.427740] workqueue knbd0-recv: flags=0x1a [12470.434085] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12470.437278] in-flight: 658:recv_work [12470.444145] workqueue ptlrpc_pinger: flags=0x2 [12470.454969] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12470.462906] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12470.466318] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10539s workers=4 idle: 32984 37 8 [12470.469313] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12455s workers=2 idle: 661 [12472.057259] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [12472.062922] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12472.084020] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12472.088526] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12472.091986] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12472.094267] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12472.101124] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12472.103731] RAX: 0000000000002f0e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12472.106053] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [12472.108523] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [12472.110862] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [12472.113330] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [12472.116224] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [12472.119522] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12472.121823] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [12472.124526] Call Trace: [12472.125546] [12472.126356] ? show_regs.cold.9+0x22/0x2f [12472.127723] ? watchdog_timer_fn.cold.6+0x62/0x10a [12472.129357] ? __hrtimer_run_queues+0x1c2/0x4d0 [12472.130984] ? watchdog+0x40/0x40 [12472.132278] ? hrtimer_interrupt+0x128/0x2c0 [12472.133959] ? smp_apic_timer_interrupt+0xa6/0x270 [12472.135954] ? apic_timer_interrupt+0xf/0x20 [12472.137762] [12472.138658] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12472.140506] do_raw_spin_lock+0x177/0x190 [12472.141986] _raw_spin_lock+0x21/0x30 [12472.143474] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12472.145845] ? do_raw_spin_unlock+0x75/0x190 [12472.147320] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12472.149547] ? do_wait_intr+0xf0/0xf0 [12472.150880] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12472.152970] mdc_close+0x53b/0xd70 [mdc] [12472.154456] lmv_close+0x18d/0x490 [lmv] [12472.156165] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12472.158319] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12472.160517] ll_md_real_close+0xa5/0x2a0 [lustre] [12472.162258] ll_md_close+0x274/0x800 [lustre] [12472.164043] ll_file_release+0x128/0xc50 [lustre] [12472.166045] __fput+0x10d/0x440 [12472.167291] ____fput+0x12/0x20 [12472.168658] task_work_run+0xc6/0x110 [12472.169918] exit_to_usermode_loop+0x1dd/0x1f0 [12472.171638] do_syscall_64+0x3d6/0x440 [12472.173163] entry_SYSCALL_64_after_hwframe+0x49/0xae [12472.175310] RIP: 0033:0x7f151bee3cd8 [12472.176720] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12472.183665] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12472.186051] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [12472.188347] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [12472.191265] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12472.195135] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12472.197744] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12480.055128] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [12480.058330] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12480.075913] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12480.080221] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12480.084155] RIP: 0010:native_safe_halt+0x2e/0x50 [12480.086169] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12480.094230] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12480.097422] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [12480.100577] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [12480.103650] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [12480.106733] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [12480.109657] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12480.112630] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [12480.115843] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12480.117699] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12480.119838] Call Trace: [12480.120649] [12480.121275] ? show_regs.cold.9+0x22/0x2f [12480.122623] ? watchdog_timer_fn.cold.6+0x62/0x10a [12480.124720] ? __hrtimer_run_queues+0x1c2/0x4d0 [12480.126747] ? watchdog+0x40/0x40 [12480.127811] ? hrtimer_interrupt+0x128/0x2c0 [12480.129347] ? smp_apic_timer_interrupt+0xa6/0x270 [12480.131384] ? apic_timer_interrupt+0xf/0x20 [12480.133494] [12480.134396] ? native_safe_halt+0x2e/0x50 [12480.135848] kvm_wait+0xcd/0xe0 [12480.137354] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12480.139502] do_raw_spin_lock+0x177/0x190 [12480.141576] _raw_spin_lock+0x21/0x30 [12480.143397] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12480.145897] obd_mod_free+0x46/0x140 [ptlrpc] [12480.148374] mdc_commit_open+0x91/0xb0 [mdc] [12480.150323] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12480.152903] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12480.155708] after_reply+0xe2c/0x1780 [ptlrpc] [12480.158286] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12480.160692] ? __next_timer_interrupt+0x160/0x160 [12480.162811] ptlrpcd+0xa82/0xe40 [ptlrpc] [12480.164787] ? do_wait_intr+0xf0/0xf0 [12480.166771] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12480.169572] kthread+0x1d1/0x200 [12480.170979] ? set_kthread_struct+0x70/0x70 [12480.172623] ret_from_fork+0x1f/0x30 [12496.052168] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [12496.055237] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12496.056073] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12496.086355] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12496.093332] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12496.093411] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12496.093419] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12496.093425] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12496.093684] RIP: 0010:native_safe_halt+0x2e/0x50 [12496.093695] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12496.093700] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12496.093704] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12496.093706] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [12496.093709] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [12496.093711] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [12496.093713] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [12496.093716] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [12496.093719] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12496.093720] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12496.093729] Call Trace: [12496.093735] [12496.094288] ? show_regs.cold.9+0x22/0x2f [12496.094308] ? watchdog_timer_fn.cold.6+0x62/0x10a [12496.094316] ? __hrtimer_run_queues+0x1c2/0x4d0 [12496.094320] ? watchdog+0x40/0x40 [12496.094328] ? hrtimer_interrupt+0x128/0x2c0 [12496.094336] ? smp_apic_timer_interrupt+0xa6/0x270 [12496.094343] ? apic_timer_interrupt+0xf/0x20 [12496.094348] [12496.094351] ? native_safe_halt+0x2e/0x50 [12496.094355] kvm_wait+0xcd/0xe0 [12496.094367] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12496.094375] do_raw_spin_lock+0x177/0x190 [12496.094380] _raw_spin_lock+0x21/0x30 [12496.094384] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12496.094662] process_one_work+0x2c8/0x700 [12496.094672] worker_thread+0x296/0x6e0 [12496.094677] ? rescuer_thread+0x570/0x570 [12496.094681] kthread+0x1d1/0x200 [12496.094685] ? set_kthread_struct+0x70/0x70 [12496.094689] ret_from_fork+0x1f/0x30 [12496.327217] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12496.330350] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12496.332277] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12496.337093] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12496.343430] RAX: 0000000000001e12 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12496.345340] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [12496.348056] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [12496.349819] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [12496.351955] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [12496.353788] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [12496.356966] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12496.359891] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [12496.362716] Call Trace: [12496.363785] [12496.364662] ? show_regs.cold.9+0x22/0x2f [12496.368645] ? watchdog_timer_fn.cold.6+0x62/0x10a [12496.378605] ? __hrtimer_run_queues+0x1c2/0x4d0 [12496.384453] ? watchdog+0x40/0x40 [12496.389490] ? hrtimer_interrupt+0x128/0x2c0 [12496.398258] ? smp_apic_timer_interrupt+0xa6/0x270 [12496.399910] ? apic_timer_interrupt+0xf/0x20 [12496.408992] [12496.409739] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12496.416610] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12496.421011] do_raw_spin_lock+0x177/0x190 [12496.422116] _raw_spin_lock+0x21/0x30 [12496.423356] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12496.432093] ? do_raw_spin_unlock+0x75/0x190 [12496.434552] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12496.437527] ? do_wait_intr+0xf0/0xf0 [12496.440483] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12496.446571] mdc_reint+0x2d/0x110 [mdc] [12496.448750] mdc_unlink+0x658/0x870 [mdc] [12496.451283] lmv_unlink+0x34d/0xbd0 [lmv] [12496.452928] ll_unlink+0x24d/0x7f0 [lustre] [12496.455422] vfs_unlink+0x202/0x330 [12496.458591] do_unlinkat+0x2fa/0x460 [12496.459798] __x64_sys_unlink+0x37/0x50 [12496.461138] do_syscall_64+0xc1/0x440 [12496.469535] entry_SYSCALL_64_after_hwframe+0x49/0xae [12496.475394] RIP: 0033:0x7f151bdee10b [12496.477681] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [12496.486241] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [12496.488643] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [12496.491010] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [12496.493373] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12496.495538] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12496.497890] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12500.057012] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [12500.060850] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12500.101847] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12500.116409] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12500.122490] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12500.131381] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12500.152771] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12500.165503] RAX: 0000000000006993 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12500.172071] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [12500.177908] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [12500.191544] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [12500.195834] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [12500.202810] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [12500.206762] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12500.214735] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [12500.220328] Call Trace: [12500.223430] [12500.225140] ? show_regs.cold.9+0x22/0x2f [12500.227527] ? watchdog_timer_fn.cold.6+0x62/0x10a [12500.231273] ? __hrtimer_run_queues+0x1c2/0x4d0 [12500.243537] ? watchdog+0x40/0x40 [12500.253067] ? hrtimer_interrupt+0x128/0x2c0 [12500.254678] ? smp_apic_timer_interrupt+0xa6/0x270 [12500.260020] ? apic_timer_interrupt+0xf/0x20 [12500.267678] [12500.272671] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12500.274569] do_raw_spin_lock+0x177/0x190 [12500.277624] _raw_spin_lock+0x21/0x30 [12500.278539] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12500.283200] ? do_raw_spin_unlock+0x75/0x190 [12500.285152] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12500.291121] ? do_wait_intr+0xf0/0xf0 [12500.292622] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12500.296446] mdc_close+0x53b/0xd70 [mdc] [12500.302344] lmv_close+0x18d/0x490 [lmv] [12500.306272] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12500.309389] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12500.312950] ll_md_real_close+0xa5/0x2a0 [lustre] [12500.315359] ll_md_close+0x274/0x800 [lustre] [12500.320540] ll_file_release+0x128/0xc50 [lustre] [12500.329891] __fput+0x10d/0x440 [12500.331391] ____fput+0x12/0x20 [12500.332561] task_work_run+0xc6/0x110 [12500.333822] exit_to_usermode_loop+0x1dd/0x1f0 [12500.341505] do_syscall_64+0x3d6/0x440 [12500.349266] entry_SYSCALL_64_after_hwframe+0x49/0xae [12500.356376] RIP: 0033:0x7f151bee3cd8 [12500.359035] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12500.373415] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12500.377422] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [12500.385248] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [12500.396031] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12500.403508] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12500.411396] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12500.963101] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10570s! [12500.968399] Showing busy workqueues and worker pools: [12500.971696] workqueue events: flags=0x0 [12500.973625] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12500.978176] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [12500.983545] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12500.986365] pending: kfree_rcu_monitor [12500.988406] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12500.993386] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [12500.999764] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12501.004369] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [12501.010157] workqueue events_long: flags=0x0 [12501.012847] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12501.015389] pending: lu_objects_flush [obdclass] [12501.020520] workqueue events_unbound: flags=0x2 [12501.023338] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12501.028318] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12501.038794] workqueue events_power_efficient: flags=0x80 [12501.042925] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12501.047978] pending: do_cache_clean, neigh_periodic_work, check_lifetime [12501.052764] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12501.058146] pending: neigh_periodic_work [12501.061161] workqueue mm_percpu_wq: flags=0x8 [12501.064147] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12501.068952] pending: vmstat_update, lru_add_drain_per_cpu [12501.073433] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12501.077323] pending: vmstat_update [12501.078712] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12501.084967] pending: lru_add_drain_per_cpu [12501.088566] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [12501.093654] pending: lru_add_drain_per_cpu BAR(48) [12501.097590] workqueue writeback: flags=0x4a [12501.100415] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [12501.111324] pending: wb_workfn [12501.114707] workqueue xprtiod: flags=0xa [12501.119576] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12501.133147] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [12501.147237] workqueue ipv6_addrconf: flags=0x40008 [12501.154850] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12501.164288] pending: addrconf_verify_work [12501.171895] workqueue knbd0-recv: flags=0x1a [12501.175272] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12501.181486] in-flight: 658:recv_work [12501.185198] workqueue ptlrpc_pinger: flags=0x2 [12501.188215] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12501.195474] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12501.207948] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10570s workers=4 idle: 32984 37 8 [12501.220264] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12485s workers=2 idle: 661 [12508.057557] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [12508.067131] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12508.122312] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12508.134822] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12508.148014] RIP: 0010:native_safe_halt+0x2e/0x50 [12508.153533] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12508.168571] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12508.176253] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [12508.178524] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [12508.186894] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [12508.196318] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [12508.204082] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12508.206715] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [12508.212655] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12508.216565] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12508.226700] Call Trace: [12508.228303] [12508.229452] ? show_regs.cold.9+0x22/0x2f [12508.231251] ? watchdog_timer_fn.cold.6+0x62/0x10a [12508.237131] ? __hrtimer_run_queues+0x1c2/0x4d0 [12508.240270] ? watchdog+0x40/0x40 [12508.241233] ? hrtimer_interrupt+0x128/0x2c0 [12508.242371] ? smp_apic_timer_interrupt+0xa6/0x270 [12508.246814] ? apic_timer_interrupt+0xf/0x20 [12508.253470] [12508.254696] ? native_safe_halt+0x2e/0x50 [12508.259840] kvm_wait+0xcd/0xe0 [12508.261585] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12508.267566] do_raw_spin_lock+0x177/0x190 [12508.270295] _raw_spin_lock+0x21/0x30 [12508.275508] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12508.278557] obd_mod_free+0x46/0x140 [ptlrpc] [12508.280935] mdc_commit_open+0x91/0xb0 [mdc] [12508.286765] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12508.298857] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12508.312603] after_reply+0xe2c/0x1780 [ptlrpc] [12508.318935] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12508.324857] ? __next_timer_interrupt+0x160/0x160 [12508.331486] ptlrpcd+0xa82/0xe40 [ptlrpc] [12508.336559] ? do_wait_intr+0xf0/0xf0 [12508.343023] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12508.347870] kthread+0x1d1/0x200 [12508.353976] ? set_kthread_struct+0x70/0x70 [12508.356821] ret_from_fork+0x1f/0x30 [12524.056020] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12524.058403] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12524.089450] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12524.100401] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12524.106814] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12524.117772] RIP: 0010:native_safe_halt+0x2e/0x50 [12524.123850] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12524.149431] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12524.160429] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12524.166258] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [12524.173861] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [12524.184301] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [12524.191274] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [12524.197104] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [12524.199873] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12524.211165] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12524.219394] Call Trace: [12524.221453] [12524.223791] ? show_regs.cold.9+0x22/0x2f [12524.228413] ? watchdog_timer_fn.cold.6+0x62/0x10a [12524.235392] ? __hrtimer_run_queues+0x1c2/0x4d0 [12524.243681] ? watchdog+0x40/0x40 [12524.249990] ? hrtimer_interrupt+0x128/0x2c0 [12524.254983] ? smp_apic_timer_interrupt+0xa6/0x270 [12524.261363] ? apic_timer_interrupt+0xf/0x20 [12524.264638] [12524.266101] ? native_safe_halt+0x2e/0x50 [12524.268862] kvm_wait+0xcd/0xe0 [12524.270790] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12524.274243] do_raw_spin_lock+0x177/0x190 [12524.276946] _raw_spin_lock+0x21/0x30 [12524.278988] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12524.282640] process_one_work+0x2c8/0x700 [12524.284860] worker_thread+0x296/0x6e0 [12524.287379] ? rescuer_thread+0x570/0x570 [12524.289809] kthread+0x1d1/0x200 [12524.291667] ? set_kthread_struct+0x70/0x70 [12524.294698] ret_from_fork+0x1f/0x30 [12528.054287] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [12528.058031] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [12528.066998] Modules linked in: lustre(O) osp(O) ofd(O) [12528.076592] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12528.076680] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12528.076686] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12528.076689] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12528.076702] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12528.076707] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12528.076711] RAX: 0000000000005b07 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12528.076714] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [12528.076716] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [12528.076718] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [12528.076721] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [12528.076724] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [12528.076728] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12528.076730] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [12528.076739] Call Trace: [12528.076745] [12528.076750] ? show_regs.cold.9+0x22/0x2f [12528.076760] ? watchdog_timer_fn.cold.6+0x62/0x10a [12528.076766] ? __hrtimer_run_queues+0x1c2/0x4d0 [12528.076770] ? watchdog+0x40/0x40 [12528.076775] ? hrtimer_interrupt+0x128/0x2c0 [12528.076780] ? smp_apic_timer_interrupt+0xa6/0x270 [12528.076788] ? apic_timer_interrupt+0xf/0x20 [12528.076793] [12528.076795] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12528.076801] do_raw_spin_lock+0x177/0x190 [12528.076806] _raw_spin_lock+0x21/0x30 [12528.076811] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12528.077078] ? do_raw_spin_unlock+0x75/0x190 [12528.077083] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12528.077292] ? do_wait_intr+0xf0/0xf0 [12528.077301] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12528.077511] mdc_close+0x53b/0xd70 [mdc] [12528.077551] lmv_close+0x18d/0x490 [lmv] [12528.077579] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12528.077764] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12528.077881] ll_md_real_close+0xa5/0x2a0 [lustre] [12528.077964] ll_md_close+0x274/0x800 [lustre] [12528.078048] ll_file_release+0x128/0xc50 [lustre] [12528.078136] __fput+0x10d/0x440 [12528.078145] ____fput+0x12/0x20 [12528.078149] task_work_run+0xc6/0x110 [12528.078157] exit_to_usermode_loop+0x1dd/0x1f0 [12528.078163] do_syscall_64+0x3d6/0x440 [12528.078168] entry_SYSCALL_64_after_hwframe+0x49/0xae [12528.078175] RIP: 0033:0x7f151bee3cd8 [12528.078181] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12528.078185] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12528.078189] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [12528.078191] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [12528.078193] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12528.078195] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12528.078197] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12528.365901] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12528.394219] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12528.404864] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12528.419090] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12528.436595] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12528.455180] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12528.462620] RAX: 00000000000057fc RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12528.465338] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [12528.467984] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [12528.471002] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [12528.477620] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [12528.480496] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [12528.485190] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12528.489692] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [12528.494376] Call Trace: [12528.495834] [12528.496744] ? show_regs.cold.9+0x22/0x2f [12528.498245] ? watchdog_timer_fn.cold.6+0x62/0x10a [12528.500291] ? __hrtimer_run_queues+0x1c2/0x4d0 [12528.507665] ? watchdog+0x40/0x40 [12528.509057] ? hrtimer_interrupt+0x128/0x2c0 [12528.512351] ? smp_apic_timer_interrupt+0xa6/0x270 [12528.514626] ? apic_timer_interrupt+0xf/0x20 [12528.518418] [12528.519414] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12528.521041] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12528.523405] do_raw_spin_lock+0x177/0x190 [12528.524946] _raw_spin_lock+0x21/0x30 [12528.526253] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12528.528447] ? do_raw_spin_unlock+0x75/0x190 [12528.530379] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12528.532889] ? do_wait_intr+0xf0/0xf0 [12528.534390] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12528.536469] mdc_reint+0x2d/0x110 [mdc] [12528.538248] mdc_unlink+0x658/0x870 [mdc] [12528.539431] lmv_unlink+0x34d/0xbd0 [lmv] [12528.541164] ll_unlink+0x24d/0x7f0 [lustre] [12528.543017] vfs_unlink+0x202/0x330 [12528.544282] do_unlinkat+0x2fa/0x460 [12528.546111] __x64_sys_unlink+0x37/0x50 [12528.547420] do_syscall_64+0xc1/0x440 [12528.548797] entry_SYSCALL_64_after_hwframe+0x49/0xae [12528.551028] RIP: 0033:0x7f151bdee10b [12528.552178] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [12528.559498] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [12528.567314] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [12528.574213] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [12528.591241] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12528.598682] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12528.612414] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12531.687991] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10601s! [12531.691083] Showing busy workqueues and worker pools: [12531.695045] workqueue events: flags=0x0 [12531.696937] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12531.700375] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [12531.715559] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12531.722336] pending: kfree_rcu_monitor [12531.724166] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12531.728749] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [12531.737027] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12531.743851] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [12531.747135] workqueue events_long: flags=0x0 [12531.752987] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12531.756858] pending: lu_objects_flush [obdclass] [12531.762013] workqueue events_unbound: flags=0x2 [12531.768647] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12531.772217] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12531.777727] workqueue events_power_efficient: flags=0x80 [12531.780239] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12531.783326] pending: do_cache_clean, neigh_periodic_work, check_lifetime [12531.789386] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12531.797062] pending: neigh_periodic_work [12531.801979] workqueue mm_percpu_wq: flags=0x8 [12531.806232] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12531.812048] pending: vmstat_update, lru_add_drain_per_cpu [12531.816873] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12531.818746] pending: vmstat_update [12531.819924] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12531.825965] pending: lru_add_drain_per_cpu [12531.827964] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [12531.833332] pending: lru_add_drain_per_cpu BAR(48) [12531.838032] workqueue writeback: flags=0x4a [12531.840012] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [12531.843016] pending: wb_workfn [12531.848271] workqueue xprtiod: flags=0xa [12531.849751] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12531.859258] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [12531.861625] workqueue ipv6_addrconf: flags=0x40008 [12531.871021] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12531.875780] pending: addrconf_verify_work [12531.878610] workqueue knbd0-recv: flags=0x1a [12531.880842] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12531.883085] in-flight: 658:recv_work [12531.885459] workqueue ptlrpc_pinger: flags=0x2 [12531.888050] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12531.893168] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12531.895470] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10601s workers=4 idle: 32984 37 8 [12531.903940] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12516s workers=2 idle: 661 [12536.055724] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [12536.058797] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12536.106303] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12536.113582] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12536.127943] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12536.133969] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12536.158463] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12536.164780] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [12536.170821] RDX: 00000000000014bb RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [12536.182381] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [12536.190326] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [12536.196968] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12536.206830] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [12536.215421] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12536.220315] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12536.231297] Call Trace: [12536.235316] [12536.241110] ? show_regs.cold.9+0x22/0x2f [12536.243902] ? watchdog_timer_fn.cold.6+0x62/0x10a [12536.249333] ? __hrtimer_run_queues+0x1c2/0x4d0 [12536.256845] ? watchdog+0x40/0x40 [12536.259064] ? hrtimer_interrupt+0x128/0x2c0 [12536.265271] ? smp_apic_timer_interrupt+0xa6/0x270 [12536.270840] ? apic_timer_interrupt+0xf/0x20 [12536.278139] [12536.280278] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12536.283566] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12536.287958] do_raw_spin_lock+0x177/0x190 [12536.290306] _raw_spin_lock+0x21/0x30 [12536.297817] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12536.308730] obd_mod_free+0x46/0x140 [ptlrpc] [12536.315582] mdc_commit_open+0x91/0xb0 [mdc] [12536.321344] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12536.327800] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12536.338404] after_reply+0xe2c/0x1780 [ptlrpc] [12536.350101] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12536.353325] ? __next_timer_interrupt+0x160/0x160 [12536.364994] ptlrpcd+0xa82/0xe40 [ptlrpc] [12536.367319] ? do_wait_intr+0xf0/0xf0 [12536.369822] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12536.379949] kthread+0x1d1/0x200 [12536.383912] ? set_kthread_struct+0x70/0x70 [12536.389732] ret_from_fork+0x1f/0x30 [12552.056589] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [12552.061335] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12552.110712] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12552.120335] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12552.129432] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12552.135898] RIP: 0010:native_safe_halt+0x2e/0x50 [12552.139667] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12552.152064] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12552.163198] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12552.169055] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [12552.172658] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [12552.176214] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [12552.179280] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [12552.184479] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [12552.193002] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12552.195237] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12552.198514] Call Trace: [12552.200602] [12552.201789] ? show_regs.cold.9+0x22/0x2f [12552.207262] ? watchdog_timer_fn.cold.6+0x62/0x10a [12552.209585] ? __hrtimer_run_queues+0x1c2/0x4d0 [12552.212260] ? watchdog+0x40/0x40 [12552.213450] ? hrtimer_interrupt+0x128/0x2c0 [12552.216427] ? smp_apic_timer_interrupt+0xa6/0x270 [12552.222854] ? apic_timer_interrupt+0xf/0x20 [12552.226251] [12552.229815] ? native_safe_halt+0x2e/0x50 [12552.235658] kvm_wait+0xcd/0xe0 [12552.238150] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12552.241062] do_raw_spin_lock+0x177/0x190 [12552.243908] _raw_spin_lock+0x21/0x30 [12552.247355] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12552.250253] process_one_work+0x2c8/0x700 [12552.252170] worker_thread+0x296/0x6e0 [12552.254782] ? rescuer_thread+0x570/0x570 [12552.256021] kthread+0x1d1/0x200 [12552.257411] ? set_kthread_struct+0x70/0x70 [12552.262264] ret_from_fork+0x1f/0x30 [12556.052410] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [12556.057104] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [12556.057798] Modules linked in: [12556.063721] Modules linked in: [12556.063722] lustre(O) osp(O) ofd(O) lod(O) mdt(O) [12556.066680] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12556.066765] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12556.066774] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12556.066778] RIP: 0010:native_safe_halt+0x2e/0x50 [12556.067593] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12556.067599] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12556.067604] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12556.067606] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [12556.067609] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [12556.067611] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [12556.067614] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [12556.067618] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [12556.067622] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12556.067625] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [12556.067635] Call Trace: [12556.067643] [12556.067651] ? show_regs.cold.9+0x22/0x2f [12556.067664] ? watchdog_timer_fn.cold.6+0x62/0x10a [12556.067672] ? __hrtimer_run_queues+0x1c2/0x4d0 [12556.067678] ? watchdog+0x40/0x40 [12556.067682] ? hrtimer_interrupt+0x128/0x2c0 [12556.067686] ? smp_apic_timer_interrupt+0xa6/0x270 [12556.067693] ? apic_timer_interrupt+0xf/0x20 [12556.067698] [12556.067701] ? native_safe_halt+0x2e/0x50 [12556.067706] kvm_wait+0xcd/0xe0 [12556.067715] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12556.067726] do_raw_spin_lock+0x177/0x190 [12556.067731] _raw_spin_lock+0x21/0x30 [12556.067736] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12556.068740] ? do_raw_spin_unlock+0x75/0x190 [12556.068749] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12556.103555] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12556.103637] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12556.103648] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12556.103651] RIP: 0010:native_safe_halt+0x2e/0x50 [12556.103670] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12556.103675] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12556.103680] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12556.103682] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [12556.103685] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [12556.103687] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [12556.103690] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [12556.103694] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [12556.103698] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12556.103700] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [12556.103709] Call Trace: [12556.103717] [12556.103725] ? show_regs.cold.9+0x22/0x2f [12556.103737] ? watchdog_timer_fn.cold.6+0x62/0x10a [12556.103744] ? __hrtimer_run_queues+0x1c2/0x4d0 [12556.103749] ? watchdog+0x40/0x40 [12556.103753] ? hrtimer_interrupt+0x128/0x2c0 [12556.103757] ? smp_apic_timer_interrupt+0xa6/0x270 [12556.103764] ? apic_timer_interrupt+0xf/0x20 [12556.103769] [12556.103772] ? native_safe_halt+0x2e/0x50 [12556.103776] kvm_wait+0xcd/0xe0 [12556.103786] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12556.103985] do_raw_spin_lock+0x177/0x190 [12556.103990] _raw_spin_lock+0x21/0x30 [12556.103997] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12556.104354] ? do_raw_spin_unlock+0x75/0x190 [12556.104360] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12556.104566] ? do_wait_intr+0xf0/0xf0 [12556.104576] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12556.104912] mdc_reint+0x2d/0x110 [mdc] [12556.107854] mdc_unlink+0x658/0x870 [mdc] [12556.108554] lmv_unlink+0x34d/0xbd0 [lmv] [12556.108592] ll_unlink+0x24d/0x7f0 [lustre] [12556.123689] ? do_wait_intr+0xf0/0xf0 [12556.123856] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12556.124235] mdc_close+0x53b/0xd70 [mdc] [12556.124356] lmv_close+0x18d/0x490 [lmv] [12556.124861] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12556.125176] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12556.133643] vfs_unlink+0x202/0x330 [12556.143070] ll_md_real_close+0xa5/0x2a0 [lustre] [12556.166685] do_unlinkat+0x2fa/0x460 [12556.166710] __x64_sys_unlink+0x37/0x50 [12556.166715] do_syscall_64+0xc1/0x440 [12556.166723] entry_SYSCALL_64_after_hwframe+0x49/0xae [12556.166734] RIP: 0033:0x7f151bdee10b [12556.166746] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [12556.166751] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [12556.166758] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [12556.166760] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [12556.166763] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12556.166765] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12556.166767] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12556.599166] ll_md_close+0x274/0x800 [lustre] [12556.604029] ll_file_release+0x128/0xc50 [lustre] [12556.609369] __fput+0x10d/0x440 [12556.613462] ____fput+0x12/0x20 [12556.616855] task_work_run+0xc6/0x110 [12556.621431] exit_to_usermode_loop+0x1dd/0x1f0 [12556.624453] do_syscall_64+0x3d6/0x440 [12556.629566] entry_SYSCALL_64_after_hwframe+0x49/0xae [12556.633683] RIP: 0033:0x7f151bee3cd8 [12556.635812] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12556.650893] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12556.654654] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [12556.658546] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [12556.661046] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12556.664284] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12556.666621] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12562.412356] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10631s! [12562.421481] Showing busy workqueues and worker pools: [12562.423835] workqueue events: flags=0x0 [12562.424921] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12562.432417] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [12562.438066] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12562.441207] pending: kfree_rcu_monitor [12562.450562] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12562.460873] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [12562.464961] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12562.473635] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [12562.481781] workqueue events_long: flags=0x0 [12562.484249] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12562.490141] pending: lu_objects_flush [obdclass] [12562.499639] workqueue events_unbound: flags=0x2 [12562.501778] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12562.503963] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12562.508251] workqueue events_power_efficient: flags=0x80 [12562.509873] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12562.511909] pending: do_cache_clean, neigh_periodic_work, check_lifetime [12562.514655] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12562.516822] pending: neigh_periodic_work [12562.518434] workqueue mm_percpu_wq: flags=0x8 [12562.520029] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12562.522177] pending: vmstat_update, lru_add_drain_per_cpu [12562.524372] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12562.526673] pending: vmstat_update [12562.528091] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12562.530635] pending: lru_add_drain_per_cpu [12562.532136] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [12562.534659] pending: lru_add_drain_per_cpu BAR(48) [12562.536569] workqueue writeback: flags=0x4a [12562.538173] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [12562.540406] pending: wb_workfn [12562.541765] workqueue xprtiod: flags=0xa [12562.543171] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12562.545677] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [12562.548612] workqueue ipv6_addrconf: flags=0x40008 [12562.550394] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12562.552897] pending: addrconf_verify_work [12562.554606] workqueue knbd0-recv: flags=0x1a [12562.555989] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12562.558349] in-flight: 658:recv_work [12562.559785] workqueue ptlrpc_pinger: flags=0x2 [12562.561464] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12562.579596] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12562.582623] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10631s workers=4 idle: 32984 37 8 [12562.585926] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12547s workers=2 idle: 661 [12564.055027] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [12564.058157] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12564.076539] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12564.081801] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12564.088123] RIP: 0010:native_safe_halt+0x2e/0x50 [12564.090085] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12564.096805] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12564.103294] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [12564.111011] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [12564.117440] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [12564.122530] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [12564.125731] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12564.136021] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [12564.145739] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12564.153667] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12564.166362] Call Trace: [12564.170360] [12564.171510] ? show_regs.cold.9+0x22/0x2f [12564.174787] ? watchdog_timer_fn.cold.6+0x62/0x10a [12564.180513] ? __hrtimer_run_queues+0x1c2/0x4d0 [12564.186907] ? watchdog+0x40/0x40 [12564.189764] ? hrtimer_interrupt+0x128/0x2c0 [12564.193707] ? smp_apic_timer_interrupt+0xa6/0x270 [12564.206576] ? apic_timer_interrupt+0xf/0x20 [12564.209371] [12564.210511] ? native_safe_halt+0x2e/0x50 [12564.220401] kvm_wait+0xcd/0xe0 [12564.222714] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12564.228439] do_raw_spin_lock+0x177/0x190 [12564.231103] _raw_spin_lock+0x21/0x30 [12564.236401] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12564.240569] obd_mod_free+0x46/0x140 [ptlrpc] [12564.246941] mdc_commit_open+0x91/0xb0 [mdc] [12564.249230] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12564.256087] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12564.260936] after_reply+0xe2c/0x1780 [ptlrpc] [12564.264148] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12564.269014] ? __next_timer_interrupt+0x160/0x160 [12564.281947] ptlrpcd+0xa82/0xe40 [ptlrpc] [12564.291717] ? do_wait_intr+0xf0/0xf0 [12564.294369] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12564.320967] kthread+0x1d1/0x200 [12564.331801] ? set_kthread_struct+0x70/0x70 [12564.334926] ret_from_fork+0x1f/0x30 [12580.056083] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [12580.060533] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12580.111926] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12580.123349] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12580.131363] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12580.137593] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12580.146815] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12580.172675] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12580.179841] RAX: 0000000000007f8b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12580.185650] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [12580.195627] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [12580.198908] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [12580.203142] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [12580.205591] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [12580.209882] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12580.215612] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12580.223754] Call Trace: [12580.224730] [12580.225446] ? show_regs.cold.9+0x22/0x2f [12580.228348] ? watchdog_timer_fn.cold.6+0x62/0x10a [12580.235901] ? __hrtimer_run_queues+0x1c2/0x4d0 [12580.239729] ? watchdog+0x40/0x40 [12580.241857] ? hrtimer_interrupt+0x128/0x2c0 [12580.247584] ? smp_apic_timer_interrupt+0xa6/0x270 [12580.251538] ? apic_timer_interrupt+0xf/0x20 [12580.254575] [12580.255688] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12580.258752] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12580.260947] do_raw_spin_lock+0x177/0x190 [12580.263352] _raw_spin_lock+0x21/0x30 [12580.265251] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12580.267830] process_one_work+0x2c8/0x700 [12580.269625] worker_thread+0x296/0x6e0 [12580.270795] ? rescuer_thread+0x570/0x570 [12580.272309] kthread+0x1d1/0x200 [12580.273512] ? set_kthread_struct+0x70/0x70 [12580.275373] ret_from_fork+0x1f/0x30 [12584.057081] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [12584.062903] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12584.126770] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12584.137283] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12584.150068] RIP: 0010:native_safe_halt+0x2e/0x50 [12584.154550] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12584.183160] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12584.202196] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12584.226075] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [12584.239933] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [12584.258334] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [12584.260796] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [12584.290164] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [12584.311306] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12584.319483] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [12584.327306] Call Trace: [12584.346082] [12584.352184] ? show_regs.cold.9+0x22/0x2f [12584.361553] ? watchdog_timer_fn.cold.6+0x62/0x10a [12584.367923] ? __hrtimer_run_queues+0x1c2/0x4d0 [12584.382591] ? watchdog+0x40/0x40 [12584.385679] ? hrtimer_interrupt+0x128/0x2c0 [12584.393580] ? smp_apic_timer_interrupt+0xa6/0x270 [12584.399686] ? apic_timer_interrupt+0xf/0x20 [12584.404537] [12584.407223] ? native_safe_halt+0x2e/0x50 [12584.413586] kvm_wait+0xcd/0xe0 [12584.414665] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12584.417478] do_raw_spin_lock+0x177/0x190 [12584.426790] _raw_spin_lock+0x21/0x30 [12584.435741] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12584.445786] ? do_raw_spin_unlock+0x75/0x190 [12584.450540] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12584.457665] ? do_wait_intr+0xf0/0xf0 [12584.458835] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12584.466186] mdc_close+0x53b/0xd70 [mdc] [12584.470688] lmv_close+0x18d/0x490 [lmv] [12584.472072] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12584.477940] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12584.502658] ll_md_real_close+0xa5/0x2a0 [lustre] [12584.508748] ll_md_close+0x274/0x800 [lustre] [12584.516666] ll_file_release+0x128/0xc50 [lustre] [12584.522061] __fput+0x10d/0x440 [12584.523225] ____fput+0x12/0x20 [12584.531344] task_work_run+0xc6/0x110 [12584.533381] exit_to_usermode_loop+0x1dd/0x1f0 [12584.536925] do_syscall_64+0x3d6/0x440 [12584.539151] entry_SYSCALL_64_after_hwframe+0x49/0xae [12584.542491] RIP: 0033:0x7f151bee3cd8 [12584.548121] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12584.561122] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12584.570160] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [12584.575224] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [12584.580714] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12584.586786] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12584.590754] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12588.052364] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [12588.055838] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12588.083567] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12588.088892] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12588.093115] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12588.096021] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12588.105630] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12588.110339] RAX: 0000000000005987 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12588.113843] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [12588.117459] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [12588.120629] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [12588.123816] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [12588.126907] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [12588.131444] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12588.134564] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [12588.137974] Call Trace: [12588.139034] [12588.139929] ? show_regs.cold.9+0x22/0x2f [12588.141547] ? watchdog_timer_fn.cold.6+0x62/0x10a [12588.143761] ? __hrtimer_run_queues+0x1c2/0x4d0 [12588.145584] ? watchdog+0x40/0x40 [12588.146868] ? hrtimer_interrupt+0x128/0x2c0 [12588.148340] ? smp_apic_timer_interrupt+0xa6/0x270 [12588.150224] ? apic_timer_interrupt+0xf/0x20 [12588.151839] [12588.153048] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12588.155308] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12588.158106] do_raw_spin_lock+0x177/0x190 [12588.159831] _raw_spin_lock+0x21/0x30 [12588.161671] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12588.165724] ? do_raw_spin_unlock+0x75/0x190 [12588.169886] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12588.173235] ? do_wait_intr+0xf0/0xf0 [12588.175006] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12588.178359] mdc_reint+0x2d/0x110 [mdc] [12588.182088] mdc_unlink+0x658/0x870 [mdc] [12588.183886] lmv_unlink+0x34d/0xbd0 [lmv] [12588.187147] ll_unlink+0x24d/0x7f0 [lustre] [12588.189619] vfs_unlink+0x202/0x330 [12588.196014] do_unlinkat+0x2fa/0x460 [12588.197825] __x64_sys_unlink+0x37/0x50 [12588.202123] do_syscall_64+0xc1/0x440 [12588.204100] entry_SYSCALL_64_after_hwframe+0x49/0xae [12588.208129] RIP: 0033:0x7f151bdee10b [12588.210706] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [12588.230468] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [12588.238207] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [12588.244903] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [12588.249839] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12588.255195] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12588.258046] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12592.058535] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [12592.068464] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12592.125128] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12592.135903] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12592.143444] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12592.147395] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12592.156976] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12592.162128] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [12592.165998] RDX: 0000000000003371 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [12592.169506] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [12592.173478] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [12592.177108] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12592.180301] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [12592.187250] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12592.194024] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12592.201507] Call Trace: [12592.204806] [12592.205536] ? show_regs.cold.9+0x22/0x2f [12592.208889] ? watchdog_timer_fn.cold.6+0x62/0x10a [12592.211480] ? __hrtimer_run_queues+0x1c2/0x4d0 [12592.218345] ? watchdog+0x40/0x40 [12592.222161] ? hrtimer_interrupt+0x128/0x2c0 [12592.224471] ? smp_apic_timer_interrupt+0xa6/0x270 [12592.226456] ? apic_timer_interrupt+0xf/0x20 [12592.231401] [12592.232317] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12592.234644] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12592.237465] do_raw_spin_lock+0x177/0x190 [12592.239100] _raw_spin_lock+0x21/0x30 [12592.240011] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12592.241830] obd_mod_free+0x46/0x140 [ptlrpc] [12592.243527] mdc_commit_open+0x91/0xb0 [mdc] [12592.244863] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12592.246776] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12592.248630] after_reply+0xe2c/0x1780 [ptlrpc] [12592.250402] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12592.252238] ? __next_timer_interrupt+0x160/0x160 [12592.253829] ptlrpcd+0xa82/0xe40 [ptlrpc] [12592.255394] ? do_wait_intr+0xf0/0xf0 [12592.256592] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12592.258624] kthread+0x1d1/0x200 [12592.259785] ? set_kthread_struct+0x70/0x70 [12592.263709] ret_from_fork+0x1f/0x30 [12592.615133] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10662s! [12592.620430] Showing busy workqueues and worker pools: [12592.622837] workqueue events: flags=0x0 [12592.624825] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12592.628581] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [12592.633097] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12592.636747] pending: kfree_rcu_monitor [12592.638781] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12592.642803] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [12592.649138] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12592.652741] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [12592.655598] workqueue events_long: flags=0x0 [12592.658286] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12592.660707] pending: lu_objects_flush [obdclass] [12592.664809] workqueue events_unbound: flags=0x2 [12592.667286] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12592.671279] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12592.677118] workqueue events_power_efficient: flags=0x80 [12592.679918] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12592.682256] pending: do_cache_clean, neigh_periodic_work, check_lifetime [12592.687344] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12592.691768] pending: neigh_periodic_work [12592.694505] workqueue mm_percpu_wq: flags=0x8 [12592.696056] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12592.700718] pending: vmstat_update, lru_add_drain_per_cpu [12592.703982] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12592.706888] pending: vmstat_update [12592.709166] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12592.712717] pending: lru_add_drain_per_cpu [12592.716358] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [12592.720290] pending: lru_add_drain_per_cpu BAR(48) [12592.723782] workqueue writeback: flags=0x4a [12592.725876] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [12592.729717] pending: wb_workfn [12592.731316] workqueue xprtiod: flags=0xa [12592.732315] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12592.733866] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [12592.738623] workqueue ipv6_addrconf: flags=0x40008 [12592.741737] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12592.746344] pending: addrconf_verify_work [12592.750199] workqueue knbd0-recv: flags=0x1a [12592.753372] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12592.755750] in-flight: 658:recv_work [12592.757364] workqueue ptlrpc_pinger: flags=0x2 [12592.759832] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12592.764748] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12592.769925] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10662s workers=4 idle: 32984 37 8 [12592.775352] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12577s workers=2 idle: 661 [12604.880058] rcu: INFO: rcu_sched self-detected stall on CPU [12604.883237] rcu: 1-...!: (6177015 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [12604.890276] (t=10680233 jiffies g=249657 q=5588) [12604.894618] rcu: rcu_sched kthread starved for 10675185 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [12604.901307] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [12604.905593] rcu: RCU grace-period kthread stack dump: [12604.908059] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [12604.913335] Call Trace: [12604.914468] __schedule+0x351/0xcb0 [12604.916393] schedule+0xc0/0x180 [12604.918947] schedule_timeout+0xb4/0x190 [12604.921133] ? do_raw_spin_unlock+0x75/0x190 [12604.923249] ? __next_timer_interrupt+0x160/0x160 [12604.924682] ? prepare_to_swait_event+0xf3/0x140 [12604.929293] rcu_gp_kthread+0x8dd/0x15b0 [12604.931440] ? force_qs_rnp+0x2c0/0x2c0 [12604.934991] kthread+0x1d1/0x200 [12604.937143] ? set_kthread_struct+0x70/0x70 [12604.940263] ret_from_fork+0x1f/0x30 [12604.941960] rcu: Stack dump where RCU GP kthread last ran: [12604.946382] Sending NMI from CPU 1 to CPUs 3: [12604.950940] NMI backtrace for cpu 3 skipped: idling at native_safe_halt+0x2e/0x50 [12604.951099] NMI backtrace for cpu 1 [12604.957960] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12604.965623] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12604.971084] Call Trace: [12604.973022] [12604.973732] ? dump_stack+0xbb/0x10e [12604.977375] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [12604.988517] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [12604.992401] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [12604.996412] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [12605.002482] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [12605.007312] ? rcu_dump_cpu_stacks+0x11b/0x171 [12605.011286] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [12605.017088] ? update_process_times+0x6d/0xc0 [12605.018700] ? tick_sched_handle.isra.18+0x27/0x90 [12605.025576] ? tick_sched_timer+0x53/0xb0 [12605.028269] ? __hrtimer_run_queues+0x1c2/0x4d0 [12605.032427] ? tick_sched_handle.isra.18+0x90/0x90 [12605.036242] ? hrtimer_interrupt+0x128/0x2c0 [12605.040056] ? smp_apic_timer_interrupt+0xa6/0x270 [12605.044500] ? apic_timer_interrupt+0xf/0x20 [12605.049721] [12605.053523] ? native_safe_halt+0x2e/0x50 [12605.059489] ? kvm_wait+0xcd/0xe0 [12605.062635] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12605.070588] ? do_raw_spin_lock+0x177/0x190 [12605.074700] ? _raw_spin_lock+0x21/0x30 [12605.077756] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12605.081591] ? obd_mod_free+0x46/0x140 [ptlrpc] [12605.086046] ? mdc_commit_open+0x91/0xb0 [mdc] [12605.087496] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12605.099208] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12605.101579] ? after_reply+0xe2c/0x1780 [ptlrpc] [12605.104582] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12605.111232] ? __next_timer_interrupt+0x160/0x160 [12605.115116] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [12605.119058] ? do_wait_intr+0xf0/0xf0 [12605.120695] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12605.127334] ? kthread+0x1d1/0x200 [12605.130309] ? set_kthread_struct+0x70/0x70 [12605.133307] ? ret_from_fork+0x1f/0x30 [12608.056014] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [12608.058794] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12608.075682] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12608.079913] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12608.083907] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12608.086902] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12608.089724] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12608.095911] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12608.098694] RAX: 0000000000001f6d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12608.101854] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [12608.104788] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [12608.107724] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [12608.110897] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [12608.114443] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [12608.118063] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12608.120355] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12608.122965] Call Trace: [12608.124198] [12608.125221] ? show_regs.cold.9+0x22/0x2f [12608.126972] ? watchdog_timer_fn.cold.6+0x62/0x10a [12608.129011] ? __hrtimer_run_queues+0x1c2/0x4d0 [12608.131090] ? watchdog+0x40/0x40 [12608.132498] ? hrtimer_interrupt+0x128/0x2c0 [12608.134028] ? smp_apic_timer_interrupt+0xa6/0x270 [12608.135857] ? apic_timer_interrupt+0xf/0x20 [12608.137198] [12608.137727] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12608.139697] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12608.141529] do_raw_spin_lock+0x177/0x190 [12608.143274] _raw_spin_lock+0x21/0x30 [12608.144932] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12608.147931] process_one_work+0x2c8/0x700 [12608.149463] worker_thread+0x296/0x6e0 [12608.153047] ? rescuer_thread+0x570/0x570 [12608.155391] kthread+0x1d1/0x200 [12608.157617] ? set_kthread_struct+0x70/0x70 [12608.159849] ret_from_fork+0x1f/0x30 [12620.052030] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [12620.054679] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12620.098793] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12620.106107] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12620.114114] RIP: 0010:native_safe_halt+0x2e/0x50 [12620.118060] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12620.146416] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12620.153684] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12620.158887] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [12620.161500] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [12620.176295] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [12620.191923] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [12620.203827] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [12620.209148] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12620.211374] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [12620.214862] Call Trace: [12620.215839] [12620.219909] ? show_regs.cold.9+0x22/0x2f [12620.222279] ? watchdog_timer_fn.cold.6+0x62/0x10a [12620.228344] ? __hrtimer_run_queues+0x1c2/0x4d0 [12620.233106] ? watchdog+0x40/0x40 [12620.234276] ? hrtimer_interrupt+0x128/0x2c0 [12620.235739] ? smp_apic_timer_interrupt+0xa6/0x270 [12620.237320] ? apic_timer_interrupt+0xf/0x20 [12620.238555] [12620.239496] ? native_safe_halt+0x2e/0x50 [12620.243569] kvm_wait+0xcd/0xe0 [12620.244536] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12620.247490] do_raw_spin_lock+0x177/0x190 [12620.248993] _raw_spin_lock+0x21/0x30 [12620.251828] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12620.254715] ? do_raw_spin_unlock+0x75/0x190 [12620.257340] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12620.261314] ? do_wait_intr+0xf0/0xf0 [12620.262562] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12620.265859] mdc_reint+0x2d/0x110 [mdc] [12620.268276] mdc_unlink+0x658/0x870 [mdc] [12620.270553] lmv_unlink+0x34d/0xbd0 [lmv] [12620.272724] ll_unlink+0x24d/0x7f0 [lustre] [12620.274542] vfs_unlink+0x202/0x330 [12620.279078] do_unlinkat+0x2fa/0x460 [12620.280883] __x64_sys_unlink+0x37/0x50 [12620.282800] do_syscall_64+0xc1/0x440 [12620.286406] entry_SYSCALL_64_after_hwframe+0x49/0xae [12620.290532] RIP: 0033:0x7f151bdee10b [12620.293996] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [12620.310067] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [12620.316569] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [12620.325872] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [12620.334169] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12620.341899] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12620.349463] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12622.817044] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10692s! [12622.820452] Showing busy workqueues and worker pools: [12622.823088] workqueue events: flags=0x0 [12622.825757] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12622.829762] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [12622.837897] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12622.841502] pending: kfree_rcu_monitor [12622.845026] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12622.848531] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [12622.855334] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12622.860426] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [12622.865141] workqueue events_long: flags=0x0 [12622.868882] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12622.872768] pending: lu_objects_flush [obdclass] [12622.876243] workqueue events_unbound: flags=0x2 [12622.880206] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12622.885410] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12622.893743] workqueue events_power_efficient: flags=0x80 [12622.897146] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12622.901774] pending: do_cache_clean, neigh_periodic_work, check_lifetime [12622.906177] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12622.910291] pending: neigh_periodic_work [12622.916300] workqueue mm_percpu_wq: flags=0x8 [12622.921819] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12622.928771] pending: vmstat_update, lru_add_drain_per_cpu [12622.934191] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12622.939918] pending: vmstat_update [12622.943022] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12622.949160] pending: lru_add_drain_per_cpu [12622.955616] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [12622.967120] pending: lru_add_drain_per_cpu BAR(48) [12622.976303] workqueue writeback: flags=0x4a [12622.979143] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [12622.987165] pending: wb_workfn [12622.992212] workqueue xprtiod: flags=0xa [12622.995186] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12623.000708] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [12623.006260] workqueue ipv6_addrconf: flags=0x40008 [12623.009460] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12623.013479] pending: addrconf_verify_work [12623.015724] workqueue knbd0-recv: flags=0x1a [12623.018226] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12623.022877] in-flight: 658:recv_work [12623.027494] workqueue ptlrpc_pinger: flags=0x2 [12623.030146] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12623.033546] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12623.039674] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10692s workers=4 idle: 32984 37 8 [12623.046470] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12607s workers=2 idle: 661 [12632.055519] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [12632.057078] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [12632.060019] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata [12632.063168] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12632.063226] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12632.063232] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12632.063235] RIP: 0010:native_safe_halt+0x2e/0x50 [12632.063246] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12632.063249] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12632.063253] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12632.063255] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [12632.063256] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [12632.063258] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [12632.063259] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [12632.063262] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [12632.063265] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12632.063268] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [12632.063275] Call Trace: [12632.063281] [12632.063287] ? show_regs.cold.9+0x22/0x2f [12632.063297] ? watchdog_timer_fn.cold.6+0x62/0x10a [12632.063303] ? __hrtimer_run_queues+0x1c2/0x4d0 [12632.063308] ? watchdog+0x40/0x40 [12632.063313] ? hrtimer_interrupt+0x128/0x2c0 [12632.063317] ? smp_apic_timer_interrupt+0xa6/0x270 [12632.063325] ? apic_timer_interrupt+0xf/0x20 [12632.063329] [12632.063330] ? native_safe_halt+0x2e/0x50 [12632.063333] kvm_wait+0xcd/0xe0 [12632.063339] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12632.063345] do_raw_spin_lock+0x177/0x190 [12632.063349] _raw_spin_lock+0x21/0x30 [12632.063352] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12632.089261] dm_mirror [12632.110117] ? do_raw_spin_unlock+0x75/0x190 [12632.114092] dm_region_hash [12632.117859] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12632.119204] dm_log [12632.136621] ? do_wait_intr+0xf0/0xf0 [12632.138962] dm_mod [12632.142548] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12632.144534] sha512_ssse3 sha512_generic [12632.144546] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12632.144552] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12632.144554] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12632.144567] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12632.144571] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12632.144576] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [12632.144578] RDX: 0000000000007e13 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [12632.144581] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [12632.144583] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [12632.144586] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12632.144589] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [12632.144592] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12632.144594] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12632.144603] Call Trace: [12632.144607] [12632.144612] ? show_regs.cold.9+0x22/0x2f [12632.144621] ? watchdog_timer_fn.cold.6+0x62/0x10a [12632.144627] ? __hrtimer_run_queues+0x1c2/0x4d0 [12632.144631] ? watchdog+0x40/0x40 [12632.144635] ? hrtimer_interrupt+0x128/0x2c0 [12632.144647] ? smp_apic_timer_interrupt+0xa6/0x270 [12632.144656] ? apic_timer_interrupt+0xf/0x20 [12632.144721] [12632.144724] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12632.144729] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12632.144733] do_raw_spin_lock+0x177/0x190 [12632.144737] _raw_spin_lock+0x21/0x30 [12632.144744] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12632.145069] obd_mod_free+0x46/0x140 [ptlrpc] [12632.145214] mdc_commit_open+0x91/0xb0 [mdc] [12632.145241] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12632.145389] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12632.145575] after_reply+0xe2c/0x1780 [ptlrpc] [12632.145838] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12632.154445] mdc_close+0x53b/0xd70 [mdc] [12632.159171] ? __next_timer_interrupt+0x160/0x160 [12632.159192] ptlrpcd+0xa82/0xe40 [ptlrpc] [12632.159475] ? do_wait_intr+0xf0/0xf0 [12632.159485] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12632.159890] kthread+0x1d1/0x200 [12632.159896] ? set_kthread_struct+0x70/0x70 [12632.159900] ret_from_fork+0x1f/0x30 [12632.339744] lmv_close+0x18d/0x490 [lmv] [12632.341750] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12632.344109] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12632.346157] ll_md_real_close+0xa5/0x2a0 [lustre] [12632.347973] ll_md_close+0x274/0x800 [lustre] [12632.349293] ll_file_release+0x128/0xc50 [lustre] [12632.351220] __fput+0x10d/0x440 [12632.352454] ____fput+0x12/0x20 [12632.353822] task_work_run+0xc6/0x110 [12632.355292] exit_to_usermode_loop+0x1dd/0x1f0 [12632.357135] do_syscall_64+0x3d6/0x440 [12632.358489] entry_SYSCALL_64_after_hwframe+0x49/0xae [12632.360764] RIP: 0033:0x7f151bee3cd8 [12632.362256] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12632.370067] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12632.372934] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [12632.375788] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [12632.378958] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12632.382002] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12632.384964] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12636.056033] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [12636.059256] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12636.077816] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12636.083362] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12636.087202] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12636.091219] RIP: 0010:native_safe_halt+0x2e/0x50 [12636.094250] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12636.105060] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12636.109292] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12636.113015] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [12636.117067] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [12636.125690] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [12636.131895] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [12636.137744] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [12636.147303] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12636.154951] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12636.160013] Call Trace: [12636.161963] [12636.162549] ? show_regs.cold.9+0x22/0x2f [12636.164474] ? watchdog_timer_fn.cold.6+0x62/0x10a [12636.167339] ? __hrtimer_run_queues+0x1c2/0x4d0 [12636.169840] ? watchdog+0x40/0x40 [12636.171460] ? hrtimer_interrupt+0x128/0x2c0 [12636.174981] ? smp_apic_timer_interrupt+0xa6/0x270 [12636.177447] ? apic_timer_interrupt+0xf/0x20 [12636.180620] [12636.181938] ? native_safe_halt+0x2e/0x50 [12636.184921] kvm_wait+0xcd/0xe0 [12636.186896] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12636.189514] do_raw_spin_lock+0x177/0x190 [12636.191424] _raw_spin_lock+0x21/0x30 [12636.194122] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12636.198008] process_one_work+0x2c8/0x700 [12636.199929] worker_thread+0x296/0x6e0 [12636.202137] ? rescuer_thread+0x570/0x570 [12636.204228] kthread+0x1d1/0x200 [12636.206260] ? set_kthread_struct+0x70/0x70 [12636.209611] ret_from_fork+0x1f/0x30 [12648.052043] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [12648.056217] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12648.082797] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12648.094174] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12648.099054] RIP: 0010:native_safe_halt+0x2e/0x50 [12648.101838] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12648.110831] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12648.114521] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12648.117657] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [12648.121830] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [12648.125413] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [12648.128924] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [12648.133655] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [12648.139850] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12648.144100] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [12648.149514] Call Trace: [12648.151460] [12648.152778] ? show_regs.cold.9+0x22/0x2f [12648.155175] ? watchdog_timer_fn.cold.6+0x62/0x10a [12648.158238] ? __hrtimer_run_queues+0x1c2/0x4d0 [12648.160609] ? watchdog+0x40/0x40 [12648.162819] ? hrtimer_interrupt+0x128/0x2c0 [12648.165938] ? smp_apic_timer_interrupt+0xa6/0x270 [12648.168374] ? apic_timer_interrupt+0xf/0x20 [12648.172218] [12648.173529] ? native_safe_halt+0x2e/0x50 [12648.181320] kvm_wait+0xcd/0xe0 [12648.182466] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12648.189940] do_raw_spin_lock+0x177/0x190 [12648.194900] _raw_spin_lock+0x21/0x30 [12648.199350] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12648.207605] ? do_raw_spin_unlock+0x75/0x190 [12648.210698] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12648.219567] ? do_wait_intr+0xf0/0xf0 [12648.224735] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12648.232264] mdc_reint+0x2d/0x110 [mdc] [12648.236926] mdc_unlink+0x658/0x870 [mdc] [12648.241008] lmv_unlink+0x34d/0xbd0 [lmv] [12648.250925] ll_unlink+0x24d/0x7f0 [lustre] [12648.256335] vfs_unlink+0x202/0x330 [12648.260785] do_unlinkat+0x2fa/0x460 [12648.263651] __x64_sys_unlink+0x37/0x50 [12648.269051] do_syscall_64+0xc1/0x440 [12648.275032] entry_SYSCALL_64_after_hwframe+0x49/0xae [12648.280085] RIP: 0033:0x7f151bdee10b [12648.286813] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [12648.296628] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [12648.299575] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [12648.302592] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [12648.306690] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12648.309810] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12648.313080] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12653.537491] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10722s! [12653.543280] Showing busy workqueues and worker pools: [12653.547075] workqueue events: flags=0x0 [12653.549032] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12653.554254] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [12653.567945] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12653.571168] pending: kfree_rcu_monitor [12653.572765] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12653.575142] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [12653.580668] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12653.583846] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [12653.587761] workqueue events_long: flags=0x0 [12653.589669] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12653.593071] pending: lu_objects_flush [obdclass] [12653.595840] workqueue events_unbound: flags=0x2 [12653.597527] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12653.600765] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12653.606241] workqueue events_power_efficient: flags=0x80 [12653.609177] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12653.612994] pending: do_cache_clean, neigh_periodic_work, check_lifetime [12653.616651] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12653.623937] pending: neigh_periodic_work [12653.626285] workqueue mm_percpu_wq: flags=0x8 [12653.632489] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12653.644946] pending: vmstat_update, lru_add_drain_per_cpu [12653.652475] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12653.664129] pending: vmstat_update [12653.668164] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12653.679827] pending: lru_add_drain_per_cpu [12653.682089] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [12653.687113] pending: lru_add_drain_per_cpu BAR(48) [12653.697256] workqueue writeback: flags=0x4a [12653.704275] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [12653.711035] pending: wb_workfn [12653.714208] workqueue xprtiod: flags=0xa [12653.718301] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12653.723838] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [12653.727944] workqueue ipv6_addrconf: flags=0x40008 [12653.730376] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12653.737080] pending: addrconf_verify_work [12653.743099] workqueue knbd0-recv: flags=0x1a [12653.744561] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12653.751679] in-flight: 658:recv_work [12653.755794] workqueue ptlrpc_pinger: flags=0x2 [12653.759012] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12653.762147] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12653.767130] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10723s workers=4 idle: 32984 37 8 [12653.770897] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12638s workers=2 idle: 661 [12660.055056] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [12660.057142] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [12660.062961] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12660.063056] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12660.063065] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12660.063068] RIP: 0010:native_safe_halt+0x2e/0x50 [12660.063082] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12660.063087] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12660.063095] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [12660.063098] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [12660.063100] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [12660.063102] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [12660.063105] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12660.063108] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [12660.063110] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12660.063113] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12660.063121] Call Trace: [12660.063127] [12660.063132] ? show_regs.cold.9+0x22/0x2f [12660.063142] ? watchdog_timer_fn.cold.6+0x62/0x10a [12660.063148] ? __hrtimer_run_queues+0x1c2/0x4d0 [12660.063152] ? watchdog+0x40/0x40 [12660.063157] ? hrtimer_interrupt+0x128/0x2c0 [12660.063160] ? smp_apic_timer_interrupt+0xa6/0x270 [12660.063166] ? apic_timer_interrupt+0xf/0x20 [12660.063178] [12660.063181] ? native_safe_halt+0x2e/0x50 [12660.063186] kvm_wait+0xcd/0xe0 [12660.063194] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12660.063202] do_raw_spin_lock+0x177/0x190 [12660.063207] _raw_spin_lock+0x21/0x30 [12660.063213] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12660.063502] obd_mod_free+0x46/0x140 [ptlrpc] [12660.063785] mdc_commit_open+0x91/0xb0 [mdc] [12660.063833] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12660.064074] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12660.064301] after_reply+0xe2c/0x1780 [ptlrpc] [12660.064530] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12660.064797] ? __next_timer_interrupt+0x160/0x160 [12660.064807] ptlrpcd+0xa82/0xe40 [ptlrpc] [12660.065062] ? do_wait_intr+0xf0/0xf0 [12660.065071] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12660.065323] kthread+0x1d1/0x200 [12660.065329] ? set_kthread_struct+0x70/0x70 [12660.065333] ret_from_fork+0x1f/0x30 [12660.209586] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12660.232036] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12660.238321] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12660.243019] RIP: 0010:native_safe_halt+0x2e/0x50 [12660.245067] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12660.254039] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12660.258762] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12660.262191] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [12660.266120] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [12660.271539] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [12660.275635] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [12660.278075] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [12660.281333] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12660.284295] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [12660.287988] Call Trace: [12660.289322] [12660.290923] ? show_regs.cold.9+0x22/0x2f [12660.292786] ? watchdog_timer_fn.cold.6+0x62/0x10a [12660.296026] ? __hrtimer_run_queues+0x1c2/0x4d0 [12660.298489] ? watchdog+0x40/0x40 [12660.299814] ? hrtimer_interrupt+0x128/0x2c0 [12660.302017] ? smp_apic_timer_interrupt+0xa6/0x270 [12660.304108] ? apic_timer_interrupt+0xf/0x20 [12660.306066] [12660.307023] ? native_safe_halt+0x2e/0x50 [12660.309064] kvm_wait+0xcd/0xe0 [12660.310483] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12660.313374] do_raw_spin_lock+0x177/0x190 [12660.315092] _raw_spin_lock+0x21/0x30 [12660.316917] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12660.319597] ? do_raw_spin_unlock+0x75/0x190 [12660.321798] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12660.325164] ? do_wait_intr+0xf0/0xf0 [12660.327506] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12660.330525] mdc_close+0x53b/0xd70 [mdc] [12660.332475] lmv_close+0x18d/0x490 [lmv] [12660.334749] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12660.338069] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12660.340523] ll_md_real_close+0xa5/0x2a0 [lustre] [12660.342519] ll_md_close+0x274/0x800 [lustre] [12660.344274] ll_file_release+0x128/0xc50 [lustre] [12660.346677] __fput+0x10d/0x440 [12660.347996] ____fput+0x12/0x20 [12660.349151] task_work_run+0xc6/0x110 [12660.350751] exit_to_usermode_loop+0x1dd/0x1f0 [12660.352480] do_syscall_64+0x3d6/0x440 [12660.353885] entry_SYSCALL_64_after_hwframe+0x49/0xae [12660.356198] RIP: 0033:0x7f151bee3cd8 [12660.357876] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12660.364069] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12660.366791] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [12660.369338] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [12660.371854] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12660.374511] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12660.376940] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12664.056048] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [12664.060104] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12664.095125] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12664.107965] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12664.115135] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12664.126391] RIP: 0010:native_safe_halt+0x2e/0x50 [12664.129586] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12664.152346] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12664.155737] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12664.162314] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [12664.166436] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [12664.170318] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [12664.177887] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [12664.183905] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [12664.192384] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12664.198448] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12664.206152] Call Trace: [12664.210946] [12664.211837] ? show_regs.cold.9+0x22/0x2f [12664.217494] ? watchdog_timer_fn.cold.6+0x62/0x10a [12664.219882] ? __hrtimer_run_queues+0x1c2/0x4d0 [12664.222133] ? watchdog+0x40/0x40 [12664.223849] ? hrtimer_interrupt+0x128/0x2c0 [12664.229304] ? smp_apic_timer_interrupt+0xa6/0x270 [12664.231915] ? apic_timer_interrupt+0xf/0x20 [12664.234428] [12664.235894] ? native_safe_halt+0x2e/0x50 [12664.238064] kvm_wait+0xcd/0xe0 [12664.240316] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12664.243074] do_raw_spin_lock+0x177/0x190 [12664.245185] _raw_spin_lock+0x21/0x30 [12664.246538] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12664.249244] process_one_work+0x2c8/0x700 [12664.251354] worker_thread+0x296/0x6e0 [12664.252919] ? rescuer_thread+0x570/0x570 [12664.255468] kthread+0x1d1/0x200 [12664.257232] ? set_kthread_struct+0x70/0x70 [12664.259358] ret_from_fork+0x1f/0x30 [12680.052028] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [12680.061454] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12680.091437] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12680.103770] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12680.109755] RIP: 0010:native_safe_halt+0x2e/0x50 [12680.112911] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12680.127437] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12680.130368] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12680.133625] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [12680.141722] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [12680.146386] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [12680.154455] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [12680.160000] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [12680.166901] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12680.173307] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [12680.177890] Call Trace: [12680.180111] [12680.181212] ? show_regs.cold.9+0x22/0x2f [12680.182956] ? watchdog_timer_fn.cold.6+0x62/0x10a [12680.185374] ? __hrtimer_run_queues+0x1c2/0x4d0 [12680.187506] ? watchdog+0x40/0x40 [12680.188720] ? hrtimer_interrupt+0x128/0x2c0 [12680.190731] ? smp_apic_timer_interrupt+0xa6/0x270 [12680.192678] ? apic_timer_interrupt+0xf/0x20 [12680.194587] [12680.195616] ? native_safe_halt+0x2e/0x50 [12680.197332] kvm_wait+0xcd/0xe0 [12680.198478] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12680.200719] do_raw_spin_lock+0x177/0x190 [12680.202465] _raw_spin_lock+0x21/0x30 [12680.204791] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12680.207003] ? do_raw_spin_unlock+0x75/0x190 [12680.208491] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12680.210867] ? do_wait_intr+0xf0/0xf0 [12680.212371] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12680.215478] mdc_reint+0x2d/0x110 [mdc] [12680.217084] mdc_unlink+0x658/0x870 [mdc] [12680.218957] lmv_unlink+0x34d/0xbd0 [lmv] [12680.220696] ll_unlink+0x24d/0x7f0 [lustre] [12680.222768] vfs_unlink+0x202/0x330 [12680.224523] do_unlinkat+0x2fa/0x460 [12680.226245] __x64_sys_unlink+0x37/0x50 [12680.238768] do_syscall_64+0xc1/0x440 [12680.244079] entry_SYSCALL_64_after_hwframe+0x49/0xae [12680.251230] RIP: 0033:0x7f151bdee10b [12680.260367] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [12680.285102] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [12680.292796] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [12680.301404] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [12680.308559] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12680.317827] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12680.322905] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12684.258305] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10753s! [12684.266965] Showing busy workqueues and worker pools: [12684.271786] workqueue events: flags=0x0 [12684.275230] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12684.280917] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [12684.288712] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12684.295197] pending: kfree_rcu_monitor [12684.299397] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12684.307935] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [12684.317826] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12684.323756] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [12684.327951] workqueue events_long: flags=0x0 [12684.329809] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12684.335996] pending: lu_objects_flush [obdclass] [12684.340892] workqueue events_unbound: flags=0x2 [12684.344737] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12684.352898] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12684.360889] workqueue events_power_efficient: flags=0x80 [12684.366223] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12684.370995] pending: do_cache_clean, neigh_periodic_work, check_lifetime [12684.377858] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12684.383874] pending: neigh_periodic_work [12684.388654] workqueue mm_percpu_wq: flags=0x8 [12684.394223] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12684.397639] pending: vmstat_update, lru_add_drain_per_cpu [12684.402161] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12684.408732] pending: vmstat_update [12684.411897] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12684.417771] pending: lru_add_drain_per_cpu [12684.421807] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [12684.425696] pending: lru_add_drain_per_cpu BAR(48) [12684.431831] workqueue writeback: flags=0x4a [12684.433777] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [12684.439927] pending: wb_workfn [12684.443922] workqueue xprtiod: flags=0xa [12684.447983] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12684.460213] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [12684.467927] workqueue ipv6_addrconf: flags=0x40008 [12684.472965] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12684.480161] pending: addrconf_verify_work [12684.483130] workqueue knbd0-recv: flags=0x1a [12684.486076] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12684.491465] in-flight: 658:recv_work [12684.493826] workqueue ptlrpc_pinger: flags=0x2 [12684.498889] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12684.502391] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12684.508193] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10753s workers=4 idle: 32984 37 8 [12684.516025] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12669s workers=2 idle: 661 [12688.055342] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [12688.059082] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [12688.076614] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12688.076694] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12688.076703] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12688.076706] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12688.076722] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12688.076727] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12688.076732] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [12688.076735] RDX: 0000000000005341 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [12688.076737] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [12688.076739] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [12688.076742] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12688.076745] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [12688.076748] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12688.076750] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12688.076760] Call Trace: [12688.076766] [12688.076772] ? show_regs.cold.9+0x22/0x2f [12688.076782] ? watchdog_timer_fn.cold.6+0x62/0x10a [12688.076789] ? __hrtimer_run_queues+0x1c2/0x4d0 [12688.076793] ? watchdog+0x40/0x40 [12688.076798] ? hrtimer_interrupt+0x128/0x2c0 [12688.076802] ? smp_apic_timer_interrupt+0xa6/0x270 [12688.076810] ? apic_timer_interrupt+0xf/0x20 [12688.076815] [12688.076817] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12688.076822] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12688.076827] do_raw_spin_lock+0x177/0x190 [12688.076831] _raw_spin_lock+0x21/0x30 [12688.076837] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12688.081168] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12688.081252] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12688.081259] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12688.081262] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12688.081276] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12688.081281] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12688.081284] RAX: 0000000000002f0c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12688.081287] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [12688.081289] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [12688.081291] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [12688.081293] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [12688.081296] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [12688.081299] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12688.081302] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [12688.081312] Call Trace: [12688.081317] [12688.081322] ? show_regs.cold.9+0x22/0x2f [12688.081331] ? watchdog_timer_fn.cold.6+0x62/0x10a [12688.081338] ? __hrtimer_run_queues+0x1c2/0x4d0 [12688.081343] ? watchdog+0x40/0x40 [12688.081346] ? hrtimer_interrupt+0x128/0x2c0 [12688.081348] ? smp_apic_timer_interrupt+0xa6/0x270 [12688.081354] ? apic_timer_interrupt+0xf/0x20 [12688.081357] [12688.081359] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12688.081364] do_raw_spin_lock+0x177/0x190 [12688.081368] _raw_spin_lock+0x21/0x30 [12688.081371] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12688.083974] ? do_raw_spin_unlock+0x75/0x190 [12688.083983] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12688.084256] ? do_wait_intr+0xf0/0xf0 [12688.084264] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12688.084458] mdc_close+0x53b/0xd70 [mdc] [12688.084497] lmv_close+0x18d/0x490 [lmv] [12688.084526] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12688.085064] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12688.140006] obd_mod_free+0x46/0x140 [ptlrpc] [12688.143861] mdc_commit_open+0x91/0xb0 [mdc] [12688.144829] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12688.173633] ll_md_real_close+0xa5/0x2a0 [lustre] [12688.186311] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12688.196157] ll_md_close+0x274/0x800 [lustre] [12688.231750] after_reply+0xe2c/0x1780 [ptlrpc] [12688.232091] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12688.232445] ? __next_timer_interrupt+0x160/0x160 [12688.232457] ptlrpcd+0xa82/0xe40 [ptlrpc] [12688.232670] ? do_wait_intr+0xf0/0xf0 [12688.232680] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12688.232891] kthread+0x1d1/0x200 [12688.232897] ? set_kthread_struct+0x70/0x70 [12688.232901] ret_from_fork+0x1f/0x30 [12688.887525] ll_file_release+0x128/0xc50 [lustre] [12688.899778] __fput+0x10d/0x440 [12688.901893] ____fput+0x12/0x20 [12688.904819] task_work_run+0xc6/0x110 [12688.913468] exit_to_usermode_loop+0x1dd/0x1f0 [12688.929914] do_syscall_64+0x3d6/0x440 [12688.936284] entry_SYSCALL_64_after_hwframe+0x49/0xae [12688.941976] RIP: 0033:0x7f151bee3cd8 [12688.944859] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12688.976961] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12688.986501] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [12688.990923] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [12688.997627] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12689.003937] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12689.015843] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12692.057512] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12692.062917] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12692.132470] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12692.144512] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12692.151185] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12692.155891] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12692.160379] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12692.180437] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12692.196792] RAX: 0000000000001482 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12692.215203] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [12692.227103] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [12692.240314] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [12692.248554] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [12692.259741] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [12692.279393] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12692.287283] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12692.302245] Call Trace: [12692.307421] [12692.309438] ? show_regs.cold.9+0x22/0x2f [12692.312385] ? watchdog_timer_fn.cold.6+0x62/0x10a [12692.316975] ? __hrtimer_run_queues+0x1c2/0x4d0 [12692.320257] ? watchdog+0x40/0x40 [12692.323632] ? hrtimer_interrupt+0x128/0x2c0 [12692.327457] ? smp_apic_timer_interrupt+0xa6/0x270 [12692.329988] ? apic_timer_interrupt+0xf/0x20 [12692.332307] [12692.333291] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12692.336295] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12692.338818] do_raw_spin_lock+0x177/0x190 [12692.341292] _raw_spin_lock+0x21/0x30 [12692.343276] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12692.347654] process_one_work+0x2c8/0x700 [12692.349810] worker_thread+0x296/0x6e0 [12692.351477] ? rescuer_thread+0x570/0x570 [12692.353735] kthread+0x1d1/0x200 [12692.355228] ? set_kthread_struct+0x70/0x70 [12692.356984] ret_from_fork+0x1f/0x30 [12712.053252] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [12712.060328] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12712.111896] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12712.119582] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12712.127855] RIP: 0010:native_safe_halt+0x2e/0x50 [12712.131316] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12712.146353] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12712.156370] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12712.166596] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [12712.176382] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [12712.187898] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [12712.194289] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [12712.203641] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [12712.215015] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12712.224714] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [12712.235812] Call Trace: [12712.237635] [12712.239659] ? show_regs.cold.9+0x22/0x2f [12712.242712] ? watchdog_timer_fn.cold.6+0x62/0x10a [12712.247551] ? __hrtimer_run_queues+0x1c2/0x4d0 [12712.253725] ? watchdog+0x40/0x40 [12712.257262] ? hrtimer_interrupt+0x128/0x2c0 [12712.263980] ? smp_apic_timer_interrupt+0xa6/0x270 [12712.274499] ? apic_timer_interrupt+0xf/0x20 [12712.279801] [12712.280582] ? native_safe_halt+0x2e/0x50 [12712.283428] kvm_wait+0xcd/0xe0 [12712.286276] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12712.289990] do_raw_spin_lock+0x177/0x190 [12712.292717] _raw_spin_lock+0x21/0x30 [12712.296430] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12712.301394] ? do_raw_spin_unlock+0x75/0x190 [12712.305223] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12712.311264] ? do_wait_intr+0xf0/0xf0 [12712.317431] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12712.325674] mdc_reint+0x2d/0x110 [mdc] [12712.330290] mdc_unlink+0x658/0x870 [mdc] [12712.334340] lmv_unlink+0x34d/0xbd0 [lmv] [12712.337447] ll_unlink+0x24d/0x7f0 [lustre] [12712.340381] vfs_unlink+0x202/0x330 [12712.341485] do_unlinkat+0x2fa/0x460 [12712.342562] __x64_sys_unlink+0x37/0x50 [12712.352425] do_syscall_64+0xc1/0x440 [12712.357655] entry_SYSCALL_64_after_hwframe+0x49/0xae [12712.364984] RIP: 0033:0x7f151bdee10b [12712.369682] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [12712.385969] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [12712.391989] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [12712.398903] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [12712.404300] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12712.408984] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12712.414393] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12714.977172] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10784s! [12714.982565] Showing busy workqueues and worker pools: [12714.986362] workqueue events: flags=0x0 [12714.988515] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12714.999261] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [12715.008685] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12715.016166] pending: kfree_rcu_monitor [12715.019409] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12715.025244] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [12715.031436] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12715.036681] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [12715.042991] workqueue events_long: flags=0x0 [12715.049029] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12715.051342] pending: lu_objects_flush [obdclass] [12715.067693] workqueue events_unbound: flags=0x2 [12715.071577] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12715.076468] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12715.083530] workqueue events_power_efficient: flags=0x80 [12715.087502] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12715.091637] pending: do_cache_clean, neigh_periodic_work, check_lifetime [12715.100431] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12715.106409] pending: neigh_periodic_work [12715.109508] workqueue mm_percpu_wq: flags=0x8 [12715.112687] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12715.118529] pending: vmstat_update, lru_add_drain_per_cpu [12715.123194] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12715.131532] pending: vmstat_update [12715.137518] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12715.141432] pending: lru_add_drain_per_cpu [12715.145291] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [12715.157452] pending: lru_add_drain_per_cpu BAR(48) [12715.163650] workqueue writeback: flags=0x4a [12715.167686] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [12715.188045] pending: wb_workfn [12715.189334] workqueue xprtiod: flags=0xa [12715.192903] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12715.197037] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [12715.205181] workqueue ipv6_addrconf: flags=0x40008 [12715.209208] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12715.212364] pending: addrconf_verify_work [12715.215210] workqueue knbd0-recv: flags=0x1a [12715.217921] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12715.224383] in-flight: 658:recv_work [12715.228323] workqueue ptlrpc_pinger: flags=0x2 [12715.231742] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12715.242753] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12715.246228] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10784s workers=4 idle: 32984 37 8 [12715.257609] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12700s workers=2 idle: 661 [12716.055056] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [12716.058051] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12716.061882] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [12716.077046] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12716.077063] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12716.077066] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12716.077085] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12716.077093] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12716.077102] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [12716.077105] RDX: 00000000000003c7 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [12716.077107] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [12716.077109] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [12716.077111] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12716.077114] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [12716.077116] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12716.077118] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12716.077129] Call Trace: [12716.077138] [12716.077144] ? show_regs.cold.9+0x22/0x2f [12716.077157] ? watchdog_timer_fn.cold.6+0x62/0x10a [12716.077163] ? __hrtimer_run_queues+0x1c2/0x4d0 [12716.077168] ? watchdog+0x40/0x40 [12716.077177] ? hrtimer_interrupt+0x128/0x2c0 [12716.077181] ? smp_apic_timer_interrupt+0xa6/0x270 [12716.077196] ? apic_timer_interrupt+0xf/0x20 [12716.077202] [12716.077204] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12716.077209] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12716.077214] do_raw_spin_lock+0x177/0x190 [12716.077221] _raw_spin_lock+0x21/0x30 [12716.077229] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12716.077572] obd_mod_free+0x46/0x140 [ptlrpc] [12716.077786] mdc_commit_open+0x91/0xb0 [mdc] [12716.077822] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12716.078018] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12716.078218] after_reply+0xe2c/0x1780 [ptlrpc] [12716.078477] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12716.078890] ? __next_timer_interrupt+0x160/0x160 [12716.078900] ptlrpcd+0xa82/0xe40 [ptlrpc] [12716.079300] ? do_wait_intr+0xf0/0xf0 [12716.079310] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12716.079595] kthread+0x1d1/0x200 [12716.079601] ? set_kthread_struct+0x70/0x70 [12716.079605] ret_from_fork+0x1f/0x30 [12716.181147] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12716.201598] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12716.206446] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12716.210626] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12716.212841] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12716.221671] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12716.224973] RAX: 0000000000004d83 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12716.228286] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [12716.231491] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [12716.234614] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [12716.237329] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [12716.240304] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [12716.243974] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12716.246369] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [12716.249281] Call Trace: [12716.250600] [12716.251355] ? show_regs.cold.9+0x22/0x2f [12716.253017] ? watchdog_timer_fn.cold.6+0x62/0x10a [12716.255060] ? __hrtimer_run_queues+0x1c2/0x4d0 [12716.257115] ? watchdog+0x40/0x40 [12716.258335] ? hrtimer_interrupt+0x128/0x2c0 [12716.260933] ? smp_apic_timer_interrupt+0xa6/0x270 [12716.262788] ? apic_timer_interrupt+0xf/0x20 [12716.264470] [12716.265466] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12716.267806] do_raw_spin_lock+0x177/0x190 [12716.269344] _raw_spin_lock+0x21/0x30 [12716.271312] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12716.274025] ? do_raw_spin_unlock+0x75/0x190 [12716.275649] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12716.277999] ? do_wait_intr+0xf0/0xf0 [12716.279701] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12716.282132] mdc_close+0x53b/0xd70 [mdc] [12716.283952] lmv_close+0x18d/0x490 [lmv] [12716.285578] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12716.288051] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12716.290576] ll_md_real_close+0xa5/0x2a0 [lustre] [12716.292546] ll_md_close+0x274/0x800 [lustre] [12716.294660] ll_file_release+0x128/0xc50 [lustre] [12716.297234] __fput+0x10d/0x440 [12716.298511] ____fput+0x12/0x20 [12716.299834] task_work_run+0xc6/0x110 [12716.301206] exit_to_usermode_loop+0x1dd/0x1f0 [12716.303004] do_syscall_64+0x3d6/0x440 [12716.305631] entry_SYSCALL_64_after_hwframe+0x49/0xae [12716.307815] RIP: 0033:0x7f151bee3cd8 [12716.309183] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12716.318339] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12716.321600] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [12716.324639] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [12716.328149] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12716.331447] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12716.334469] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12720.056037] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12720.058865] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12720.118571] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12720.136109] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12720.153487] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12720.170246] RIP: 0010:native_safe_halt+0x2e/0x50 [12720.174771] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12720.197589] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [12720.202085] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12720.207170] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [12720.215853] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [12720.225131] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [12720.234892] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [12720.241240] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [12720.244567] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12720.250128] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12720.256059] Call Trace: [12720.258169] [12720.261999] ? show_regs.cold.9+0x22/0x2f [12720.267168] ? watchdog_timer_fn.cold.6+0x62/0x10a [12720.268978] ? __hrtimer_run_queues+0x1c2/0x4d0 [12720.276841] ? watchdog+0x40/0x40 [12720.278216] ? hrtimer_interrupt+0x128/0x2c0 [12720.286035] ? smp_apic_timer_interrupt+0xa6/0x270 [12720.290283] ? apic_timer_interrupt+0xf/0x20 [12720.292257] [12720.293443] ? native_safe_halt+0x2e/0x50 [12720.296370] kvm_wait+0xcd/0xe0 [12720.298988] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12720.302008] do_raw_spin_lock+0x177/0x190 [12720.304193] _raw_spin_lock+0x21/0x30 [12720.307004] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12720.310343] process_one_work+0x2c8/0x700 [12720.313282] worker_thread+0x296/0x6e0 [12720.315606] ? rescuer_thread+0x570/0x570 [12720.316918] kthread+0x1d1/0x200 [12720.318552] ? set_kthread_struct+0x70/0x70 [12720.320458] ret_from_fork+0x1f/0x30 [12740.053717] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [12740.061250] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12740.103406] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12740.112862] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12740.121185] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12740.125888] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12740.145207] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12740.149243] RAX: 0000000000007909 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12740.156357] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [12740.163167] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [12740.170745] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [12740.174726] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [12740.182277] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [12740.191160] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12740.198698] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [12740.201631] Call Trace: [12740.203318] [12740.204239] ? show_regs.cold.9+0x22/0x2f [12740.205836] ? watchdog_timer_fn.cold.6+0x62/0x10a [12740.213364] ? __hrtimer_run_queues+0x1c2/0x4d0 [12740.217571] ? watchdog+0x40/0x40 [12740.218992] ? hrtimer_interrupt+0x128/0x2c0 [12740.224605] ? smp_apic_timer_interrupt+0xa6/0x270 [12740.234373] ? apic_timer_interrupt+0xf/0x20 [12740.246089] [12740.247566] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12740.249454] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12740.256334] do_raw_spin_lock+0x177/0x190 [12740.258909] _raw_spin_lock+0x21/0x30 [12740.265194] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12740.270874] ? do_raw_spin_unlock+0x75/0x190 [12740.276145] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12740.280354] ? do_wait_intr+0xf0/0xf0 [12740.285140] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12740.291849] mdc_reint+0x2d/0x110 [mdc] [12740.296938] mdc_unlink+0x658/0x870 [mdc] [12740.304348] lmv_unlink+0x34d/0xbd0 [lmv] [12740.311924] ll_unlink+0x24d/0x7f0 [lustre] [12740.315917] vfs_unlink+0x202/0x330 [12740.321191] do_unlinkat+0x2fa/0x460 [12740.326631] __x64_sys_unlink+0x37/0x50 [12740.329129] do_syscall_64+0xc1/0x440 [12740.335262] entry_SYSCALL_64_after_hwframe+0x49/0xae [12740.343332] RIP: 0033:0x7f151bdee10b [12740.345122] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [12740.359509] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [12740.368153] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [12740.373443] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [12740.379503] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12740.384565] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12740.388806] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12744.060379] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [12744.065266] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [12744.071067] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12744.071144] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12744.071151] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12744.071154] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12744.071167] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12744.071171] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12744.071176] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [12744.071179] RDX: 000000000000646e RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [12744.071181] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [12744.071184] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [12744.071186] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12744.071189] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [12744.071193] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12744.071195] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12744.071215] Call Trace: [12744.071220] [12744.071225] ? show_regs.cold.9+0x22/0x2f [12744.071234] ? watchdog_timer_fn.cold.6+0x62/0x10a [12744.071241] ? __hrtimer_run_queues+0x1c2/0x4d0 [12744.071245] ? watchdog+0x40/0x40 [12744.071250] ? hrtimer_interrupt+0x128/0x2c0 [12744.071254] ? smp_apic_timer_interrupt+0xa6/0x270 [12744.071263] ? apic_timer_interrupt+0xf/0x20 [12744.071267] [12744.071270] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12744.071275] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12744.071280] do_raw_spin_lock+0x177/0x190 [12744.071285] _raw_spin_lock+0x21/0x30 [12744.071291] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12744.071549] obd_mod_free+0x46/0x140 [ptlrpc] [12744.185688] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12744.185769] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12744.185775] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12744.185779] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12744.185799] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12744.185803] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12744.185808] RAX: 0000000000001b8e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12744.185810] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [12744.185812] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [12744.185815] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [12744.185817] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [12744.185820] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [12744.185823] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12744.185825] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [12744.185834] Call Trace: [12744.185839] [12744.185845] ? show_regs.cold.9+0x22/0x2f [12744.185853] ? watchdog_timer_fn.cold.6+0x62/0x10a [12744.185860] ? __hrtimer_run_queues+0x1c2/0x4d0 [12744.185864] ? watchdog+0x40/0x40 [12744.185869] ? hrtimer_interrupt+0x128/0x2c0 [12744.185873] ? smp_apic_timer_interrupt+0xa6/0x270 [12744.185881] ? apic_timer_interrupt+0xf/0x20 [12744.185886] [12744.185889] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12744.185896] do_raw_spin_lock+0x177/0x190 [12744.185900] _raw_spin_lock+0x21/0x30 [12744.185905] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12744.186185] ? do_raw_spin_unlock+0x75/0x190 [12744.186191] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12744.187021] ? do_wait_intr+0xf0/0xf0 [12744.187031] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12744.187278] mdc_close+0x53b/0xd70 [mdc] [12744.187314] lmv_close+0x18d/0x490 [lmv] [12744.187341] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12744.187527] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12744.188324] ll_md_real_close+0xa5/0x2a0 [lustre] [12744.188412] ll_md_close+0x274/0x800 [lustre] [12744.188502] ll_file_release+0x128/0xc50 [lustre] [12744.188960] __fput+0x10d/0x440 [12744.188969] ____fput+0x12/0x20 [12744.188980] task_work_run+0xc6/0x110 [12744.188990] exit_to_usermode_loop+0x1dd/0x1f0 [12744.188996] do_syscall_64+0x3d6/0x440 [12744.189001] entry_SYSCALL_64_after_hwframe+0x49/0xae [12744.189009] RIP: 0033:0x7f151bee3cd8 [12744.189017] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12744.189022] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12744.189027] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [12744.189029] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [12744.189031] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12744.189034] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12744.189036] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12745.110268] mdc_commit_open+0x91/0xb0 [mdc] [12745.113304] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12745.120225] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12745.128283] after_reply+0xe2c/0x1780 [ptlrpc] [12745.129869] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12745.131952] ? __next_timer_interrupt+0x160/0x160 [12745.133283] ptlrpcd+0xa82/0xe40 [ptlrpc] [12745.135364] ? do_wait_intr+0xf0/0xf0 [12745.136783] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12745.138750] kthread+0x1d1/0x200 [12745.140029] ? set_kthread_struct+0x70/0x70 [12745.142670] ret_from_fork+0x1f/0x30 [12745.700966] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10815s! [12745.709660] Showing busy workqueues and worker pools: [12745.712043] workqueue events: flags=0x0 [12745.717790] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12745.727602] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [12745.735121] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12745.748962] pending: kfree_rcu_monitor [12745.756347] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12745.768746] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [12745.779355] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12745.794363] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [12745.808872] workqueue events_long: flags=0x0 [12745.817961] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12745.822426] pending: lu_objects_flush [obdclass] [12745.824934] workqueue events_unbound: flags=0x2 [12745.827244] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12745.830324] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12745.835732] workqueue events_power_efficient: flags=0x80 [12745.838271] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12745.842217] pending: do_cache_clean, neigh_periodic_work, check_lifetime [12745.845897] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12745.849231] pending: neigh_periodic_work [12745.851244] workqueue mm_percpu_wq: flags=0x8 [12745.854138] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12745.858689] pending: vmstat_update, lru_add_drain_per_cpu [12745.861618] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12745.865168] pending: vmstat_update [12745.867425] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12745.871110] pending: lru_add_drain_per_cpu [12745.873231] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [12745.876919] pending: lru_add_drain_per_cpu BAR(48) [12745.880318] workqueue writeback: flags=0x4a [12745.882779] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [12745.891801] pending: wb_workfn [12745.893115] workqueue xprtiod: flags=0xa [12745.898732] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12745.904095] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [12745.909928] workqueue ipv6_addrconf: flags=0x40008 [12745.918488] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12745.928958] pending: addrconf_verify_work [12745.932037] workqueue knbd0-recv: flags=0x1a [12745.937571] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12745.941253] in-flight: 658:recv_work [12745.943507] workqueue ptlrpc_pinger: flags=0x2 [12745.951060] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12745.957908] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12745.961173] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10815s workers=4 idle: 32984 37 8 [12745.965158] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12730s workers=2 idle: 661 [12748.065066] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12748.074947] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12748.146369] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12748.158923] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12748.166333] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12748.180716] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12748.186295] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12748.197978] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12748.206374] RAX: 00000000000009c5 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12748.215276] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [12748.218645] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [12748.224365] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [12748.235369] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [12748.242401] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [12748.251678] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12748.254872] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12748.259068] Call Trace: [12748.268801] [12748.269767] ? show_regs.cold.9+0x22/0x2f [12748.274750] ? watchdog_timer_fn.cold.6+0x62/0x10a [12748.280269] ? __hrtimer_run_queues+0x1c2/0x4d0 [12748.285841] ? watchdog+0x40/0x40 [12748.289293] ? hrtimer_interrupt+0x128/0x2c0 [12748.295093] ? smp_apic_timer_interrupt+0xa6/0x270 [12748.296791] ? apic_timer_interrupt+0xf/0x20 [12748.307073] [12748.309916] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12748.313859] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12748.321686] do_raw_spin_lock+0x177/0x190 [12748.325315] _raw_spin_lock+0x21/0x30 [12748.328821] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12748.336779] process_one_work+0x2c8/0x700 [12748.338772] worker_thread+0x296/0x6e0 [12748.340535] ? rescuer_thread+0x570/0x570 [12748.347648] kthread+0x1d1/0x200 [12748.354748] ? set_kthread_struct+0x70/0x70 [12748.357870] ret_from_fork+0x1f/0x30 [12772.052195] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [12772.055069] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [12772.055569] Modules linked in: [12772.058183] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [12772.058197] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12772.058270] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12772.058276] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12772.058279] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12772.058306] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12772.058310] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12772.058315] RAX: 0000000000005ec2 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12772.058317] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [12772.058320] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [12772.058322] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [12772.058325] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [12772.058328] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [12772.058331] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12772.058333] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [12772.058342] Call Trace: [12772.058347] [12772.058353] ? show_regs.cold.9+0x22/0x2f [12772.058363] ? watchdog_timer_fn.cold.6+0x62/0x10a [12772.058370] ? __hrtimer_run_queues+0x1c2/0x4d0 [12772.058374] ? watchdog+0x40/0x40 [12772.058379] ? hrtimer_interrupt+0x128/0x2c0 [12772.058383] ? smp_apic_timer_interrupt+0xa6/0x270 [12772.058391] ? apic_timer_interrupt+0xf/0x20 [12772.058396] [12772.058399] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12772.058405] do_raw_spin_lock+0x177/0x190 [12772.058410] _raw_spin_lock+0x21/0x30 [12772.058415] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12772.058968] ? do_raw_spin_unlock+0x75/0x190 [12772.058974] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12772.059172] ? do_wait_intr+0xf0/0xf0 [12772.059184] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12772.059415] mdc_close+0x53b/0xd70 [mdc] [12772.059455] lmv_close+0x18d/0x490 [lmv] [12772.059565] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12772.059815] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12772.059932] ll_md_real_close+0xa5/0x2a0 [lustre] [12772.060029] ll_md_close+0x274/0x800 [lustre] [12772.060115] ll_file_release+0x128/0xc50 [lustre] [12772.060195] __fput+0x10d/0x440 [12772.060204] ____fput+0x12/0x20 [12772.060208] task_work_run+0xc6/0x110 [12772.060215] exit_to_usermode_loop+0x1dd/0x1f0 [12772.060221] do_syscall_64+0x3d6/0x440 [12772.060225] entry_SYSCALL_64_after_hwframe+0x49/0xae [12772.060233] RIP: 0033:0x7f151bee3cd8 [12772.060240] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12772.060245] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12772.061056] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12772.061139] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12772.061145] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12772.061147] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12772.061161] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12772.061164] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12772.061168] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [12772.061170] RDX: 0000000000001691 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [12772.061173] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [12772.061175] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [12772.061177] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12772.061180] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [12772.061183] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12772.061186] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12772.061201] Call Trace: [12772.061206] [12772.061210] ? show_regs.cold.9+0x22/0x2f [12772.061223] ? watchdog_timer_fn.cold.6+0x62/0x10a [12772.061230] ? __hrtimer_run_queues+0x1c2/0x4d0 [12772.061238] ? watchdog+0x40/0x40 [12772.061242] ? hrtimer_interrupt+0x128/0x2c0 [12772.061246] ? smp_apic_timer_interrupt+0xa6/0x270 [12772.061256] ? apic_timer_interrupt+0xf/0x20 [12772.061261] [12772.061263] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12772.061268] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12772.061273] do_raw_spin_lock+0x177/0x190 [12772.061277] _raw_spin_lock+0x21/0x30 [12772.061291] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12772.060249] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [12772.061777] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [12772.061780] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12772.061782] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12772.061785] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12772.061945] obd_mod_free+0x46/0x140 [ptlrpc] [12772.062378] mdc_commit_open+0x91/0xb0 [mdc] [12772.062432] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12772.063067] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12772.066890] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12772.066981] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12772.066989] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12772.066992] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12772.067008] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12772.067013] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12772.067017] RAX: 0000000000005f9e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12772.067019] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [12772.067021] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [12772.067023] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [12772.067025] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [12772.067029] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [12772.067032] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12772.067034] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [12772.067043] Call Trace: [12772.067049] [12772.067056] ? show_regs.cold.9+0x22/0x2f [12772.067065] ? watchdog_timer_fn.cold.6+0x62/0x10a [12772.067072] ? __hrtimer_run_queues+0x1c2/0x4d0 [12772.067076] ? watchdog+0x40/0x40 [12772.067080] ? hrtimer_interrupt+0x128/0x2c0 [12772.067084] ? smp_apic_timer_interrupt+0xa6/0x270 [12772.067093] ? apic_timer_interrupt+0xf/0x20 [12772.067098] [12772.067100] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12772.067105] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12772.067110] do_raw_spin_lock+0x177/0x190 [12772.067115] _raw_spin_lock+0x21/0x30 [12772.067121] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12772.067578] ? do_raw_spin_unlock+0x75/0x190 [12772.067584] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12772.071090] after_reply+0xe2c/0x1780 [ptlrpc] [12772.071500] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12772.071841] ? __next_timer_interrupt+0x160/0x160 [12772.071852] ptlrpcd+0xa82/0xe40 [ptlrpc] [12772.072077] ? do_wait_intr+0xf0/0xf0 [12772.072087] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12772.072319] kthread+0x1d1/0x200 [12772.072325] ? set_kthread_struct+0x70/0x70 [12772.072329] ret_from_fork+0x1f/0x30 [12772.561594] ? do_wait_intr+0xf0/0xf0 [12772.563055] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12772.565807] mdc_reint+0x2d/0x110 [mdc] [12772.567647] mdc_unlink+0x658/0x870 [mdc] [12772.569373] lmv_unlink+0x34d/0xbd0 [lmv] [12772.571559] ll_unlink+0x24d/0x7f0 [lustre] [12772.573628] vfs_unlink+0x202/0x330 [12772.575222] do_unlinkat+0x2fa/0x460 [12772.576643] __x64_sys_unlink+0x37/0x50 [12772.578898] do_syscall_64+0xc1/0x440 [12772.580554] entry_SYSCALL_64_after_hwframe+0x49/0xae [12772.583213] RIP: 0033:0x7f151bdee10b [12772.584627] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [12772.592024] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [12772.596789] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [12772.600511] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [12772.604621] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12772.607621] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12772.610561] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12776.056025] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12776.059583] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12776.081641] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12776.087079] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12776.091230] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12776.095081] RIP: 0010:native_safe_halt+0x2e/0x50 [12776.097581] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12776.108082] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12776.112030] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12776.117591] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [12776.122358] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [12776.126366] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [12776.130366] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [12776.136029] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [12776.140655] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12776.143181] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12776.146223] Call Trace: [12776.147730] [12776.148486] ? show_regs.cold.9+0x22/0x2f [12776.150553] ? watchdog_timer_fn.cold.6+0x62/0x10a [12776.153281] ? __hrtimer_run_queues+0x1c2/0x4d0 [12776.155335] ? watchdog+0x40/0x40 [12776.157233] ? hrtimer_interrupt+0x128/0x2c0 [12776.158586] ? smp_apic_timer_interrupt+0xa6/0x270 [12776.160162] ? apic_timer_interrupt+0xf/0x20 [12776.162118] [12776.163156] ? native_safe_halt+0x2e/0x50 [12776.164865] kvm_wait+0xcd/0xe0 [12776.166469] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12776.169075] do_raw_spin_lock+0x177/0x190 [12776.171008] _raw_spin_lock+0x21/0x30 [12776.172981] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12776.175782] process_one_work+0x2c8/0x700 [12776.177977] worker_thread+0x296/0x6e0 [12776.179793] ? rescuer_thread+0x570/0x570 [12776.181617] kthread+0x1d1/0x200 [12776.183157] ? set_kthread_struct+0x70/0x70 [12776.185382] ret_from_fork+0x1f/0x30 [12776.418483] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10845s! [12776.428050] Showing busy workqueues and worker pools: [12776.431943] workqueue events: flags=0x0 [12776.433957] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12776.436880] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [12776.447987] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12776.454504] pending: kfree_rcu_monitor [12776.461815] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12776.466822] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [12776.484319] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12776.486295] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [12776.494477] workqueue events_long: flags=0x0 [12776.498362] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12776.504438] pending: lu_objects_flush [obdclass] [12776.507538] workqueue events_unbound: flags=0x2 [12776.511076] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12776.516395] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12776.525741] workqueue events_power_efficient: flags=0x80 [12776.527476] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12776.533760] pending: do_cache_clean, neigh_periodic_work, check_lifetime [12776.538782] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12776.541042] pending: neigh_periodic_work [12776.542517] workqueue mm_percpu_wq: flags=0x8 [12776.546749] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12776.552448] pending: vmstat_update, lru_add_drain_per_cpu [12776.555294] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12776.558763] pending: vmstat_update [12776.560861] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12776.564163] pending: lru_add_drain_per_cpu [12776.570930] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [12776.576457] pending: lru_add_drain_per_cpu BAR(48) [12776.578257] workqueue writeback: flags=0x4a [12776.586317] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [12776.589075] pending: wb_workfn [12776.592014] workqueue xprtiod: flags=0xa [12776.593527] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12776.598303] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [12776.604017] workqueue ipv6_addrconf: flags=0x40008 [12776.606815] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12776.611685] pending: addrconf_verify_work [12776.614332] workqueue knbd0-recv: flags=0x1a [12776.619804] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12776.623892] in-flight: 658:recv_work [12776.626319] workqueue ptlrpc_pinger: flags=0x2 [12776.629478] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12776.633692] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12776.638890] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10846s workers=4 idle: 32984 37 8 [12776.645349] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12761s workers=2 idle: 661 [12784.883022] rcu: INFO: rcu_sched self-detected stall on CPU [12784.884624] rcu: 1-...!: (6292005 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [12784.888258] (t=10860236 jiffies g=249657 q=5649) [12784.890628] rcu: rcu_sched kthread starved for 10855188 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [12784.904117] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [12784.911478] rcu: RCU grace-period kthread stack dump: [12784.918803] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [12784.928628] Call Trace: [12784.932609] __schedule+0x351/0xcb0 [12784.934357] schedule+0xc0/0x180 [12784.936474] schedule_timeout+0xb4/0x190 [12784.944769] ? do_raw_spin_unlock+0x75/0x190 [12784.949751] ? __next_timer_interrupt+0x160/0x160 [12784.951541] ? prepare_to_swait_event+0xf3/0x140 [12784.964761] rcu_gp_kthread+0x8dd/0x15b0 [12784.967818] ? force_qs_rnp+0x2c0/0x2c0 [12784.969612] kthread+0x1d1/0x200 [12784.970870] ? set_kthread_struct+0x70/0x70 [12784.975583] ret_from_fork+0x1f/0x30 [12784.981135] rcu: Stack dump where RCU GP kthread last ran: [12784.984844] Sending NMI from CPU 1 to CPUs 0: [12784.991570] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [12784.996088] NMI backtrace for cpu 1 [12785.004975] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12785.014072] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12785.021732] Call Trace: [12785.022693] [12785.024261] ? dump_stack+0xbb/0x10e [12785.026243] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [12785.030535] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [12785.033467] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [12785.036437] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [12785.041979] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [12785.046313] ? rcu_dump_cpu_stacks+0x11b/0x171 [12785.051685] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [12785.056257] ? update_process_times+0x6d/0xc0 [12785.061899] ? tick_sched_handle.isra.18+0x27/0x90 [12785.064612] ? tick_sched_timer+0x53/0xb0 [12785.067675] ? __hrtimer_run_queues+0x1c2/0x4d0 [12785.070501] ? tick_sched_handle.isra.18+0x90/0x90 [12785.073400] ? hrtimer_interrupt+0x128/0x2c0 [12785.075633] ? smp_apic_timer_interrupt+0xa6/0x270 [12785.078140] ? apic_timer_interrupt+0xf/0x20 [12785.081751] [12785.083943] ? native_safe_halt+0x2e/0x50 [12785.093291] ? kvm_wait+0xcd/0xe0 [12785.097908] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12785.103011] ? do_raw_spin_lock+0x177/0x190 [12785.107843] ? _raw_spin_lock+0x21/0x30 [12785.112922] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12785.119264] ? obd_mod_free+0x46/0x140 [ptlrpc] [12785.122785] ? mdc_commit_open+0x91/0xb0 [mdc] [12785.129647] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12785.139850] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12785.152234] ? after_reply+0xe2c/0x1780 [ptlrpc] [12785.159836] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12785.163787] ? __next_timer_interrupt+0x160/0x160 [12785.165954] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [12785.173230] ? do_wait_intr+0xf0/0xf0 [12785.176276] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12785.182977] ? kthread+0x1d1/0x200 [12785.185519] ? set_kthread_struct+0x70/0x70 [12785.187996] ? ret_from_fork+0x1f/0x30 [12800.059119] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [12800.069869] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12800.140692] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12800.145325] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12800.156678] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12800.166244] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12800.177634] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12800.185552] RAX: 0000000000005008 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12800.194720] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [12800.199165] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [12800.204678] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [12800.208580] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [12800.217455] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [12800.225196] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12800.228579] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [12800.236436] Call Trace: [12800.237516] [12800.238662] ? show_regs.cold.9+0x22/0x2f [12800.241615] ? watchdog_timer_fn.cold.6+0x62/0x10a [12800.247084] ? __hrtimer_run_queues+0x1c2/0x4d0 [12800.253061] ? watchdog+0x40/0x40 [12800.253927] ? hrtimer_interrupt+0x128/0x2c0 [12800.266179] ? smp_apic_timer_interrupt+0xa6/0x270 [12800.267692] ? apic_timer_interrupt+0xf/0x20 [12800.268918] [12800.279764] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12800.281517] do_raw_spin_lock+0x177/0x190 [12800.290260] _raw_spin_lock+0x21/0x30 [12800.294832] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12800.300370] ? do_raw_spin_unlock+0x75/0x190 [12800.303997] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12800.306851] ? do_wait_intr+0xf0/0xf0 [12800.309032] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12800.311674] mdc_close+0x53b/0xd70 [mdc] [12800.313338] lmv_close+0x18d/0x490 [lmv] [12800.315035] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12800.319405] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12800.329866] ll_md_real_close+0xa5/0x2a0 [lustre] [12800.332319] ll_md_close+0x274/0x800 [lustre] [12800.333860] ll_file_release+0x128/0xc50 [lustre] [12800.342423] __fput+0x10d/0x440 [12800.348577] ____fput+0x12/0x20 [12800.351086] task_work_run+0xc6/0x110 [12800.353650] exit_to_usermode_loop+0x1dd/0x1f0 [12800.354686] do_syscall_64+0x3d6/0x440 [12800.355887] entry_SYSCALL_64_after_hwframe+0x49/0xae [12800.362676] RIP: 0033:0x7f151bee3cd8 [12800.363683] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12800.380603] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12800.385546] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [12800.388701] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [12800.393841] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12800.405242] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12800.414475] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12804.058020] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12804.063648] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12804.132964] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12804.149654] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12804.156337] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12804.168781] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12804.175246] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12804.218285] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 [12804.221324] RAX: 0000000000006500 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12804.224701] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [12804.228831] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [12804.232296] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [12804.235502] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [12804.239382] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [12804.243374] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12804.246669] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12804.249631] Call Trace: [12804.251830] [12804.252562] ? show_regs.cold.9+0x22/0x2f [12804.254491] ? watchdog_timer_fn.cold.6+0x62/0x10a [12804.256436] ? __hrtimer_run_queues+0x1c2/0x4d0 [12804.258193] ? watchdog+0x40/0x40 [12804.259365] ? hrtimer_interrupt+0x128/0x2c0 [12804.260986] ? smp_apic_timer_interrupt+0xa6/0x270 [12804.262571] ? apic_timer_interrupt+0xf/0x20 [12804.264678] [12804.265620] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12804.268725] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12804.271395] do_raw_spin_lock+0x177/0x190 [12804.273216] _raw_spin_lock+0x21/0x30 [12804.274820] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12804.277576] process_one_work+0x2c8/0x700 [12804.279660] worker_thread+0x296/0x6e0 [12804.281523] ? rescuer_thread+0x570/0x570 [12804.283610] kthread+0x1d1/0x200 [12804.284627] ? set_kthread_struct+0x70/0x70 [12804.287522] ret_from_fork+0x1f/0x30 [12807.142843] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10876s! [12807.154227] Showing busy workqueues and worker pools: [12807.163578] workqueue events: flags=0x0 [12807.169801] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12807.174940] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [12807.180458] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12807.189156] pending: kfree_rcu_monitor [12807.193270] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12807.201282] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [12807.212293] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12807.224573] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [12807.238024] workqueue events_long: flags=0x0 [12807.243352] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12807.249314] pending: lu_objects_flush [obdclass] [12807.255372] workqueue events_unbound: flags=0x2 [12807.264299] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12807.276214] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12807.290301] workqueue events_power_efficient: flags=0x80 [12807.293727] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12807.302518] pending: do_cache_clean, neigh_periodic_work, check_lifetime [12807.311106] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12807.317271] pending: neigh_periodic_work [12807.320398] workqueue mm_percpu_wq: flags=0x8 [12807.327258] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12807.335684] pending: vmstat_update, lru_add_drain_per_cpu [12807.340633] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12807.347746] pending: vmstat_update [12807.357429] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12807.363551] pending: lru_add_drain_per_cpu [12807.369150] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [12807.383933] pending: lru_add_drain_per_cpu BAR(48) [12807.394228] workqueue writeback: flags=0x4a [12807.396312] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [12807.404681] pending: wb_workfn [12807.408704] workqueue xprtiod: flags=0xa [12807.413422] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12807.420574] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [12807.436108] workqueue ipv6_addrconf: flags=0x40008 [12807.440968] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12807.448557] pending: addrconf_verify_work [12807.451234] workqueue knbd0-recv: flags=0x1a [12807.453661] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12807.457299] in-flight: 658:recv_work [12807.463158] workqueue ptlrpc_pinger: flags=0x2 [12807.465260] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12807.473496] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12807.483243] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10876s workers=4 idle: 32984 37 8 [12807.487486] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12792s workers=2 idle: 661 [12812.057801] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [12812.067520] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12812.122207] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12812.133795] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12812.143918] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12812.154665] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12812.168514] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12812.173433] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [12812.177739] RDX: 0000000000005f2f RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [12812.182005] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [12812.186455] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [12812.190450] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12812.195478] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [12812.203908] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12812.208692] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12812.212958] Call Trace: [12812.214374] [12812.216130] ? show_regs.cold.9+0x22/0x2f [12812.218623] ? watchdog_timer_fn.cold.6+0x62/0x10a [12812.221403] ? __hrtimer_run_queues+0x1c2/0x4d0 [12812.226696] ? watchdog+0x40/0x40 [12812.230245] ? hrtimer_interrupt+0x128/0x2c0 [12812.239231] ? smp_apic_timer_interrupt+0xa6/0x270 [12812.247916] ? apic_timer_interrupt+0xf/0x20 [12812.254099] [12812.255973] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12812.262746] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12812.268456] do_raw_spin_lock+0x177/0x190 [12812.274464] _raw_spin_lock+0x21/0x30 [12812.275940] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12812.285646] obd_mod_free+0x46/0x140 [ptlrpc] [12812.293051] mdc_commit_open+0x91/0xb0 [mdc] [12812.297900] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12812.305756] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12812.314693] after_reply+0xe2c/0x1780 [ptlrpc] [12812.318976] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12812.335349] ? __next_timer_interrupt+0x160/0x160 [12812.339293] ptlrpcd+0xa82/0xe40 [ptlrpc] [12812.344659] ? do_wait_intr+0xf0/0xf0 [12812.351145] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12812.359439] kthread+0x1d1/0x200 [12812.363592] ? set_kthread_struct+0x70/0x70 [12812.369104] ret_from_fork+0x1f/0x30 [12828.058098] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [12828.063057] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12828.122954] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12828.138429] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12828.150593] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12828.153295] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12828.167251] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12828.169748] RAX: 0000000000006906 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12828.172875] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [12828.175537] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [12828.183727] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [12828.186461] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [12828.193734] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [12828.197958] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12828.201318] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [12828.207805] Call Trace: [12828.211304] [12828.216075] ? show_regs.cold.9+0x22/0x2f [12828.220040] ? watchdog_timer_fn.cold.6+0x62/0x10a [12828.224606] ? __hrtimer_run_queues+0x1c2/0x4d0 [12828.229233] ? watchdog+0x40/0x40 [12828.235519] ? hrtimer_interrupt+0x128/0x2c0 [12828.242652] ? smp_apic_timer_interrupt+0xa6/0x270 [12828.245973] ? apic_timer_interrupt+0xf/0x20 [12828.250378] [12828.253430] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12828.258491] do_raw_spin_lock+0x177/0x190 [12828.261593] _raw_spin_lock+0x21/0x30 [12828.262817] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12828.268259] ? do_raw_spin_unlock+0x75/0x190 [12828.271272] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12828.278505] ? do_wait_intr+0xf0/0xf0 [12828.280118] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12828.284558] mdc_close+0x53b/0xd70 [mdc] [12828.286251] lmv_close+0x18d/0x490 [lmv] [12828.290249] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12828.298284] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12828.301445] ll_md_real_close+0xa5/0x2a0 [lustre] [12828.307500] ll_md_close+0x274/0x800 [lustre] [12828.312880] ll_file_release+0x128/0xc50 [lustre] [12828.317468] __fput+0x10d/0x440 [12828.319208] ____fput+0x12/0x20 [12828.321313] task_work_run+0xc6/0x110 [12828.323192] exit_to_usermode_loop+0x1dd/0x1f0 [12828.328170] do_syscall_64+0x3d6/0x440 [12828.331787] entry_SYSCALL_64_after_hwframe+0x49/0xae [12828.335499] RIP: 0033:0x7f151bee3cd8 [12828.342142] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12828.368236] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12828.372777] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [12828.378950] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [12828.393512] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12828.404233] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12828.414748] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12832.052028] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [12832.056596] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12832.063293] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) [12832.068648] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12832.068748] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12832.068755] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12832.068761] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12832.069090] RIP: 0010:native_safe_halt+0x2e/0x50 [12832.069100] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12832.069105] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12832.069109] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12832.069112] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [12832.069114] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [12832.069116] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [12832.069118] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [12832.069122] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [12832.069124] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12832.069127] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12832.069139] Call Trace: [12832.069144] [12832.069150] ? show_regs.cold.9+0x22/0x2f [12832.069161] ? watchdog_timer_fn.cold.6+0x62/0x10a [12832.069168] ? __hrtimer_run_queues+0x1c2/0x4d0 [12832.069179] ? watchdog+0x40/0x40 [12832.069184] ? hrtimer_interrupt+0x128/0x2c0 [12832.069188] ? smp_apic_timer_interrupt+0xa6/0x270 [12832.069195] ? apic_timer_interrupt+0xf/0x20 [12832.069200] [12832.069207] ? native_safe_halt+0x2e/0x50 [12832.069211] kvm_wait+0xcd/0xe0 [12832.069218] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12832.069226] do_raw_spin_lock+0x177/0x190 [12832.069231] _raw_spin_lock+0x21/0x30 [12832.069235] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12832.069733] process_one_work+0x2c8/0x700 [12832.069740] worker_thread+0x296/0x6e0 [12832.069745] ? rescuer_thread+0x570/0x570 [12832.069749] kthread+0x1d1/0x200 [12832.069753] ? set_kthread_struct+0x70/0x70 [12832.069756] ret_from_fork+0x1f/0x30 [12832.223870] obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12832.241108] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12832.246767] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12832.250786] RIP: 0010:native_safe_halt+0x2e/0x50 [12832.252923] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12832.261652] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12832.264857] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12832.269287] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [12832.272727] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [12832.275839] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [12832.278960] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [12832.282138] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [12832.285849] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12832.287911] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [12832.291866] Call Trace: [12832.293258] [12832.294651] ? show_regs.cold.9+0x22/0x2f [12832.296511] ? watchdog_timer_fn.cold.6+0x62/0x10a [12832.298672] ? __hrtimer_run_queues+0x1c2/0x4d0 [12832.300717] ? watchdog+0x40/0x40 [12832.302207] ? hrtimer_interrupt+0x128/0x2c0 [12832.304197] ? smp_apic_timer_interrupt+0xa6/0x270 [12832.306689] ? apic_timer_interrupt+0xf/0x20 [12832.308633] [12832.309518] ? native_safe_halt+0x2e/0x50 [12832.311279] kvm_wait+0xcd/0xe0 [12832.312777] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12832.315143] do_raw_spin_lock+0x177/0x190 [12832.316995] _raw_spin_lock+0x21/0x30 [12832.319282] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12832.322252] ? do_raw_spin_unlock+0x75/0x190 [12832.324476] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12832.330279] ? do_wait_intr+0xf0/0xf0 [12832.333786] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12832.338128] mdc_reint+0x2d/0x110 [mdc] [12832.340934] mdc_unlink+0x658/0x870 [mdc] [12832.344802] lmv_unlink+0x34d/0xbd0 [lmv] [12832.347376] ll_unlink+0x24d/0x7f0 [lustre] [12832.352859] vfs_unlink+0x202/0x330 [12832.354884] do_unlinkat+0x2fa/0x460 [12832.357539] __x64_sys_unlink+0x37/0x50 [12832.360845] do_syscall_64+0xc1/0x440 [12832.363560] entry_SYSCALL_64_after_hwframe+0x49/0xae [12832.366957] RIP: 0033:0x7f151bdee10b [12832.369843] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [12832.379493] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [12832.384008] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [12832.387557] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [12832.395415] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12832.401437] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12832.405472] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12837.857394] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10907s! [12837.861663] Showing busy workqueues and worker pools: [12837.864075] workqueue events: flags=0x0 [12837.868324] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12837.872496] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [12837.878455] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12837.885671] pending: kfree_rcu_monitor [12837.889213] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12837.894036] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [12837.906177] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12837.910491] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [12837.914845] workqueue events_long: flags=0x0 [12837.917307] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12837.920484] pending: lu_objects_flush [obdclass] [12837.923191] workqueue events_unbound: flags=0x2 [12837.925565] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12837.928539] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12837.934113] workqueue events_power_efficient: flags=0x80 [12837.942598] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12837.945252] pending: do_cache_clean, neigh_periodic_work, check_lifetime [12837.948675] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12837.952729] pending: neigh_periodic_work [12837.954904] workqueue mm_percpu_wq: flags=0x8 [12837.957338] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12837.961844] pending: vmstat_update, lru_add_drain_per_cpu [12837.964577] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12837.967400] pending: vmstat_update [12837.969286] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12837.972234] pending: lru_add_drain_per_cpu [12837.974240] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [12837.978229] pending: lru_add_drain_per_cpu BAR(48) [12837.981342] workqueue writeback: flags=0x4a [12837.983293] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [12837.986039] pending: wb_workfn [12837.987613] workqueue xprtiod: flags=0xa [12837.989524] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12837.992340] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [12837.995946] workqueue ipv6_addrconf: flags=0x40008 [12837.997766] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12838.001136] pending: addrconf_verify_work [12838.002923] workqueue knbd0-recv: flags=0x1a [12838.004324] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12838.006812] in-flight: 658:recv_work [12838.008291] workqueue ptlrpc_pinger: flags=0x2 [12838.009667] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12838.012169] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12838.014855] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10907s workers=4 idle: 32984 37 8 [12838.018059] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12822s workers=2 idle: 661 [12840.056379] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [12840.061105] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12840.097985] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12840.108730] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12840.112734] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12840.119225] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12840.131605] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12840.139164] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [12840.144163] RDX: 00000000000022f4 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [12840.149020] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [12840.154501] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [12840.159484] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12840.164615] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [12840.172799] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12840.176175] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12840.184727] Call Trace: [12840.186066] [12840.187106] ? show_regs.cold.9+0x22/0x2f [12840.188926] ? watchdog_timer_fn.cold.6+0x62/0x10a [12840.195776] ? __hrtimer_run_queues+0x1c2/0x4d0 [12840.197814] ? watchdog+0x40/0x40 [12840.201916] ? hrtimer_interrupt+0x128/0x2c0 [12840.206683] ? smp_apic_timer_interrupt+0xa6/0x270 [12840.211737] ? apic_timer_interrupt+0xf/0x20 [12840.213241] [12840.217421] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12840.219384] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12840.224214] do_raw_spin_lock+0x177/0x190 [12840.226198] _raw_spin_lock+0x21/0x30 [12840.229799] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12840.233535] obd_mod_free+0x46/0x140 [ptlrpc] [12840.235720] mdc_commit_open+0x91/0xb0 [mdc] [12840.237219] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12840.244873] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12840.256945] after_reply+0xe2c/0x1780 [ptlrpc] [12840.259957] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12840.264786] ? __next_timer_interrupt+0x160/0x160 [12840.269108] ptlrpcd+0xa82/0xe40 [ptlrpc] [12840.271485] ? do_wait_intr+0xf0/0xf0 [12840.274947] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12840.278284] kthread+0x1d1/0x200 [12840.280236] ? set_kthread_struct+0x70/0x70 [12840.284117] ret_from_fork+0x1f/0x30 [12856.058055] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [12856.065125] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12856.115772] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12856.127842] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12856.133683] RIP: 0010:native_safe_halt+0x2e/0x50 [12856.136634] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12856.167993] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12856.174110] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12856.182679] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [12856.193692] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [12856.201568] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [12856.205624] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [12856.208980] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [12856.212483] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12856.215226] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [12856.218192] Call Trace: [12856.219226] [12856.220299] ? show_regs.cold.9+0x22/0x2f [12856.222257] ? watchdog_timer_fn.cold.6+0x62/0x10a [12856.224686] ? __hrtimer_run_queues+0x1c2/0x4d0 [12856.227029] ? watchdog+0x40/0x40 [12856.228516] ? hrtimer_interrupt+0x128/0x2c0 [12856.230965] ? smp_apic_timer_interrupt+0xa6/0x270 [12856.234648] ? apic_timer_interrupt+0xf/0x20 [12856.239656] [12856.240648] ? native_safe_halt+0x2e/0x50 [12856.243502] kvm_wait+0xcd/0xe0 [12856.245654] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12856.250171] do_raw_spin_lock+0x177/0x190 [12856.255287] _raw_spin_lock+0x21/0x30 [12856.257282] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12856.260128] ? do_raw_spin_unlock+0x75/0x190 [12856.262404] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12856.265178] ? do_wait_intr+0xf0/0xf0 [12856.266575] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12856.269215] mdc_close+0x53b/0xd70 [mdc] [12856.271098] lmv_close+0x18d/0x490 [lmv] [12856.273570] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12856.276672] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12856.280013] ll_md_real_close+0xa5/0x2a0 [lustre] [12856.287311] ll_md_close+0x274/0x800 [lustre] [12856.289809] ll_file_release+0x128/0xc50 [lustre] [12856.295805] __fput+0x10d/0x440 [12856.297622] ____fput+0x12/0x20 [12856.302206] task_work_run+0xc6/0x110 [12856.303498] exit_to_usermode_loop+0x1dd/0x1f0 [12856.305217] do_syscall_64+0x3d6/0x440 [12856.309695] entry_SYSCALL_64_after_hwframe+0x49/0xae [12856.313389] RIP: 0033:0x7f151bee3cd8 [12856.314356] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12856.323324] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12856.328171] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [12856.333726] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [12856.338115] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12856.342308] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12856.350248] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12860.056128] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12860.059704] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12860.075641] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12860.081642] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12860.087188] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12860.091833] RIP: 0010:native_safe_halt+0x2e/0x50 [12860.094460] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12860.104866] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12860.108154] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12860.111649] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [12860.114575] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [12860.117042] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [12860.119721] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [12860.122542] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [12860.125768] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12860.128003] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12860.131390] Call Trace: [12860.132959] [12860.133784] ? show_regs.cold.9+0x22/0x2f [12860.135648] ? watchdog_timer_fn.cold.6+0x62/0x10a [12860.137999] ? __hrtimer_run_queues+0x1c2/0x4d0 [12860.140414] ? watchdog+0x40/0x40 [12860.141856] ? hrtimer_interrupt+0x128/0x2c0 [12860.143698] ? smp_apic_timer_interrupt+0xa6/0x270 [12860.145110] ? apic_timer_interrupt+0xf/0x20 [12860.146540] [12860.147287] ? native_safe_halt+0x2e/0x50 [12860.148971] kvm_wait+0xcd/0xe0 [12860.150460] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12860.152764] do_raw_spin_lock+0x177/0x190 [12860.155047] _raw_spin_lock+0x21/0x30 [12860.156686] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12860.159173] process_one_work+0x2c8/0x700 [12860.161017] worker_thread+0x296/0x6e0 [12860.162637] ? rescuer_thread+0x570/0x570 [12860.164537] kthread+0x1d1/0x200 [12860.165794] ? set_kthread_struct+0x70/0x70 [12860.167508] ret_from_fork+0x1f/0x30 [12864.052315] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [12864.058563] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12864.084120] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12864.089639] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12864.094894] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12864.098275] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12864.110111] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [12864.114270] RAX: 0000000000006065 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12864.117502] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [12864.120972] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [12864.124361] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [12864.128587] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [12864.133057] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [12864.137247] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12864.139619] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [12864.144525] Call Trace: [12864.146319] [12864.147233] ? show_regs.cold.9+0x22/0x2f [12864.149661] ? watchdog_timer_fn.cold.6+0x62/0x10a [12864.152316] ? __hrtimer_run_queues+0x1c2/0x4d0 [12864.155138] ? watchdog+0x40/0x40 [12864.156526] ? hrtimer_interrupt+0x128/0x2c0 [12864.159129] ? smp_apic_timer_interrupt+0xa6/0x270 [12864.160989] ? apic_timer_interrupt+0xf/0x20 [12864.163061] [12864.164007] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12864.166448] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12864.169015] do_raw_spin_lock+0x177/0x190 [12864.170871] _raw_spin_lock+0x21/0x30 [12864.172993] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12864.175095] ? do_raw_spin_unlock+0x75/0x190 [12864.176941] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12864.179668] ? do_wait_intr+0xf0/0xf0 [12864.181223] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12864.184435] mdc_reint+0x2d/0x110 [mdc] [12864.186885] mdc_unlink+0x658/0x870 [mdc] [12864.189148] lmv_unlink+0x34d/0xbd0 [lmv] [12864.192051] ll_unlink+0x24d/0x7f0 [lustre] [12864.194142] vfs_unlink+0x202/0x330 [12864.195618] do_unlinkat+0x2fa/0x460 [12864.197013] __x64_sys_unlink+0x37/0x50 [12864.198251] do_syscall_64+0xc1/0x440 [12864.199769] entry_SYSCALL_64_after_hwframe+0x49/0xae [12864.201746] RIP: 0033:0x7f151bdee10b [12864.203093] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [12864.210352] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [12864.213764] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [12864.216746] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [12864.219307] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12864.222165] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12864.225052] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12868.055025] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [12868.058482] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12868.086912] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12868.098241] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12868.108220] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12868.118325] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12868.132806] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12868.140651] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [12868.144679] RDX: 00000000000058db RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [12868.147696] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [12868.154706] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [12868.170512] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12868.184371] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [12868.191599] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12868.195692] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12868.203073] Call Trace: [12868.204500] [12868.205396] ? show_regs.cold.9+0x22/0x2f [12868.207489] ? watchdog_timer_fn.cold.6+0x62/0x10a [12868.211992] ? __hrtimer_run_queues+0x1c2/0x4d0 [12868.215839] ? watchdog+0x40/0x40 [12868.221881] ? hrtimer_interrupt+0x128/0x2c0 [12868.224632] ? smp_apic_timer_interrupt+0xa6/0x270 [12868.230086] ? apic_timer_interrupt+0xf/0x20 [12868.232828] [12868.234958] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12868.237830] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12868.240465] do_raw_spin_lock+0x177/0x190 [12868.244064] _raw_spin_lock+0x21/0x30 [12868.247748] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12868.249957] obd_mod_free+0x46/0x140 [ptlrpc] [12868.252864] mdc_commit_open+0x91/0xb0 [mdc] [12868.254996] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12868.257485] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12868.259325] after_reply+0xe2c/0x1780 [ptlrpc] [12868.261227] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12868.263700] ? __next_timer_interrupt+0x160/0x160 [12868.265272] ptlrpcd+0xa82/0xe40 [ptlrpc] [12868.268584] ? do_wait_intr+0xf0/0xf0 [12868.273208] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12868.275321] kthread+0x1d1/0x200 [12868.284121] ? set_kthread_struct+0x70/0x70 [12868.287034] ret_from_fork+0x1f/0x30 [12868.289704] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10937s! [12868.298795] Showing busy workqueues and worker pools: [12868.301038] workqueue events: flags=0x0 [12868.307431] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12868.315149] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [12868.323878] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12868.331053] pending: kfree_rcu_monitor [12868.340719] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12868.354828] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [12868.363483] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12868.369743] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [12868.382377] workqueue events_long: flags=0x0 [12868.384847] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12868.392103] pending: lu_objects_flush [obdclass] [12868.398791] workqueue events_unbound: flags=0x2 [12868.403732] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12868.407797] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12868.419570] workqueue events_power_efficient: flags=0x80 [12868.425031] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12868.432060] pending: do_cache_clean, neigh_periodic_work, check_lifetime [12868.439526] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12868.447373] pending: neigh_periodic_work [12868.449274] workqueue mm_percpu_wq: flags=0x8 [12868.456755] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12868.464242] pending: vmstat_update, lru_add_drain_per_cpu [12868.468300] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12868.474075] pending: vmstat_update [12868.476050] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12868.485901] pending: lru_add_drain_per_cpu [12868.489835] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [12868.493922] pending: lru_add_drain_per_cpu BAR(48) [12868.499116] workqueue writeback: flags=0x4a [12868.502186] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [12868.505099] pending: wb_workfn [12868.508490] workqueue xprtiod: flags=0xa [12868.510662] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12868.514891] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [12868.523188] workqueue ipv6_addrconf: flags=0x40008 [12868.530190] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12868.539943] pending: addrconf_verify_work [12868.542920] workqueue knbd0-recv: flags=0x1a [12868.548276] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12868.552063] in-flight: 658:recv_work [12868.556929] workqueue ptlrpc_pinger: flags=0x2 [12868.562341] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12868.565002] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12868.573422] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10937s workers=4 idle: 32984 37 8 [12868.580939] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12853s workers=2 idle: 661 [12884.059436] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [12884.067754] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12884.132892] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12884.146047] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12884.163025] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12884.171541] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12884.194608] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12884.204956] RAX: 0000000000006719 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12884.217488] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [12884.232821] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [12884.244951] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [12884.259338] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [12884.274537] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [12884.290257] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12884.297523] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [12884.302574] Call Trace: [12884.303988] [12884.306362] ? show_regs.cold.9+0x22/0x2f [12884.316847] ? watchdog_timer_fn.cold.6+0x62/0x10a [12884.320487] ? __hrtimer_run_queues+0x1c2/0x4d0 [12884.328742] ? watchdog+0x40/0x40 [12884.341490] ? hrtimer_interrupt+0x128/0x2c0 [12884.344479] ? smp_apic_timer_interrupt+0xa6/0x270 [12884.348459] ? apic_timer_interrupt+0xf/0x20 [12884.350145] [12884.356997] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12884.364720] do_raw_spin_lock+0x177/0x190 [12884.368698] _raw_spin_lock+0x21/0x30 [12884.372761] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12884.392044] ? do_raw_spin_unlock+0x75/0x190 [12884.399314] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12884.407351] ? do_wait_intr+0xf0/0xf0 [12884.408916] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12884.422063] mdc_close+0x53b/0xd70 [mdc] [12884.425007] lmv_close+0x18d/0x490 [lmv] [12884.440817] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12884.449674] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12884.458327] ll_md_real_close+0xa5/0x2a0 [lustre] [12884.466070] ll_md_close+0x274/0x800 [lustre] [12884.472799] ll_file_release+0x128/0xc50 [lustre] [12884.480539] __fput+0x10d/0x440 [12884.482894] ____fput+0x12/0x20 [12884.490976] task_work_run+0xc6/0x110 [12884.496787] exit_to_usermode_loop+0x1dd/0x1f0 [12884.505940] do_syscall_64+0x3d6/0x440 [12884.510744] entry_SYSCALL_64_after_hwframe+0x49/0xae [12884.519732] RIP: 0033:0x7f151bee3cd8 [12884.522027] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12884.547889] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12884.556910] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [12884.563804] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [12884.570103] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12884.572654] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12884.591997] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12888.058048] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [12888.069325] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12888.122660] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12888.128517] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12888.135621] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12888.138938] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12888.141932] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12888.166894] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12888.174808] RAX: 0000000000002c04 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12888.181705] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [12888.186570] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [12888.190893] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [12888.201174] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [12888.204702] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [12888.212908] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12888.220768] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12888.229281] Call Trace: [12888.230913] [12888.232850] ? show_regs.cold.9+0x22/0x2f [12888.235183] ? watchdog_timer_fn.cold.6+0x62/0x10a [12888.238589] ? __hrtimer_run_queues+0x1c2/0x4d0 [12888.241045] ? watchdog+0x40/0x40 [12888.242184] ? hrtimer_interrupt+0x128/0x2c0 [12888.244529] ? smp_apic_timer_interrupt+0xa6/0x270 [12888.249460] ? apic_timer_interrupt+0xf/0x20 [12888.256537] [12888.258849] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12888.265673] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12888.270760] do_raw_spin_lock+0x177/0x190 [12888.273960] _raw_spin_lock+0x21/0x30 [12888.276035] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12888.280866] process_one_work+0x2c8/0x700 [12888.287324] worker_thread+0x296/0x6e0 [12888.290496] ? rescuer_thread+0x570/0x570 [12888.298168] kthread+0x1d1/0x200 [12888.305969] ? set_kthread_struct+0x70/0x70 [12888.312759] ret_from_fork+0x1f/0x30 [12896.052758] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [12896.055792] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [12896.059056] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12896.059152] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12896.059161] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12896.059163] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12896.059177] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12896.059181] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12896.059186] RAX: 0000000000006fe6 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12896.059189] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [12896.059192] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [12896.059194] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [12896.059197] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [12896.059201] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [12896.059204] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12896.059207] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [12896.059216] Call Trace: [12896.059221] [12896.059227] ? show_regs.cold.9+0x22/0x2f [12896.059236] ? watchdog_timer_fn.cold.6+0x62/0x10a [12896.059243] ? __hrtimer_run_queues+0x1c2/0x4d0 [12896.059247] ? watchdog+0x40/0x40 [12896.059251] ? hrtimer_interrupt+0x128/0x2c0 [12896.059256] ? smp_apic_timer_interrupt+0xa6/0x270 [12896.059265] ? apic_timer_interrupt+0xf/0x20 [12896.059271] [12896.059274] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12896.059279] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12896.059284] do_raw_spin_lock+0x177/0x190 [12896.059289] _raw_spin_lock+0x21/0x30 [12896.059295] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12896.059572] ? do_raw_spin_unlock+0x75/0x190 [12896.059578] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12896.059791] ? do_wait_intr+0xf0/0xf0 [12896.059798] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12896.060009] mdc_reint+0x2d/0x110 [mdc] [12896.060047] mdc_unlink+0x658/0x870 [mdc] [12896.060076] lmv_unlink+0x34d/0xbd0 [lmv] [12896.060129] ll_unlink+0x24d/0x7f0 [lustre] [12896.060253] vfs_unlink+0x202/0x330 [12896.060261] do_unlinkat+0x2fa/0x460 [12896.060266] __x64_sys_unlink+0x37/0x50 [12896.060270] do_syscall_64+0xc1/0x440 [12896.060277] entry_SYSCALL_64_after_hwframe+0x49/0xae [12896.060284] RIP: 0033:0x7f151bdee10b [12896.060320] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [12896.060325] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [12896.060330] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [12896.060333] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [12896.060335] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12896.060337] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12896.060340] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12896.366368] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12896.403176] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12896.410385] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12896.418468] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12896.428718] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12896.452853] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12896.460208] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [12896.463835] RDX: 0000000000007d50 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [12896.469444] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [12896.474702] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [12896.479701] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12896.492818] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [12896.504158] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12896.509287] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12896.517741] Call Trace: [12896.520270] [12896.522424] ? show_regs.cold.9+0x22/0x2f [12896.528347] ? watchdog_timer_fn.cold.6+0x62/0x10a [12896.533830] ? __hrtimer_run_queues+0x1c2/0x4d0 [12896.537949] ? watchdog+0x40/0x40 [12896.539481] ? hrtimer_interrupt+0x128/0x2c0 [12896.543743] ? smp_apic_timer_interrupt+0xa6/0x270 [12896.546452] ? apic_timer_interrupt+0xf/0x20 [12896.550743] [12896.552233] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12896.555374] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12896.569559] do_raw_spin_lock+0x177/0x190 [12896.571214] _raw_spin_lock+0x21/0x30 [12896.575797] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12896.583284] obd_mod_free+0x46/0x140 [ptlrpc] [12896.589216] mdc_commit_open+0x91/0xb0 [mdc] [12896.593311] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12896.595305] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12896.600378] after_reply+0xe2c/0x1780 [ptlrpc] [12896.603525] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12896.607167] ? __next_timer_interrupt+0x160/0x160 [12896.610531] ptlrpcd+0xa82/0xe40 [ptlrpc] [12896.613993] ? do_wait_intr+0xf0/0xf0 [12896.616157] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12896.621796] kthread+0x1d1/0x200 [12896.629137] ? set_kthread_struct+0x70/0x70 [12896.632562] ret_from_fork+0x1f/0x30 [12898.787309] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10968s! [12898.793606] Showing busy workqueues and worker pools: [12898.797557] workqueue events: flags=0x0 [12898.801275] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12898.806876] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [12898.812551] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12898.818731] pending: kfree_rcu_monitor [12898.821423] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12898.825334] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [12898.832755] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12898.836118] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [12898.841387] workqueue events_long: flags=0x0 [12898.844767] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12898.848899] pending: lu_objects_flush [obdclass] [12898.851203] workqueue events_unbound: flags=0x2 [12898.853426] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12898.856305] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12898.862169] workqueue events_power_efficient: flags=0x80 [12898.864719] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12898.868089] pending: do_cache_clean, neigh_periodic_work, check_lifetime [12898.871719] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12898.875032] pending: neigh_periodic_work [12898.877865] workqueue mm_percpu_wq: flags=0x8 [12898.880150] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12898.883540] pending: vmstat_update, lru_add_drain_per_cpu [12898.886735] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12898.890536] pending: vmstat_update [12898.892478] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12898.895916] pending: lru_add_drain_per_cpu [12898.898664] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [12898.902447] pending: lru_add_drain_per_cpu BAR(48) [12898.905145] workqueue writeback: flags=0x4a [12898.907077] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [12898.910175] pending: wb_workfn [12898.912129] workqueue xprtiod: flags=0xa [12898.914100] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12898.917257] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [12898.921551] workqueue ipv6_addrconf: flags=0x40008 [12898.923946] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12898.927496] pending: addrconf_verify_work [12898.929495] workqueue knbd0-recv: flags=0x1a [12898.931768] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12898.935301] in-flight: 658:recv_work [12898.937200] workqueue ptlrpc_pinger: flags=0x2 [12898.939309] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12898.942253] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12898.945542] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10968s workers=4 idle: 32984 37 8 [12898.948213] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12883s workers=2 idle: 661 [12912.057224] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [12912.061587] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12912.086233] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12912.092174] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12912.096850] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12912.100223] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12912.109886] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12912.114171] RAX: 00000000000033a7 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12912.118220] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [12912.122477] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [12912.125782] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [12912.129519] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [12912.132863] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [12912.137154] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12912.140172] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [12912.144024] Call Trace: [12912.145263] [12912.146413] ? show_regs.cold.9+0x22/0x2f [12912.148557] ? watchdog_timer_fn.cold.6+0x62/0x10a [12912.151141] ? __hrtimer_run_queues+0x1c2/0x4d0 [12912.153454] ? watchdog+0x40/0x40 [12912.155562] ? hrtimer_interrupt+0x128/0x2c0 [12912.157681] ? smp_apic_timer_interrupt+0xa6/0x270 [12912.160246] ? apic_timer_interrupt+0xf/0x20 [12912.162381] [12912.163493] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12912.168173] do_raw_spin_lock+0x177/0x190 [12912.171113] _raw_spin_lock+0x21/0x30 [12912.173200] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12912.176427] ? do_raw_spin_unlock+0x75/0x190 [12912.178158] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12912.181334] ? do_wait_intr+0xf0/0xf0 [12912.183574] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12912.186603] mdc_close+0x53b/0xd70 [mdc] [12912.190562] lmv_close+0x18d/0x490 [lmv] [12912.193664] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12912.204350] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12912.212604] ll_md_real_close+0xa5/0x2a0 [lustre] [12912.216476] ll_md_close+0x274/0x800 [lustre] [12912.223350] ll_file_release+0x128/0xc50 [lustre] [12912.226474] __fput+0x10d/0x440 [12912.228308] ____fput+0x12/0x20 [12912.230665] task_work_run+0xc6/0x110 [12912.232345] exit_to_usermode_loop+0x1dd/0x1f0 [12912.235277] do_syscall_64+0x3d6/0x440 [12912.237454] entry_SYSCALL_64_after_hwframe+0x49/0xae [12912.240352] RIP: 0033:0x7f151bee3cd8 [12912.242096] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12912.257212] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12912.262454] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [12912.271497] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [12912.280079] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12912.284033] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12912.288015] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12916.056050] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [12916.088516] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12916.166763] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12916.184651] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12916.200547] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12916.214683] RIP: 0010:native_safe_halt+0x2e/0x50 [12916.217525] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12916.240219] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12916.265048] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12916.273221] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [12916.278022] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [12916.285393] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [12916.287767] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [12916.297335] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [12916.304600] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12916.312307] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12916.317049] Call Trace: [12916.319452] [12916.320544] ? show_regs.cold.9+0x22/0x2f [12916.325635] ? watchdog_timer_fn.cold.6+0x62/0x10a [12916.333120] ? __hrtimer_run_queues+0x1c2/0x4d0 [12916.337210] ? watchdog+0x40/0x40 [12916.344387] ? hrtimer_interrupt+0x128/0x2c0 [12916.348523] ? smp_apic_timer_interrupt+0xa6/0x270 [12916.353355] ? apic_timer_interrupt+0xf/0x20 [12916.361024] [12916.364189] ? native_safe_halt+0x2e/0x50 [12916.369684] kvm_wait+0xcd/0xe0 [12916.373699] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12916.377558] do_raw_spin_lock+0x177/0x190 [12916.379090] _raw_spin_lock+0x21/0x30 [12916.380420] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12916.388273] process_one_work+0x2c8/0x700 [12916.392535] worker_thread+0x296/0x6e0 [12916.395408] ? rescuer_thread+0x570/0x570 [12916.397722] kthread+0x1d1/0x200 [12916.398858] ? set_kthread_struct+0x70/0x70 [12916.402620] ret_from_fork+0x1f/0x30 [12924.052988] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [12924.055012] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [12924.063137] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver [12924.076132] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12924.076212] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12924.076220] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12924.076224] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12924.076243] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12924.076247] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12924.076252] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [12924.076259] RDX: 000000000000428b RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [12924.076264] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [12924.076266] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [12924.076269] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12924.076271] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [12924.076274] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12924.076276] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12924.076283] Call Trace: [12924.076293] [12924.076299] ? show_regs.cold.9+0x22/0x2f [12924.076309] ? watchdog_timer_fn.cold.6+0x62/0x10a [12924.076320] ? __hrtimer_run_queues+0x1c2/0x4d0 [12924.076324] ? watchdog+0x40/0x40 [12924.076328] ? hrtimer_interrupt+0x128/0x2c0 [12924.076331] ? smp_apic_timer_interrupt+0xa6/0x270 [12924.076339] ? apic_timer_interrupt+0xf/0x20 [12924.076344] [12924.076347] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12924.076351] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12924.076356] do_raw_spin_lock+0x177/0x190 [12924.076360] _raw_spin_lock+0x21/0x30 [12924.076365] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12924.079494] obd_mod_free+0x46/0x140 [ptlrpc] [12924.080955] mdc_commit_open+0x91/0xb0 [mdc] [12924.080999] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12924.081199] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12924.081400] after_reply+0xe2c/0x1780 [ptlrpc] [12924.081596] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12924.082339] ? __next_timer_interrupt+0x160/0x160 [12924.082352] ptlrpcd+0xa82/0xe40 [ptlrpc] [12924.082563] ? do_wait_intr+0xf0/0xf0 [12924.082573] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12924.082805] kthread+0x1d1/0x200 [12924.082820] ? set_kthread_struct+0x70/0x70 [12924.082824] ret_from_fork+0x1f/0x30 [12924.112225] intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12924.112267] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12924.112275] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12924.112277] RIP: 0010:native_safe_halt+0x2e/0x50 [12924.112293] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12924.112298] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12924.112302] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12924.112304] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [12924.112307] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [12924.112309] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [12924.112311] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [12924.112314] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [12924.112316] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12924.112318] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [12924.112326] Call Trace: [12924.112332] [12924.112338] ? show_regs.cold.9+0x22/0x2f [12924.112349] ? watchdog_timer_fn.cold.6+0x62/0x10a [12924.112355] ? __hrtimer_run_queues+0x1c2/0x4d0 [12924.112359] ? watchdog+0x40/0x40 [12924.112363] ? hrtimer_interrupt+0x128/0x2c0 [12924.112367] ? smp_apic_timer_interrupt+0xa6/0x270 [12924.112374] ? apic_timer_interrupt+0xf/0x20 [12924.112379] [12924.112382] ? native_safe_halt+0x2e/0x50 [12924.112385] kvm_wait+0xcd/0xe0 [12924.112393] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12924.112400] do_raw_spin_lock+0x177/0x190 [12924.112405] _raw_spin_lock+0x21/0x30 [12924.112411] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12924.113200] ? do_raw_spin_unlock+0x75/0x190 [12924.113208] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12924.113424] ? do_wait_intr+0xf0/0xf0 [12924.113431] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12924.704396] mdc_reint+0x2d/0x110 [mdc] [12924.710273] mdc_unlink+0x658/0x870 [mdc] [12924.714148] lmv_unlink+0x34d/0xbd0 [lmv] [12924.719531] ll_unlink+0x24d/0x7f0 [lustre] [12924.726725] vfs_unlink+0x202/0x330 [12924.727876] do_unlinkat+0x2fa/0x460 [12924.737283] __x64_sys_unlink+0x37/0x50 [12924.744785] do_syscall_64+0xc1/0x440 [12924.749857] entry_SYSCALL_64_after_hwframe+0x49/0xae [12924.757724] RIP: 0033:0x7f151bdee10b [12924.762567] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [12924.801889] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [12924.810554] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [12924.813835] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [12924.822496] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12924.826639] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12924.832089] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12928.994471] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10998s! [12929.001377] Showing busy workqueues and worker pools: [12929.008369] workqueue events: flags=0x0 [12929.010390] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12929.016740] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [12929.029731] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12929.037280] pending: kfree_rcu_monitor [12929.041089] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12929.047692] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [12929.057540] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12929.062521] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [12929.076362] workqueue events_long: flags=0x0 [12929.088496] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12929.090941] pending: lu_objects_flush [obdclass] [12929.100328] workqueue events_unbound: flags=0x2 [12929.105603] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12929.107987] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12929.135441] workqueue events_power_efficient: flags=0x80 [12929.137510] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12929.149409] pending: do_cache_clean, neigh_periodic_work, check_lifetime [12929.159608] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12929.166629] pending: neigh_periodic_work [12929.171337] workqueue mm_percpu_wq: flags=0x8 [12929.173509] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12929.181915] pending: vmstat_update, lru_add_drain_per_cpu [12929.186908] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12929.192579] pending: vmstat_update [12929.194060] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12929.207169] pending: lru_add_drain_per_cpu [12929.208600] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [12929.213549] pending: lru_add_drain_per_cpu BAR(48) [12929.216513] workqueue writeback: flags=0x4a [12929.218085] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [12929.232441] pending: wb_workfn [12929.233695] workqueue xprtiod: flags=0xa [12929.247523] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12929.252733] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [12929.269889] workqueue ipv6_addrconf: flags=0x40008 [12929.273227] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12929.282930] pending: addrconf_verify_work [12929.288146] workqueue knbd0-recv: flags=0x1a [12929.297894] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12929.311437] in-flight: 658:recv_work [12929.322913] workqueue ptlrpc_pinger: flags=0x2 [12929.324657] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12929.334962] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12929.346872] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10998s workers=4 idle: 32984 37 8 [12929.358382] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12914s workers=2 idle: 661 [12940.057022] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [12940.060277] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12940.078188] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12940.080730] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12940.082722] RIP: 0010:native_safe_halt+0x2e/0x50 [12940.083802] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12940.089718] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12940.092016] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12940.094209] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [12940.096001] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [12940.098010] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [12940.101865] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [12940.104260] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [12940.106182] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12940.109346] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [12940.112480] Call Trace: [12940.113928] [12940.118963] ? show_regs.cold.9+0x22/0x2f [12940.124379] ? watchdog_timer_fn.cold.6+0x62/0x10a [12940.126275] ? __hrtimer_run_queues+0x1c2/0x4d0 [12940.127389] ? watchdog+0x40/0x40 [12940.128541] ? hrtimer_interrupt+0x128/0x2c0 [12940.130224] ? smp_apic_timer_interrupt+0xa6/0x270 [12940.131432] ? apic_timer_interrupt+0xf/0x20 [12940.132696] [12940.133322] ? native_safe_halt+0x2e/0x50 [12940.134216] kvm_wait+0xcd/0xe0 [12940.135023] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12940.136254] do_raw_spin_lock+0x177/0x190 [12940.137202] _raw_spin_lock+0x21/0x30 [12940.138298] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12940.140234] ? do_raw_spin_unlock+0x75/0x190 [12940.141225] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12940.143157] ? do_wait_intr+0xf0/0xf0 [12940.144366] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12940.145813] mdc_close+0x53b/0xd70 [mdc] [12940.147341] lmv_close+0x18d/0x490 [lmv] [12940.148555] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12940.153166] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12940.154547] ll_md_real_close+0xa5/0x2a0 [lustre] [12940.155863] ll_md_close+0x274/0x800 [lustre] [12940.156980] ll_file_release+0x128/0xc50 [lustre] [12940.158248] __fput+0x10d/0x440 [12940.159027] ____fput+0x12/0x20 [12940.159984] task_work_run+0xc6/0x110 [12940.160902] exit_to_usermode_loop+0x1dd/0x1f0 [12940.161971] do_syscall_64+0x3d6/0x440 [12940.162934] entry_SYSCALL_64_after_hwframe+0x49/0xae [12940.164174] RIP: 0033:0x7f151bee3cd8 [12940.165125] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12940.169351] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12940.171224] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [12940.172954] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [12940.174622] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12940.176273] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12940.178080] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12944.056404] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [12944.064400] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12944.111975] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12944.119546] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12944.136668] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12944.140244] RIP: 0010:native_safe_halt+0x2e/0x50 [12944.145076] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12944.165374] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12944.171445] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12944.179800] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [12944.185446] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [12944.193975] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [12944.202799] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [12944.208270] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [12944.213579] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12944.216561] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12944.222036] Call Trace: [12944.225716] [12944.234130] ? show_regs.cold.9+0x22/0x2f [12944.238151] ? watchdog_timer_fn.cold.6+0x62/0x10a [12944.245406] ? __hrtimer_run_queues+0x1c2/0x4d0 [12944.248821] ? watchdog+0x40/0x40 [12944.252290] ? hrtimer_interrupt+0x128/0x2c0 [12944.256314] ? smp_apic_timer_interrupt+0xa6/0x270 [12944.260704] ? apic_timer_interrupt+0xf/0x20 [12944.263789] [12944.266587] ? native_safe_halt+0x2e/0x50 [12944.268674] kvm_wait+0xcd/0xe0 [12944.270406] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12944.276571] do_raw_spin_lock+0x177/0x190 [12944.279308] _raw_spin_lock+0x21/0x30 [12944.284306] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12944.290857] process_one_work+0x2c8/0x700 [12944.294418] worker_thread+0x296/0x6e0 [12944.296818] ? rescuer_thread+0x570/0x570 [12944.298283] kthread+0x1d1/0x200 [12944.306088] ? set_kthread_struct+0x70/0x70 [12944.308332] ret_from_fork+0x1f/0x30 [12952.055180] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [12952.059856] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12952.104607] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12952.114871] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12952.122592] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12952.133238] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12952.148687] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12952.153144] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [12952.162331] RDX: 00000000000070d8 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [12952.165734] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [12952.169738] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [12952.173307] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12952.176237] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [12952.188587] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12952.193808] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12952.197637] Call Trace: [12952.199202] [12952.200625] ? show_regs.cold.9+0x22/0x2f [12952.201747] ? watchdog_timer_fn.cold.6+0x62/0x10a [12952.203750] ? __hrtimer_run_queues+0x1c2/0x4d0 [12952.205352] ? watchdog+0x40/0x40 [12952.207362] ? hrtimer_interrupt+0x128/0x2c0 [12952.208625] ? smp_apic_timer_interrupt+0xa6/0x270 [12952.210128] ? apic_timer_interrupt+0xf/0x20 [12952.212411] [12952.213156] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12952.215508] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12952.219509] do_raw_spin_lock+0x177/0x190 [12952.221128] _raw_spin_lock+0x21/0x30 [12952.222785] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12952.224797] obd_mod_free+0x46/0x140 [ptlrpc] [12952.226375] mdc_commit_open+0x91/0xb0 [mdc] [12952.227715] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12952.229375] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12952.232951] after_reply+0xe2c/0x1780 [ptlrpc] [12952.235794] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12952.238920] ? __next_timer_interrupt+0x160/0x160 [12952.242462] ptlrpcd+0xa82/0xe40 [ptlrpc] [12952.245582] ? do_wait_intr+0xf0/0xf0 [12952.247879] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12952.251830] kthread+0x1d1/0x200 [12952.253904] ? set_kthread_struct+0x70/0x70 [12952.257796] ret_from_fork+0x1f/0x30 [12956.052010] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [12956.054988] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12956.074287] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12956.079210] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12956.082795] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12956.084754] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12956.091276] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12956.094346] RAX: 000000000000205c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12956.097041] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [12956.099621] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [12956.102191] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [12956.105157] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [12956.107890] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [12956.111110] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12956.113442] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [12956.116530] Call Trace: [12956.117598] [12956.118935] ? show_regs.cold.9+0x22/0x2f [12956.120217] ? watchdog_timer_fn.cold.6+0x62/0x10a [12956.122240] ? __hrtimer_run_queues+0x1c2/0x4d0 [12956.124450] ? watchdog+0x40/0x40 [12956.125950] ? hrtimer_interrupt+0x128/0x2c0 [12956.127925] ? smp_apic_timer_interrupt+0xa6/0x270 [12956.130056] ? apic_timer_interrupt+0xf/0x20 [12956.133658] [12956.134262] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12956.137031] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12956.140948] do_raw_spin_lock+0x177/0x190 [12956.143164] _raw_spin_lock+0x21/0x30 [12956.144471] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12956.147811] ? do_raw_spin_unlock+0x75/0x190 [12956.149616] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12956.152078] ? do_wait_intr+0xf0/0xf0 [12956.153572] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12956.156232] mdc_reint+0x2d/0x110 [mdc] [12956.158134] mdc_unlink+0x658/0x870 [mdc] [12956.160122] lmv_unlink+0x34d/0xbd0 [lmv] [12956.162071] ll_unlink+0x24d/0x7f0 [lustre] [12956.164104] vfs_unlink+0x202/0x330 [12956.165613] do_unlinkat+0x2fa/0x460 [12956.166717] __x64_sys_unlink+0x37/0x50 [12956.168340] do_syscall_64+0xc1/0x440 [12956.169929] entry_SYSCALL_64_after_hwframe+0x49/0xae [12956.172155] RIP: 0033:0x7f151bdee10b [12956.173047] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [12956.180452] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [12956.181983] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [12956.184738] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [12956.187800] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12956.189588] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12956.192064] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12959.722813] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11029s! [12959.727618] Showing busy workqueues and worker pools: [12959.730104] workqueue events: flags=0x0 [12959.733932] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12959.741536] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [12959.750311] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12959.758376] pending: kfree_rcu_monitor [12959.761719] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12959.765886] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [12959.772229] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12959.775254] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [12959.778869] workqueue events_long: flags=0x0 [12959.780951] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12959.784557] pending: lu_objects_flush [obdclass] [12959.787779] workqueue events_unbound: flags=0x2 [12959.791012] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12959.795279] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12959.804318] workqueue events_power_efficient: flags=0x80 [12959.807874] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12959.820635] pending: do_cache_clean, neigh_periodic_work, check_lifetime [12959.829761] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12959.837439] pending: neigh_periodic_work [12959.843139] workqueue mm_percpu_wq: flags=0x8 [12959.846768] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12959.851954] pending: vmstat_update, lru_add_drain_per_cpu [12959.855522] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12959.861227] pending: vmstat_update [12959.863405] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12959.866670] pending: lru_add_drain_per_cpu [12959.868843] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [12959.871552] pending: lru_add_drain_per_cpu BAR(48) [12959.874297] workqueue writeback: flags=0x4a [12959.876387] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [12959.879302] pending: wb_workfn [12959.881722] workqueue xprtiod: flags=0xa [12959.883401] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12959.887543] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [12959.890808] workqueue ipv6_addrconf: flags=0x40008 [12959.893114] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12959.896070] pending: addrconf_verify_work [12959.899203] workqueue knbd0-recv: flags=0x1a [12959.901044] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12959.904341] in-flight: 658:recv_work [12959.906389] workqueue ptlrpc_pinger: flags=0x2 [12959.908125] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12959.910973] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12959.913609] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11029s workers=4 idle: 32984 37 8 [12959.917557] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12944s workers=2 idle: 661 [12964.886055] rcu: INFO: rcu_sched self-detected stall on CPU [12964.888558] rcu: 1-...!: (6406295 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [12964.893590] (t=11040239 jiffies g=249657 q=5748) [12964.895388] rcu: rcu_sched kthread starved for 11035191 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [12964.899625] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [12964.903108] rcu: RCU grace-period kthread stack dump: [12964.905168] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [12964.909111] Call Trace: [12964.909891] __schedule+0x351/0xcb0 [12964.911053] schedule+0xc0/0x180 [12964.912057] schedule_timeout+0xb4/0x190 [12964.913365] ? do_raw_spin_unlock+0x75/0x190 [12964.914939] ? __next_timer_interrupt+0x160/0x160 [12964.916755] ? prepare_to_swait_event+0xf3/0x140 [12964.918202] rcu_gp_kthread+0x8dd/0x15b0 [12964.919683] ? force_qs_rnp+0x2c0/0x2c0 [12964.920860] kthread+0x1d1/0x200 [12964.921992] ? set_kthread_struct+0x70/0x70 [12964.926770] ret_from_fork+0x1f/0x30 [12964.929359] rcu: Stack dump where RCU GP kthread last ran: [12964.931638] NMI backtrace for cpu 1 [12964.933350] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12964.937954] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12964.944675] Call Trace: [12964.949163] [12964.951595] ? dump_stack+0xbb/0x10e [12964.952884] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [12964.963338] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [12964.969683] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [12964.980223] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [12964.981928] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [12964.990355] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [12964.995905] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [12965.002854] ? update_process_times+0x6d/0xc0 [12965.005271] ? tick_sched_handle.isra.18+0x27/0x90 [12965.009026] ? tick_sched_timer+0x53/0xb0 [12965.011711] ? __hrtimer_run_queues+0x1c2/0x4d0 [12965.013941] ? tick_sched_handle.isra.18+0x90/0x90 [12965.020660] ? hrtimer_interrupt+0x128/0x2c0 [12965.022553] ? smp_apic_timer_interrupt+0xa6/0x270 [12965.028148] ? apic_timer_interrupt+0xf/0x20 [12965.032058] [12965.033518] ? native_safe_halt+0x2e/0x50 [12965.036794] ? kvm_wait+0xcd/0xe0 [12965.038393] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12965.044252] ? do_raw_spin_lock+0x177/0x190 [12965.046788] ? _raw_spin_lock+0x21/0x30 [12965.060132] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12965.062362] ? obd_mod_free+0x46/0x140 [ptlrpc] [12965.066745] ? mdc_commit_open+0x91/0xb0 [mdc] [12965.068981] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12965.075813] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12965.081683] ? after_reply+0xe2c/0x1780 [ptlrpc] [12965.090145] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12965.094365] ? __next_timer_interrupt+0x160/0x160 [12965.098200] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [12965.103113] ? do_wait_intr+0xf0/0xf0 [12965.104806] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12965.111246] ? kthread+0x1d1/0x200 [12965.117561] ? set_kthread_struct+0x70/0x70 [12965.121238] ? ret_from_fork+0x1f/0x30 [12965.124891] NMI backtrace for cpu 1 [12965.127801] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12965.143137] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12965.148502] Call Trace: [12965.150802] [12965.151913] ? dump_stack+0xbb/0x10e [12965.155539] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [12965.161078] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [12965.165199] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [12965.169538] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [12965.176981] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [12965.179345] ? rcu_dump_cpu_stacks+0x11b/0x171 [12965.180850] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [12965.189269] ? update_process_times+0x6d/0xc0 [12965.191483] ? tick_sched_handle.isra.18+0x27/0x90 [12965.194350] ? tick_sched_timer+0x53/0xb0 [12965.196527] ? __hrtimer_run_queues+0x1c2/0x4d0 [12965.199403] ? tick_sched_handle.isra.18+0x90/0x90 [12965.209069] ? hrtimer_interrupt+0x128/0x2c0 [12965.213835] ? smp_apic_timer_interrupt+0xa6/0x270 [12965.218224] ? apic_timer_interrupt+0xf/0x20 [12965.222373] [12965.223376] ? native_safe_halt+0x2e/0x50 [12965.229895] ? kvm_wait+0xcd/0xe0 [12965.233810] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12965.236360] ? do_raw_spin_lock+0x177/0x190 [12965.238204] ? _raw_spin_lock+0x21/0x30 [12965.242925] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12965.246196] ? obd_mod_free+0x46/0x140 [ptlrpc] [12965.251436] ? mdc_commit_open+0x91/0xb0 [mdc] [12965.255894] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12965.258853] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12965.261818] ? after_reply+0xe2c/0x1780 [ptlrpc] [12965.268379] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12965.271853] ? __next_timer_interrupt+0x160/0x160 [12965.277536] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [12965.281098] ? do_wait_intr+0xf0/0xf0 [12965.284123] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12965.290859] ? kthread+0x1d1/0x200 [12965.294150] ? set_kthread_struct+0x70/0x70 [12965.298808] ? ret_from_fork+0x1f/0x30 [12968.057020] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [12968.063357] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12968.098469] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12968.108937] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12968.120878] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12968.126299] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12968.137395] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12968.142697] RAX: 0000000000001406 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12968.150752] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [12968.156910] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [12968.167916] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [12968.178967] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [12968.184126] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [12968.194788] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12968.199222] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [12968.203585] Call Trace: [12968.204769] [12968.205760] ? show_regs.cold.9+0x22/0x2f [12968.207954] ? watchdog_timer_fn.cold.6+0x62/0x10a [12968.210814] ? __hrtimer_run_queues+0x1c2/0x4d0 [12968.216124] ? watchdog+0x40/0x40 [12968.220070] ? hrtimer_interrupt+0x128/0x2c0 [12968.222993] ? smp_apic_timer_interrupt+0xa6/0x270 [12968.229754] ? apic_timer_interrupt+0xf/0x20 [12968.234964] [12968.237361] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12968.243214] do_raw_spin_lock+0x177/0x190 [12968.250992] _raw_spin_lock+0x21/0x30 [12968.253762] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12968.261960] ? do_raw_spin_unlock+0x75/0x190 [12968.264831] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12968.271923] ? do_wait_intr+0xf0/0xf0 [12968.277842] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12968.282013] mdc_close+0x53b/0xd70 [mdc] [12968.283571] lmv_close+0x18d/0x490 [lmv] [12968.287729] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12968.292174] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12968.297956] ll_md_real_close+0xa5/0x2a0 [lustre] [12968.301254] ll_md_close+0x274/0x800 [lustre] [12968.305202] ll_file_release+0x128/0xc50 [lustre] [12968.311949] __fput+0x10d/0x440 [12968.314374] ____fput+0x12/0x20 [12968.318755] task_work_run+0xc6/0x110 [12968.321221] exit_to_usermode_loop+0x1dd/0x1f0 [12968.327050] do_syscall_64+0x3d6/0x440 [12968.329103] entry_SYSCALL_64_after_hwframe+0x49/0xae [12968.334736] RIP: 0033:0x7f151bee3cd8 [12968.336088] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12968.353757] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12968.358255] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [12968.368946] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [12968.373028] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12968.378355] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12968.385258] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12972.056374] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [12972.061616] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12972.098175] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12972.104092] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12972.109414] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12972.114436] RIP: 0010:native_safe_halt+0x2e/0x50 [12972.116950] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12972.127548] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12972.134110] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12972.147078] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [12972.154017] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [12972.159146] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [12972.167896] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [12972.171578] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [12972.175482] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12972.177163] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12972.179675] Call Trace: [12972.180580] [12972.181188] ? show_regs.cold.9+0x22/0x2f [12972.182527] ? watchdog_timer_fn.cold.6+0x62/0x10a [12972.184099] ? __hrtimer_run_queues+0x1c2/0x4d0 [12972.185963] ? watchdog+0x40/0x40 [12972.186883] ? hrtimer_interrupt+0x128/0x2c0 [12972.188863] ? smp_apic_timer_interrupt+0xa6/0x270 [12972.190173] ? apic_timer_interrupt+0xf/0x20 [12972.191701] [12972.192869] ? native_safe_halt+0x2e/0x50 [12972.194088] kvm_wait+0xcd/0xe0 [12972.195448] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12972.197067] do_raw_spin_lock+0x177/0x190 [12972.198267] _raw_spin_lock+0x21/0x30 [12972.199578] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12972.201432] process_one_work+0x2c8/0x700 [12972.203119] worker_thread+0x296/0x6e0 [12972.205462] ? rescuer_thread+0x570/0x570 [12972.207569] kthread+0x1d1/0x200 [12972.209547] ? set_kthread_struct+0x70/0x70 [12972.211056] ret_from_fork+0x1f/0x30 [12984.052042] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [12984.055416] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12984.091418] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12984.102242] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12984.107884] RIP: 0010:native_safe_halt+0x2e/0x50 [12984.111210] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12984.128376] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12984.132401] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12984.135925] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [12984.140125] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [12984.143964] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [12984.148929] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [12984.153952] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [12984.161592] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12984.166788] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [12984.171109] Call Trace: [12984.173648] [12984.174560] ? show_regs.cold.9+0x22/0x2f [12984.176437] ? watchdog_timer_fn.cold.6+0x62/0x10a [12984.182237] ? __hrtimer_run_queues+0x1c2/0x4d0 [12984.188005] ? watchdog+0x40/0x40 [12984.189287] ? hrtimer_interrupt+0x128/0x2c0 [12984.192561] ? smp_apic_timer_interrupt+0xa6/0x270 [12984.195565] ? apic_timer_interrupt+0xf/0x20 [12984.199160] [12984.200059] ? native_safe_halt+0x2e/0x50 [12984.202425] kvm_wait+0xcd/0xe0 [12984.203980] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12984.207422] do_raw_spin_lock+0x177/0x190 [12984.215940] _raw_spin_lock+0x21/0x30 [12984.217671] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12984.221583] ? do_raw_spin_unlock+0x75/0x190 [12984.224637] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12984.227407] ? do_wait_intr+0xf0/0xf0 [12984.230173] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12984.232595] mdc_reint+0x2d/0x110 [mdc] [12984.233820] mdc_unlink+0x658/0x870 [mdc] [12984.235147] lmv_unlink+0x34d/0xbd0 [lmv] [12984.236423] ll_unlink+0x24d/0x7f0 [lustre] [12984.238821] vfs_unlink+0x202/0x330 [12984.241864] do_unlinkat+0x2fa/0x460 [12984.243386] __x64_sys_unlink+0x37/0x50 [12984.245606] do_syscall_64+0xc1/0x440 [12984.247319] entry_SYSCALL_64_after_hwframe+0x49/0xae [12984.250977] RIP: 0033:0x7f151bdee10b [12984.252787] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [12984.263974] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [12984.271037] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [12984.275745] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [12984.280590] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12984.284780] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12984.288391] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [12990.433101] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11059s! [12990.436229] Showing busy workqueues and worker pools: [12990.438399] workqueue events: flags=0x0 [12990.439810] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12990.442712] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [12990.446542] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12990.449266] pending: kfree_rcu_monitor [12990.451356] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12990.453780] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [12990.457544] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12990.460352] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [12990.466604] workqueue events_long: flags=0x0 [12990.471682] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12990.480292] pending: lu_objects_flush [obdclass] [12990.482511] workqueue events_unbound: flags=0x2 [12990.487609] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [12990.491198] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [12990.497021] workqueue events_power_efficient: flags=0x80 [12990.500125] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12990.502696] pending: do_cache_clean, neigh_periodic_work, check_lifetime [12990.508599] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12990.512322] pending: neigh_periodic_work [12990.514064] workqueue mm_percpu_wq: flags=0x8 [12990.516397] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12990.519479] pending: vmstat_update, lru_add_drain_per_cpu [12990.525068] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12990.528111] pending: vmstat_update [12990.529663] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12990.532344] pending: lru_add_drain_per_cpu [12990.533830] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [12990.536108] pending: lru_add_drain_per_cpu BAR(48) [12990.538116] workqueue writeback: flags=0x4a [12990.539599] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [12990.542759] pending: wb_workfn [12990.544574] workqueue xprtiod: flags=0xa [12990.546489] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [12990.551889] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [12990.558707] workqueue ipv6_addrconf: flags=0x40008 [12990.560838] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12990.564894] pending: addrconf_verify_work [12990.568438] workqueue knbd0-recv: flags=0x1a [12990.569833] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12990.573101] in-flight: 658:recv_work [12990.575529] workqueue ptlrpc_pinger: flags=0x2 [12990.577601] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12990.580499] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [12990.587497] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11059s workers=4 idle: 32984 37 8 [12990.591662] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12975s workers=2 idle: 661 [12992.057541] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [12992.070076] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12992.120099] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12992.131526] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12992.137816] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12992.146949] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12992.168515] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12992.178304] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [12992.185102] RDX: 0000000000001037 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [12992.197764] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [12992.207073] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [12992.215501] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12992.220838] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [12992.237632] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12992.248848] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [12992.256864] Call Trace: [12992.259099] [12992.262277] ? show_regs.cold.9+0x22/0x2f [12992.264874] ? watchdog_timer_fn.cold.6+0x62/0x10a [12992.270304] ? __hrtimer_run_queues+0x1c2/0x4d0 [12992.271957] ? watchdog+0x40/0x40 [12992.273133] ? hrtimer_interrupt+0x128/0x2c0 [12992.283080] ? smp_apic_timer_interrupt+0xa6/0x270 [12992.289748] ? apic_timer_interrupt+0xf/0x20 [12992.293136] [12992.296891] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12992.303113] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12992.308860] do_raw_spin_lock+0x177/0x190 [12992.312833] _raw_spin_lock+0x21/0x30 [12992.318429] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12992.325997] obd_mod_free+0x46/0x140 [ptlrpc] [12992.332006] mdc_commit_open+0x91/0xb0 [mdc] [12992.334824] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12992.340389] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12992.343974] after_reply+0xe2c/0x1780 [ptlrpc] [12992.347803] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12992.351171] ? __next_timer_interrupt+0x160/0x160 [12992.354291] ptlrpcd+0xa82/0xe40 [ptlrpc] [12992.357141] ? do_wait_intr+0xf0/0xf0 [12992.358956] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12992.361999] kthread+0x1d1/0x200 [12992.364849] ? set_kthread_struct+0x70/0x70 [12992.367228] ret_from_fork+0x1f/0x30 [12996.060300] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [12996.066099] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12996.125899] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12996.140623] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12996.149901] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12996.163849] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12996.218934] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12996.229582] RAX: 000000000000720d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [12996.242417] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [12996.250231] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [12996.259755] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [12996.264030] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [12996.268466] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [12996.277688] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12996.284236] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [12996.291891] Call Trace: [12996.294627] [12996.297385] ? show_regs.cold.9+0x22/0x2f [12996.301674] ? watchdog_timer_fn.cold.6+0x62/0x10a [12996.303555] ? __hrtimer_run_queues+0x1c2/0x4d0 [12996.305744] ? watchdog+0x40/0x40 [12996.307417] ? hrtimer_interrupt+0x128/0x2c0 [12996.311514] ? smp_apic_timer_interrupt+0xa6/0x270 [12996.315588] ? apic_timer_interrupt+0xf/0x20 [12996.319946] [12996.321364] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12996.326619] do_raw_spin_lock+0x177/0x190 [12996.328891] _raw_spin_lock+0x21/0x30 [12996.331519] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12996.338326] ? do_raw_spin_unlock+0x75/0x190 [12996.343549] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12996.348380] ? do_wait_intr+0xf0/0xf0 [12996.350528] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12996.354141] mdc_close+0x53b/0xd70 [mdc] [12996.360828] lmv_close+0x18d/0x490 [lmv] [12996.363886] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [12996.370661] ll_close_inode_openhandle+0x493/0xea0 [lustre] [12996.374624] ll_md_real_close+0xa5/0x2a0 [lustre] [12996.379674] ll_md_close+0x274/0x800 [lustre] [12996.384277] ll_file_release+0x128/0xc50 [lustre] [12996.386464] __fput+0x10d/0x440 [12996.388194] ____fput+0x12/0x20 [12996.389967] task_work_run+0xc6/0x110 [12996.392416] exit_to_usermode_loop+0x1dd/0x1f0 [12996.397763] do_syscall_64+0x3d6/0x440 [12996.401587] entry_SYSCALL_64_after_hwframe+0x49/0xae [12996.407192] RIP: 0033:0x7f151bee3cd8 [12996.410643] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [12996.423513] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [12996.429803] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [12996.435878] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [12996.443900] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [12996.452911] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [12996.463736] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13000.056010] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [13000.062827] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13000.113084] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13000.122557] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13000.131748] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13000.144439] RIP: 0010:native_safe_halt+0x2e/0x50 [13000.146201] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13000.153443] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13000.157226] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13000.159428] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [13000.167086] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [13000.171601] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [13000.179479] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [13000.186477] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [13000.189195] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13000.193044] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13000.201680] Call Trace: [13000.202818] [13000.206612] ? show_regs.cold.9+0x22/0x2f [13000.208359] ? watchdog_timer_fn.cold.6+0x62/0x10a [13000.210728] ? __hrtimer_run_queues+0x1c2/0x4d0 [13000.212744] ? watchdog+0x40/0x40 [13000.215424] ? hrtimer_interrupt+0x128/0x2c0 [13000.220008] ? smp_apic_timer_interrupt+0xa6/0x270 [13000.224382] ? apic_timer_interrupt+0xf/0x20 [13000.228835] [13000.231180] ? native_safe_halt+0x2e/0x50 [13000.235788] kvm_wait+0xcd/0xe0 [13000.239409] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13000.243772] do_raw_spin_lock+0x177/0x190 [13000.251710] _raw_spin_lock+0x21/0x30 [13000.256708] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13000.261362] process_one_work+0x2c8/0x700 [13000.264356] worker_thread+0x296/0x6e0 [13000.269048] ? rescuer_thread+0x570/0x570 [13000.273179] kthread+0x1d1/0x200 [13000.276117] ? set_kthread_struct+0x70/0x70 [13000.279755] ret_from_fork+0x1f/0x30 [13016.053479] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [13016.055773] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13016.074038] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13016.078476] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13016.081768] RIP: 0010:native_safe_halt+0x2e/0x50 [13016.083233] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13016.089843] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13016.092198] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13016.094168] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [13016.096109] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [13016.098247] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [13016.100064] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [13016.103064] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [13016.105126] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13016.106989] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [13016.108738] Call Trace: [13016.109362] [13016.109973] ? show_regs.cold.9+0x22/0x2f [13016.111000] ? watchdog_timer_fn.cold.6+0x62/0x10a [13016.112291] ? __hrtimer_run_queues+0x1c2/0x4d0 [13016.113431] ? watchdog+0x40/0x40 [13016.114391] ? hrtimer_interrupt+0x128/0x2c0 [13016.115682] ? smp_apic_timer_interrupt+0xa6/0x270 [13016.117050] ? apic_timer_interrupt+0xf/0x20 [13016.118662] [13016.119265] ? native_safe_halt+0x2e/0x50 [13016.120480] kvm_wait+0xcd/0xe0 [13016.121382] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13016.122869] do_raw_spin_lock+0x177/0x190 [13016.124145] _raw_spin_lock+0x21/0x30 [13016.125283] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13016.127153] ? do_raw_spin_unlock+0x75/0x190 [13016.128395] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13016.130109] ? do_wait_intr+0xf0/0xf0 [13016.131226] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13016.132866] mdc_reint+0x2d/0x110 [mdc] [13016.134246] mdc_unlink+0x658/0x870 [mdc] [13016.135341] lmv_unlink+0x34d/0xbd0 [lmv] [13016.136878] ll_unlink+0x24d/0x7f0 [lustre] [13016.138108] vfs_unlink+0x202/0x330 [13016.139241] do_unlinkat+0x2fa/0x460 [13016.140171] __x64_sys_unlink+0x37/0x50 [13016.141161] do_syscall_64+0xc1/0x440 [13016.142289] entry_SYSCALL_64_after_hwframe+0x49/0xae [13016.143698] RIP: 0033:0x7f151bdee10b [13016.144887] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [13016.151433] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [13016.153719] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [13016.155838] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [13016.158204] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13016.160580] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13016.162975] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13020.055725] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [13020.060809] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13020.081205] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13020.088269] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13020.093737] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13020.097669] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13020.111446] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13020.115109] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [13020.118493] RDX: 0000000000007f39 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [13020.122343] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [13020.127857] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [13020.130341] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13020.135458] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [13020.138277] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13020.142888] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13020.146456] Call Trace: [13020.147866] [13020.148896] ? show_regs.cold.9+0x22/0x2f [13020.151015] ? watchdog_timer_fn.cold.6+0x62/0x10a [13020.153063] ? __hrtimer_run_queues+0x1c2/0x4d0 [13020.155099] ? watchdog+0x40/0x40 [13020.156973] ? hrtimer_interrupt+0x128/0x2c0 [13020.158830] ? smp_apic_timer_interrupt+0xa6/0x270 [13020.160399] ? apic_timer_interrupt+0xf/0x20 [13020.162156] [13020.163029] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13020.164330] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13020.165547] do_raw_spin_lock+0x177/0x190 [13020.167088] _raw_spin_lock+0x21/0x30 [13020.168954] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13020.171937] obd_mod_free+0x46/0x140 [ptlrpc] [13020.173998] mdc_commit_open+0x91/0xb0 [mdc] [13020.175564] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13020.177927] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13020.180387] after_reply+0xe2c/0x1780 [ptlrpc] [13020.182624] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13020.184365] ? __next_timer_interrupt+0x160/0x160 [13020.185715] ptlrpcd+0xa82/0xe40 [ptlrpc] [13020.186966] ? do_wait_intr+0xf0/0xf0 [13020.187960] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13020.190246] kthread+0x1d1/0x200 [13020.191638] ? set_kthread_struct+0x70/0x70 [13020.193418] ret_from_fork+0x1f/0x30 [13020.641260] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11090s! [13020.649212] Showing busy workqueues and worker pools: [13020.653347] workqueue events: flags=0x0 [13020.656361] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13020.658911] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [13020.663973] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13020.666930] pending: kfree_rcu_monitor [13020.669105] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13020.671808] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [13020.675505] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13020.678383] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [13020.681793] workqueue events_long: flags=0x0 [13020.683057] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13020.685718] pending: lu_objects_flush [obdclass] [13020.688592] workqueue events_unbound: flags=0x2 [13020.690460] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13020.693313] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13020.697388] workqueue events_power_efficient: flags=0x80 [13020.699624] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13020.701939] pending: do_cache_clean, neigh_periodic_work, check_lifetime [13020.704696] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13020.707914] pending: neigh_periodic_work [13020.710397] workqueue mm_percpu_wq: flags=0x8 [13020.712580] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13020.716059] pending: vmstat_update, lru_add_drain_per_cpu [13020.718405] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13020.721627] pending: vmstat_update [13020.723356] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13020.726449] pending: lru_add_drain_per_cpu [13020.732806] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [13020.736049] pending: lru_add_drain_per_cpu BAR(48) [13020.738311] workqueue writeback: flags=0x4a [13020.739902] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [13020.742245] pending: wb_workfn [13020.743558] workqueue xprtiod: flags=0xa [13020.744803] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13020.747458] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [13020.750235] workqueue ipv6_addrconf: flags=0x40008 [13020.751895] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13020.754393] pending: addrconf_verify_work [13020.756330] workqueue knbd0-recv: flags=0x1a [13020.757914] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13020.760330] in-flight: 658:recv_work [13020.761843] workqueue ptlrpc_pinger: flags=0x2 [13020.763551] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13020.765859] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13020.768404] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11090s workers=4 idle: 32984 37 8 [13020.770758] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13005s workers=2 idle: 661 [13024.057065] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [13024.059880] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13024.079271] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13024.083382] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13024.087211] RIP: 0010:native_safe_halt+0x2e/0x50 [13024.089184] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13024.097635] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13024.101341] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13024.104467] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [13024.107693] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [13024.116127] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [13024.119220] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [13024.122346] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [13024.125685] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13024.128169] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [13024.131281] Call Trace: [13024.132286] [13024.133018] ? show_regs.cold.9+0x22/0x2f [13024.134975] ? watchdog_timer_fn.cold.6+0x62/0x10a [13024.136836] ? __hrtimer_run_queues+0x1c2/0x4d0 [13024.138983] ? watchdog+0x40/0x40 [13024.140270] ? hrtimer_interrupt+0x128/0x2c0 [13024.142127] ? smp_apic_timer_interrupt+0xa6/0x270 [13024.146003] ? apic_timer_interrupt+0xf/0x20 [13024.149504] [13024.150222] ? native_safe_halt+0x2e/0x50 [13024.152250] kvm_wait+0xcd/0xe0 [13024.153510] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13024.155361] do_raw_spin_lock+0x177/0x190 [13024.157374] _raw_spin_lock+0x21/0x30 [13024.159127] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13024.161860] ? do_raw_spin_unlock+0x75/0x190 [13024.163536] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13024.165952] ? do_wait_intr+0xf0/0xf0 [13024.167613] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13024.170230] mdc_close+0x53b/0xd70 [mdc] [13024.172127] lmv_close+0x18d/0x490 [lmv] [13024.174095] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13024.176816] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13024.179821] ll_md_real_close+0xa5/0x2a0 [lustre] [13024.181966] ll_md_close+0x274/0x800 [lustre] [13024.184683] ll_file_release+0x128/0xc50 [lustre] [13024.187668] __fput+0x10d/0x440 [13024.189071] ____fput+0x12/0x20 [13024.190881] task_work_run+0xc6/0x110 [13024.192697] exit_to_usermode_loop+0x1dd/0x1f0 [13024.194447] do_syscall_64+0x3d6/0x440 [13024.196052] entry_SYSCALL_64_after_hwframe+0x49/0xae [13024.197817] RIP: 0033:0x7f151bee3cd8 [13024.199428] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13024.217656] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13024.226933] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [13024.233455] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [13024.245751] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13024.252529] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13024.258620] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13028.056013] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [13028.059990] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13028.108849] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13028.118374] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13028.125779] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13028.134107] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13028.137925] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13028.158134] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13028.163457] RAX: 0000000000004fa7 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13028.166490] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [13028.169558] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [13028.172348] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [13028.175373] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [13028.178551] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [13028.181748] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13028.184062] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13028.186787] Call Trace: [13028.188028] [13028.191484] ? show_regs.cold.9+0x22/0x2f [13028.193228] ? watchdog_timer_fn.cold.6+0x62/0x10a [13028.200541] ? __hrtimer_run_queues+0x1c2/0x4d0 [13028.209202] ? watchdog+0x40/0x40 [13028.210216] ? hrtimer_interrupt+0x128/0x2c0 [13028.213181] ? smp_apic_timer_interrupt+0xa6/0x270 [13028.216621] ? apic_timer_interrupt+0xf/0x20 [13028.218419] [13028.219474] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13028.225215] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13028.230387] do_raw_spin_lock+0x177/0x190 [13028.232138] _raw_spin_lock+0x21/0x30 [13028.234631] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13028.237603] process_one_work+0x2c8/0x700 [13028.239419] worker_thread+0x296/0x6e0 [13028.241748] ? rescuer_thread+0x570/0x570 [13028.243523] kthread+0x1d1/0x200 [13028.245398] ? set_kthread_struct+0x70/0x70 [13028.247260] ret_from_fork+0x1f/0x30 [13048.055019] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [13048.058219] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss [13048.058281] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [13048.058286] nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13048.058333] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13048.058338] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13048.058341] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13048.091249] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac [13048.095123] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13048.095135] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13048.095142] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [13048.095145] RDX: 000000000000043c RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [13048.095147] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [13048.095150] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [13048.095152] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13048.095155] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [13048.095158] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13048.095159] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13048.095167] Call Trace: [13048.095173] [13048.095178] ? show_regs.cold.9+0x22/0x2f [13048.095188] ? watchdog_timer_fn.cold.6+0x62/0x10a [13048.095195] ? __hrtimer_run_queues+0x1c2/0x4d0 [13048.095198] ? watchdog+0x40/0x40 [13048.095202] ? hrtimer_interrupt+0x128/0x2c0 [13048.095205] ? smp_apic_timer_interrupt+0xa6/0x270 [13048.095212] ? apic_timer_interrupt+0xf/0x20 [13048.095216] [13048.095219] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13048.095224] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13048.095229] do_raw_spin_lock+0x177/0x190 [13048.095233] _raw_spin_lock+0x21/0x30 [13048.095240] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13048.095500] obd_mod_free+0x46/0x140 [ptlrpc] [13048.095816] mdc_commit_open+0x91/0xb0 [mdc] [13048.095854] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13048.096047] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13048.096241] after_reply+0xe2c/0x1780 [ptlrpc] [13048.096601] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13048.096806] ? __next_timer_interrupt+0x160/0x160 [13048.096816] ptlrpcd+0xa82/0xe40 [ptlrpc] [13048.097019] ? do_wait_intr+0xf0/0xf0 [13048.097029] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13048.097234] kthread+0x1d1/0x200 [13048.097240] ? set_kthread_struct+0x70/0x70 [13048.097244] ret_from_fork+0x1f/0x30 [13048.240586] rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13048.249903] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13048.255004] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13048.258700] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13048.262117] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13048.270958] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13048.274284] RAX: 0000000000006f89 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13048.277640] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [13048.281055] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [13048.284216] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [13048.287114] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [13048.290510] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [13048.294076] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13048.296504] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [13048.299114] Call Trace: [13048.300294] [13048.301021] ? show_regs.cold.9+0x22/0x2f [13048.302752] ? watchdog_timer_fn.cold.6+0x62/0x10a [13048.305112] ? __hrtimer_run_queues+0x1c2/0x4d0 [13048.307300] ? watchdog+0x40/0x40 [13048.308929] ? hrtimer_interrupt+0x128/0x2c0 [13048.310837] ? smp_apic_timer_interrupt+0xa6/0x270 [13048.316309] ? apic_timer_interrupt+0xf/0x20 [13048.321469] [13048.322868] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13048.325623] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13048.328187] do_raw_spin_lock+0x177/0x190 [13048.330228] _raw_spin_lock+0x21/0x30 [13048.332126] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13048.334947] ? do_raw_spin_unlock+0x75/0x190 [13048.337109] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13048.339711] ? do_wait_intr+0xf0/0xf0 [13048.340870] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13048.343190] mdc_reint+0x2d/0x110 [mdc] [13048.345359] mdc_unlink+0x658/0x870 [mdc] [13048.347127] lmv_unlink+0x34d/0xbd0 [lmv] [13048.349216] ll_unlink+0x24d/0x7f0 [lustre] [13048.351193] vfs_unlink+0x202/0x330 [13048.352990] do_unlinkat+0x2fa/0x460 [13048.354613] __x64_sys_unlink+0x37/0x50 [13048.356918] do_syscall_64+0xc1/0x440 [13048.358976] entry_SYSCALL_64_after_hwframe+0x49/0xae [13048.361569] RIP: 0033:0x7f151bdee10b [13048.363300] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [13048.372865] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [13048.376189] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [13048.379589] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [13048.383352] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13048.386922] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13048.390506] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13050.858674] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11120s! [13050.872789] Showing busy workqueues and worker pools: [13050.879558] workqueue events: flags=0x0 [13050.885231] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13050.898495] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [13050.910065] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13050.922253] pending: kfree_rcu_monitor [13050.929511] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13050.938825] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [13050.949044] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13050.954095] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [13050.961031] workqueue events_long: flags=0x0 [13050.972024] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13050.980511] pending: lu_objects_flush [obdclass] [13050.983371] workqueue events_unbound: flags=0x2 [13050.984938] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13050.987786] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13050.992857] workqueue events_power_efficient: flags=0x80 [13050.996405] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13051.000120] pending: do_cache_clean, neigh_periodic_work, check_lifetime [13051.004107] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13051.009748] pending: neigh_periodic_work [13051.015551] workqueue mm_percpu_wq: flags=0x8 [13051.021118] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13051.030658] pending: vmstat_update, lru_add_drain_per_cpu [13051.033695] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13051.037269] pending: vmstat_update [13051.039341] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13051.043086] pending: lru_add_drain_per_cpu [13051.045162] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [13051.048929] pending: lru_add_drain_per_cpu BAR(48) [13051.051408] workqueue writeback: flags=0x4a [13051.053089] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [13051.056798] pending: wb_workfn [13051.058429] workqueue xprtiod: flags=0xa [13051.059695] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13051.063414] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [13051.070796] workqueue ipv6_addrconf: flags=0x40008 [13051.079505] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13051.088089] pending: addrconf_verify_work [13051.092979] workqueue knbd0-recv: flags=0x1a [13051.099334] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13051.108925] in-flight: 658:recv_work [13051.116855] workqueue ptlrpc_pinger: flags=0x2 [13051.120100] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13051.123887] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13051.132658] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11120s workers=4 idle: 32984 37 8 [13051.146866] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13035s workers=2 idle: 661 [13052.062149] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [13052.079201] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13052.160089] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13052.172212] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13052.202562] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13052.215835] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13052.277272] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13052.295200] RAX: 0000000000002511 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13052.307812] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [13052.310382] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [13052.338887] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [13052.349286] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [13052.367442] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [13052.386254] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13052.400293] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [13052.419120] Call Trace: [13052.426250] [13052.427155] ? show_regs.cold.9+0x22/0x2f [13052.430297] ? watchdog_timer_fn.cold.6+0x62/0x10a [13052.435900] ? __hrtimer_run_queues+0x1c2/0x4d0 [13052.445040] ? watchdog+0x40/0x40 [13052.450391] ? hrtimer_interrupt+0x128/0x2c0 [13052.463019] ? smp_apic_timer_interrupt+0xa6/0x270 [13052.470363] ? apic_timer_interrupt+0xf/0x20 [13052.480760] [13052.481558] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13052.510386] do_raw_spin_lock+0x177/0x190 [13052.512414] _raw_spin_lock+0x21/0x30 [13052.530185] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13052.532239] ? do_raw_spin_unlock+0x75/0x190 [13052.547636] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13052.556967] ? do_wait_intr+0xf0/0xf0 [13052.565347] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13052.575656] mdc_close+0x53b/0xd70 [mdc] [13052.577046] lmv_close+0x18d/0x490 [lmv] [13052.590986] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13052.603263] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13052.605373] ll_md_real_close+0xa5/0x2a0 [lustre] [13052.621169] ll_md_close+0x274/0x800 [lustre] [13052.627233] ll_file_release+0x128/0xc50 [lustre] [13052.632973] __fput+0x10d/0x440 [13052.639092] ____fput+0x12/0x20 [13052.643243] task_work_run+0xc6/0x110 [13052.649951] exit_to_usermode_loop+0x1dd/0x1f0 [13052.659049] do_syscall_64+0x3d6/0x440 [13052.660346] entry_SYSCALL_64_after_hwframe+0x49/0xae [13052.676901] RIP: 0033:0x7f151bee3cd8 [13052.685108] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13052.726388] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13052.737319] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [13052.750157] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [13052.759110] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13052.763037] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13052.772847] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13056.056095] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13056.059971] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13056.084290] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13056.090715] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13056.095646] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13056.099090] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13056.103134] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13056.113839] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13056.118229] RAX: 000000000000689a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13056.122016] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [13056.126295] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [13056.130346] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [13056.134338] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [13056.138079] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [13056.142170] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13056.145892] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13056.150249] Call Trace: [13056.151741] [13056.153299] ? show_regs.cold.9+0x22/0x2f [13056.155084] ? watchdog_timer_fn.cold.6+0x62/0x10a [13056.160191] ? __hrtimer_run_queues+0x1c2/0x4d0 [13056.162773] ? watchdog+0x40/0x40 [13056.164559] ? hrtimer_interrupt+0x128/0x2c0 [13056.167281] ? smp_apic_timer_interrupt+0xa6/0x270 [13056.170019] ? apic_timer_interrupt+0xf/0x20 [13056.173956] [13056.174993] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13056.178581] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13056.182037] do_raw_spin_lock+0x177/0x190 [13056.183985] _raw_spin_lock+0x21/0x30 [13056.186195] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13056.190013] process_one_work+0x2c8/0x700 [13056.191975] worker_thread+0x296/0x6e0 [13056.194349] ? rescuer_thread+0x570/0x570 [13056.196554] kthread+0x1d1/0x200 [13056.199201] ? set_kthread_struct+0x70/0x70 [13056.201291] ret_from_fork+0x1f/0x30 [13076.054670] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [13076.055072] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [13076.063204] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13076.063284] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13076.063290] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13076.063293] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13076.063307] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13076.063312] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13076.063316] RAX: 0000000000006438 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13076.063319] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [13076.063321] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [13076.063323] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [13076.063326] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [13076.063330] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [13076.063342] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13076.063345] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [13076.063352] Call Trace: [13076.063357] [13076.063362] ? show_regs.cold.9+0x22/0x2f [13076.063371] ? watchdog_timer_fn.cold.6+0x62/0x10a [13076.063378] ? __hrtimer_run_queues+0x1c2/0x4d0 [13076.063382] ? watchdog+0x40/0x40 [13076.063386] ? hrtimer_interrupt+0x128/0x2c0 [13076.063391] ? smp_apic_timer_interrupt+0xa6/0x270 [13076.063399] ? apic_timer_interrupt+0xf/0x20 [13076.063404] [13076.063407] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13076.063412] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13076.063417] do_raw_spin_lock+0x177/0x190 [13076.063421] _raw_spin_lock+0x21/0x30 [13076.069036] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13076.069116] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13076.069124] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13076.069126] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13076.069141] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13076.069146] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13076.069150] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [13076.069153] RDX: 00000000000050c0 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [13076.069155] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [13076.069158] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [13076.069160] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13076.069164] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [13076.069167] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13076.069169] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13076.069177] Call Trace: [13076.069182] [13076.069187] ? show_regs.cold.9+0x22/0x2f [13076.069198] ? watchdog_timer_fn.cold.6+0x62/0x10a [13076.069205] ? __hrtimer_run_queues+0x1c2/0x4d0 [13076.069209] ? watchdog+0x40/0x40 [13076.069214] ? hrtimer_interrupt+0x128/0x2c0 [13076.069219] ? smp_apic_timer_interrupt+0xa6/0x270 [13076.069228] ? apic_timer_interrupt+0xf/0x20 [13076.069233] [13076.069236] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13076.069241] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13076.069439] do_raw_spin_lock+0x177/0x190 [13076.069445] _raw_spin_lock+0x21/0x30 [13076.069451] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13076.069727] obd_mod_free+0x46/0x140 [ptlrpc] [13076.069943] mdc_commit_open+0x91/0xb0 [mdc] [13076.069985] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13076.070196] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13076.116279] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13076.116608] ? do_raw_spin_unlock+0x75/0x190 [13076.116617] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13076.116852] ? do_wait_intr+0xf0/0xf0 [13076.116861] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13076.117169] mdc_reint+0x2d/0x110 [mdc] [13076.117224] mdc_unlink+0x658/0x870 [mdc] [13076.117417] lmv_unlink+0x34d/0xbd0 [lmv] [13076.117455] ll_unlink+0x24d/0x7f0 [lustre] [13076.117588] vfs_unlink+0x202/0x330 [13076.117598] do_unlinkat+0x2fa/0x460 [13076.117604] __x64_sys_unlink+0x37/0x50 [13076.117608] do_syscall_64+0xc1/0x440 [13076.117616] entry_SYSCALL_64_after_hwframe+0x49/0xae [13076.117738] RIP: 0033:0x7f151bdee10b [13076.117746] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [13076.117750] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [13076.117755] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [13076.117758] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [13076.117760] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13076.117762] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13076.117764] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13076.595722] after_reply+0xe2c/0x1780 [ptlrpc] [13076.608256] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13076.611676] ? __next_timer_interrupt+0x160/0x160 [13076.615550] ptlrpcd+0xa82/0xe40 [ptlrpc] [13076.617383] ? do_wait_intr+0xf0/0xf0 [13076.618651] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13076.621873] kthread+0x1d1/0x200 [13076.623116] ? set_kthread_struct+0x70/0x70 [13076.627175] ret_from_fork+0x1f/0x30 [13080.059856] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [13080.068940] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13080.144911] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13080.167546] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13080.179851] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13080.189697] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13080.207018] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13080.213531] RAX: 0000000000007f95 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13080.219009] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [13080.221928] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [13080.225567] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [13080.231869] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [13080.233606] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [13080.239968] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13080.244873] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [13080.250802] Call Trace: [13080.253558] [13080.254805] ? show_regs.cold.9+0x22/0x2f [13080.256826] ? watchdog_timer_fn.cold.6+0x62/0x10a [13080.261787] ? __hrtimer_run_queues+0x1c2/0x4d0 [13080.264954] ? watchdog+0x40/0x40 [13080.267700] ? hrtimer_interrupt+0x128/0x2c0 [13080.270835] ? smp_apic_timer_interrupt+0xa6/0x270 [13080.273678] ? apic_timer_interrupt+0xf/0x20 [13080.276732] [13080.277797] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13080.280259] do_raw_spin_lock+0x177/0x190 [13080.281767] _raw_spin_lock+0x21/0x30 [13080.283719] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13080.290825] ? do_raw_spin_unlock+0x75/0x190 [13080.294444] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13080.297703] ? do_wait_intr+0xf0/0xf0 [13080.299636] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13080.302705] mdc_close+0x53b/0xd70 [mdc] [13080.304814] lmv_close+0x18d/0x490 [lmv] [13080.306969] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13080.316618] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13080.325936] ll_md_real_close+0xa5/0x2a0 [lustre] [13080.328963] ll_md_close+0x274/0x800 [lustre] [13080.333169] ll_file_release+0x128/0xc50 [lustre] [13080.337657] __fput+0x10d/0x440 [13080.338828] ____fput+0x12/0x20 [13080.347824] task_work_run+0xc6/0x110 [13080.349969] exit_to_usermode_loop+0x1dd/0x1f0 [13080.356638] do_syscall_64+0x3d6/0x440 [13080.359571] entry_SYSCALL_64_after_hwframe+0x49/0xae [13080.363689] RIP: 0033:0x7f151bee3cd8 [13080.366807] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13080.379740] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13080.383915] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [13080.385888] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [13080.393449] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13080.397724] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13080.401655] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13081.569039] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11150s! [13081.574887] Showing busy workqueues and worker pools: [13081.578572] workqueue events: flags=0x0 [13081.581152] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13081.584495] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [13081.589853] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13081.593296] pending: kfree_rcu_monitor [13081.595454] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13081.598944] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [13081.603218] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13081.607610] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [13081.614242] workqueue events_long: flags=0x0 [13081.616330] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13081.619482] pending: lu_objects_flush [obdclass] [13081.621457] workqueue events_unbound: flags=0x2 [13081.622949] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13081.625796] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13081.631900] workqueue events_power_efficient: flags=0x80 [13081.634343] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13081.636976] pending: do_cache_clean, neigh_periodic_work, check_lifetime [13081.638966] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13081.640797] pending: neigh_periodic_work [13081.642163] workqueue mm_percpu_wq: flags=0x8 [13081.643418] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13081.645369] pending: vmstat_update, lru_add_drain_per_cpu [13081.647531] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13081.650247] pending: vmstat_update [13081.651338] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13081.654124] pending: lru_add_drain_per_cpu [13081.656265] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [13081.665499] pending: lru_add_drain_per_cpu BAR(48) [13081.668661] workqueue writeback: flags=0x4a [13081.669953] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [13081.672455] pending: wb_workfn [13081.673922] workqueue xprtiod: flags=0xa [13081.675711] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13081.678584] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [13081.682476] workqueue ipv6_addrconf: flags=0x40008 [13081.684636] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13081.687604] pending: addrconf_verify_work [13081.689729] workqueue knbd0-recv: flags=0x1a [13081.691677] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13081.694216] in-flight: 658:recv_work [13081.695978] workqueue ptlrpc_pinger: flags=0x2 [13081.697720] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13081.700291] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13081.702407] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11151s workers=4 idle: 32984 37 8 [13081.705491] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13066s workers=2 idle: 661 [13084.060076] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13084.092635] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13084.155714] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13084.182407] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13084.200910] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13084.215582] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13084.230883] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13084.243911] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13084.248020] RAX: 0000000000004c56 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13084.252038] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [13084.259014] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [13084.262858] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [13084.267636] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [13084.273055] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [13084.284752] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13084.288708] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13084.291779] Call Trace: [13084.292893] [13084.293872] ? show_regs.cold.9+0x22/0x2f [13084.295681] ? watchdog_timer_fn.cold.6+0x62/0x10a [13084.297732] ? __hrtimer_run_queues+0x1c2/0x4d0 [13084.299787] ? watchdog+0x40/0x40 [13084.301323] ? hrtimer_interrupt+0x128/0x2c0 [13084.303213] ? smp_apic_timer_interrupt+0xa6/0x270 [13084.310867] ? apic_timer_interrupt+0xf/0x20 [13084.317223] [13084.319786] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13084.322719] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13084.334456] do_raw_spin_lock+0x177/0x190 [13084.335787] _raw_spin_lock+0x21/0x30 [13084.337936] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13084.339925] process_one_work+0x2c8/0x700 [13084.344411] worker_thread+0x296/0x6e0 [13084.346600] ? rescuer_thread+0x570/0x570 [13084.348086] kthread+0x1d1/0x200 [13084.350988] ? set_kthread_struct+0x70/0x70 [13084.354857] ret_from_fork+0x1f/0x30 [13104.055019] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [13104.057219] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13104.072139] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13104.076112] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13104.078878] RIP: 0010:native_safe_halt+0x2e/0x50 [13104.080770] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13104.087560] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13104.090106] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [13104.092329] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [13104.095010] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [13104.097481] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [13104.099749] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13104.102251] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [13104.104710] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13104.106539] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13104.108773] Call Trace: [13104.109605] [13104.110303] ? show_regs.cold.9+0x22/0x2f [13104.111735] ? watchdog_timer_fn.cold.6+0x62/0x10a [13104.113472] ? __hrtimer_run_queues+0x1c2/0x4d0 [13104.114907] ? watchdog+0x40/0x40 [13104.116107] ? hrtimer_interrupt+0x128/0x2c0 [13104.117467] ? smp_apic_timer_interrupt+0xa6/0x270 [13104.118997] ? apic_timer_interrupt+0xf/0x20 [13104.120483] [13104.121206] ? native_safe_halt+0x2e/0x50 [13104.122728] kvm_wait+0xcd/0xe0 [13104.123771] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13104.125535] do_raw_spin_lock+0x177/0x190 [13104.126825] _raw_spin_lock+0x21/0x30 [13104.128142] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13104.130163] obd_mod_free+0x46/0x140 [ptlrpc] [13104.131916] mdc_commit_open+0x91/0xb0 [mdc] [13104.133566] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13104.135662] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13104.137670] after_reply+0xe2c/0x1780 [ptlrpc] [13104.139552] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13104.141591] ? __next_timer_interrupt+0x160/0x160 [13104.143454] ptlrpcd+0xa82/0xe40 [ptlrpc] [13104.145137] ? do_wait_intr+0xf0/0xf0 [13104.146453] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13104.148485] kthread+0x1d1/0x200 [13104.149665] ? set_kthread_struct+0x70/0x70 [13104.151187] ret_from_fork+0x1f/0x30 [13108.057209] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [13108.060978] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [13108.067402] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13108.067477] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13108.067484] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13108.067487] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13108.067680] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13108.067686] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13108.067691] RAX: 0000000000006f46 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13108.067693] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [13108.067695] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [13108.067697] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [13108.067699] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [13108.067703] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [13108.067705] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13108.067708] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [13108.067715] Call Trace: [13108.067729] [13108.067735] ? show_regs.cold.9+0x22/0x2f [13108.067744] ? watchdog_timer_fn.cold.6+0x62/0x10a [13108.067751] ? __hrtimer_run_queues+0x1c2/0x4d0 [13108.067755] ? watchdog+0x40/0x40 [13108.067759] ? hrtimer_interrupt+0x128/0x2c0 [13108.067763] ? smp_apic_timer_interrupt+0xa6/0x270 [13108.067773] ? apic_timer_interrupt+0xf/0x20 [13108.067777] [13108.067780] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13108.067788] do_raw_spin_lock+0x177/0x190 [13108.067793] _raw_spin_lock+0x21/0x30 [13108.067798] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13108.072698] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) [13108.113384] ? do_raw_spin_unlock+0x75/0x190 [13108.116332] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13108.116382] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13108.116388] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13108.116390] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13108.116400] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13108.116403] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13108.116407] RAX: 0000000000006f91 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13108.116408] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [13108.116410] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [13108.116411] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [13108.116413] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [13108.116416] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [13108.116417] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13108.116419] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [13108.116425] Call Trace: [13108.116428] [13108.116433] ? show_regs.cold.9+0x22/0x2f [13108.116440] ? watchdog_timer_fn.cold.6+0x62/0x10a [13108.116445] ? __hrtimer_run_queues+0x1c2/0x4d0 [13108.116447] ? watchdog+0x40/0x40 [13108.116450] ? hrtimer_interrupt+0x128/0x2c0 [13108.116452] ? smp_apic_timer_interrupt+0xa6/0x270 [13108.116458] ? apic_timer_interrupt+0xf/0x20 [13108.116461] [13108.116464] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13108.116467] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13108.116470] do_raw_spin_lock+0x177/0x190 [13108.116473] _raw_spin_lock+0x21/0x30 [13108.116477] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13108.116774] ? do_raw_spin_unlock+0x75/0x190 [13108.116778] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13108.116903] ? do_wait_intr+0xf0/0xf0 [13108.116908] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13108.117088] mdc_reint+0x2d/0x110 [mdc] [13108.117126] mdc_unlink+0x658/0x870 [mdc] [13108.117152] lmv_unlink+0x34d/0xbd0 [lmv] [13108.117187] ll_unlink+0x24d/0x7f0 [lustre] [13108.117285] vfs_unlink+0x202/0x330 [13108.117291] do_unlinkat+0x2fa/0x460 [13108.117295] __x64_sys_unlink+0x37/0x50 [13108.117297] do_syscall_64+0xc1/0x440 [13108.117302] entry_SYSCALL_64_after_hwframe+0x49/0xae [13108.117306] RIP: 0033:0x7f151bdee10b [13108.117312] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [13108.117314] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [13108.117318] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [13108.117320] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [13108.117321] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13108.117323] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13108.117324] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13108.434286] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13108.436670] ? do_wait_intr+0xf0/0xf0 [13108.438472] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13108.440821] mdc_close+0x53b/0xd70 [mdc] [13108.442748] lmv_close+0x18d/0x490 [lmv] [13108.444605] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13108.450986] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13108.454695] ll_md_real_close+0xa5/0x2a0 [lustre] [13108.463477] ll_md_close+0x274/0x800 [lustre] [13108.467710] ll_file_release+0x128/0xc50 [lustre] [13108.474289] __fput+0x10d/0x440 [13108.477288] ____fput+0x12/0x20 [13108.480127] task_work_run+0xc6/0x110 [13108.482405] exit_to_usermode_loop+0x1dd/0x1f0 [13108.484797] do_syscall_64+0x3d6/0x440 [13108.486957] entry_SYSCALL_64_after_hwframe+0x49/0xae [13108.490033] RIP: 0033:0x7f151bee3cd8 [13108.492025] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13108.502336] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13108.506057] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [13108.509572] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [13108.513337] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13108.516761] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13108.524937] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13111.776826] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11181s! [13111.785628] Showing busy workqueues and worker pools: [13111.790700] workqueue events: flags=0x0 [13111.794291] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13111.797605] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [13111.802863] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13111.808504] pending: kfree_rcu_monitor [13111.811285] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13111.814685] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [13111.819086] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13111.822525] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [13111.827476] workqueue events_long: flags=0x0 [13111.829082] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13111.833170] pending: lu_objects_flush [obdclass] [13111.836231] workqueue events_unbound: flags=0x2 [13111.838613] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13111.844572] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13111.857848] workqueue events_power_efficient: flags=0x80 [13111.863394] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13111.865664] pending: do_cache_clean, neigh_periodic_work, check_lifetime [13111.870643] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13111.873636] pending: neigh_periodic_work [13111.875678] workqueue mm_percpu_wq: flags=0x8 [13111.883628] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13111.891467] pending: vmstat_update, lru_add_drain_per_cpu [13111.895351] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13111.899691] pending: vmstat_update [13111.902546] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13111.911290] pending: lru_add_drain_per_cpu [13111.915499] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [13111.922764] pending: lru_add_drain_per_cpu BAR(48) [13111.925526] workqueue writeback: flags=0x4a [13111.931734] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [13111.936302] pending: wb_workfn [13111.938399] workqueue xprtiod: flags=0xa [13111.941741] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13111.948327] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [13111.953042] workqueue ipv6_addrconf: flags=0x40008 [13111.958169] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13111.963141] pending: addrconf_verify_work [13111.971549] workqueue knbd0-recv: flags=0x1a [13111.974660] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13111.983416] in-flight: 658:recv_work [13111.986140] workqueue ptlrpc_pinger: flags=0x2 [13111.990302] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13111.994510] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13112.000366] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11181s workers=4 idle: 32984 37 8 [13112.013280] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13096s workers=2 idle: 661 [13112.056198] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13112.059224] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13112.099478] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13112.105507] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13112.117892] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13112.120360] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13112.131382] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13112.144162] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13112.148298] RAX: 0000000000002ff7 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13112.152413] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [13112.156627] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [13112.160080] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [13112.165509] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [13112.169817] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [13112.174326] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13112.177325] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13112.189505] Call Trace: [13112.190183] [13112.190681] ? show_regs.cold.9+0x22/0x2f [13112.191657] ? watchdog_timer_fn.cold.6+0x62/0x10a [13112.197164] ? __hrtimer_run_queues+0x1c2/0x4d0 [13112.199529] ? watchdog+0x40/0x40 [13112.202829] ? hrtimer_interrupt+0x128/0x2c0 [13112.207154] ? smp_apic_timer_interrupt+0xa6/0x270 [13112.209483] ? apic_timer_interrupt+0xf/0x20 [13112.211310] [13112.213013] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13112.215646] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13112.220754] do_raw_spin_lock+0x177/0x190 [13112.222671] _raw_spin_lock+0x21/0x30 [13112.225930] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13112.227973] process_one_work+0x2c8/0x700 [13112.230924] worker_thread+0x296/0x6e0 [13112.233429] ? rescuer_thread+0x570/0x570 [13112.236197] kthread+0x1d1/0x200 [13112.239839] ? set_kthread_struct+0x70/0x70 [13112.242661] ret_from_fork+0x1f/0x30 [13132.055050] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [13132.057689] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13132.074480] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13132.078206] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13132.081128] RIP: 0010:native_safe_halt+0x2e/0x50 [13132.082860] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13132.089224] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13132.092299] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [13132.095140] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [13132.097598] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [13132.099721] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [13132.102273] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13132.104878] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [13132.107710] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13132.109501] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13132.112274] Call Trace: [13132.113239] [13132.114038] ? show_regs.cold.9+0x22/0x2f [13132.115521] ? watchdog_timer_fn.cold.6+0x62/0x10a [13132.117399] ? __hrtimer_run_queues+0x1c2/0x4d0 [13132.119247] ? watchdog+0x40/0x40 [13132.120221] ? hrtimer_interrupt+0x128/0x2c0 [13132.121825] ? smp_apic_timer_interrupt+0xa6/0x270 [13132.123431] ? apic_timer_interrupt+0xf/0x20 [13132.125383] [13132.126322] ? native_safe_halt+0x2e/0x50 [13132.127702] kvm_wait+0xcd/0xe0 [13132.129103] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13132.131376] do_raw_spin_lock+0x177/0x190 [13132.132940] _raw_spin_lock+0x21/0x30 [13132.134593] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13132.136758] obd_mod_free+0x46/0x140 [ptlrpc] [13132.139070] mdc_commit_open+0x91/0xb0 [mdc] [13132.140469] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13132.142416] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13132.144894] after_reply+0xe2c/0x1780 [ptlrpc] [13132.147308] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13132.149811] ? __next_timer_interrupt+0x160/0x160 [13132.151418] ptlrpcd+0xa82/0xe40 [ptlrpc] [13132.153390] ? do_wait_intr+0xf0/0xf0 [13132.155062] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13132.157450] kthread+0x1d1/0x200 [13132.158926] ? set_kthread_struct+0x70/0x70 [13132.160608] ret_from_fork+0x1f/0x30 [13136.052021] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [13136.054972] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13136.057791] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [13136.075182] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13136.075194] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13136.075197] RIP: 0010:native_safe_halt+0x2e/0x50 [13136.075209] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13136.075212] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13136.075217] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13136.075219] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [13136.075221] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [13136.075223] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [13136.075225] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [13136.075228] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [13136.075230] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13136.075232] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [13136.075240] Call Trace: [13136.075244] [13136.075248] ? show_regs.cold.9+0x22/0x2f [13136.075256] ? watchdog_timer_fn.cold.6+0x62/0x10a [13136.075262] ? __hrtimer_run_queues+0x1c2/0x4d0 [13136.075266] ? watchdog+0x40/0x40 [13136.075269] ? hrtimer_interrupt+0x128/0x2c0 [13136.075272] ? smp_apic_timer_interrupt+0xa6/0x270 [13136.075278] ? apic_timer_interrupt+0xf/0x20 [13136.075281] [13136.075284] ? native_safe_halt+0x2e/0x50 [13136.075287] kvm_wait+0xcd/0xe0 [13136.075293] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13136.075299] do_raw_spin_lock+0x177/0x190 [13136.075302] _raw_spin_lock+0x21/0x30 [13136.075308] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13136.075747] ? do_raw_spin_unlock+0x75/0x190 [13136.075762] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13136.080146] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata [13136.084256] ? do_wait_intr+0xf0/0xf0 [13136.084282] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13136.084583] mdc_reint+0x2d/0x110 [mdc] [13136.084636] mdc_unlink+0x658/0x870 [mdc] [13136.084663] lmv_unlink+0x34d/0xbd0 [lmv] [13136.084696] ll_unlink+0x24d/0x7f0 [lustre] [13136.084830] vfs_unlink+0x202/0x330 [13136.084839] do_unlinkat+0x2fa/0x460 [13136.084844] __x64_sys_unlink+0x37/0x50 [13136.084847] do_syscall_64+0xc1/0x440 [13136.084854] entry_SYSCALL_64_after_hwframe+0x49/0xae [13136.084862] RIP: 0033:0x7f151bdee10b [13136.084912] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [13136.084918] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [13136.084923] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [13136.084926] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [13136.084928] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13136.084930] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13136.084932] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13136.195045] dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13136.198300] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13136.202298] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13136.205551] RIP: 0010:native_safe_halt+0x2e/0x50 [13136.207869] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13136.215665] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13136.219049] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13136.221566] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [13136.224708] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [13136.227439] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [13136.229829] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [13136.232667] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [13136.235370] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13136.237745] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [13136.240399] Call Trace: [13136.241268] [13136.241808] ? show_regs.cold.9+0x22/0x2f [13136.243558] ? watchdog_timer_fn.cold.6+0x62/0x10a [13136.245592] ? __hrtimer_run_queues+0x1c2/0x4d0 [13136.247465] ? watchdog+0x40/0x40 [13136.248818] ? hrtimer_interrupt+0x128/0x2c0 [13136.250357] ? smp_apic_timer_interrupt+0xa6/0x270 [13136.254856] ? apic_timer_interrupt+0xf/0x20 [13136.258283] [13136.259072] ? native_safe_halt+0x2e/0x50 [13136.260645] kvm_wait+0xcd/0xe0 [13136.261827] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13136.263606] do_raw_spin_lock+0x177/0x190 [13136.265331] _raw_spin_lock+0x21/0x30 [13136.266592] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13136.269211] ? do_raw_spin_unlock+0x75/0x190 [13136.270970] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13136.272934] ? do_wait_intr+0xf0/0xf0 [13136.274279] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13136.276352] mdc_close+0x53b/0xd70 [mdc] [13136.278273] lmv_close+0x18d/0x490 [lmv] [13136.279712] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13136.281731] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13136.284170] ll_md_real_close+0xa5/0x2a0 [lustre] [13136.285898] ll_md_close+0x274/0x800 [lustre] [13136.287663] ll_file_release+0x128/0xc50 [lustre] [13136.289893] __fput+0x10d/0x440 [13136.290867] ____fput+0x12/0x20 [13136.292230] task_work_run+0xc6/0x110 [13136.293526] exit_to_usermode_loop+0x1dd/0x1f0 [13136.295667] do_syscall_64+0x3d6/0x440 [13136.296877] entry_SYSCALL_64_after_hwframe+0x49/0xae [13136.298429] RIP: 0033:0x7f151bee3cd8 [13136.300196] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13136.306964] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13136.309576] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [13136.311912] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [13136.315270] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13136.317497] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13136.320427] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13140.056508] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13140.058892] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13140.074804] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13140.079918] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13140.083091] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13140.086041] RIP: 0010:native_safe_halt+0x2e/0x50 [13140.087667] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13140.096242] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13140.099957] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13140.103284] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [13140.106287] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [13140.109792] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [13140.113003] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [13140.116636] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [13140.120214] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13140.122930] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13140.126257] Call Trace: [13140.127429] [13140.128503] ? show_regs.cold.9+0x22/0x2f [13140.130483] ? watchdog_timer_fn.cold.6+0x62/0x10a [13140.132510] ? __hrtimer_run_queues+0x1c2/0x4d0 [13140.134299] ? watchdog+0x40/0x40 [13140.135924] ? hrtimer_interrupt+0x128/0x2c0 [13140.137892] ? smp_apic_timer_interrupt+0xa6/0x270 [13140.139989] ? apic_timer_interrupt+0xf/0x20 [13140.141307] [13140.142013] ? native_safe_halt+0x2e/0x50 [13140.143567] kvm_wait+0xcd/0xe0 [13140.145015] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13140.147012] do_raw_spin_lock+0x177/0x190 [13140.148478] _raw_spin_lock+0x21/0x30 [13140.150028] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13140.152349] process_one_work+0x2c8/0x700 [13140.154016] worker_thread+0x296/0x6e0 [13140.155559] ? rescuer_thread+0x570/0x570 [13140.157169] kthread+0x1d1/0x200 [13140.159459] ? set_kthread_struct+0x70/0x70 [13140.161251] ret_from_fork+0x1f/0x30 [13142.496105] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11211s! [13142.499383] Showing busy workqueues and worker pools: [13142.501914] workqueue events: flags=0x0 [13142.503748] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13142.507079] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [13142.510431] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13142.514657] pending: kfree_rcu_monitor [13142.517243] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13142.520331] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [13142.525280] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13142.528913] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [13142.532847] workqueue events_long: flags=0x0 [13142.534976] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13142.538064] pending: lu_objects_flush [obdclass] [13142.540534] workqueue events_unbound: flags=0x2 [13142.542955] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13142.546228] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13142.551890] workqueue events_power_efficient: flags=0x80 [13142.554671] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13142.557881] pending: do_cache_clean, neigh_periodic_work, check_lifetime [13142.561097] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13142.565338] pending: neigh_periodic_work [13142.568318] workqueue mm_percpu_wq: flags=0x8 [13142.570269] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13142.574075] pending: vmstat_update, lru_add_drain_per_cpu [13142.576868] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13142.580193] pending: vmstat_update [13142.582053] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13142.588915] pending: lru_add_drain_per_cpu [13142.590955] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [13142.594045] pending: lru_add_drain_per_cpu BAR(48) [13142.597108] workqueue writeback: flags=0x4a [13142.599042] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [13142.602818] pending: wb_workfn [13142.604453] workqueue xprtiod: flags=0xa [13142.606881] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13142.609810] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [13142.614187] workqueue ipv6_addrconf: flags=0x40008 [13142.616130] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13142.620231] pending: addrconf_verify_work [13142.622414] workqueue knbd0-recv: flags=0x1a [13142.624704] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13142.628858] in-flight: 658:recv_work [13142.630710] workqueue ptlrpc_pinger: flags=0x2 [13142.632876] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13142.636037] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13142.639037] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11212s workers=4 idle: 32984 37 8 [13142.643203] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13127s workers=2 idle: 661 [13144.889545] rcu: INFO: rcu_sched self-detected stall on CPU [13144.893511] rcu: 1-...!: (6522148 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [13144.900207] (t=11220242 jiffies g=249657 q=5852) [13144.904136] rcu: rcu_sched kthread starved for 11215194 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [13144.916370] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [13144.923022] rcu: RCU grace-period kthread stack dump: [13144.925315] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [13144.934259] Call Trace: [13144.935035] __schedule+0x351/0xcb0 [13144.936393] schedule+0xc0/0x180 [13144.942880] schedule_timeout+0xb4/0x190 [13144.944304] ? do_raw_spin_unlock+0x75/0x190 [13144.955095] ? __next_timer_interrupt+0x160/0x160 [13144.956808] ? prepare_to_swait_event+0xf3/0x140 [13144.960946] rcu_gp_kthread+0x8dd/0x15b0 [13144.962417] ? force_qs_rnp+0x2c0/0x2c0 [13144.965477] kthread+0x1d1/0x200 [13144.972252] ? set_kthread_struct+0x70/0x70 [13144.976203] ret_from_fork+0x1f/0x30 [13144.978224] rcu: Stack dump where RCU GP kthread last ran: [13144.983433] Sending NMI from CPU 1 to CPUs 2: [13144.994872] NMI backtrace for cpu 2 skipped: idling at native_safe_halt+0x2e/0x50 [13144.996003] NMI backtrace for cpu 1 [13145.004376] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13145.021509] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13145.031835] Call Trace: [13145.033093] [13145.034150] ? dump_stack+0xbb/0x10e [13145.036409] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [13145.043061] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [13145.046234] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [13145.054336] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [13145.056301] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [13145.062387] ? rcu_dump_cpu_stacks+0x11b/0x171 [13145.063978] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [13145.075653] ? update_process_times+0x6d/0xc0 [13145.077030] ? tick_sched_handle.isra.18+0x27/0x90 [13145.083870] ? tick_sched_timer+0x53/0xb0 [13145.085253] ? __hrtimer_run_queues+0x1c2/0x4d0 [13145.091440] ? tick_sched_handle.isra.18+0x90/0x90 [13145.093211] ? hrtimer_interrupt+0x128/0x2c0 [13145.096205] ? smp_apic_timer_interrupt+0xa6/0x270 [13145.097826] ? apic_timer_interrupt+0xf/0x20 [13145.102431] [13145.103319] ? native_safe_halt+0x2e/0x50 [13145.105810] ? kvm_wait+0xcd/0xe0 [13145.107283] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13145.109214] ? do_raw_spin_lock+0x177/0x190 [13145.114660] ? _raw_spin_lock+0x21/0x30 [13145.116197] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13145.126111] ? obd_mod_free+0x46/0x140 [ptlrpc] [13145.127828] ? mdc_commit_open+0x91/0xb0 [mdc] [13145.133976] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13145.135919] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13145.148302] ? after_reply+0xe2c/0x1780 [ptlrpc] [13145.166368] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13145.168292] ? __next_timer_interrupt+0x160/0x160 [13145.177103] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [13145.181333] ? do_wait_intr+0xf0/0xf0 [13145.183440] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13145.188143] ? kthread+0x1d1/0x200 [13145.189647] ? set_kthread_struct+0x70/0x70 [13145.190999] ? ret_from_fork+0x1f/0x30 [13164.057689] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [13164.062809] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13164.075830] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13164.078949] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13164.082580] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13164.084935] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13164.092043] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13164.094695] RAX: 0000000000003414 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13164.097208] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [13164.100208] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [13164.103233] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [13164.105906] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [13164.108758] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [13164.111914] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13164.113955] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [13164.116570] Call Trace: [13164.117285] [13164.117897] ? show_regs.cold.9+0x22/0x2f [13164.119087] ? watchdog_timer_fn.cold.6+0x62/0x10a [13164.120698] ? __hrtimer_run_queues+0x1c2/0x4d0 [13164.122201] ? watchdog+0x40/0x40 [13164.123173] ? hrtimer_interrupt+0x128/0x2c0 [13164.124524] ? smp_apic_timer_interrupt+0xa6/0x270 [13164.125973] ? apic_timer_interrupt+0xf/0x20 [13164.127314] [13164.127934] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13164.130030] do_raw_spin_lock+0x177/0x190 [13164.131567] _raw_spin_lock+0x21/0x30 [13164.132919] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13164.135432] ? do_raw_spin_unlock+0x75/0x190 [13164.136999] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13164.138904] ? do_wait_intr+0xf0/0xf0 [13164.140020] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13164.145211] mdc_close+0x53b/0xd70 [mdc] [13164.146436] lmv_close+0x18d/0x490 [lmv] [13164.153315] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13164.155751] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13164.158651] ll_md_real_close+0xa5/0x2a0 [lustre] [13164.160497] ll_md_close+0x274/0x800 [lustre] [13164.162130] ll_file_release+0x128/0xc50 [lustre] [13164.164137] __fput+0x10d/0x440 [13164.165176] ____fput+0x12/0x20 [13164.166070] task_work_run+0xc6/0x110 [13164.167505] exit_to_usermode_loop+0x1dd/0x1f0 [13164.169246] do_syscall_64+0x3d6/0x440 [13164.170652] entry_SYSCALL_64_after_hwframe+0x49/0xae [13164.174321] RIP: 0033:0x7f151bee3cd8 [13164.179681] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13164.189762] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13164.192696] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [13164.195550] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [13164.198395] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13164.201230] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13164.204593] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13168.053588] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [13168.061673] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13168.088610] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13168.099422] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13168.105751] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13168.109790] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13168.121262] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13168.125853] RAX: 0000000000005d8d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13168.131566] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [13168.138413] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [13168.143564] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [13168.147466] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [13168.154134] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [13168.158231] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13168.160635] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [13168.165853] Call Trace: [13168.166846] [13168.167757] ? show_regs.cold.9+0x22/0x2f [13168.169835] ? watchdog_timer_fn.cold.6+0x62/0x10a [13168.173057] ? __hrtimer_run_queues+0x1c2/0x4d0 [13168.175117] ? watchdog+0x40/0x40 [13168.176782] ? hrtimer_interrupt+0x128/0x2c0 [13168.179128] ? smp_apic_timer_interrupt+0xa6/0x270 [13168.182154] ? apic_timer_interrupt+0xf/0x20 [13168.187587] [13168.188669] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13168.192324] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13168.194971] do_raw_spin_lock+0x177/0x190 [13168.200154] _raw_spin_lock+0x21/0x30 [13168.204234] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13168.208498] ? do_raw_spin_unlock+0x75/0x190 [13168.214045] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13168.220185] ? do_wait_intr+0xf0/0xf0 [13168.221803] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13168.225514] mdc_reint+0x2d/0x110 [mdc] [13168.227845] mdc_unlink+0x658/0x870 [mdc] [13168.234403] lmv_unlink+0x34d/0xbd0 [lmv] [13168.241743] ll_unlink+0x24d/0x7f0 [lustre] [13168.245559] vfs_unlink+0x202/0x330 [13168.252400] do_unlinkat+0x2fa/0x460 [13168.254744] __x64_sys_unlink+0x37/0x50 [13168.261806] do_syscall_64+0xc1/0x440 [13168.264254] entry_SYSCALL_64_after_hwframe+0x49/0xae [13168.271908] RIP: 0033:0x7f151bdee10b [13168.280549] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [13168.304281] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [13168.316498] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [13168.323668] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [13168.331725] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13168.338132] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13168.348189] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13172.057428] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [13172.063423] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13172.066045] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13172.066126] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13172.066134] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13172.066137] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13172.066151] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13172.066156] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13172.066160] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [13172.066163] RDX: 00000000000036f3 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [13172.066166] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [13172.066169] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [13172.066171] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13172.066174] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [13172.066177] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13172.066180] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13172.066190] Call Trace: [13172.066195] [13172.066201] ? show_regs.cold.9+0x22/0x2f [13172.066210] ? watchdog_timer_fn.cold.6+0x62/0x10a [13172.066217] ? __hrtimer_run_queues+0x1c2/0x4d0 [13172.066221] ? watchdog+0x40/0x40 [13172.066226] ? hrtimer_interrupt+0x128/0x2c0 [13172.066229] ? smp_apic_timer_interrupt+0xa6/0x270 [13172.066237] ? apic_timer_interrupt+0xf/0x20 [13172.066242] [13172.066245] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13172.066250] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13172.066255] do_raw_spin_lock+0x177/0x190 [13172.066259] _raw_spin_lock+0x21/0x30 [13172.066265] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13172.067482] obd_mod_free+0x46/0x140 [ptlrpc] [13172.072238] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13172.072321] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13172.072329] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13172.072335] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13172.072663] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13172.072676] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13172.072681] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13172.072686] RAX: 0000000000006b09 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13172.072688] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [13172.072690] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [13172.072693] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [13172.072695] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [13172.072699] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [13172.072702] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13172.072705] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13172.072716] Call Trace: [13172.072722] [13172.072729] ? show_regs.cold.9+0x22/0x2f [13172.072742] ? watchdog_timer_fn.cold.6+0x62/0x10a [13172.072749] ? __hrtimer_run_queues+0x1c2/0x4d0 [13172.072754] ? watchdog+0x40/0x40 [13172.072758] ? hrtimer_interrupt+0x128/0x2c0 [13172.072762] ? smp_apic_timer_interrupt+0xa6/0x270 [13172.072774] ? apic_timer_interrupt+0xf/0x20 [13172.072779] [13172.072782] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13172.072788] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13172.072793] do_raw_spin_lock+0x177/0x190 [13172.072798] _raw_spin_lock+0x21/0x30 [13172.072804] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13172.073011] process_one_work+0x2c8/0x700 [13172.073019] worker_thread+0x296/0x6e0 [13172.073023] ? rescuer_thread+0x570/0x570 [13172.073027] kthread+0x1d1/0x200 [13172.073032] ? set_kthread_struct+0x70/0x70 [13172.073035] ret_from_fork+0x1f/0x30 [13172.251982] mdc_commit_open+0x91/0xb0 [mdc] [13172.253645] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13172.255771] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13172.257693] after_reply+0xe2c/0x1780 [ptlrpc] [13172.259431] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13172.261614] ? __next_timer_interrupt+0x160/0x160 [13172.263231] ptlrpcd+0xa82/0xe40 [ptlrpc] [13172.264979] ? do_wait_intr+0xf0/0xf0 [13172.266503] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13172.269387] kthread+0x1d1/0x200 [13172.270583] ? set_kthread_struct+0x70/0x70 [13172.272108] ret_from_fork+0x1f/0x30 [13172.707359] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11242s! [13172.715611] Showing busy workqueues and worker pools: [13172.718984] workqueue events: flags=0x0 [13172.726821] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13172.733518] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [13172.743796] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13172.749081] pending: kfree_rcu_monitor [13172.758592] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13172.767840] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [13172.781981] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13172.795458] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [13172.803566] workqueue events_long: flags=0x0 [13172.808452] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13172.813474] pending: lu_objects_flush [obdclass] [13172.817868] workqueue events_unbound: flags=0x2 [13172.821989] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13172.826422] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13172.837119] workqueue events_power_efficient: flags=0x80 [13172.844515] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13172.847323] pending: do_cache_clean, neigh_periodic_work, check_lifetime [13172.858656] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13172.867391] pending: neigh_periodic_work [13172.869680] workqueue mm_percpu_wq: flags=0x8 [13172.872734] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13172.877066] pending: vmstat_update, lru_add_drain_per_cpu [13172.880440] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13172.883406] pending: vmstat_update [13172.885417] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13172.889114] pending: lru_add_drain_per_cpu [13172.891372] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [13172.894321] pending: lru_add_drain_per_cpu BAR(48) [13172.896641] workqueue writeback: flags=0x4a [13172.898024] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [13172.900320] pending: wb_workfn [13172.901765] workqueue xprtiod: flags=0xa [13172.903089] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13172.905577] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [13172.908839] workqueue ipv6_addrconf: flags=0x40008 [13172.911015] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13172.914109] pending: addrconf_verify_work [13172.916240] workqueue knbd0-recv: flags=0x1a [13172.918366] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13172.922019] in-flight: 658:recv_work [13172.923956] workqueue ptlrpc_pinger: flags=0x2 [13172.926514] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13172.929602] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13172.932458] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11242s workers=4 idle: 32984 37 8 [13172.935973] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13157s workers=2 idle: 661 [13192.057025] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [13192.062238] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13192.093143] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13192.101513] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13192.108013] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13192.113626] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13192.127955] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13192.133634] RAX: 0000000000004f33 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13192.139298] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [13192.143998] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [13192.149362] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [13192.154000] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [13192.158176] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [13192.165991] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13192.169387] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [13192.172850] Call Trace: [13192.174050] [13192.176328] ? show_regs.cold.9+0x22/0x2f [13192.178103] ? watchdog_timer_fn.cold.6+0x62/0x10a [13192.181266] ? __hrtimer_run_queues+0x1c2/0x4d0 [13192.184024] ? watchdog+0x40/0x40 [13192.185960] ? hrtimer_interrupt+0x128/0x2c0 [13192.189321] ? smp_apic_timer_interrupt+0xa6/0x270 [13192.192771] ? apic_timer_interrupt+0xf/0x20 [13192.195318] [13192.196591] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13192.200053] do_raw_spin_lock+0x177/0x190 [13192.202283] _raw_spin_lock+0x21/0x30 [13192.204683] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13192.207899] ? do_raw_spin_unlock+0x75/0x190 [13192.210357] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13192.213900] ? do_wait_intr+0xf0/0xf0 [13192.215760] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13192.218569] mdc_close+0x53b/0xd70 [mdc] [13192.220786] lmv_close+0x18d/0x490 [lmv] [13192.222848] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13192.226058] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13192.229913] ll_md_real_close+0xa5/0x2a0 [lustre] [13192.232320] ll_md_close+0x274/0x800 [lustre] [13192.235022] ll_file_release+0x128/0xc50 [lustre] [13192.237625] __fput+0x10d/0x440 [13192.238800] ____fput+0x12/0x20 [13192.240769] task_work_run+0xc6/0x110 [13192.242758] exit_to_usermode_loop+0x1dd/0x1f0 [13192.244939] do_syscall_64+0x3d6/0x440 [13192.246737] entry_SYSCALL_64_after_hwframe+0x49/0xae [13192.249087] RIP: 0033:0x7f151bee3cd8 [13192.250759] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13192.260799] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13192.265086] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [13192.269189] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [13192.273008] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13192.276706] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13192.281863] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13200.053162] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [13200.056018] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13200.061408] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [13200.061423] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13200.061505] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13200.062139] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13200.062143] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13200.062156] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13200.062160] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13200.062164] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [13200.062166] RDX: 0000000000005062 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [13200.062169] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [13200.062171] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [13200.062173] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13200.062176] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [13200.062179] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13200.062181] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13200.062190] Call Trace: [13200.062195] [13200.062200] ? show_regs.cold.9+0x22/0x2f [13200.062210] ? watchdog_timer_fn.cold.6+0x62/0x10a [13200.062216] ? __hrtimer_run_queues+0x1c2/0x4d0 [13200.062220] ? watchdog+0x40/0x40 [13200.062225] ? hrtimer_interrupt+0x128/0x2c0 [13200.062228] ? smp_apic_timer_interrupt+0xa6/0x270 [13200.062235] ? apic_timer_interrupt+0xf/0x20 [13200.062239] [13200.062241] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13200.062246] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13200.062250] do_raw_spin_lock+0x177/0x190 [13200.062255] _raw_spin_lock+0x21/0x30 [13200.062261] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13200.062547] obd_mod_free+0x46/0x140 [ptlrpc] [13200.065752] mdc_commit_open+0x91/0xb0 [mdc] [13200.065795] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13200.066117] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13200.066335] after_reply+0xe2c/0x1780 [ptlrpc] [13200.066528] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13200.066725] ? __next_timer_interrupt+0x160/0x160 [13200.066735] ptlrpcd+0xa82/0xe40 [ptlrpc] [13200.066937] ? do_wait_intr+0xf0/0xf0 [13200.067866] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13200.067946] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13200.067953] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13200.067955] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13200.067966] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13200.067971] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13200.067975] RAX: 0000000000001411 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13200.067978] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [13200.067980] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [13200.067983] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [13200.067985] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [13200.067988] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [13200.067992] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13200.067994] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [13200.068003] Call Trace: [13200.068007] [13200.068010] ? show_regs.cold.9+0x22/0x2f [13200.068020] ? watchdog_timer_fn.cold.6+0x62/0x10a [13200.068027] ? __hrtimer_run_queues+0x1c2/0x4d0 [13200.068031] ? watchdog+0x40/0x40 [13200.068036] ? hrtimer_interrupt+0x128/0x2c0 [13200.068040] ? smp_apic_timer_interrupt+0xa6/0x270 [13200.068049] ? apic_timer_interrupt+0xf/0x20 [13200.068054] [13200.068058] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13200.068063] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13200.068068] do_raw_spin_lock+0x177/0x190 [13200.068074] _raw_spin_lock+0x21/0x30 [13200.066947] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13200.068859] kthread+0x1d1/0x200 [13200.068867] ? set_kthread_struct+0x70/0x70 [13200.068871] ret_from_fork+0x1f/0x30 [13200.073635] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13200.073725] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13200.073732] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13200.073738] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13200.074199] RIP: 0010:native_safe_halt+0x2e/0x50 [13200.074209] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13200.074213] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13200.074219] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13200.074221] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [13200.074223] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [13200.074226] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [13200.074228] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [13200.074232] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [13200.074235] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13200.074237] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13200.074244] Call Trace: [13200.074249] [13200.074253] ? show_regs.cold.9+0x22/0x2f [13200.074262] ? watchdog_timer_fn.cold.6+0x62/0x10a [13200.074268] ? __hrtimer_run_queues+0x1c2/0x4d0 [13200.074272] ? watchdog+0x40/0x40 [13200.074277] ? hrtimer_interrupt+0x128/0x2c0 [13200.074281] ? smp_apic_timer_interrupt+0xa6/0x270 [13200.074287] ? apic_timer_interrupt+0xf/0x20 [13200.074296] [13200.074299] ? native_safe_halt+0x2e/0x50 [13200.074303] kvm_wait+0xcd/0xe0 [13200.074310] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13200.074317] do_raw_spin_lock+0x177/0x190 [13200.074321] _raw_spin_lock+0x21/0x30 [13200.074325] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13200.074637] process_one_work+0x2c8/0x700 [13200.074645] worker_thread+0x296/0x6e0 [13200.074650] ? rescuer_thread+0x570/0x570 [13200.074654] kthread+0x1d1/0x200 [13200.074659] ? set_kthread_struct+0x70/0x70 [13200.074663] ret_from_fork+0x1f/0x30 [13200.505178] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13200.506975] ? do_raw_spin_unlock+0x75/0x190 [13200.508517] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13200.510204] ? do_wait_intr+0xf0/0xf0 [13200.511748] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13200.516084] mdc_reint+0x2d/0x110 [mdc] [13200.520836] mdc_unlink+0x658/0x870 [mdc] [13200.522490] lmv_unlink+0x34d/0xbd0 [lmv] [13200.524697] ll_unlink+0x24d/0x7f0 [lustre] [13200.527525] vfs_unlink+0x202/0x330 [13200.528519] do_unlinkat+0x2fa/0x460 [13200.529653] __x64_sys_unlink+0x37/0x50 [13200.530838] do_syscall_64+0xc1/0x440 [13200.531932] entry_SYSCALL_64_after_hwframe+0x49/0xae [13200.533328] RIP: 0033:0x7f151bdee10b [13200.534514] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [13200.539667] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [13200.542147] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [13200.544064] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [13200.546113] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13200.548277] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13200.550499] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13203.426825] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11272s! [13203.444899] Showing busy workqueues and worker pools: [13203.449716] workqueue events: flags=0x0 [13203.450863] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13203.459852] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [13203.469383] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13203.473680] pending: kfree_rcu_monitor [13203.475584] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13203.478203] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [13203.486668] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13203.490983] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [13203.496431] workqueue events_long: flags=0x0 [13203.501667] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13203.505871] pending: lu_objects_flush [obdclass] [13203.511406] workqueue events_unbound: flags=0x2 [13203.516745] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13203.519945] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13203.532539] workqueue events_power_efficient: flags=0x80 [13203.537106] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13203.545752] pending: do_cache_clean, neigh_periodic_work, check_lifetime [13203.562539] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13203.569223] pending: neigh_periodic_work [13203.583671] workqueue mm_percpu_wq: flags=0x8 [13203.591916] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13203.601776] pending: vmstat_update, lru_add_drain_per_cpu [13203.608332] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13203.619764] pending: vmstat_update [13203.624886] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13203.632597] pending: lru_add_drain_per_cpu [13203.640774] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [13203.651368] pending: lru_add_drain_per_cpu BAR(48) [13203.660145] workqueue writeback: flags=0x4a [13203.663785] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [13203.674051] pending: wb_workfn [13203.681471] workqueue xprtiod: flags=0xa [13203.685654] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13203.695009] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [13203.703538] workqueue ipv6_addrconf: flags=0x40008 [13203.711819] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13203.717500] pending: addrconf_verify_work [13203.721778] workqueue knbd0-recv: flags=0x1a [13203.732661] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13203.736873] in-flight: 658:recv_work [13203.745281] workqueue ptlrpc_pinger: flags=0x2 [13203.751212] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13203.755177] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13203.758188] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11273s workers=4 idle: 32984 37 8 [13203.761423] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13188s workers=2 idle: 661 [13220.057186] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [13220.059409] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13220.113591] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13220.125198] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13220.131474] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13220.136024] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13220.155299] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13220.158215] RAX: 0000000000000112 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13220.161590] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [13220.164395] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [13220.167171] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [13220.171554] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [13220.178817] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [13220.185789] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13220.189287] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [13220.196603] Call Trace: [13220.197453] [13220.199428] ? show_regs.cold.9+0x22/0x2f [13220.205912] ? watchdog_timer_fn.cold.6+0x62/0x10a [13220.209808] ? __hrtimer_run_queues+0x1c2/0x4d0 [13220.211976] ? watchdog+0x40/0x40 [13220.213486] ? hrtimer_interrupt+0x128/0x2c0 [13220.215943] ? smp_apic_timer_interrupt+0xa6/0x270 [13220.219284] ? apic_timer_interrupt+0xf/0x20 [13220.220791] [13220.221605] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13220.224890] do_raw_spin_lock+0x177/0x190 [13220.228349] _raw_spin_lock+0x21/0x30 [13220.229745] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13220.232179] ? do_raw_spin_unlock+0x75/0x190 [13220.235201] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13220.237590] ? do_wait_intr+0xf0/0xf0 [13220.239110] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13220.241656] mdc_close+0x53b/0xd70 [mdc] [13220.244421] lmv_close+0x18d/0x490 [lmv] [13220.248322] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13220.251649] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13220.255370] ll_md_real_close+0xa5/0x2a0 [lustre] [13220.259093] ll_md_close+0x274/0x800 [lustre] [13220.261026] ll_file_release+0x128/0xc50 [lustre] [13220.264036] __fput+0x10d/0x440 [13220.267557] ____fput+0x12/0x20 [13220.271622] task_work_run+0xc6/0x110 [13220.272539] exit_to_usermode_loop+0x1dd/0x1f0 [13220.275093] do_syscall_64+0x3d6/0x440 [13220.276186] entry_SYSCALL_64_after_hwframe+0x49/0xae [13220.279633] RIP: 0033:0x7f151bee3cd8 [13220.281271] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13220.292352] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13220.296454] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [13220.308257] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [13220.312617] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13220.317779] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13220.320020] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13228.052036] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [13228.055142] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [13228.055605] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13228.055674] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13228.055680] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13228.055683] RIP: 0010:native_safe_halt+0x2e/0x50 [13228.055696] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13228.055700] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13228.055705] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13228.055707] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [13228.055709] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [13228.055711] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [13228.055714] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [13228.055717] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [13228.055720] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13228.055722] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [13228.055781] Call Trace: [13228.055787] [13228.055794] ? show_regs.cold.9+0x22/0x2f [13228.055804] ? watchdog_timer_fn.cold.6+0x62/0x10a [13228.055810] ? __hrtimer_run_queues+0x1c2/0x4d0 [13228.055815] ? watchdog+0x40/0x40 [13228.055819] ? hrtimer_interrupt+0x128/0x2c0 [13228.055823] ? smp_apic_timer_interrupt+0xa6/0x270 [13228.055829] ? apic_timer_interrupt+0xf/0x20 [13228.055834] [13228.055836] ? native_safe_halt+0x2e/0x50 [13228.055840] kvm_wait+0xcd/0xe0 [13228.055847] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13228.055945] do_raw_spin_lock+0x177/0x190 [13228.055949] _raw_spin_lock+0x21/0x30 [13228.056021] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13228.056026] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13228.056080] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13228.056083] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13228.056087] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13228.056325] RIP: 0010:native_safe_halt+0x2e/0x50 [13228.056329] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13228.056332] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [13228.056334] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13228.056335] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [13228.056337] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [13228.056338] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [13228.056340] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [13228.056342] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [13228.056344] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13228.056345] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13228.056351] Call Trace: [13228.056352] [13228.056354] ? show_regs.cold.9+0x22/0x2f [13228.056358] ? watchdog_timer_fn.cold.6+0x62/0x10a [13228.056361] ? __hrtimer_run_queues+0x1c2/0x4d0 [13228.056363] ? watchdog+0x40/0x40 [13228.056365] ? hrtimer_interrupt+0x128/0x2c0 [13228.056368] ? smp_apic_timer_interrupt+0xa6/0x270 [13228.056371] ? apic_timer_interrupt+0xf/0x20 [13228.056374] [13228.056376] ? native_safe_halt+0x2e/0x50 [13228.056378] kvm_wait+0xcd/0xe0 [13228.056382] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13228.056386] do_raw_spin_lock+0x177/0x190 [13228.056388] _raw_spin_lock+0x21/0x30 [13228.056391] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13228.056558] process_one_work+0x2c8/0x700 [13228.056566] worker_thread+0x296/0x6e0 [13228.056571] ? rescuer_thread+0x570/0x570 [13228.056575] kthread+0x1d1/0x200 [13228.056579] ? set_kthread_struct+0x70/0x70 [13228.056583] ret_from_fork+0x1f/0x30 [13228.055954] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13228.060342] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13228.060395] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13228.060400] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13228.060402] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13228.060412] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13228.060416] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13228.060419] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [13228.060421] RDX: 0000000000003df4 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [13228.060423] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [13228.060425] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [13228.060426] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13228.060429] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [13228.060431] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13228.060433] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13228.060439] Call Trace: [13228.060443] [13228.060446] ? show_regs.cold.9+0x22/0x2f [13228.060453] ? watchdog_timer_fn.cold.6+0x62/0x10a [13228.060457] ? __hrtimer_run_queues+0x1c2/0x4d0 [13228.060459] ? watchdog+0x40/0x40 [13228.060462] ? hrtimer_interrupt+0x128/0x2c0 [13228.060465] ? smp_apic_timer_interrupt+0xa6/0x270 [13228.060470] ? apic_timer_interrupt+0xf/0x20 [13228.060473] [13228.060475] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13228.060479] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13228.060484] do_raw_spin_lock+0x177/0x190 [13228.060489] _raw_spin_lock+0x21/0x30 [13228.060494] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13228.082524] ? do_raw_spin_unlock+0x75/0x190 [13228.082545] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13228.088275] obd_mod_free+0x46/0x140 [ptlrpc] [13228.088474] mdc_commit_open+0x91/0xb0 [mdc] [13228.088512] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13228.093714] ? do_wait_intr+0xf0/0xf0 [13228.093978] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13228.094336] mdc_reint+0x2d/0x110 [mdc] [13228.094370] mdc_unlink+0x658/0x870 [mdc] [13228.094389] lmv_unlink+0x34d/0xbd0 [lmv] [13228.094412] ll_unlink+0x24d/0x7f0 [lustre] [13228.096476] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13228.096800] after_reply+0xe2c/0x1780 [ptlrpc] [13228.097128] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13228.097367] ? __next_timer_interrupt+0x160/0x160 [13228.097374] ptlrpcd+0xa82/0xe40 [ptlrpc] [13228.097555] ? do_wait_intr+0xf0/0xf0 [13228.097569] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13228.098133] kthread+0x1d1/0x200 [13228.098138] ? set_kthread_struct+0x70/0x70 [13228.098142] ret_from_fork+0x1f/0x30 [13228.355618] vfs_unlink+0x202/0x330 [13228.356936] do_unlinkat+0x2fa/0x460 [13228.358099] __x64_sys_unlink+0x37/0x50 [13228.359217] do_syscall_64+0xc1/0x440 [13228.360425] entry_SYSCALL_64_after_hwframe+0x49/0xae [13228.362181] RIP: 0033:0x7f151bdee10b [13228.363417] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [13228.370564] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [13228.373408] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [13228.375708] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [13228.378287] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13228.380677] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13228.382982] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13234.144667] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11303s! [13234.151835] Showing busy workqueues and worker pools: [13234.159368] workqueue events: flags=0x0 [13234.164439] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13234.174317] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [13234.185406] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13234.196044] pending: kfree_rcu_monitor [13234.206051] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13234.220781] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [13234.240158] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13234.254189] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [13234.261440] workqueue events_long: flags=0x0 [13234.270750] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13234.272402] pending: lu_objects_flush [obdclass] [13234.273775] workqueue events_unbound: flags=0x2 [13234.288100] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13234.294294] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13234.301414] workqueue events_power_efficient: flags=0x80 [13234.311643] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13234.320425] pending: do_cache_clean, neigh_periodic_work, check_lifetime [13234.327029] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13234.332077] pending: neigh_periodic_work [13234.334868] workqueue mm_percpu_wq: flags=0x8 [13234.340683] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13234.354104] pending: vmstat_update, lru_add_drain_per_cpu [13234.364729] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13234.383486] pending: vmstat_update [13234.384943] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13234.388357] pending: lru_add_drain_per_cpu [13234.395011] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [13234.403114] pending: lru_add_drain_per_cpu BAR(48) [13234.408500] workqueue writeback: flags=0x4a [13234.411879] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [13234.418066] pending: wb_workfn [13234.422401] workqueue xprtiod: flags=0xa [13234.426160] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13234.432645] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [13234.439077] workqueue ipv6_addrconf: flags=0x40008 [13234.444083] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13234.448476] pending: addrconf_verify_work [13234.456614] workqueue knbd0-recv: flags=0x1a [13234.464207] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13234.471952] in-flight: 658:recv_work [13234.473719] workqueue ptlrpc_pinger: flags=0x2 [13234.475415] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13234.486990] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13234.493922] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11303s workers=4 idle: 32984 37 8 [13234.504034] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13219s workers=2 idle: 661 [13248.057309] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [13248.061303] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13248.087398] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13248.092176] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13248.096271] RIP: 0010:native_safe_halt+0x2e/0x50 [13248.098425] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13248.106371] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13248.109012] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13248.111466] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [13248.114213] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [13248.116655] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [13248.119439] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [13248.122168] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [13248.125081] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13248.126818] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [13248.128767] Call Trace: [13248.129400] [13248.130101] ? show_regs.cold.9+0x22/0x2f [13248.131724] ? watchdog_timer_fn.cold.6+0x62/0x10a [13248.133460] ? __hrtimer_run_queues+0x1c2/0x4d0 [13248.135230] ? watchdog+0x40/0x40 [13248.136738] ? hrtimer_interrupt+0x128/0x2c0 [13248.138269] ? smp_apic_timer_interrupt+0xa6/0x270 [13248.140016] ? apic_timer_interrupt+0xf/0x20 [13248.141733] [13248.142412] ? native_safe_halt+0x2e/0x50 [13248.144013] kvm_wait+0xcd/0xe0 [13248.145307] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13248.147321] do_raw_spin_lock+0x177/0x190 [13248.149056] _raw_spin_lock+0x21/0x30 [13248.150262] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13248.152526] ? do_raw_spin_unlock+0x75/0x190 [13248.153981] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13248.156090] ? do_wait_intr+0xf0/0xf0 [13248.157527] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13248.159821] mdc_close+0x53b/0xd70 [mdc] [13248.161318] lmv_close+0x18d/0x490 [lmv] [13248.162774] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13248.164735] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13248.167038] ll_md_real_close+0xa5/0x2a0 [lustre] [13248.168998] ll_md_close+0x274/0x800 [lustre] [13248.170878] ll_file_release+0x128/0xc50 [lustre] [13248.172704] __fput+0x10d/0x440 [13248.173978] ____fput+0x12/0x20 [13248.175236] task_work_run+0xc6/0x110 [13248.176442] exit_to_usermode_loop+0x1dd/0x1f0 [13248.178133] do_syscall_64+0x3d6/0x440 [13248.179716] entry_SYSCALL_64_after_hwframe+0x49/0xae [13248.181474] RIP: 0033:0x7f151bee3cd8 [13248.183095] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13248.190085] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13248.193464] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [13248.196800] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [13248.199978] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13248.203780] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13248.207151] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13256.055256] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [13256.056200] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13256.061017] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13256.061097] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13256.061105] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13256.061107] RIP: 0010:native_safe_halt+0x2e/0x50 [13256.061120] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13256.061125] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13256.061131] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [13256.061133] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [13256.061136] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [13256.061138] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [13256.061141] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13256.061145] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [13256.061148] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13256.061151] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13256.061160] Call Trace: [13256.061166] [13256.061171] ? show_regs.cold.9+0x22/0x2f [13256.061182] ? watchdog_timer_fn.cold.6+0x62/0x10a [13256.061189] ? __hrtimer_run_queues+0x1c2/0x4d0 [13256.061194] ? watchdog+0x40/0x40 [13256.061199] ? hrtimer_interrupt+0x128/0x2c0 [13256.061203] ? smp_apic_timer_interrupt+0xa6/0x270 [13256.061210] ? apic_timer_interrupt+0xf/0x20 [13256.061215] [13256.061219] ? native_safe_halt+0x2e/0x50 [13256.061223] kvm_wait+0xcd/0xe0 [13256.061230] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13256.061237] do_raw_spin_lock+0x177/0x190 [13256.061241] _raw_spin_lock+0x21/0x30 [13256.061247] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13256.067093] Modules linked in: [13256.083428] obd_mod_free+0x46/0x140 [ptlrpc] [13256.088194] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod [13256.091934] mdc_commit_open+0x91/0xb0 [mdc] [13256.094387] sha512_ssse3 [13256.101245] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13256.104703] sha512_generic [13256.107516] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13256.110987] [13256.113745] after_reply+0xe2c/0x1780 [ptlrpc] [13256.117212] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13256.120057] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13256.123870] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13256.126065] ? __next_timer_interrupt+0x160/0x160 [13256.129979] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13256.131003] ptlrpcd+0xa82/0xe40 [ptlrpc] [13256.131788] [13256.133295] ? do_wait_intr+0xf0/0xf0 [13256.133314] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13256.135731] RIP: 0010:native_safe_halt+0x2e/0x50 [13256.137415] kthread+0x1d1/0x200 [13256.139863] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13256.141359] ? set_kthread_struct+0x70/0x70 [13256.144071] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 [13256.145790] ret_from_fork+0x1f/0x30 [13256.146916] ORIG_RAX: ffffffffffffff13 [13256.226066] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13256.228689] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [13256.232051] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [13256.234834] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [13256.237657] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [13256.240912] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [13256.246463] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13256.252940] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13256.255665] Call Trace: [13256.256938] [13256.257673] ? show_regs.cold.9+0x22/0x2f [13256.258926] ? watchdog_timer_fn.cold.6+0x62/0x10a [13256.261316] ? __hrtimer_run_queues+0x1c2/0x4d0 [13256.263273] ? watchdog+0x40/0x40 [13256.264762] ? hrtimer_interrupt+0x128/0x2c0 [13256.266126] ? smp_apic_timer_interrupt+0xa6/0x270 [13256.268316] ? apic_timer_interrupt+0xf/0x20 [13256.269921] [13256.271032] ? native_safe_halt+0x2e/0x50 [13256.272498] kvm_wait+0xcd/0xe0 [13256.273795] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13256.276193] do_raw_spin_lock+0x177/0x190 [13256.277816] _raw_spin_lock+0x21/0x30 [13256.279426] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13256.281861] process_one_work+0x2c8/0x700 [13256.283862] worker_thread+0x296/0x6e0 [13256.285383] ? rescuer_thread+0x570/0x570 [13256.287374] kthread+0x1d1/0x200 [13256.289110] ? set_kthread_struct+0x70/0x70 [13256.291317] ret_from_fork+0x1f/0x30 [13260.052009] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [13260.059642] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13260.102728] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13260.115666] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13260.126718] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13260.134918] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13260.159583] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13260.164516] RAX: 00000000000011d8 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13260.171984] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [13260.180799] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [13260.188787] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [13260.198271] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [13260.202175] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [13260.212422] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13260.214976] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [13260.218938] Call Trace: [13260.220285] [13260.221647] ? show_regs.cold.9+0x22/0x2f [13260.223179] ? watchdog_timer_fn.cold.6+0x62/0x10a [13260.226973] ? __hrtimer_run_queues+0x1c2/0x4d0 [13260.233383] ? watchdog+0x40/0x40 [13260.238045] ? hrtimer_interrupt+0x128/0x2c0 [13260.239540] ? smp_apic_timer_interrupt+0xa6/0x270 [13260.249939] ? apic_timer_interrupt+0xf/0x20 [13260.253343] [13260.255388] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13260.261362] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13260.265806] do_raw_spin_lock+0x177/0x190 [13260.271290] _raw_spin_lock+0x21/0x30 [13260.272675] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13260.281315] ? do_raw_spin_unlock+0x75/0x190 [13260.284998] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13260.294921] ? do_wait_intr+0xf0/0xf0 [13260.302408] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13260.311943] mdc_reint+0x2d/0x110 [mdc] [13260.314495] mdc_unlink+0x658/0x870 [mdc] [13260.323197] lmv_unlink+0x34d/0xbd0 [lmv] [13260.329079] ll_unlink+0x24d/0x7f0 [lustre] [13260.340329] vfs_unlink+0x202/0x330 [13260.341621] do_unlinkat+0x2fa/0x460 [13260.345922] __x64_sys_unlink+0x37/0x50 [13260.351061] do_syscall_64+0xc1/0x440 [13260.356845] entry_SYSCALL_64_after_hwframe+0x49/0xae [13260.358894] RIP: 0033:0x7f151bdee10b [13260.364495] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [13260.378186] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [13260.387376] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [13260.397827] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [13260.401428] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13260.411210] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13260.423181] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13264.867263] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11334s! [13264.873908] Showing busy workqueues and worker pools: [13264.880385] workqueue events: flags=0x0 [13264.884225] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13264.892111] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [13264.906232] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13264.916255] pending: kfree_rcu_monitor [13264.923763] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13264.932672] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [13264.939827] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13264.942979] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [13264.948667] workqueue events_long: flags=0x0 [13264.951735] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13264.956371] pending: lu_objects_flush [obdclass] [13264.957555] workqueue events_unbound: flags=0x2 [13264.962966] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13264.966733] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13264.972109] workqueue events_power_efficient: flags=0x80 [13264.977076] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13264.983211] pending: do_cache_clean, neigh_periodic_work, check_lifetime [13264.989627] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13264.997709] pending: neigh_periodic_work [13265.001269] workqueue mm_percpu_wq: flags=0x8 [13265.006073] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13265.012636] pending: vmstat_update, lru_add_drain_per_cpu [13265.018616] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13265.027228] pending: vmstat_update [13265.029059] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13265.038324] pending: lru_add_drain_per_cpu [13265.039974] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [13265.046617] pending: lru_add_drain_per_cpu BAR(48) [13265.051176] workqueue writeback: flags=0x4a [13265.055262] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [13265.061596] pending: wb_workfn [13265.064229] workqueue xprtiod: flags=0xa [13265.069164] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13265.075138] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [13265.083929] workqueue ipv6_addrconf: flags=0x40008 [13265.088013] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13265.097104] pending: addrconf_verify_work [13265.103711] workqueue knbd0-recv: flags=0x1a [13265.108356] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13265.116792] in-flight: 658:recv_work [13265.118437] workqueue ptlrpc_pinger: flags=0x2 [13265.123065] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13265.126130] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13265.131022] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11334s workers=4 idle: 32984 37 8 [13265.141260] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13249s workers=2 idle: 661 [13276.060798] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [13276.063228] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13276.096909] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13276.111430] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13276.119905] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13276.124755] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13276.137743] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13276.145204] RAX: 0000000000006785 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13276.153084] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [13276.163801] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [13276.169657] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [13276.177648] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [13276.185817] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [13276.194364] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13276.200643] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [13276.205653] Call Trace: [13276.207800] [13276.210852] ? show_regs.cold.9+0x22/0x2f [13276.224159] ? watchdog_timer_fn.cold.6+0x62/0x10a [13276.237675] ? __hrtimer_run_queues+0x1c2/0x4d0 [13276.243439] ? watchdog+0x40/0x40 [13276.244694] ? hrtimer_interrupt+0x128/0x2c0 [13276.246263] ? smp_apic_timer_interrupt+0xa6/0x270 [13276.251822] ? apic_timer_interrupt+0xf/0x20 [13276.254641] [13276.255291] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13276.257291] do_raw_spin_lock+0x177/0x190 [13276.261053] _raw_spin_lock+0x21/0x30 [13276.262869] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13276.268836] ? do_raw_spin_unlock+0x75/0x190 [13276.270853] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13276.275886] ? do_wait_intr+0xf0/0xf0 [13276.278419] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13276.282299] mdc_close+0x53b/0xd70 [mdc] [13276.283782] lmv_close+0x18d/0x490 [lmv] [13276.285269] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13276.293093] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13276.299682] ll_md_real_close+0xa5/0x2a0 [lustre] [13276.305870] ll_md_close+0x274/0x800 [lustre] [13276.308380] ll_file_release+0x128/0xc50 [lustre] [13276.317845] __fput+0x10d/0x440 [13276.322868] ____fput+0x12/0x20 [13276.330813] task_work_run+0xc6/0x110 [13276.332130] exit_to_usermode_loop+0x1dd/0x1f0 [13276.333222] do_syscall_64+0x3d6/0x440 [13276.340648] entry_SYSCALL_64_after_hwframe+0x49/0xae [13276.345636] RIP: 0033:0x7f151bee3cd8 [13276.352178] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13276.384521] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13276.390512] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [13276.403869] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [13276.413913] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13276.418839] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13276.423601] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13284.055090] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [13284.056042] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13284.059028] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash [13284.061528] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13284.061611] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13284.061618] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13284.061627] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13284.061976] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13284.061986] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13284.061989] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13284.061993] RAX: 0000000000002b07 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13284.061995] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [13284.061997] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [13284.061999] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [13284.062001] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [13284.062004] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [13284.062006] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13284.062008] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13284.062015] Call Trace: [13284.062020] [13284.062024] ? show_regs.cold.9+0x22/0x2f [13284.062031] ? watchdog_timer_fn.cold.6+0x62/0x10a [13284.062036] ? __hrtimer_run_queues+0x1c2/0x4d0 [13284.062040] ? watchdog+0x40/0x40 [13284.062044] ? hrtimer_interrupt+0x128/0x2c0 [13284.062048] ? smp_apic_timer_interrupt+0xa6/0x270 [13284.062055] ? apic_timer_interrupt+0xf/0x20 [13284.062063] [13284.062066] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13284.062069] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13284.062074] do_raw_spin_lock+0x177/0x190 [13284.062078] _raw_spin_lock+0x21/0x30 [13284.062083] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13284.062292] process_one_work+0x2c8/0x700 [13284.062299] worker_thread+0x296/0x6e0 [13284.062303] ? rescuer_thread+0x570/0x570 [13284.062427] kthread+0x1d1/0x200 [13284.062434] ? set_kthread_struct+0x70/0x70 [13284.062437] ret_from_fork+0x1f/0x30 [13284.232972] dm_log dm_mod sha512_ssse3 sha512_generic [13284.235091] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13284.241132] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13284.245094] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13284.247898] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13284.263425] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13284.269250] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [13284.273270] RDX: 0000000000002a57 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [13284.278985] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [13284.285093] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [13284.289029] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13284.292279] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [13284.296782] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13284.299655] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13284.302836] Call Trace: [13284.303861] [13284.304944] ? show_regs.cold.9+0x22/0x2f [13284.306925] ? watchdog_timer_fn.cold.6+0x62/0x10a [13284.309294] ? __hrtimer_run_queues+0x1c2/0x4d0 [13284.311439] ? watchdog+0x40/0x40 [13284.313087] ? hrtimer_interrupt+0x128/0x2c0 [13284.315033] ? smp_apic_timer_interrupt+0xa6/0x270 [13284.317113] ? apic_timer_interrupt+0xf/0x20 [13284.319908] [13284.320880] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13284.323225] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13284.326068] do_raw_spin_lock+0x177/0x190 [13284.328099] _raw_spin_lock+0x21/0x30 [13284.330040] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13284.332701] obd_mod_free+0x46/0x140 [ptlrpc] [13284.335459] mdc_commit_open+0x91/0xb0 [mdc] [13284.337410] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13284.339919] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13284.342462] after_reply+0xe2c/0x1780 [ptlrpc] [13284.345091] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13284.347875] ? __next_timer_interrupt+0x160/0x160 [13284.349877] ptlrpcd+0xa82/0xe40 [ptlrpc] [13284.352024] ? do_wait_intr+0xf0/0xf0 [13284.353463] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13284.355867] kthread+0x1d1/0x200 [13284.357142] ? set_kthread_struct+0x70/0x70 [13284.358965] ret_from_fork+0x1f/0x30 [13292.052020] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [13292.053923] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13292.066202] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13292.068349] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13292.070824] RIP: 0010:native_safe_halt+0x2e/0x50 [13292.072622] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13292.078149] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13292.080301] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13292.082415] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [13292.084921] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [13292.087590] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [13292.090209] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [13292.092737] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [13292.095801] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13292.097940] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [13292.100814] Call Trace: [13292.102302] [13292.102893] ? show_regs.cold.9+0x22/0x2f [13292.104043] ? watchdog_timer_fn.cold.6+0x62/0x10a [13292.107378] ? __hrtimer_run_queues+0x1c2/0x4d0 [13292.108680] ? watchdog+0x40/0x40 [13292.109882] ? hrtimer_interrupt+0x128/0x2c0 [13292.111352] ? smp_apic_timer_interrupt+0xa6/0x270 [13292.112975] ? apic_timer_interrupt+0xf/0x20 [13292.114401] [13292.115158] ? native_safe_halt+0x2e/0x50 [13292.116445] kvm_wait+0xcd/0xe0 [13292.117532] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13292.119455] do_raw_spin_lock+0x177/0x190 [13292.120629] _raw_spin_lock+0x21/0x30 [13292.121747] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13292.124206] ? do_raw_spin_unlock+0x75/0x190 [13292.125621] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13292.127634] ? do_wait_intr+0xf0/0xf0 [13292.129311] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13292.131247] mdc_reint+0x2d/0x110 [mdc] [13292.132460] mdc_unlink+0x658/0x870 [mdc] [13292.134099] lmv_unlink+0x34d/0xbd0 [lmv] [13292.135749] ll_unlink+0x24d/0x7f0 [lustre] [13292.137315] vfs_unlink+0x202/0x330 [13292.138458] do_unlinkat+0x2fa/0x460 [13292.139550] __x64_sys_unlink+0x37/0x50 [13292.140789] do_syscall_64+0xc1/0x440 [13292.142541] entry_SYSCALL_64_after_hwframe+0x49/0xae [13292.144609] RIP: 0033:0x7f151bdee10b [13292.145941] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [13292.152074] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [13292.154872] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [13292.158328] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [13292.163197] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13292.167171] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13292.170374] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13295.589621] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11364s! [13295.600996] Showing busy workqueues and worker pools: [13295.606338] workqueue events: flags=0x0 [13295.608951] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13295.611688] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [13295.616221] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13295.626911] pending: kfree_rcu_monitor [13295.628818] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13295.635266] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [13295.650460] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13295.654373] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [13295.666562] workqueue events_long: flags=0x0 [13295.670342] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13295.679534] pending: lu_objects_flush [obdclass] [13295.683364] workqueue events_unbound: flags=0x2 [13295.686326] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13295.689708] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13295.695505] workqueue events_power_efficient: flags=0x80 [13295.697889] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13295.701172] pending: do_cache_clean, neigh_periodic_work, check_lifetime [13295.704364] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13295.707986] pending: neigh_periodic_work [13295.710699] workqueue mm_percpu_wq: flags=0x8 [13295.712313] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13295.715722] pending: vmstat_update, lru_add_drain_per_cpu [13295.718414] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13295.721630] pending: vmstat_update [13295.723306] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13295.731494] pending: lru_add_drain_per_cpu [13295.736295] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [13295.743305] pending: lru_add_drain_per_cpu BAR(48) [13295.749180] workqueue writeback: flags=0x4a [13295.754304] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [13295.764613] pending: wb_workfn [13295.765992] workqueue xprtiod: flags=0xa [13295.773049] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13295.776581] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [13295.780484] workqueue ipv6_addrconf: flags=0x40008 [13295.785777] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13295.789815] pending: addrconf_verify_work [13295.792049] workqueue knbd0-recv: flags=0x1a [13295.793804] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13295.802436] in-flight: 658:recv_work [13295.807835] workqueue ptlrpc_pinger: flags=0x2 [13295.810105] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13295.816503] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13295.822237] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11365s workers=4 idle: 32984 37 8 [13295.824384] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13280s workers=2 idle: 661 [13304.058368] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [13304.069856] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13304.140780] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13304.155679] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13304.169893] RIP: 0010:native_safe_halt+0x2e/0x50 [13304.172461] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13304.202666] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [13304.211654] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13304.219596] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [13304.228539] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [13304.238236] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [13304.242401] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [13304.250666] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [13304.267762] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13304.272528] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [13304.282260] Call Trace: [13304.284563] [13304.287591] ? show_regs.cold.9+0x22/0x2f [13304.292440] ? watchdog_timer_fn.cold.6+0x62/0x10a [13304.296973] ? __hrtimer_run_queues+0x1c2/0x4d0 [13304.303759] ? watchdog+0x40/0x40 [13304.305761] ? hrtimer_interrupt+0x128/0x2c0 [13304.309974] ? smp_apic_timer_interrupt+0xa6/0x270 [13304.317715] ? apic_timer_interrupt+0xf/0x20 [13304.324275] [13304.329861] ? native_safe_halt+0x2e/0x50 [13304.340140] kvm_wait+0xcd/0xe0 [13304.347436] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13304.355793] do_raw_spin_lock+0x177/0x190 [13304.363266] _raw_spin_lock+0x21/0x30 [13304.365449] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13304.373714] ? do_raw_spin_unlock+0x75/0x190 [13304.379354] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13304.386457] ? do_wait_intr+0xf0/0xf0 [13304.390201] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13304.398444] mdc_close+0x53b/0xd70 [mdc] [13304.402290] lmv_close+0x18d/0x490 [lmv] [13304.408392] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13304.416398] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13304.425629] ll_md_real_close+0xa5/0x2a0 [lustre] [13304.435177] ll_md_close+0x274/0x800 [lustre] [13304.440537] ll_file_release+0x128/0xc50 [lustre] [13304.447763] __fput+0x10d/0x440 [13304.453299] ____fput+0x12/0x20 [13304.454466] task_work_run+0xc6/0x110 [13304.455826] exit_to_usermode_loop+0x1dd/0x1f0 [13304.462697] do_syscall_64+0x3d6/0x440 [13304.467433] entry_SYSCALL_64_after_hwframe+0x49/0xae [13304.473652] RIP: 0033:0x7f151bee3cd8 [13304.478794] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13304.497897] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13304.502915] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [13304.513249] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [13304.517633] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13304.526856] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13304.536809] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13312.056568] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13312.061413] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13312.063953] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [13312.082570] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13312.082584] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13312.082589] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13312.083136] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13312.083148] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13312.083152] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13312.083157] RAX: 0000000000007c1a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13312.083159] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [13312.083161] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [13312.083164] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [13312.083166] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [13312.083169] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [13312.083173] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13312.083175] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13312.083183] Call Trace: [13312.083189] [13312.083195] ? show_regs.cold.9+0x22/0x2f [13312.083204] ? watchdog_timer_fn.cold.6+0x62/0x10a [13312.083211] ? __hrtimer_run_queues+0x1c2/0x4d0 [13312.083215] ? watchdog+0x40/0x40 [13312.083219] ? hrtimer_interrupt+0x128/0x2c0 [13312.083223] ? smp_apic_timer_interrupt+0xa6/0x270 [13312.083232] ? apic_timer_interrupt+0xf/0x20 [13312.083237] [13312.083240] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13312.083245] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13312.083250] do_raw_spin_lock+0x177/0x190 [13312.083255] _raw_spin_lock+0x21/0x30 [13312.083260] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13312.090777] Modules linked in: lustre(O) osp(O) [13312.100971] process_one_work+0x2c8/0x700 [13312.100993] worker_thread+0x296/0x6e0 [13312.100998] ? rescuer_thread+0x570/0x570 [13312.101002] kthread+0x1d1/0x200 [13312.101008] ? set_kthread_struct+0x70/0x70 [13312.101012] ret_from_fork+0x1f/0x30 [13312.232767] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13312.261625] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13312.266580] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13312.271601] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13312.276923] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13312.292159] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13312.301124] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [13312.306896] RDX: 00000000000079ac RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [13312.312409] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [13312.318450] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [13312.321593] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13312.325219] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [13312.329480] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13312.332404] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13312.335829] Call Trace: [13312.338791] [13312.340064] ? show_regs.cold.9+0x22/0x2f [13312.346901] ? watchdog_timer_fn.cold.6+0x62/0x10a [13312.351701] ? __hrtimer_run_queues+0x1c2/0x4d0 [13312.355731] ? watchdog+0x40/0x40 [13312.358200] ? hrtimer_interrupt+0x128/0x2c0 [13312.361557] ? smp_apic_timer_interrupt+0xa6/0x270 [13312.364032] ? apic_timer_interrupt+0xf/0x20 [13312.365272] [13312.366127] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13312.367733] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13312.369727] do_raw_spin_lock+0x177/0x190 [13312.377573] _raw_spin_lock+0x21/0x30 [13312.379057] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13312.383339] obd_mod_free+0x46/0x140 [ptlrpc] [13312.387224] mdc_commit_open+0x91/0xb0 [mdc] [13312.389397] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13312.391793] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13312.393703] after_reply+0xe2c/0x1780 [ptlrpc] [13312.395872] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13312.398291] ? __next_timer_interrupt+0x160/0x160 [13312.404658] ptlrpcd+0xa82/0xe40 [ptlrpc] [13312.407111] ? do_wait_intr+0xf0/0xf0 [13312.409362] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13312.412100] kthread+0x1d1/0x200 [13312.413519] ? set_kthread_struct+0x70/0x70 [13312.419949] ret_from_fork+0x1f/0x30 [13320.052025] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [13320.055932] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13320.102760] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13320.109482] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13320.116875] RIP: 0010:native_safe_halt+0x2e/0x50 [13320.118907] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13320.132977] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13320.138594] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13320.142903] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [13320.150502] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [13320.157902] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [13320.162377] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [13320.169772] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [13320.174457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13320.181134] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [13320.190056] Call Trace: [13320.191310] [13320.192134] ? show_regs.cold.9+0x22/0x2f [13320.194095] ? watchdog_timer_fn.cold.6+0x62/0x10a [13320.196955] ? __hrtimer_run_queues+0x1c2/0x4d0 [13320.200107] ? watchdog+0x40/0x40 [13320.201314] ? hrtimer_interrupt+0x128/0x2c0 [13320.203251] ? smp_apic_timer_interrupt+0xa6/0x270 [13320.209074] ? apic_timer_interrupt+0xf/0x20 [13320.212029] [13320.213102] ? native_safe_halt+0x2e/0x50 [13320.215507] kvm_wait+0xcd/0xe0 [13320.221000] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13320.223650] do_raw_spin_lock+0x177/0x190 [13320.228012] _raw_spin_lock+0x21/0x30 [13320.230445] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13320.235918] ? do_raw_spin_unlock+0x75/0x190 [13320.238956] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13320.243068] ? do_wait_intr+0xf0/0xf0 [13320.247750] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13320.253961] mdc_reint+0x2d/0x110 [mdc] [13320.257100] mdc_unlink+0x658/0x870 [mdc] [13320.260027] lmv_unlink+0x34d/0xbd0 [lmv] [13320.267003] ll_unlink+0x24d/0x7f0 [lustre] [13320.269437] vfs_unlink+0x202/0x330 [13320.271165] do_unlinkat+0x2fa/0x460 [13320.272773] __x64_sys_unlink+0x37/0x50 [13320.275067] do_syscall_64+0xc1/0x440 [13320.278598] entry_SYSCALL_64_after_hwframe+0x49/0xae [13320.281622] RIP: 0033:0x7f151bdee10b [13320.283397] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [13320.298041] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [13320.303997] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [13320.308722] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [13320.312900] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13320.318620] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13320.323328] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13324.893971] rcu: INFO: rcu_sched self-detected stall on CPU [13324.899674] rcu: 1-...!: (6638975 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [13324.911396] (t=11400246 jiffies g=249657 q=5920) [13324.920382] rcu: rcu_sched kthread starved for 11395198 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [13324.930741] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [13324.938445] rcu: RCU grace-period kthread stack dump: [13324.943637] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [13324.953257] Call Trace: [13324.954408] __schedule+0x351/0xcb0 [13324.957195] schedule+0xc0/0x180 [13324.958608] schedule_timeout+0xb4/0x190 [13324.963286] ? do_raw_spin_unlock+0x75/0x190 [13324.967518] ? __next_timer_interrupt+0x160/0x160 [13324.971715] ? prepare_to_swait_event+0xf3/0x140 [13324.978344] rcu_gp_kthread+0x8dd/0x15b0 [13324.982597] ? force_qs_rnp+0x2c0/0x2c0 [13324.985711] kthread+0x1d1/0x200 [13324.987422] ? set_kthread_struct+0x70/0x70 [13324.992134] ret_from_fork+0x1f/0x30 [13324.994258] rcu: Stack dump where RCU GP kthread last ran: [13324.996764] Sending NMI from CPU 1 to CPUs 3: [13325.005980] NMI backtrace for cpu 3 [13325.005991] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13325.005994] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13325.005996] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13325.005997] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13325.005999] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 [13325.006002] RAX: 0000000000006b0c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13325.006007] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [13325.006008] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [13325.006010] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [13325.006011] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [13325.006012] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [13325.006013] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13325.006014] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [13325.006015] Call Trace: [13325.006016] [13325.006017] ? show_regs.cold.9+0x22/0x2f [13325.006018] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [13325.006019] ? nmi_cpu_backtrace_handler+0x15/0x40 [13325.006020] ? nmi_handle+0xbd/0x210 [13325.006021] ? default_do_nmi+0x4b/0x1b0 [13325.006022] ? do_nmi+0x280/0x380 [13325.006023] ? end_repeat_nmi+0x16/0x1f [13325.006024] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13325.006025] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13325.006026] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13325.006027] [13325.006028] do_raw_spin_lock+0x177/0x190 [13325.006029] _raw_spin_lock+0x21/0x30 [13325.006030] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13325.006034] ? do_raw_spin_unlock+0x75/0x190 [13325.006036] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13325.006036] ? do_wait_intr+0xf0/0xf0 [13325.006038] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13325.006039] mdc_close+0x53b/0xd70 [mdc] [13325.006040] lmv_close+0x18d/0x490 [lmv] [13325.006041] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13325.006042] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13325.006043] ll_md_real_close+0xa5/0x2a0 [lustre] [13325.006044] ll_md_close+0x274/0x800 [lustre] [13325.006046] ll_file_release+0x128/0xc50 [lustre] [13325.006047] __fput+0x10d/0x440 [13325.006048] ____fput+0x12/0x20 [13325.006048] task_work_run+0xc6/0x110 [13325.006050] exit_to_usermode_loop+0x1dd/0x1f0 [13325.006051] do_syscall_64+0x3d6/0x440 [13325.006052] entry_SYSCALL_64_after_hwframe+0x49/0xae [13325.006053] RIP: 0033:0x7f151bee3cd8 [13325.006058] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13325.006060] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13325.006062] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [13325.006063] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [13325.006064] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13325.006065] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13325.006066] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13325.008432] NMI backtrace for cpu 1 [13325.207255] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13325.214636] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13325.220026] Call Trace: [13325.221536] [13325.223041] ? dump_stack+0xbb/0x10e [13325.224985] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [13325.227659] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [13325.230243] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [13325.232587] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [13325.235286] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [13325.238596] ? rcu_dump_cpu_stacks+0x11b/0x171 [13325.241011] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [13325.243380] ? update_process_times+0x6d/0xc0 [13325.245457] ? tick_sched_handle.isra.18+0x27/0x90 [13325.248119] ? tick_sched_timer+0x53/0xb0 [13325.255178] ? __hrtimer_run_queues+0x1c2/0x4d0 [13325.259534] ? tick_sched_handle.isra.18+0x90/0x90 [13325.264441] ? hrtimer_interrupt+0x128/0x2c0 [13325.266006] ? smp_apic_timer_interrupt+0xa6/0x270 [13325.267789] ? apic_timer_interrupt+0xf/0x20 [13325.275280] [13325.276666] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13325.281010] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13325.286262] ? do_raw_spin_lock+0x177/0x190 [13325.290133] ? _raw_spin_lock+0x21/0x30 [13325.292695] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13325.296455] ? obd_mod_free+0x46/0x140 [ptlrpc] [13325.300129] ? mdc_commit_open+0x91/0xb0 [mdc] [13325.302661] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13325.307133] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13325.311129] ? after_reply+0xe2c/0x1780 [ptlrpc] [13325.314270] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13325.317445] ? __next_timer_interrupt+0x160/0x160 [13325.320249] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [13325.323116] ? do_wait_intr+0xf0/0xf0 [13325.332560] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13325.337136] ? kthread+0x1d1/0x200 [13325.343418] ? set_kthread_struct+0x70/0x70 [13325.351453] ? ret_from_fork+0x1f/0x30 [13326.307351] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11395s! [13326.320033] Showing busy workqueues and worker pools: [13326.324050] workqueue events: flags=0x0 [13326.328268] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13326.333469] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [13326.343318] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13326.354401] pending: kfree_rcu_monitor [13326.356635] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13326.360494] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [13326.365761] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13326.369690] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [13326.373449] workqueue events_long: flags=0x0 [13326.377485] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13326.385359] pending: lu_objects_flush [obdclass] [13326.390684] workqueue events_unbound: flags=0x2 [13326.393817] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13326.397085] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13326.403372] workqueue events_power_efficient: flags=0x80 [13326.406989] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13326.415591] pending: do_cache_clean, neigh_periodic_work, check_lifetime [13326.429318] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13326.435927] pending: neigh_periodic_work [13326.439537] workqueue mm_percpu_wq: flags=0x8 [13326.442421] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13326.448085] pending: vmstat_update, lru_add_drain_per_cpu [13326.451636] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13326.456521] pending: vmstat_update [13326.459361] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13326.463510] pending: lru_add_drain_per_cpu [13326.468306] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [13326.474519] pending: lru_add_drain_per_cpu BAR(48) [13326.479403] workqueue writeback: flags=0x4a [13326.482394] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [13326.487172] pending: wb_workfn [13326.489494] workqueue xprtiod: flags=0xa [13326.491744] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13326.496777] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [13326.508628] workqueue ipv6_addrconf: flags=0x40008 [13326.515868] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13326.522942] pending: addrconf_verify_work [13326.530254] workqueue knbd0-recv: flags=0x1a [13326.533216] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13326.541766] in-flight: 658:recv_work [13326.547634] workqueue ptlrpc_pinger: flags=0x2 [13326.549323] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13326.553667] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13326.560884] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11395s workers=4 idle: 32984 37 8 [13326.568444] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13311s workers=2 idle: 661 [13340.056149] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13340.058749] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13340.097707] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13340.104633] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13340.108701] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13340.115697] RIP: 0010:native_safe_halt+0x2e/0x50 [13340.120239] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13340.133216] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13340.143016] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13340.154728] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [13340.164915] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [13340.170924] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [13340.186316] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [13340.196761] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [13340.201558] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13340.205723] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13340.208958] Call Trace: [13340.211890] [13340.213096] ? show_regs.cold.9+0x22/0x2f [13340.218118] ? watchdog_timer_fn.cold.6+0x62/0x10a [13340.226789] ? __hrtimer_run_queues+0x1c2/0x4d0 [13340.232322] ? watchdog+0x40/0x40 [13340.239686] ? hrtimer_interrupt+0x128/0x2c0 [13340.243583] ? smp_apic_timer_interrupt+0xa6/0x270 [13340.250243] ? apic_timer_interrupt+0xf/0x20 [13340.253218] [13340.253881] ? native_safe_halt+0x2e/0x50 [13340.257118] kvm_wait+0xcd/0xe0 [13340.258386] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13340.262807] do_raw_spin_lock+0x177/0x190 [13340.264333] _raw_spin_lock+0x21/0x30 [13340.265463] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13340.270855] process_one_work+0x2c8/0x700 [13340.275469] worker_thread+0x296/0x6e0 [13340.278756] ? rescuer_thread+0x570/0x570 [13340.280822] kthread+0x1d1/0x200 [13340.281822] ? set_kthread_struct+0x70/0x70 [13340.284950] ret_from_fork+0x1f/0x30 [13352.053888] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [13352.055252] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [13352.058386] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) [13352.058959] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [13352.058966] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13352.059033] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13352.059039] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13352.059042] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13352.059056] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13352.059060] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13352.059065] RAX: 0000000000001f11 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13352.059067] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [13352.059069] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [13352.059071] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [13352.059074] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [13352.059077] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [13352.059080] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13352.059083] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [13352.059092] Call Trace: [13352.059098] [13352.059225] obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13352.059258] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13352.059263] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13352.059266] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13352.059275] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13352.059279] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13352.059283] RAX: 0000000000003842 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13352.059286] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [13352.059288] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [13352.059291] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [13352.059293] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [13352.059296] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [13352.059298] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13352.059301] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [13352.059308] Call Trace: [13352.059311] [13352.059314] ? show_regs.cold.9+0x22/0x2f [13352.059324] ? watchdog_timer_fn.cold.6+0x62/0x10a [13352.059330] ? __hrtimer_run_queues+0x1c2/0x4d0 [13352.059335] ? watchdog+0x40/0x40 [13352.059339] ? hrtimer_interrupt+0x128/0x2c0 [13352.059344] ? smp_apic_timer_interrupt+0xa6/0x270 [13352.059353] ? apic_timer_interrupt+0xf/0x20 [13352.059358] [13352.059361] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13352.059365] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13352.059369] do_raw_spin_lock+0x177/0x190 [13352.059374] _raw_spin_lock+0x21/0x30 [13352.059380] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13352.059821] ? do_raw_spin_unlock+0x75/0x190 [13352.059827] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13352.060025] ? do_wait_intr+0xf0/0xf0 [13352.060034] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13352.060227] mdc_reint+0x2d/0x110 [mdc] [13352.060265] mdc_unlink+0x658/0x870 [mdc] [13352.060294] lmv_unlink+0x34d/0xbd0 [lmv] [13352.060323] ll_unlink+0x24d/0x7f0 [lustre] [13352.060635] vfs_unlink+0x202/0x330 [13352.061002] do_unlinkat+0x2fa/0x460 [13352.061010] __x64_sys_unlink+0x37/0x50 [13352.061014] do_syscall_64+0xc1/0x440 [13352.061021] entry_SYSCALL_64_after_hwframe+0x49/0xae [13352.061028] RIP: 0033:0x7f151bdee10b [13352.061035] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [13352.061039] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [13352.061044] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [13352.061047] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [13352.061050] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13352.061052] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13352.061055] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13352.061189] ? show_regs.cold.9+0x22/0x2f [13352.061202] ? watchdog_timer_fn.cold.6+0x62/0x10a [13352.061209] ? __hrtimer_run_queues+0x1c2/0x4d0 [13352.061213] ? watchdog+0x40/0x40 [13352.061217] ? hrtimer_interrupt+0x128/0x2c0 [13352.061222] ? smp_apic_timer_interrupt+0xa6/0x270 [13352.061229] ? apic_timer_interrupt+0xf/0x20 [13352.061234] [13352.061238] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13352.061245] do_raw_spin_lock+0x177/0x190 [13352.061250] _raw_spin_lock+0x21/0x30 [13352.061254] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13352.061516] ? do_raw_spin_unlock+0x75/0x190 [13352.061523] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13352.061733] ? do_wait_intr+0xf0/0xf0 [13352.061742] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13352.061939] mdc_close+0x53b/0xd70 [mdc] [13352.061976] lmv_close+0x18d/0x490 [lmv] [13352.062002] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13352.069128] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13352.069188] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13352.069193] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13352.069196] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13352.069209] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13352.069213] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13352.069218] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [13352.069220] RDX: 0000000000006277 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [13352.069222] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [13352.069224] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [13352.069226] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13352.069229] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [13352.069232] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13352.069235] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13352.069244] Call Trace: [13352.069250] [13352.069255] ? show_regs.cold.9+0x22/0x2f [13352.069264] ? watchdog_timer_fn.cold.6+0x62/0x10a [13352.069269] ? __hrtimer_run_queues+0x1c2/0x4d0 [13352.069272] ? watchdog+0x40/0x40 [13352.069275] ? hrtimer_interrupt+0x128/0x2c0 [13352.069277] ? smp_apic_timer_interrupt+0xa6/0x270 [13352.069284] ? apic_timer_interrupt+0xf/0x20 [13352.069288] [13352.069290] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13352.069293] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13352.069296] do_raw_spin_lock+0x177/0x190 [13352.069299] _raw_spin_lock+0x21/0x30 [13352.069303] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13352.069493] obd_mod_free+0x46/0x140 [ptlrpc] [13352.071238] mdc_commit_open+0x91/0xb0 [mdc] [13352.071274] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13352.071460] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13352.075085] after_reply+0xe2c/0x1780 [ptlrpc] [13352.075299] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13352.075424] ? __next_timer_interrupt+0x160/0x160 [13352.075431] ptlrpcd+0xa82/0xe40 [ptlrpc] [13352.075608] ? do_wait_intr+0xf0/0xf0 [13352.075615] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13352.075831] kthread+0x1d1/0x200 [13352.075836] ? set_kthread_struct+0x70/0x70 [13352.075838] ret_from_fork+0x1f/0x30 [13352.961751] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13352.965086] ll_md_real_close+0xa5/0x2a0 [lustre] [13352.971439] ll_md_close+0x274/0x800 [lustre] [13352.973242] ll_file_release+0x128/0xc50 [lustre] [13352.976722] __fput+0x10d/0x440 [13352.978819] ____fput+0x12/0x20 [13352.980095] task_work_run+0xc6/0x110 [13352.982148] exit_to_usermode_loop+0x1dd/0x1f0 [13352.984189] do_syscall_64+0x3d6/0x440 [13352.985856] entry_SYSCALL_64_after_hwframe+0x49/0xae [13352.989414] RIP: 0033:0x7f151bee3cd8 [13352.990898] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13353.008904] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13353.011454] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [13353.027462] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [13353.031679] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13353.034946] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13353.037674] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13357.025051] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11426s! [13357.030296] Showing busy workqueues and worker pools: [13357.032985] workqueue events: flags=0x0 [13357.034681] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13357.037982] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [13357.043197] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13357.046604] pending: kfree_rcu_monitor [13357.049178] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13357.053199] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [13357.058172] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13357.061429] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [13357.065953] workqueue events_long: flags=0x0 [13357.068414] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13357.071733] pending: lu_objects_flush [obdclass] [13357.074339] workqueue events_unbound: flags=0x2 [13357.077087] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13357.080204] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13357.086047] workqueue events_power_efficient: flags=0x80 [13357.088320] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13357.091574] pending: do_cache_clean, neigh_periodic_work, check_lifetime [13357.094970] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13357.098102] pending: neigh_periodic_work [13357.100071] workqueue mm_percpu_wq: flags=0x8 [13357.102252] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13357.105706] pending: vmstat_update, lru_add_drain_per_cpu [13357.108380] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13357.112166] pending: vmstat_update [13357.113967] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13357.117031] pending: lru_add_drain_per_cpu [13357.119134] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [13357.122231] pending: lru_add_drain_per_cpu BAR(48) [13357.124786] workqueue writeback: flags=0x4a [13357.126802] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [13357.130072] pending: wb_workfn [13357.132110] workqueue xprtiod: flags=0xa [13357.134364] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13357.138287] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [13357.142290] workqueue ipv6_addrconf: flags=0x40008 [13357.144992] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13357.148208] pending: addrconf_verify_work [13357.150402] workqueue knbd0-recv: flags=0x1a [13357.153275] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13357.156703] in-flight: 658:recv_work [13357.158759] workqueue ptlrpc_pinger: flags=0x2 [13357.161291] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13357.164339] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13357.167672] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11426s workers=4 idle: 32984 37 8 [13357.171164] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13341s workers=2 idle: 661 [13368.056665] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13368.059771] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13368.086219] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13368.093139] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13368.100643] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13368.103293] RIP: 0010:native_safe_halt+0x2e/0x50 [13368.109323] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13368.125050] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13368.131843] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13368.135051] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [13368.139035] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [13368.142111] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [13368.151680] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [13368.155138] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [13368.158264] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13368.161937] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13368.167273] Call Trace: [13368.169732] [13368.170275] ? show_regs.cold.9+0x22/0x2f [13368.171279] ? watchdog_timer_fn.cold.6+0x62/0x10a [13368.176961] ? __hrtimer_run_queues+0x1c2/0x4d0 [13368.178622] ? watchdog+0x40/0x40 [13368.179875] ? hrtimer_interrupt+0x128/0x2c0 [13368.185160] ? smp_apic_timer_interrupt+0xa6/0x270 [13368.191326] ? apic_timer_interrupt+0xf/0x20 [13368.192896] [13368.193796] ? native_safe_halt+0x2e/0x50 [13368.195661] kvm_wait+0xcd/0xe0 [13368.201098] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13368.203069] do_raw_spin_lock+0x177/0x190 [13368.204834] _raw_spin_lock+0x21/0x30 [13368.207158] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13368.210994] process_one_work+0x2c8/0x700 [13368.214826] worker_thread+0x296/0x6e0 [13368.217969] ? rescuer_thread+0x570/0x570 [13368.220146] kthread+0x1d1/0x200 [13368.222807] ? set_kthread_struct+0x70/0x70 [13368.229262] ret_from_fork+0x1f/0x30 [13380.055025] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [13380.057027] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [13380.058559] Modules linked in: lustre(O) [13380.062790] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13380.062865] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13380.062872] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13380.062875] RIP: 0010:native_safe_halt+0x2e/0x50 [13380.062890] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13380.062895] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13380.062899] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13380.062901] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [13380.062903] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [13380.062906] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [13380.062908] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [13380.062911] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [13380.062914] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13380.062916] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [13380.062925] Call Trace: [13380.062930] [13380.062935] ? show_regs.cold.9+0x22/0x2f [13380.062944] ? watchdog_timer_fn.cold.6+0x62/0x10a [13380.062949] ? __hrtimer_run_queues+0x1c2/0x4d0 [13380.062953] ? watchdog+0x40/0x40 [13380.062958] ? hrtimer_interrupt+0x128/0x2c0 [13380.062961] ? smp_apic_timer_interrupt+0xa6/0x270 [13380.062969] ? apic_timer_interrupt+0xf/0x20 [13380.062973] [13380.062975] ? native_safe_halt+0x2e/0x50 [13380.062978] kvm_wait+0xcd/0xe0 [13380.062985] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13380.062993] do_raw_spin_lock+0x177/0x190 [13380.062996] _raw_spin_lock+0x21/0x30 [13380.063000] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13380.063341] ? do_raw_spin_unlock+0x75/0x190 [13380.063347] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13380.063574] ? do_wait_intr+0xf0/0xf0 [13380.063583] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13380.064058] mdc_close+0x53b/0xd70 [mdc] [13380.064095] lmv_close+0x18d/0x490 [lmv] [13380.064125] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13380.067610] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13380.067685] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13380.067692] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13380.067695] RIP: 0010:native_safe_halt+0x2e/0x50 [13380.067708] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13380.067713] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13380.067717] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [13380.067720] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [13380.067722] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [13380.067724] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [13380.067727] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13380.067730] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [13380.067733] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13380.067735] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13380.067745] Call Trace: [13380.067751] [13380.067756] ? show_regs.cold.9+0x22/0x2f [13380.067766] ? watchdog_timer_fn.cold.6+0x62/0x10a [13380.067772] ? __hrtimer_run_queues+0x1c2/0x4d0 [13380.067776] ? watchdog+0x40/0x40 [13380.067780] ? hrtimer_interrupt+0x128/0x2c0 [13380.067784] ? smp_apic_timer_interrupt+0xa6/0x270 [13380.067790] ? apic_timer_interrupt+0xf/0x20 [13380.067794] [13380.067797] ? native_safe_halt+0x2e/0x50 [13380.067801] kvm_wait+0xcd/0xe0 [13380.067808] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13380.067815] do_raw_spin_lock+0x177/0x190 [13380.067819] _raw_spin_lock+0x21/0x30 [13380.067824] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13380.104499] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13380.104625] ll_md_real_close+0xa5/0x2a0 [lustre] [13380.104709] ll_md_close+0x274/0x800 [lustre] [13380.104794] ll_file_release+0x128/0xc50 [lustre] [13380.104878] __fput+0x10d/0x440 [13380.104888] ____fput+0x12/0x20 [13380.104892] task_work_run+0xc6/0x110 [13380.104900] exit_to_usermode_loop+0x1dd/0x1f0 [13380.104907] do_syscall_64+0x3d6/0x440 [13380.104911] entry_SYSCALL_64_after_hwframe+0x49/0xae [13380.104920] RIP: 0033:0x7f151bee3cd8 [13380.104927] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13380.104932] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13380.104937] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [13380.104940] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [13380.104942] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13380.104945] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13380.104947] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13380.359689] obd_mod_free+0x46/0x140 [ptlrpc] [13380.362532] mdc_commit_open+0x91/0xb0 [mdc] [13380.365157] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13380.369809] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13380.373616] after_reply+0xe2c/0x1780 [ptlrpc] [13380.376921] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13380.380242] ? __next_timer_interrupt+0x160/0x160 [13380.383428] ptlrpcd+0xa82/0xe40 [ptlrpc] [13380.386371] ? do_wait_intr+0xf0/0xf0 [13380.388536] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13380.391677] kthread+0x1d1/0x200 [13380.393689] ? set_kthread_struct+0x70/0x70 [13380.396057] ret_from_fork+0x1f/0x30 [13384.052559] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [13384.058167] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13384.085535] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13384.090773] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13384.095436] RIP: 0010:native_safe_halt+0x2e/0x50 [13384.097982] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13384.117485] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13384.123550] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13384.127170] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [13384.130816] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [13384.135748] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [13384.139768] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [13384.144558] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [13384.149043] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13384.152175] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [13384.156945] Call Trace: [13384.158518] [13384.159105] ? show_regs.cold.9+0x22/0x2f [13384.161097] ? watchdog_timer_fn.cold.6+0x62/0x10a [13384.164011] ? __hrtimer_run_queues+0x1c2/0x4d0 [13384.166728] ? watchdog+0x40/0x40 [13384.168587] ? hrtimer_interrupt+0x128/0x2c0 [13384.170143] ? smp_apic_timer_interrupt+0xa6/0x270 [13384.173682] ? apic_timer_interrupt+0xf/0x20 [13384.182033] [13384.182833] ? native_safe_halt+0x2e/0x50 [13384.185155] kvm_wait+0xcd/0xe0 [13384.189511] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13384.197043] do_raw_spin_lock+0x177/0x190 [13384.200547] _raw_spin_lock+0x21/0x30 [13384.203810] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13384.207190] ? do_raw_spin_unlock+0x75/0x190 [13384.211556] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13384.214656] ? do_wait_intr+0xf0/0xf0 [13384.217006] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13384.220039] mdc_reint+0x2d/0x110 [mdc] [13384.222459] mdc_unlink+0x658/0x870 [mdc] [13384.226228] lmv_unlink+0x34d/0xbd0 [lmv] [13384.227721] ll_unlink+0x24d/0x7f0 [lustre] [13384.232494] vfs_unlink+0x202/0x330 [13384.237660] do_unlinkat+0x2fa/0x460 [13384.242675] __x64_sys_unlink+0x37/0x50 [13384.246972] do_syscall_64+0xc1/0x440 [13384.249554] entry_SYSCALL_64_after_hwframe+0x49/0xae [13384.251270] RIP: 0033:0x7f151bdee10b [13384.253322] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [13384.266324] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [13384.269660] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [13384.273053] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [13384.275806] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13384.279517] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13384.288891] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13387.233208] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11456s! [13387.238746] Showing busy workqueues and worker pools: [13387.240399] workqueue events: flags=0x0 [13387.241902] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13387.244769] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [13387.248483] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13387.251326] pending: kfree_rcu_monitor [13387.253265] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13387.255937] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [13387.260554] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13387.263443] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [13387.266295] workqueue events_long: flags=0x0 [13387.268030] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13387.270932] pending: lu_objects_flush [obdclass] [13387.272992] workqueue events_unbound: flags=0x2 [13387.274891] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13387.277596] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13387.282355] workqueue events_power_efficient: flags=0x80 [13387.284568] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13387.287608] pending: do_cache_clean, neigh_periodic_work, check_lifetime [13387.290373] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13387.293012] pending: neigh_periodic_work [13387.294704] workqueue mm_percpu_wq: flags=0x8 [13387.296569] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13387.299438] pending: vmstat_update, lru_add_drain_per_cpu [13387.301760] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13387.309643] pending: vmstat_update [13387.313665] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13387.322073] pending: lru_add_drain_per_cpu [13387.324007] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [13387.334240] pending: lru_add_drain_per_cpu BAR(48) [13387.340656] workqueue writeback: flags=0x4a [13387.342596] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [13387.349048] pending: wb_workfn [13387.352053] workqueue xprtiod: flags=0xa [13387.356121] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13387.361871] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [13387.369590] workqueue ipv6_addrconf: flags=0x40008 [13387.370920] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13387.378963] pending: addrconf_verify_work [13387.381719] workqueue knbd0-recv: flags=0x1a [13387.385696] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13387.393772] in-flight: 658:recv_work [13387.395741] workqueue ptlrpc_pinger: flags=0x2 [13387.397820] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13387.401413] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13387.412860] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11456s workers=4 idle: 32984 37 8 [13387.416669] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13372s workers=2 idle: 661 [13396.056202] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [13396.063903] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13396.119776] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13396.139330] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13396.144396] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13396.150435] RIP: 0010:native_safe_halt+0x2e/0x50 [13396.154968] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13396.178003] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13396.193802] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13396.197886] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [13396.202673] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [13396.212419] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [13396.216885] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [13396.221431] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [13396.230821] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13396.241028] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13396.248014] Call Trace: [13396.250681] [13396.254331] ? show_regs.cold.9+0x22/0x2f [13396.255764] ? watchdog_timer_fn.cold.6+0x62/0x10a [13396.267778] ? __hrtimer_run_queues+0x1c2/0x4d0 [13396.282791] ? watchdog+0x40/0x40 [13396.288467] ? hrtimer_interrupt+0x128/0x2c0 [13396.290015] ? smp_apic_timer_interrupt+0xa6/0x270 [13396.301744] ? apic_timer_interrupt+0xf/0x20 [13396.309120] [13396.311547] ? native_safe_halt+0x2e/0x50 [13396.315768] kvm_wait+0xcd/0xe0 [13396.322938] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13396.333412] do_raw_spin_lock+0x177/0x190 [13396.336933] _raw_spin_lock+0x21/0x30 [13396.338713] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13396.343480] process_one_work+0x2c8/0x700 [13396.347895] worker_thread+0x296/0x6e0 [13396.350847] ? rescuer_thread+0x570/0x570 [13396.352338] kthread+0x1d1/0x200 [13396.353363] ? set_kthread_struct+0x70/0x70 [13396.360415] ret_from_fork+0x1f/0x30 [13408.055024] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [13408.057073] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [13408.058493] Modules linked in: lustre(O) [13408.061655] Modules linked in: [13408.061659] osp(O) ofd(O) [13408.064101] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13408.064245] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13408.064253] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13408.064256] RIP: 0010:native_safe_halt+0x2e/0x50 [13408.064269] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13408.064273] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13408.064278] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13408.064280] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [13408.064283] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [13408.064285] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [13408.064288] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [13408.064291] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [13408.064294] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13408.064297] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [13408.064306] Call Trace: [13408.064313] [13408.064319] ? show_regs.cold.9+0x22/0x2f [13408.064331] ? watchdog_timer_fn.cold.6+0x62/0x10a [13408.064339] ? __hrtimer_run_queues+0x1c2/0x4d0 [13408.064343] ? watchdog+0x40/0x40 [13408.064347] ? hrtimer_interrupt+0x128/0x2c0 [13408.064351] ? smp_apic_timer_interrupt+0xa6/0x270 [13408.064357] ? apic_timer_interrupt+0xf/0x20 [13408.064362] [13408.064364] ? native_safe_halt+0x2e/0x50 [13408.064368] kvm_wait+0xcd/0xe0 [13408.064376] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13408.064385] do_raw_spin_lock+0x177/0x190 [13408.064389] _raw_spin_lock+0x21/0x30 [13408.064393] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13408.065053] ? do_raw_spin_unlock+0x75/0x190 [13408.065060] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13408.065377] ? do_wait_intr+0xf0/0xf0 [13408.065385] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13408.065581] mdc_close+0x53b/0xd70 [mdc] [13408.065624] lmv_close+0x18d/0x490 [lmv] [13408.065657] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13408.066213] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13408.066323] ll_md_real_close+0xa5/0x2a0 [lustre] [13408.066412] ll_md_close+0x274/0x800 [lustre] [13408.066504] ll_file_release+0x128/0xc50 [lustre] [13408.066596] __fput+0x10d/0x440 [13408.066605] ____fput+0x12/0x20 [13408.066609] task_work_run+0xc6/0x110 [13408.066617] exit_to_usermode_loop+0x1dd/0x1f0 [13408.066623] do_syscall_64+0x3d6/0x440 [13408.066629] entry_SYSCALL_64_after_hwframe+0x49/0xae [13408.066637] RIP: 0033:0x7f151bee3cd8 [13408.066643] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13408.066648] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13408.066652] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [13408.066654] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [13408.066656] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13408.066658] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13408.066660] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13408.313381] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13408.358816] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13408.368569] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13408.376940] RIP: 0010:native_safe_halt+0x2e/0x50 [13408.384831] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13408.407770] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13408.411791] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [13408.417982] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [13408.422620] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [13408.429704] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [13408.432350] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13408.435262] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [13408.439773] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13408.442625] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13408.445558] Call Trace: [13408.446587] [13408.447086] ? show_regs.cold.9+0x22/0x2f [13408.452755] ? watchdog_timer_fn.cold.6+0x62/0x10a [13408.456381] ? __hrtimer_run_queues+0x1c2/0x4d0 [13408.458762] ? watchdog+0x40/0x40 [13408.459785] ? hrtimer_interrupt+0x128/0x2c0 [13408.462061] ? smp_apic_timer_interrupt+0xa6/0x270 [13408.464258] ? apic_timer_interrupt+0xf/0x20 [13408.467734] [13408.468727] ? native_safe_halt+0x2e/0x50 [13408.473966] kvm_wait+0xcd/0xe0 [13408.475634] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13408.478347] do_raw_spin_lock+0x177/0x190 [13408.479735] _raw_spin_lock+0x21/0x30 [13408.489038] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13408.491839] obd_mod_free+0x46/0x140 [ptlrpc] [13408.493878] mdc_commit_open+0x91/0xb0 [mdc] [13408.496503] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13408.498951] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13408.501276] after_reply+0xe2c/0x1780 [ptlrpc] [13408.502978] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13408.509910] ? __next_timer_interrupt+0x160/0x160 [13408.512034] ptlrpcd+0xa82/0xe40 [ptlrpc] [13408.514467] ? do_wait_intr+0xf0/0xf0 [13408.515956] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13408.517874] kthread+0x1d1/0x200 [13408.519302] ? set_kthread_struct+0x70/0x70 [13408.521129] ret_from_fork+0x1f/0x30 [13412.053012] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [13412.057479] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13412.094520] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13412.101807] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13412.107204] RIP: 0010:native_safe_halt+0x2e/0x50 [13412.109618] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13412.119747] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13412.123815] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13412.127899] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [13412.134333] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [13412.145291] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [13412.152494] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [13412.158114] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [13412.162102] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13412.165377] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [13412.168898] Call Trace: [13412.170460] [13412.171354] ? show_regs.cold.9+0x22/0x2f [13412.173553] ? watchdog_timer_fn.cold.6+0x62/0x10a [13412.176394] ? __hrtimer_run_queues+0x1c2/0x4d0 [13412.178765] ? watchdog+0x40/0x40 [13412.180750] ? hrtimer_interrupt+0x128/0x2c0 [13412.183234] ? smp_apic_timer_interrupt+0xa6/0x270 [13412.184854] ? apic_timer_interrupt+0xf/0x20 [13412.187167] [13412.188270] ? native_safe_halt+0x2e/0x50 [13412.190977] kvm_wait+0xcd/0xe0 [13412.192832] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13412.194743] do_raw_spin_lock+0x177/0x190 [13412.197009] _raw_spin_lock+0x21/0x30 [13412.198705] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13412.202366] ? do_raw_spin_unlock+0x75/0x190 [13412.204915] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13412.207511] ? do_wait_intr+0xf0/0xf0 [13412.209518] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13412.212460] mdc_reint+0x2d/0x110 [mdc] [13412.214609] mdc_unlink+0x658/0x870 [mdc] [13412.216862] lmv_unlink+0x34d/0xbd0 [lmv] [13412.218718] ll_unlink+0x24d/0x7f0 [lustre] [13412.220921] vfs_unlink+0x202/0x330 [13412.222631] do_unlinkat+0x2fa/0x460 [13412.224436] __x64_sys_unlink+0x37/0x50 [13412.226301] do_syscall_64+0xc1/0x440 [13412.228216] entry_SYSCALL_64_after_hwframe+0x49/0xae [13412.230763] RIP: 0033:0x7f151bdee10b [13412.232279] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [13412.241549] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [13412.244918] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [13412.249229] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [13412.258730] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13412.268325] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13412.277141] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13417.444217] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11486s! [13417.449711] Showing busy workqueues and worker pools: [13417.454963] workqueue events: flags=0x0 [13417.457116] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13417.459794] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [13417.471009] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13417.474204] pending: kfree_rcu_monitor [13417.475468] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13417.482197] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [13417.492987] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13417.502139] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [13417.509855] workqueue events_long: flags=0x0 [13417.511199] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13417.523684] pending: lu_objects_flush [obdclass] [13417.526925] workqueue events_unbound: flags=0x2 [13417.530849] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13417.532457] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13417.537128] workqueue events_power_efficient: flags=0x80 [13417.541262] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13417.546232] pending: do_cache_clean, neigh_periodic_work, check_lifetime [13417.552258] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13417.558571] pending: neigh_periodic_work [13417.559798] workqueue mm_percpu_wq: flags=0x8 [13417.569786] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13417.571520] pending: vmstat_update, lru_add_drain_per_cpu [13417.572866] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13417.583671] pending: vmstat_update [13417.586094] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13417.587785] pending: lru_add_drain_per_cpu [13417.589010] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [13417.600674] pending: lru_add_drain_per_cpu BAR(48) [13417.604225] workqueue writeback: flags=0x4a [13417.607850] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [13417.620300] pending: wb_workfn [13417.621298] workqueue xprtiod: flags=0xa [13417.622346] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13417.628109] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [13417.638700] workqueue ipv6_addrconf: flags=0x40008 [13417.640054] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13417.644164] pending: addrconf_verify_work [13417.647661] workqueue knbd0-recv: flags=0x1a [13417.649337] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13417.651584] in-flight: 658:recv_work [13417.653857] workqueue ptlrpc_pinger: flags=0x2 [13417.655307] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13417.659577] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13417.666258] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11487s workers=4 idle: 32984 37 8 [13417.674526] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13402s workers=2 idle: 661 [13424.056334] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [13424.061344] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13424.082482] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13424.088067] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13424.092029] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13424.094892] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13424.097446] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13424.105265] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13424.108779] RAX: 0000000000004d81 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13424.111742] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [13424.114750] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [13424.118314] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [13424.121248] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [13424.126489] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [13424.131714] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13424.134650] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13424.143062] Call Trace: [13424.145596] [13424.146809] ? show_regs.cold.9+0x22/0x2f [13424.152270] ? watchdog_timer_fn.cold.6+0x62/0x10a [13424.154162] ? __hrtimer_run_queues+0x1c2/0x4d0 [13424.157514] ? watchdog+0x40/0x40 [13424.159583] ? hrtimer_interrupt+0x128/0x2c0 [13424.163444] ? smp_apic_timer_interrupt+0xa6/0x270 [13424.166297] ? apic_timer_interrupt+0xf/0x20 [13424.168080] [13424.170741] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13424.173654] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13424.180409] do_raw_spin_lock+0x177/0x190 [13424.184538] _raw_spin_lock+0x21/0x30 [13424.187271] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13424.191355] process_one_work+0x2c8/0x700 [13424.193255] worker_thread+0x296/0x6e0 [13424.196790] ? rescuer_thread+0x570/0x570 [13424.201029] kthread+0x1d1/0x200 [13424.204210] ? set_kthread_struct+0x70/0x70 [13424.206036] ret_from_fork+0x1f/0x30 [13436.058498] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [13436.062089] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [13436.065548] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13436.065771] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13436.065779] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13436.065785] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13436.065810] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13436.065814] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13436.065823] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [13436.065825] RDX: 00000000000029e6 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [13436.065828] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [13436.065830] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [13436.065832] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13436.065839] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [13436.065851] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13436.065854] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13436.065862] Call Trace: [13436.065876] [13436.065881] ? show_regs.cold.9+0x22/0x2f [13436.065890] ? watchdog_timer_fn.cold.6+0x62/0x10a [13436.065896] ? __hrtimer_run_queues+0x1c2/0x4d0 [13436.065904] ? watchdog+0x40/0x40 [13436.065909] ? hrtimer_interrupt+0x128/0x2c0 [13436.065912] ? smp_apic_timer_interrupt+0xa6/0x270 [13436.065920] ? apic_timer_interrupt+0xf/0x20 [13436.065928] [13436.065931] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13436.065936] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13436.065940] do_raw_spin_lock+0x177/0x190 [13436.065945] _raw_spin_lock+0x21/0x30 [13436.065952] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13436.066307] obd_mod_free+0x46/0x140 [ptlrpc] [13436.066523] mdc_commit_open+0x91/0xb0 [mdc] [13436.066562] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13436.066758] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13436.066983] after_reply+0xe2c/0x1780 [ptlrpc] [13436.067199] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13436.067394] ? __next_timer_interrupt+0x160/0x160 [13436.067402] ptlrpcd+0xa82/0xe40 [ptlrpc] [13436.067620] ? do_wait_intr+0xf0/0xf0 [13436.067629] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13436.067853] kthread+0x1d1/0x200 [13436.067859] ? set_kthread_struct+0x70/0x70 [13436.067864] ret_from_fork+0x1f/0x30 [13436.352699] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13436.382226] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13436.404295] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13436.413362] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13436.415830] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13436.435439] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13436.440471] RAX: 000000000000612d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13436.446428] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [13436.450420] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [13436.454458] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [13436.457855] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [13436.460949] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [13436.464475] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13436.466884] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [13436.469876] Call Trace: [13436.470983] [13436.471927] ? show_regs.cold.9+0x22/0x2f [13436.473650] ? watchdog_timer_fn.cold.6+0x62/0x10a [13436.475496] ? __hrtimer_run_queues+0x1c2/0x4d0 [13436.477394] ? watchdog+0x40/0x40 [13436.478665] ? hrtimer_interrupt+0x128/0x2c0 [13436.480533] ? smp_apic_timer_interrupt+0xa6/0x270 [13436.482857] ? apic_timer_interrupt+0xf/0x20 [13436.484573] [13436.485408] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13436.487516] do_raw_spin_lock+0x177/0x190 [13436.489564] _raw_spin_lock+0x21/0x30 [13436.491262] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13436.494398] ? do_raw_spin_unlock+0x75/0x190 [13436.497584] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13436.503355] ? do_wait_intr+0xf0/0xf0 [13436.508073] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13436.510101] mdc_close+0x53b/0xd70 [mdc] [13436.512522] lmv_close+0x18d/0x490 [lmv] [13436.514224] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13436.517529] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13436.526009] ll_md_real_close+0xa5/0x2a0 [lustre] [13436.532939] ll_md_close+0x274/0x800 [lustre] [13436.537487] ll_file_release+0x128/0xc50 [lustre] [13436.541917] __fput+0x10d/0x440 [13436.543396] ____fput+0x12/0x20 [13436.545860] task_work_run+0xc6/0x110 [13436.549111] exit_to_usermode_loop+0x1dd/0x1f0 [13436.552693] do_syscall_64+0x3d6/0x440 [13436.555572] entry_SYSCALL_64_after_hwframe+0x49/0xae [13436.560629] RIP: 0033:0x7f151bee3cd8 [13436.563356] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13436.574799] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13436.578176] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [13436.581515] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [13436.588542] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13436.591191] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13436.605329] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13444.052068] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [13444.058494] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13444.098503] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13444.104389] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13444.109393] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13444.112528] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13444.122513] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [13444.126373] RAX: 0000000000004425 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13444.129956] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [13444.133387] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [13444.137005] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [13444.140698] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [13444.144066] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [13444.147937] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13444.150969] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [13444.154355] Call Trace: [13444.155471] [13444.156619] ? show_regs.cold.9+0x22/0x2f [13444.158635] ? watchdog_timer_fn.cold.6+0x62/0x10a [13444.161036] ? __hrtimer_run_queues+0x1c2/0x4d0 [13444.163253] ? watchdog+0x40/0x40 [13444.164953] ? hrtimer_interrupt+0x128/0x2c0 [13444.167131] ? smp_apic_timer_interrupt+0xa6/0x270 [13444.170019] ? apic_timer_interrupt+0xf/0x20 [13444.172092] [13444.173265] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13444.176274] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13444.178989] do_raw_spin_lock+0x177/0x190 [13444.181043] _raw_spin_lock+0x21/0x30 [13444.182820] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13444.185891] ? do_raw_spin_unlock+0x75/0x190 [13444.187569] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13444.193621] ? do_wait_intr+0xf0/0xf0 [13444.197432] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13444.204278] mdc_reint+0x2d/0x110 [mdc] [13444.207949] mdc_unlink+0x658/0x870 [mdc] [13444.211639] lmv_unlink+0x34d/0xbd0 [lmv] [13444.214880] ll_unlink+0x24d/0x7f0 [lustre] [13444.219652] vfs_unlink+0x202/0x330 [13444.222421] do_unlinkat+0x2fa/0x460 [13444.226352] __x64_sys_unlink+0x37/0x50 [13444.230344] do_syscall_64+0xc1/0x440 [13444.231831] entry_SYSCALL_64_after_hwframe+0x49/0xae [13444.237059] RIP: 0033:0x7f151bdee10b [13444.238612] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [13444.248920] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [13444.251510] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [13444.254024] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [13444.256566] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13444.259027] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13444.261744] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13448.169151] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11517s! [13448.177909] Showing busy workqueues and worker pools: [13448.179791] workqueue events: flags=0x0 [13448.181494] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13448.184421] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [13448.205691] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13448.216156] pending: kfree_rcu_monitor [13448.219151] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13448.230174] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [13448.239898] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13448.250205] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [13448.255319] workqueue events_long: flags=0x0 [13448.256988] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13448.260107] pending: lu_objects_flush [obdclass] [13448.263159] workqueue events_unbound: flags=0x2 [13448.271127] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13448.279029] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13448.294037] workqueue events_power_efficient: flags=0x80 [13448.298757] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13448.307801] pending: do_cache_clean, neigh_periodic_work, check_lifetime [13448.312289] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13448.320145] pending: neigh_periodic_work [13448.327028] workqueue mm_percpu_wq: flags=0x8 [13448.328557] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13448.335846] pending: vmstat_update, lru_add_drain_per_cpu [13448.340415] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13448.345122] pending: vmstat_update [13448.346527] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13448.349505] pending: lru_add_drain_per_cpu [13448.351362] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [13448.356220] pending: lru_add_drain_per_cpu BAR(48) [13448.359676] workqueue writeback: flags=0x4a [13448.362119] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [13448.364579] pending: wb_workfn [13448.366170] workqueue xprtiod: flags=0xa [13448.367719] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13448.370357] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [13448.373947] workqueue ipv6_addrconf: flags=0x40008 [13448.375690] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13448.378956] pending: addrconf_verify_work [13448.380871] workqueue knbd0-recv: flags=0x1a [13448.382387] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13448.384871] in-flight: 658:recv_work [13448.386544] workqueue ptlrpc_pinger: flags=0x2 [13448.388859] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13448.391257] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13448.394016] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11517s workers=4 idle: 32984 37 8 [13448.397189] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13433s workers=2 idle: 661 [13452.057167] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [13452.064806] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13452.112814] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13452.121841] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13452.132007] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13452.143412] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13452.150633] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13452.173949] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13452.184208] RAX: 00000000000008d3 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13452.191658] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [13452.201316] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [13452.206932] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [13452.209014] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [13452.215084] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [13452.219431] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13452.224003] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13452.228810] Call Trace: [13452.230142] [13452.231083] ? show_regs.cold.9+0x22/0x2f [13452.237015] ? watchdog_timer_fn.cold.6+0x62/0x10a [13452.240749] ? __hrtimer_run_queues+0x1c2/0x4d0 [13452.242368] ? watchdog+0x40/0x40 [13452.245128] ? hrtimer_interrupt+0x128/0x2c0 [13452.250779] ? smp_apic_timer_interrupt+0xa6/0x270 [13452.257839] ? apic_timer_interrupt+0xf/0x20 [13452.270186] [13452.270993] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13452.277777] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13452.286990] do_raw_spin_lock+0x177/0x190 [13452.293461] _raw_spin_lock+0x21/0x30 [13452.296322] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13452.304352] process_one_work+0x2c8/0x700 [13452.305990] worker_thread+0x296/0x6e0 [13452.312795] ? rescuer_thread+0x570/0x570 [13452.318047] kthread+0x1d1/0x200 [13452.323835] ? set_kthread_struct+0x70/0x70 [13452.333496] ret_from_fork+0x1f/0x30 [13464.055037] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [13464.058180] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13464.058415] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [13464.076775] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13464.076788] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13464.076791] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13464.076804] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13464.076809] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13464.076814] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [13464.076817] RDX: 00000000000055cc RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [13464.076820] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [13464.076822] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [13464.076824] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13464.076827] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [13464.076830] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13464.076833] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13464.076840] Call Trace: [13464.076846] [13464.076851] ? show_regs.cold.9+0x22/0x2f [13464.076859] ? watchdog_timer_fn.cold.6+0x62/0x10a [13464.076864] ? __hrtimer_run_queues+0x1c2/0x4d0 [13464.076868] ? watchdog+0x40/0x40 [13464.076873] ? hrtimer_interrupt+0x128/0x2c0 [13464.076877] ? smp_apic_timer_interrupt+0xa6/0x270 [13464.076885] ? apic_timer_interrupt+0xf/0x20 [13464.076889] [13464.076892] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13464.076897] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13464.076902] do_raw_spin_lock+0x177/0x190 [13464.076906] _raw_spin_lock+0x21/0x30 [13464.076918] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13464.077242] obd_mod_free+0x46/0x140 [ptlrpc] [13464.077626] mdc_commit_open+0x91/0xb0 [mdc] [13464.077668] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13464.078341] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13464.078705] after_reply+0xe2c/0x1780 [ptlrpc] [13464.078938] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13464.079183] ? __next_timer_interrupt+0x160/0x160 [13464.079199] ptlrpcd+0xa82/0xe40 [ptlrpc] [13464.079430] ? do_wait_intr+0xf0/0xf0 [13464.079548] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13464.079785] kthread+0x1d1/0x200 [13464.079801] ? set_kthread_struct+0x70/0x70 [13464.079805] ret_from_fork+0x1f/0x30 [13464.222656] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13464.246228] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13464.251067] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13464.254624] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13464.257091] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13464.264634] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13464.267482] RAX: 0000000000004806 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13464.270677] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [13464.273372] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [13464.276234] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [13464.279077] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [13464.282186] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [13464.285587] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13464.287870] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [13464.292105] Call Trace: [13464.293166] [13464.293984] ? show_regs.cold.9+0x22/0x2f [13464.295728] ? watchdog_timer_fn.cold.6+0x62/0x10a [13464.297730] ? __hrtimer_run_queues+0x1c2/0x4d0 [13464.299689] ? watchdog+0x40/0x40 [13464.301172] ? hrtimer_interrupt+0x128/0x2c0 [13464.303025] ? smp_apic_timer_interrupt+0xa6/0x270 [13464.305485] ? apic_timer_interrupt+0xf/0x20 [13464.307402] [13464.308308] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13464.310679] do_raw_spin_lock+0x177/0x190 [13464.312634] _raw_spin_lock+0x21/0x30 [13464.314163] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13464.316595] ? do_raw_spin_unlock+0x75/0x190 [13464.318274] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13464.322402] ? do_wait_intr+0xf0/0xf0 [13464.324007] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13464.326898] mdc_close+0x53b/0xd70 [mdc] [13464.328707] lmv_close+0x18d/0x490 [lmv] [13464.330158] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13464.333040] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13464.335705] ll_md_real_close+0xa5/0x2a0 [lustre] [13464.337986] ll_md_close+0x274/0x800 [lustre] [13464.339963] ll_file_release+0x128/0xc50 [lustre] [13464.342111] __fput+0x10d/0x440 [13464.343216] ____fput+0x12/0x20 [13464.344889] task_work_run+0xc6/0x110 [13464.346184] exit_to_usermode_loop+0x1dd/0x1f0 [13464.348493] do_syscall_64+0x3d6/0x440 [13464.350756] entry_SYSCALL_64_after_hwframe+0x49/0xae [13464.352815] RIP: 0033:0x7f151bee3cd8 [13464.354403] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13464.362788] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13464.365920] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [13464.369619] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [13464.372851] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13464.376137] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13464.379557] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13476.052053] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [13476.053766] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13476.072620] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13476.075578] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13476.079580] RIP: 0010:native_safe_halt+0x2e/0x50 [13476.081640] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13476.090194] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13476.093381] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13476.096355] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [13476.099566] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [13476.102714] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [13476.106144] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [13476.109983] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [13476.113867] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13476.116742] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [13476.120367] Call Trace: [13476.121619] [13476.123437] ? show_regs.cold.9+0x22/0x2f [13476.125740] ? watchdog_timer_fn.cold.6+0x62/0x10a [13476.128084] ? __hrtimer_run_queues+0x1c2/0x4d0 [13476.130162] ? watchdog+0x40/0x40 [13476.131841] ? hrtimer_interrupt+0x128/0x2c0 [13476.133964] ? smp_apic_timer_interrupt+0xa6/0x270 [13476.136321] ? apic_timer_interrupt+0xf/0x20 [13476.138876] [13476.139938] ? native_safe_halt+0x2e/0x50 [13476.142099] kvm_wait+0xcd/0xe0 [13476.143698] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13476.146520] do_raw_spin_lock+0x177/0x190 [13476.148342] _raw_spin_lock+0x21/0x30 [13476.150139] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13476.153611] ? do_raw_spin_unlock+0x75/0x190 [13476.155841] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13476.158749] ? do_wait_intr+0xf0/0xf0 [13476.160775] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13476.167804] mdc_reint+0x2d/0x110 [mdc] [13476.170061] mdc_unlink+0x658/0x870 [mdc] [13476.173746] lmv_unlink+0x34d/0xbd0 [lmv] [13476.178994] ll_unlink+0x24d/0x7f0 [lustre] [13476.183430] vfs_unlink+0x202/0x330 [13476.185218] do_unlinkat+0x2fa/0x460 [13476.188809] __x64_sys_unlink+0x37/0x50 [13476.190798] do_syscall_64+0xc1/0x440 [13476.194575] entry_SYSCALL_64_after_hwframe+0x49/0xae [13476.197913] RIP: 0033:0x7f151bdee10b [13476.202877] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [13476.228807] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [13476.237737] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [13476.247686] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [13476.254946] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13476.263219] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13476.271154] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13478.881039] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11548s! [13478.884000] Showing busy workqueues and worker pools: [13478.886927] workqueue events: flags=0x0 [13478.888731] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13478.891997] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [13478.896236] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13478.899239] pending: kfree_rcu_monitor [13478.901534] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13478.904699] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [13478.908785] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13478.912498] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [13478.915928] workqueue events_long: flags=0x0 [13478.917212] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13478.919995] pending: lu_objects_flush [obdclass] [13478.921876] workqueue events_unbound: flags=0x2 [13478.923471] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13478.926064] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13478.931289] workqueue events_power_efficient: flags=0x80 [13478.934051] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13478.937540] pending: do_cache_clean, neigh_periodic_work, check_lifetime [13478.940874] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13478.945169] pending: neigh_periodic_work [13478.950151] workqueue mm_percpu_wq: flags=0x8 [13478.953184] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13478.962812] pending: vmstat_update, lru_add_drain_per_cpu [13478.966648] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13478.970494] pending: vmstat_update [13478.972223] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13478.975616] pending: lru_add_drain_per_cpu [13478.978023] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [13478.981256] pending: lru_add_drain_per_cpu BAR(48) [13478.983553] workqueue writeback: flags=0x4a [13478.985797] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [13478.991245] pending: wb_workfn [13478.992387] workqueue xprtiod: flags=0xa [13478.998161] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13479.001785] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [13479.004456] workqueue ipv6_addrconf: flags=0x40008 [13479.008348] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13479.018264] pending: addrconf_verify_work [13479.021343] workqueue knbd0-recv: flags=0x1a [13479.023253] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13479.026441] in-flight: 658:recv_work [13479.028222] workqueue ptlrpc_pinger: flags=0x2 [13479.032381] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13479.038004] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13479.043092] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11548s workers=4 idle: 32984 37 8 [13479.046741] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13463s workers=2 idle: 661 [13480.056477] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [13480.065234] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13480.096630] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13480.109197] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13480.113884] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13480.117343] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13480.129681] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13480.155866] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13480.164095] RAX: 0000000000002493 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13480.172258] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [13480.191132] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [13480.199131] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [13480.206024] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [13480.213848] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [13480.222622] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13480.225367] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13480.228894] Call Trace: [13480.230009] [13480.230753] ? show_regs.cold.9+0x22/0x2f [13480.232644] ? watchdog_timer_fn.cold.6+0x62/0x10a [13480.234390] ? __hrtimer_run_queues+0x1c2/0x4d0 [13480.244033] ? watchdog+0x40/0x40 [13480.248104] ? hrtimer_interrupt+0x128/0x2c0 [13480.252046] ? smp_apic_timer_interrupt+0xa6/0x270 [13480.255550] ? apic_timer_interrupt+0xf/0x20 [13480.258430] [13480.259676] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13480.265682] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13480.274180] do_raw_spin_lock+0x177/0x190 [13480.276996] _raw_spin_lock+0x21/0x30 [13480.282508] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13480.287110] process_one_work+0x2c8/0x700 [13480.296553] worker_thread+0x296/0x6e0 [13480.300129] ? rescuer_thread+0x570/0x570 [13480.306050] kthread+0x1d1/0x200 [13480.308917] ? set_kthread_struct+0x70/0x70 [13480.314489] ret_from_fork+0x1f/0x30 [13492.056159] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [13492.057370] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [13492.063923] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13492.064077] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13492.064085] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13492.064088] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13492.064475] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13492.064499] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13492.064504] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [13492.064506] RDX: 000000000000192d RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [13492.064509] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [13492.064511] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [13492.064513] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13492.064516] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [13492.064519] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13492.064525] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13492.064534] Call Trace: [13492.064540] [13492.064547] ? show_regs.cold.9+0x22/0x2f [13492.070956] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13492.071042] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13492.071050] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13492.071054] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13492.071067] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13492.071071] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13492.071078] RAX: 0000000000003787 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13492.071080] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [13492.071083] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [13492.071085] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [13492.071088] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [13492.096974] ? watchdog_timer_fn.cold.6+0x62/0x10a [13492.096993] ? __hrtimer_run_queues+0x1c2/0x4d0 [13492.096998] ? watchdog+0x40/0x40 [13492.097002] ? hrtimer_interrupt+0x128/0x2c0 [13492.097006] ? smp_apic_timer_interrupt+0xa6/0x270 [13492.097014] ? apic_timer_interrupt+0xf/0x20 [13492.097018] [13492.097022] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13492.097030] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13492.097034] do_raw_spin_lock+0x177/0x190 [13492.097040] _raw_spin_lock+0x21/0x30 [13492.097048] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13492.097504] obd_mod_free+0x46/0x140 [ptlrpc] [13492.097734] mdc_commit_open+0x91/0xb0 [mdc] [13492.097773] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13492.102805] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [13492.102816] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13492.102819] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [13492.102827] Call Trace: [13492.102833] [13492.102839] ? show_regs.cold.9+0x22/0x2f [13492.102850] ? watchdog_timer_fn.cold.6+0x62/0x10a [13492.102857] ? __hrtimer_run_queues+0x1c2/0x4d0 [13492.102862] ? watchdog+0x40/0x40 [13492.102867] ? hrtimer_interrupt+0x128/0x2c0 [13492.102871] ? smp_apic_timer_interrupt+0xa6/0x270 [13492.102878] ? apic_timer_interrupt+0xf/0x20 [13492.102883] [13492.102886] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13492.102894] do_raw_spin_lock+0x177/0x190 [13492.102899] _raw_spin_lock+0x21/0x30 [13492.102904] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13492.103230] ? do_raw_spin_unlock+0x75/0x190 [13492.103236] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13492.103544] ? do_wait_intr+0xf0/0xf0 [13492.103552] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13492.104501] mdc_close+0x53b/0xd70 [mdc] [13492.104540] lmv_close+0x18d/0x490 [lmv] [13492.104569] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13492.104754] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13492.109579] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13492.109884] after_reply+0xe2c/0x1780 [ptlrpc] [13492.110096] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13492.110437] ? __next_timer_interrupt+0x160/0x160 [13492.110454] ptlrpcd+0xa82/0xe40 [ptlrpc] [13492.110680] ? do_wait_intr+0xf0/0xf0 [13492.110695] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13492.110936] kthread+0x1d1/0x200 [13492.110943] ? set_kthread_struct+0x70/0x70 [13492.110948] ret_from_fork+0x1f/0x30 [13492.337103] ll_md_real_close+0xa5/0x2a0 [lustre] [13492.339745] ll_md_close+0x274/0x800 [lustre] [13492.342031] ll_file_release+0x128/0xc50 [lustre] [13492.344760] __fput+0x10d/0x440 [13492.346620] ____fput+0x12/0x20 [13492.348587] task_work_run+0xc6/0x110 [13492.349987] exit_to_usermode_loop+0x1dd/0x1f0 [13492.352486] do_syscall_64+0x3d6/0x440 [13492.353868] entry_SYSCALL_64_after_hwframe+0x49/0xae [13492.356546] RIP: 0033:0x7f151bee3cd8 [13492.358389] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13492.367563] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13492.371014] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [13492.374429] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [13492.377666] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13492.380852] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13492.384941] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13504.054471] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [13504.061875] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13504.082121] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13504.086847] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13504.089911] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13504.091932] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13504.099948] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13504.103395] RAX: 0000000000006084 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13504.107109] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [13504.113406] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [13504.123596] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [13504.125987] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [13504.140795] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [13504.158304] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13504.160447] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [13504.176055] Call Trace: [13504.182059] [13504.183096] ? show_regs.cold.9+0x22/0x2f [13504.185737] ? watchdog_timer_fn.cold.6+0x62/0x10a [13504.191051] ? __hrtimer_run_queues+0x1c2/0x4d0 [13504.192739] ? watchdog+0x40/0x40 [13504.204884] ? hrtimer_interrupt+0x128/0x2c0 [13504.206444] ? smp_apic_timer_interrupt+0xa6/0x270 [13504.208007] ? apic_timer_interrupt+0xf/0x20 [13504.210027] [13504.210759] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13504.215840] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13504.218751] do_raw_spin_lock+0x177/0x190 [13504.221476] _raw_spin_lock+0x21/0x30 [13504.222509] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13504.226161] ? do_raw_spin_unlock+0x75/0x190 [13504.230424] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13504.234660] ? do_wait_intr+0xf0/0xf0 [13504.236487] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13504.242126] mdc_reint+0x2d/0x110 [mdc] [13504.243274] mdc_unlink+0x658/0x870 [mdc] [13504.244488] lmv_unlink+0x34d/0xbd0 [lmv] [13504.249355] ll_unlink+0x24d/0x7f0 [lustre] [13504.252868] vfs_unlink+0x202/0x330 [13504.256551] do_unlinkat+0x2fa/0x460 [13504.258704] __x64_sys_unlink+0x37/0x50 [13504.260933] do_syscall_64+0xc1/0x440 [13504.264305] entry_SYSCALL_64_after_hwframe+0x49/0xae [13504.266801] RIP: 0033:0x7f151bdee10b [13504.269916] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [13504.281688] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [13504.289400] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [13504.293591] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [13504.310477] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13504.314938] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13504.324740] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13504.897736] rcu: INFO: rcu_sched self-detected stall on CPU [13504.901989] rcu: 1-...!: (6757242 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [13504.906857] (t=11580250 jiffies g=249657 q=5976) [13504.909228] rcu: rcu_sched kthread starved for 11575202 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [13504.917270] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [13504.926225] rcu: RCU grace-period kthread stack dump: [13504.927791] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [13504.933793] Call Trace: [13504.934895] __schedule+0x351/0xcb0 [13504.935997] schedule+0xc0/0x180 [13504.937170] schedule_timeout+0xb4/0x190 [13504.938119] ? do_raw_spin_unlock+0x75/0x190 [13504.939314] ? __next_timer_interrupt+0x160/0x160 [13504.941585] ? prepare_to_swait_event+0xf3/0x140 [13504.943617] rcu_gp_kthread+0x8dd/0x15b0 [13504.945158] ? force_qs_rnp+0x2c0/0x2c0 [13504.946976] kthread+0x1d1/0x200 [13504.948536] ? set_kthread_struct+0x70/0x70 [13504.954392] ret_from_fork+0x1f/0x30 [13504.957221] rcu: Stack dump where RCU GP kthread last ran: [13504.962056] Sending NMI from CPU 1 to CPUs 2: [13504.971045] NMI backtrace for cpu 2 [13504.971053] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13504.971055] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13504.971057] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13504.971065] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13504.971066] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13504.971068] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 [13504.971070] RAX: 0000000000006829 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13504.971072] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [13504.971073] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [13504.971075] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [13504.971076] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [13504.971077] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [13504.971079] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13504.971080] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13504.971082] Call Trace: [13504.971083] [13504.971083] ? show_regs.cold.9+0x22/0x2f [13504.971085] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [13504.971086] ? nmi_cpu_backtrace_handler+0x15/0x40 [13504.971087] ? nmi_handle+0xbd/0x210 [13504.971088] ? default_do_nmi+0x4b/0x1b0 [13504.971089] ? do_nmi+0x280/0x380 [13504.971090] ? end_repeat_nmi+0x16/0x1f [13504.971091] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13504.971093] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13504.971094] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13504.971095] [13504.971096] do_raw_spin_lock+0x177/0x190 [13504.971098] _raw_spin_lock+0x21/0x30 [13504.971099] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13504.971100] process_one_work+0x2c8/0x700 [13504.971101] worker_thread+0x296/0x6e0 [13504.971102] ? rescuer_thread+0x570/0x570 [13504.971102] kthread+0x1d1/0x200 [13504.971103] ? set_kthread_struct+0x70/0x70 [13504.971104] ret_from_fork+0x1f/0x30 [13504.971505] NMI backtrace for cpu 1 [13505.108870] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13505.113928] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13505.117337] Call Trace: [13505.118269] [13505.118977] ? dump_stack+0xbb/0x10e [13505.120386] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [13505.122597] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [13505.124042] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [13505.126165] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [13505.128373] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [13505.131185] ? rcu_dump_cpu_stacks+0x11b/0x171 [13505.133210] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [13505.135342] ? update_process_times+0x6d/0xc0 [13505.140214] ? tick_sched_handle.isra.18+0x27/0x90 [13505.141870] ? tick_sched_timer+0x53/0xb0 [13505.146845] ? __hrtimer_run_queues+0x1c2/0x4d0 [13505.148872] ? tick_sched_handle.isra.18+0x90/0x90 [13505.152408] ? hrtimer_interrupt+0x128/0x2c0 [13505.154672] ? smp_apic_timer_interrupt+0xa6/0x270 [13505.157928] ? apic_timer_interrupt+0xf/0x20 [13505.160141] [13505.160973] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13505.166017] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13505.168286] ? do_raw_spin_lock+0x177/0x190 [13505.169821] ? _raw_spin_lock+0x21/0x30 [13505.171012] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13505.172819] ? obd_mod_free+0x46/0x140 [ptlrpc] [13505.174371] ? mdc_commit_open+0x91/0xb0 [mdc] [13505.175752] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13505.177581] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13505.179660] ? after_reply+0xe2c/0x1780 [ptlrpc] [13505.181692] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13505.183475] ? __next_timer_interrupt+0x160/0x160 [13505.184784] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [13505.186413] ? do_wait_intr+0xf0/0xf0 [13505.187479] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13505.189438] ? kthread+0x1d1/0x200 [13505.190627] ? set_kthread_struct+0x70/0x70 [13505.191781] ? ret_from_fork+0x1f/0x30 [13509.088054] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11578s! [13509.092392] Showing busy workqueues and worker pools: [13509.094939] workqueue events: flags=0x0 [13509.097031] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13509.100594] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [13509.105082] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13509.108408] pending: kfree_rcu_monitor [13509.110180] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13509.113632] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [13509.118125] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13509.121647] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [13509.125640] workqueue events_long: flags=0x0 [13509.127920] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13509.131723] pending: lu_objects_flush [obdclass] [13509.134194] workqueue events_unbound: flags=0x2 [13509.136458] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13509.139218] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13509.144978] workqueue events_power_efficient: flags=0x80 [13509.148014] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13509.151895] pending: do_cache_clean, neigh_periodic_work, check_lifetime [13509.155161] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13509.157472] pending: neigh_periodic_work [13509.159849] workqueue mm_percpu_wq: flags=0x8 [13509.162027] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13509.165452] pending: vmstat_update, lru_add_drain_per_cpu [13509.168215] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13509.171968] pending: vmstat_update [13509.173664] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13509.176129] pending: lru_add_drain_per_cpu [13509.178034] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [13509.180722] pending: lru_add_drain_per_cpu BAR(48) [13509.182947] workqueue writeback: flags=0x4a [13509.184717] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [13509.187977] pending: wb_workfn [13509.189512] workqueue xprtiod: flags=0xa [13509.191515] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13509.193953] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [13509.197166] workqueue ipv6_addrconf: flags=0x40008 [13509.199361] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13509.201419] pending: addrconf_verify_work [13509.202701] workqueue knbd0-recv: flags=0x1a [13509.203976] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13509.206098] in-flight: 658:recv_work [13509.207283] workqueue ptlrpc_pinger: flags=0x2 [13509.208853] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13509.210838] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13509.213243] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11578s workers=4 idle: 32984 37 8 [13509.215714] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13493s workers=2 idle: 661 [13520.057594] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [13520.059948] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13520.115019] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13520.130522] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13520.147859] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13520.151870] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13520.170124] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13520.177731] RAX: 0000000000000594 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13520.185401] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [13520.193576] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [13520.206534] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [13520.215100] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [13520.228571] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [13520.233219] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13520.237315] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [13520.241602] Call Trace: [13520.244687] [13520.248325] ? show_regs.cold.9+0x22/0x2f [13520.251795] ? watchdog_timer_fn.cold.6+0x62/0x10a [13520.256768] ? __hrtimer_run_queues+0x1c2/0x4d0 [13520.259152] ? watchdog+0x40/0x40 [13520.260780] ? hrtimer_interrupt+0x128/0x2c0 [13520.276834] ? smp_apic_timer_interrupt+0xa6/0x270 [13520.282686] ? apic_timer_interrupt+0xf/0x20 [13520.287359] [13520.294427] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13520.305382] do_raw_spin_lock+0x177/0x190 [13520.316281] _raw_spin_lock+0x21/0x30 [13520.319223] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13520.332100] ? do_raw_spin_unlock+0x75/0x190 [13520.340613] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13520.346853] ? do_wait_intr+0xf0/0xf0 [13520.357068] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13520.366803] mdc_close+0x53b/0xd70 [mdc] [13520.373422] lmv_close+0x18d/0x490 [lmv] [13520.374657] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13520.384754] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13520.386977] ll_md_real_close+0xa5/0x2a0 [lustre] [13520.391870] ll_md_close+0x274/0x800 [lustre] [13520.408061] ll_file_release+0x128/0xc50 [lustre] [13520.414222] __fput+0x10d/0x440 [13520.416394] ____fput+0x12/0x20 [13520.418222] task_work_run+0xc6/0x110 [13520.420549] exit_to_usermode_loop+0x1dd/0x1f0 [13520.425472] do_syscall_64+0x3d6/0x440 [13520.428639] entry_SYSCALL_64_after_hwframe+0x49/0xae [13520.432149] RIP: 0033:0x7f151bee3cd8 [13520.434085] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13520.452590] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13520.460132] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [13520.466167] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [13520.474766] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13520.483572] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13520.490708] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13532.057058] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13532.059822] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [13532.062542] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13532.062618] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13532.062627] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13532.062633] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13532.063962] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13532.063976] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13532.063980] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13532.063985] RAX: 000000000000780d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13532.063987] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [13532.063989] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [13532.063991] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [13532.063993] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [13532.063996] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [13532.063999] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13532.064001] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13532.064009] Call Trace: [13532.064014] [13532.064021] ? show_regs.cold.9+0x22/0x2f [13532.064031] ? watchdog_timer_fn.cold.6+0x62/0x10a [13532.064037] ? __hrtimer_run_queues+0x1c2/0x4d0 [13532.064041] ? watchdog+0x40/0x40 [13532.064046] ? hrtimer_interrupt+0x128/0x2c0 [13532.064050] ? smp_apic_timer_interrupt+0xa6/0x270 [13532.064059] ? apic_timer_interrupt+0xf/0x20 [13532.064064] [13532.064067] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13532.064071] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13532.064077] do_raw_spin_lock+0x177/0x190 [13532.064082] _raw_spin_lock+0x21/0x30 [13532.064087] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13532.064305] process_one_work+0x2c8/0x700 [13532.064314] worker_thread+0x296/0x6e0 [13532.064318] ? rescuer_thread+0x570/0x570 [13532.064322] kthread+0x1d1/0x200 [13532.064326] ? set_kthread_struct+0x70/0x70 [13532.064329] ret_from_fork+0x1f/0x30 [13532.362238] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13532.421725] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13532.434364] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13532.441732] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13532.449959] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13532.471855] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13532.482669] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [13532.494268] RDX: 0000000000004474 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [13532.503578] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [13532.511706] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [13532.518356] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13532.524630] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [13532.530970] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13532.534916] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13532.540859] Call Trace: [13532.544071] [13532.546057] ? show_regs.cold.9+0x22/0x2f [13532.550297] ? watchdog_timer_fn.cold.6+0x62/0x10a [13532.556051] ? __hrtimer_run_queues+0x1c2/0x4d0 [13532.561580] ? watchdog+0x40/0x40 [13532.564698] ? hrtimer_interrupt+0x128/0x2c0 [13532.570838] ? smp_apic_timer_interrupt+0xa6/0x270 [13532.574891] ? apic_timer_interrupt+0xf/0x20 [13532.581807] [13532.583695] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13532.594255] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13532.600235] do_raw_spin_lock+0x177/0x190 [13532.604385] _raw_spin_lock+0x21/0x30 [13532.606077] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13532.610448] obd_mod_free+0x46/0x140 [ptlrpc] [13532.615330] mdc_commit_open+0x91/0xb0 [mdc] [13532.620996] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13532.628365] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13532.636545] after_reply+0xe2c/0x1780 [ptlrpc] [13532.642837] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13532.648927] ? __next_timer_interrupt+0x160/0x160 [13532.655541] ptlrpcd+0xa82/0xe40 [ptlrpc] [13532.660582] ? do_wait_intr+0xf0/0xf0 [13532.662766] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13532.665922] kthread+0x1d1/0x200 [13532.668156] ? set_kthread_struct+0x70/0x70 [13532.669822] ret_from_fork+0x1f/0x30 [13536.052028] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [13536.056531] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13536.074633] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13536.078989] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13536.082444] RIP: 0010:native_safe_halt+0x2e/0x50 [13536.084923] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13536.093233] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13536.101004] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13536.105559] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [13536.108079] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [13536.111922] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [13536.114616] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [13536.118054] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [13536.122131] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13536.125051] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [13536.128136] Call Trace: [13536.129415] [13536.130443] ? show_regs.cold.9+0x22/0x2f [13536.132285] ? watchdog_timer_fn.cold.6+0x62/0x10a [13536.134806] ? __hrtimer_run_queues+0x1c2/0x4d0 [13536.137079] ? watchdog+0x40/0x40 [13536.138932] ? hrtimer_interrupt+0x128/0x2c0 [13536.141144] ? smp_apic_timer_interrupt+0xa6/0x270 [13536.143178] ? apic_timer_interrupt+0xf/0x20 [13536.145052] [13536.145674] ? native_safe_halt+0x2e/0x50 [13536.147110] kvm_wait+0xcd/0xe0 [13536.148470] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13536.150653] do_raw_spin_lock+0x177/0x190 [13536.152349] _raw_spin_lock+0x21/0x30 [13536.153613] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13536.160255] ? do_raw_spin_unlock+0x75/0x190 [13536.162818] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13536.165703] ? do_wait_intr+0xf0/0xf0 [13536.167470] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13536.170295] mdc_reint+0x2d/0x110 [mdc] [13536.171609] mdc_unlink+0x658/0x870 [mdc] [13536.173171] lmv_unlink+0x34d/0xbd0 [lmv] [13536.174723] ll_unlink+0x24d/0x7f0 [lustre] [13536.179834] vfs_unlink+0x202/0x330 [13536.183520] do_unlinkat+0x2fa/0x460 [13536.185116] __x64_sys_unlink+0x37/0x50 [13536.189397] do_syscall_64+0xc1/0x440 [13536.191593] entry_SYSCALL_64_after_hwframe+0x49/0xae [13536.194233] RIP: 0033:0x7f151bdee10b [13536.195319] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [13536.203912] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [13536.209125] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [13536.212495] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [13536.217072] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13536.223136] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13536.229491] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13539.301314] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11608s! [13539.317574] Showing busy workqueues and worker pools: [13539.328349] workqueue events: flags=0x0 [13539.329517] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13539.334068] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [13539.342385] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13539.346322] pending: kfree_rcu_monitor [13539.348604] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13539.357427] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [13539.363931] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13539.377523] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [13539.386454] workqueue events_long: flags=0x0 [13539.390712] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13539.395444] pending: lu_objects_flush [obdclass] [13539.399741] workqueue events_unbound: flags=0x2 [13539.402621] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13539.406325] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13539.411946] workqueue events_power_efficient: flags=0x80 [13539.414576] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13539.418235] pending: do_cache_clean, neigh_periodic_work, check_lifetime [13539.424875] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13539.428679] pending: neigh_periodic_work [13539.431577] workqueue mm_percpu_wq: flags=0x8 [13539.434188] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13539.437908] pending: vmstat_update, lru_add_drain_per_cpu [13539.440696] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13539.447983] pending: vmstat_update [13539.450916] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13539.454459] pending: lru_add_drain_per_cpu [13539.459587] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [13539.465284] pending: lru_add_drain_per_cpu BAR(48) [13539.469775] workqueue writeback: flags=0x4a [13539.472231] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [13539.477399] pending: wb_workfn [13539.479666] workqueue xprtiod: flags=0xa [13539.481957] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13539.488497] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [13539.495073] workqueue ipv6_addrconf: flags=0x40008 [13539.498380] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13539.506409] pending: addrconf_verify_work [13539.510976] workqueue knbd0-recv: flags=0x1a [13539.514213] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13539.522436] in-flight: 658:recv_work [13539.524646] workqueue ptlrpc_pinger: flags=0x2 [13539.532528] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13539.538000] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13539.541909] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11608s workers=4 idle: 32984 37 8 [13539.555886] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13524s workers=2 idle: 661 [13548.057018] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [13548.059448] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13548.088280] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13548.095448] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13548.100387] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13548.103628] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13548.115552] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [13548.119106] RAX: 0000000000006482 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13548.123188] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [13548.126503] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [13548.131152] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [13548.134202] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [13548.138182] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [13548.144388] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13548.150340] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [13548.159356] Call Trace: [13548.161323] [13548.162450] ? show_regs.cold.9+0x22/0x2f [13548.165021] ? watchdog_timer_fn.cold.6+0x62/0x10a [13548.167415] ? __hrtimer_run_queues+0x1c2/0x4d0 [13548.171385] ? watchdog+0x40/0x40 [13548.173734] ? hrtimer_interrupt+0x128/0x2c0 [13548.176185] ? smp_apic_timer_interrupt+0xa6/0x270 [13548.179869] ? apic_timer_interrupt+0xf/0x20 [13548.182188] [13548.184122] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13548.189036] do_raw_spin_lock+0x177/0x190 [13548.191394] _raw_spin_lock+0x21/0x30 [13548.194590] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13548.198018] ? do_raw_spin_unlock+0x75/0x190 [13548.200376] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13548.203674] ? do_wait_intr+0xf0/0xf0 [13548.205370] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13548.207892] mdc_close+0x53b/0xd70 [mdc] [13548.210540] lmv_close+0x18d/0x490 [lmv] [13548.213590] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13548.217047] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13548.222015] ll_md_real_close+0xa5/0x2a0 [lustre] [13548.224713] ll_md_close+0x274/0x800 [lustre] [13548.226794] ll_file_release+0x128/0xc50 [lustre] [13548.232054] __fput+0x10d/0x440 [13548.234461] ____fput+0x12/0x20 [13548.238656] task_work_run+0xc6/0x110 [13548.240586] exit_to_usermode_loop+0x1dd/0x1f0 [13548.243741] do_syscall_64+0x3d6/0x440 [13548.246387] entry_SYSCALL_64_after_hwframe+0x49/0xae [13548.250172] RIP: 0033:0x7f151bee3cd8 [13548.251846] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13548.259458] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13548.262413] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [13548.264530] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [13548.268106] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13548.270934] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13548.273589] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13560.056009] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13560.059587] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [13560.065829] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13560.065911] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13560.065919] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13560.065925] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13560.075075] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13560.075157] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13560.075165] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13560.075168] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13560.075183] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13560.075187] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13560.075192] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [13560.075195] RDX: 0000000000006020 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [13560.075198] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [13560.075200] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [13560.075202] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13560.075205] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [13560.075215] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13560.075218] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13560.075225] Call Trace: [13560.075231] [13560.075237] ? show_regs.cold.9+0x22/0x2f [13560.075248] ? watchdog_timer_fn.cold.6+0x62/0x10a [13560.075254] ? __hrtimer_run_queues+0x1c2/0x4d0 [13560.075259] ? watchdog+0x40/0x40 [13560.075263] ? hrtimer_interrupt+0x128/0x2c0 [13560.075267] ? smp_apic_timer_interrupt+0xa6/0x270 [13560.075283] ? apic_timer_interrupt+0xf/0x20 [13560.075289] [13560.075293] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13560.075297] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13560.075302] do_raw_spin_lock+0x177/0x190 [13560.075307] _raw_spin_lock+0x21/0x30 [13560.075313] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13560.076224] obd_mod_free+0x46/0x140 [ptlrpc] [13560.077241] mdc_commit_open+0x91/0xb0 [mdc] [13560.077281] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13560.077621] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13560.078098] after_reply+0xe2c/0x1780 [ptlrpc] [13560.078293] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13560.078929] ? __next_timer_interrupt+0x160/0x160 [13560.078939] ptlrpcd+0xa82/0xe40 [ptlrpc] [13560.115847] [13560.115856] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13560.115874] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13560.115878] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [13560.115883] RAX: 0000000000005de4 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13560.115885] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [13560.115888] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [13560.115890] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [13560.115893] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [13560.115896] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [13560.115899] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13560.115901] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13560.115910] Call Trace: [13560.115915] [13560.115921] ? show_regs.cold.9+0x22/0x2f [13560.115930] ? watchdog_timer_fn.cold.6+0x62/0x10a [13560.115937] ? __hrtimer_run_queues+0x1c2/0x4d0 [13560.115942] ? watchdog+0x40/0x40 [13560.115947] ? hrtimer_interrupt+0x128/0x2c0 [13560.115951] ? smp_apic_timer_interrupt+0xa6/0x270 [13560.115958] ? apic_timer_interrupt+0xf/0x20 [13560.115963] [13560.115966] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13560.115971] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13560.115979] do_raw_spin_lock+0x177/0x190 [13560.115983] _raw_spin_lock+0x21/0x30 [13560.115988] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13560.116273] process_one_work+0x2c8/0x700 [13560.116281] worker_thread+0x296/0x6e0 [13560.116285] ? rescuer_thread+0x570/0x570 [13560.116289] kthread+0x1d1/0x200 [13560.116293] ? set_kthread_struct+0x70/0x70 [13560.116297] ret_from_fork+0x1f/0x30 [13560.338850] ? do_wait_intr+0xf0/0xf0 [13560.340243] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13560.342262] kthread+0x1d1/0x200 [13560.343417] ? set_kthread_struct+0x70/0x70 [13560.344980] ret_from_fork+0x1f/0x30 [13564.053376] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [13564.059218] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13564.125824] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13564.137257] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13564.151483] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13564.156651] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13564.171419] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13564.180715] RAX: 000000000000528f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13564.188579] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [13564.194678] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [13564.200891] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [13564.204757] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [13564.210766] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [13564.215403] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13564.220926] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [13564.223427] Call Trace: [13564.227191] [13564.229105] ? show_regs.cold.9+0x22/0x2f [13564.234947] ? watchdog_timer_fn.cold.6+0x62/0x10a [13564.243847] ? __hrtimer_run_queues+0x1c2/0x4d0 [13564.250273] ? watchdog+0x40/0x40 [13564.254313] ? hrtimer_interrupt+0x128/0x2c0 [13564.261261] ? smp_apic_timer_interrupt+0xa6/0x270 [13564.264270] ? apic_timer_interrupt+0xf/0x20 [13564.270083] [13564.271766] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13564.280396] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13564.289905] do_raw_spin_lock+0x177/0x190 [13564.297394] _raw_spin_lock+0x21/0x30 [13564.301243] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13564.305671] ? do_raw_spin_unlock+0x75/0x190 [13564.309225] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13564.313751] ? do_wait_intr+0xf0/0xf0 [13564.318198] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13564.328108] mdc_reint+0x2d/0x110 [mdc] [13564.330734] mdc_unlink+0x658/0x870 [mdc] [13564.333984] lmv_unlink+0x34d/0xbd0 [lmv] [13564.340390] ll_unlink+0x24d/0x7f0 [lustre] [13564.346143] vfs_unlink+0x202/0x330 [13564.349995] do_unlinkat+0x2fa/0x460 [13564.352245] __x64_sys_unlink+0x37/0x50 [13564.358743] do_syscall_64+0xc1/0x440 [13564.361749] entry_SYSCALL_64_after_hwframe+0x49/0xae [13564.365661] RIP: 0033:0x7f151bdee10b [13564.367374] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [13564.390119] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [13564.400331] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [13564.405810] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [13564.413397] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13564.423195] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13564.433785] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13570.020251] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11639s! [13570.024288] Showing busy workqueues and worker pools: [13570.026676] workqueue events: flags=0x0 [13570.028163] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13570.032265] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [13570.037528] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13570.040785] pending: kfree_rcu_monitor [13570.042945] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13570.047442] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [13570.051915] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13570.054799] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [13570.057013] workqueue events_long: flags=0x0 [13570.058374] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13570.060880] pending: lu_objects_flush [obdclass] [13570.062943] workqueue events_unbound: flags=0x2 [13570.064900] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13570.067834] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13570.073752] workqueue events_power_efficient: flags=0x80 [13570.076268] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13570.080636] pending: do_cache_clean, neigh_periodic_work, check_lifetime [13570.086234] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13570.090380] pending: neigh_periodic_work [13570.092754] workqueue mm_percpu_wq: flags=0x8 [13570.096307] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13570.100356] pending: vmstat_update, lru_add_drain_per_cpu [13570.103846] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13570.109567] pending: vmstat_update [13570.111042] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13570.115891] pending: lru_add_drain_per_cpu [13570.117779] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [13570.121191] pending: lru_add_drain_per_cpu BAR(48) [13570.123791] workqueue writeback: flags=0x4a [13570.126421] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [13570.129958] pending: wb_workfn [13570.132056] workqueue xprtiod: flags=0xa [13570.133645] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13570.137719] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [13570.143253] workqueue ipv6_addrconf: flags=0x40008 [13570.146334] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13570.149790] pending: addrconf_verify_work [13570.152549] workqueue knbd0-recv: flags=0x1a [13570.155682] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13570.160327] in-flight: 658:recv_work [13570.162190] workqueue ptlrpc_pinger: flags=0x2 [13570.164875] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13570.167996] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13570.171811] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11639s workers=4 idle: 32984 37 8 [13570.175447] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13554s workers=2 idle: 661 [13576.057480] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [13576.063416] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13576.098160] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13576.104826] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13576.108250] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13576.110426] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13576.118206] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13576.121785] RAX: 0000000000006cec RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13576.125007] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [13576.128740] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [13576.132652] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [13576.135646] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [13576.140394] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [13576.149878] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13576.153897] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [13576.159253] Call Trace: [13576.160164] [13576.160892] ? show_regs.cold.9+0x22/0x2f [13576.162424] ? watchdog_timer_fn.cold.6+0x62/0x10a [13576.164980] ? __hrtimer_run_queues+0x1c2/0x4d0 [13576.167599] ? watchdog+0x40/0x40 [13576.169554] ? hrtimer_interrupt+0x128/0x2c0 [13576.171772] ? smp_apic_timer_interrupt+0xa6/0x270 [13576.174998] ? apic_timer_interrupt+0xf/0x20 [13576.177232] [13576.178570] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13576.181644] do_raw_spin_lock+0x177/0x190 [13576.183681] _raw_spin_lock+0x21/0x30 [13576.185585] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13576.188354] ? do_raw_spin_unlock+0x75/0x190 [13576.190432] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13576.192988] ? do_wait_intr+0xf0/0xf0 [13576.194704] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13576.197109] mdc_close+0x53b/0xd70 [mdc] [13576.199611] lmv_close+0x18d/0x490 [lmv] [13576.202714] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13576.205683] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13576.208779] ll_md_real_close+0xa5/0x2a0 [lustre] [13576.211630] ll_md_close+0x274/0x800 [lustre] [13576.214870] ll_file_release+0x128/0xc50 [lustre] [13576.218268] __fput+0x10d/0x440 [13576.220811] ____fput+0x12/0x20 [13576.224648] task_work_run+0xc6/0x110 [13576.230192] exit_to_usermode_loop+0x1dd/0x1f0 [13576.235679] do_syscall_64+0x3d6/0x440 [13576.237589] entry_SYSCALL_64_after_hwframe+0x49/0xae [13576.242794] RIP: 0033:0x7f151bee3cd8 [13576.245854] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13576.268635] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13576.272921] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [13576.278270] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [13576.282714] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13576.286603] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13576.290416] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13588.055053] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [13588.056031] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13588.058378] Modules linked in: lustre(O) [13588.071162] Modules linked in: [13588.071165] osp(O) ofd(O) [13588.074136] lustre(O) [13588.075342] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13588.075417] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13588.075424] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13588.075426] RIP: 0010:native_safe_halt+0x2e/0x50 [13588.075439] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13588.075443] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13588.075448] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [13588.075451] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [13588.075453] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [13588.075455] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [13588.075457] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13588.075460] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [13588.075462] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13588.075464] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13588.075472] Call Trace: [13588.075486] [13588.075492] ? show_regs.cold.9+0x22/0x2f [13588.075502] ? watchdog_timer_fn.cold.6+0x62/0x10a [13588.075508] ? __hrtimer_run_queues+0x1c2/0x4d0 [13588.075512] ? watchdog+0x40/0x40 [13588.075516] ? hrtimer_interrupt+0x128/0x2c0 [13588.075520] ? smp_apic_timer_interrupt+0xa6/0x270 [13588.075526] ? apic_timer_interrupt+0xf/0x20 [13588.075530] [13588.075533] ? native_safe_halt+0x2e/0x50 [13588.075537] kvm_wait+0xcd/0xe0 [13588.075544] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13588.075552] do_raw_spin_lock+0x177/0x190 [13588.075557] _raw_spin_lock+0x21/0x30 [13588.075562] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13588.075846] obd_mod_free+0x46/0x140 [ptlrpc] [13588.076052] mdc_commit_open+0x91/0xb0 [mdc] [13588.076188] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13588.076406] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13588.076765] after_reply+0xe2c/0x1780 [ptlrpc] [13588.076958] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13588.077224] ? __next_timer_interrupt+0x160/0x160 [13588.077233] ptlrpcd+0xa82/0xe40 [ptlrpc] [13588.077449] ? do_wait_intr+0xf0/0xf0 [13588.077458] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13588.077775] kthread+0x1d1/0x200 [13588.077781] ? set_kthread_struct+0x70/0x70 [13588.077785] ret_from_fork+0x1f/0x30 [13588.257851] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13588.286745] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13588.292693] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13588.297303] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13588.304917] RIP: 0010:native_safe_halt+0x2e/0x50 [13588.310590] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13588.323345] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13588.339981] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13588.346711] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [13588.353024] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [13588.360786] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [13588.367810] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [13588.375844] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [13588.379088] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13588.386886] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13588.390505] Call Trace: [13588.392403] [13588.393082] ? show_regs.cold.9+0x22/0x2f [13588.396641] ? watchdog_timer_fn.cold.6+0x62/0x10a [13588.398265] ? __hrtimer_run_queues+0x1c2/0x4d0 [13588.410722] ? watchdog+0x40/0x40 [13588.411820] ? hrtimer_interrupt+0x128/0x2c0 [13588.417526] ? smp_apic_timer_interrupt+0xa6/0x270 [13588.423390] ? apic_timer_interrupt+0xf/0x20 [13588.425989] [13588.428008] ? native_safe_halt+0x2e/0x50 [13588.431720] kvm_wait+0xcd/0xe0 [13588.434152] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13588.438441] do_raw_spin_lock+0x177/0x190 [13588.439958] _raw_spin_lock+0x21/0x30 [13588.445887] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13588.449532] process_one_work+0x2c8/0x700 [13588.451583] worker_thread+0x296/0x6e0 [13588.452999] ? rescuer_thread+0x570/0x570 [13588.456806] kthread+0x1d1/0x200 [13588.459098] ? set_kthread_struct+0x70/0x70 [13588.460399] ret_from_fork+0x1f/0x30 [13596.052026] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [13596.058935] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13596.085950] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13596.091069] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13596.093994] RIP: 0010:native_safe_halt+0x2e/0x50 [13596.095612] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13596.103273] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13596.107436] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13596.110849] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [13596.114304] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [13596.117722] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [13596.121413] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [13596.124887] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [13596.130089] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13596.132938] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [13596.136240] Call Trace: [13596.137134] [13596.141583] ? show_regs.cold.9+0x22/0x2f [13596.145972] ? watchdog_timer_fn.cold.6+0x62/0x10a [13596.151770] ? __hrtimer_run_queues+0x1c2/0x4d0 [13596.154296] ? watchdog+0x40/0x40 [13596.156009] ? hrtimer_interrupt+0x128/0x2c0 [13596.158533] ? smp_apic_timer_interrupt+0xa6/0x270 [13596.161412] ? apic_timer_interrupt+0xf/0x20 [13596.164422] [13596.165666] ? native_safe_halt+0x2e/0x50 [13596.167943] kvm_wait+0xcd/0xe0 [13596.169818] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13596.172753] do_raw_spin_lock+0x177/0x190 [13596.174696] _raw_spin_lock+0x21/0x30 [13596.176453] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13596.179788] ? do_raw_spin_unlock+0x75/0x190 [13596.182073] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13596.185211] ? do_wait_intr+0xf0/0xf0 [13596.189635] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13596.193973] mdc_reint+0x2d/0x110 [mdc] [13596.198760] mdc_unlink+0x658/0x870 [mdc] [13596.202065] lmv_unlink+0x34d/0xbd0 [lmv] [13596.208114] ll_unlink+0x24d/0x7f0 [lustre] [13596.213926] vfs_unlink+0x202/0x330 [13596.215922] do_unlinkat+0x2fa/0x460 [13596.217890] __x64_sys_unlink+0x37/0x50 [13596.219678] do_syscall_64+0xc1/0x440 [13596.222130] entry_SYSCALL_64_after_hwframe+0x49/0xae [13596.224811] RIP: 0033:0x7f151bdee10b [13596.226737] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [13596.236363] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [13596.240150] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [13596.245678] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [13596.249028] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13596.251119] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13596.253741] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13600.226115] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11669s! [13600.229639] Showing busy workqueues and worker pools: [13600.234133] workqueue events: flags=0x0 [13600.245690] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13600.254017] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [13600.264881] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13600.269988] pending: kfree_rcu_monitor [13600.272994] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13600.279305] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [13600.286992] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13600.290190] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [13600.295875] workqueue events_long: flags=0x0 [13600.298804] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13600.309694] pending: lu_objects_flush [obdclass] [13600.311483] workqueue events_unbound: flags=0x2 [13600.313266] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13600.316381] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13600.322920] workqueue events_power_efficient: flags=0x80 [13600.328014] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13600.340044] pending: do_cache_clean, neigh_periodic_work, check_lifetime [13600.344787] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13600.349104] pending: neigh_periodic_work [13600.352806] workqueue mm_percpu_wq: flags=0x8 [13600.355973] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13600.366039] pending: vmstat_update, lru_add_drain_per_cpu [13600.372335] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13600.380053] pending: vmstat_update [13600.383315] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13600.388207] pending: lru_add_drain_per_cpu [13600.390680] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [13600.397986] pending: lru_add_drain_per_cpu BAR(48) [13600.401780] workqueue writeback: flags=0x4a [13600.407086] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [13600.417451] pending: wb_workfn [13600.418797] workqueue xprtiod: flags=0xa [13600.425773] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13600.429063] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [13600.434539] workqueue ipv6_addrconf: flags=0x40008 [13600.438949] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13600.443718] pending: addrconf_verify_work [13600.445780] workqueue knbd0-recv: flags=0x1a [13600.449026] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13600.455882] in-flight: 658:recv_work [13600.457922] workqueue ptlrpc_pinger: flags=0x2 [13600.461102] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13600.465829] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13600.469144] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11669s workers=4 idle: 32984 37 8 [13600.472909] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13585s workers=2 idle: 661 [13604.057083] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [13604.061687] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13604.100422] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13604.107691] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13604.112575] RIP: 0010:native_safe_halt+0x2e/0x50 [13604.115480] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13604.127803] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [13604.132183] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13604.136686] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [13604.140979] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [13604.147364] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [13604.152299] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [13604.156571] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [13604.161324] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13604.164741] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [13604.168936] Call Trace: [13604.170366] [13604.171641] ? show_regs.cold.9+0x22/0x2f [13604.173704] ? watchdog_timer_fn.cold.6+0x62/0x10a [13604.176600] ? __hrtimer_run_queues+0x1c2/0x4d0 [13604.178369] ? watchdog+0x40/0x40 [13604.179433] ? hrtimer_interrupt+0x128/0x2c0 [13604.180879] ? smp_apic_timer_interrupt+0xa6/0x270 [13604.182909] ? apic_timer_interrupt+0xf/0x20 [13604.184909] [13604.185884] ? native_safe_halt+0x2e/0x50 [13604.187472] kvm_wait+0xcd/0xe0 [13604.188611] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13604.191009] do_raw_spin_lock+0x177/0x190 [13604.193965] _raw_spin_lock+0x21/0x30 [13604.195524] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13604.198172] ? do_raw_spin_unlock+0x75/0x190 [13604.199740] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13604.203672] ? do_wait_intr+0xf0/0xf0 [13604.205584] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13604.209291] mdc_close+0x53b/0xd70 [mdc] [13604.210804] lmv_close+0x18d/0x490 [lmv] [13604.211991] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13604.213840] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13604.215681] ll_md_real_close+0xa5/0x2a0 [lustre] [13604.217229] ll_md_close+0x274/0x800 [lustre] [13604.218784] ll_file_release+0x128/0xc50 [lustre] [13604.220974] __fput+0x10d/0x440 [13604.222463] ____fput+0x12/0x20 [13604.223830] task_work_run+0xc6/0x110 [13604.226277] exit_to_usermode_loop+0x1dd/0x1f0 [13604.227680] do_syscall_64+0x3d6/0x440 [13604.229128] entry_SYSCALL_64_after_hwframe+0x49/0xae [13604.230888] RIP: 0033:0x7f151bee3cd8 [13604.232825] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13604.243102] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13604.246927] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [13604.251751] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [13604.254812] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13604.258668] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13604.262166] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13616.055021] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [13616.056662] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13616.057328] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver [13616.065746] Modules linked in: [13616.065748] intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13616.065784] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13616.065791] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13616.065794] RIP: 0010:native_safe_halt+0x2e/0x50 [13616.065807] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13616.065810] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 [13616.088401] lustre(O) [13616.089387] ORIG_RAX: ffffffffffffff13 [13616.089392] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [13616.089394] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [13616.089395] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [13616.089397] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [13616.089399] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13616.089401] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [13616.089403] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13616.089405] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13616.089411] Call Trace: [13616.089416] [13616.089420] ? show_regs.cold.9+0x22/0x2f [13616.089430] ? watchdog_timer_fn.cold.6+0x62/0x10a [13616.089434] ? __hrtimer_run_queues+0x1c2/0x4d0 [13616.089437] ? watchdog+0x40/0x40 [13616.089440] ? hrtimer_interrupt+0x128/0x2c0 [13616.089443] ? smp_apic_timer_interrupt+0xa6/0x270 [13616.089449] ? apic_timer_interrupt+0xf/0x20 [13616.089452] [13616.089454] ? native_safe_halt+0x2e/0x50 [13616.089457] kvm_wait+0xcd/0xe0 [13616.089462] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13616.089468] do_raw_spin_lock+0x177/0x190 [13616.089471] _raw_spin_lock+0x21/0x30 [13616.089476] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13616.090082] obd_mod_free+0x46/0x140 [ptlrpc] [13616.090295] mdc_commit_open+0x91/0xb0 [mdc] [13616.090332] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13616.090518] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13616.090787] after_reply+0xe2c/0x1780 [ptlrpc] [13616.090986] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13616.091190] ? __next_timer_interrupt+0x160/0x160 [13616.091199] ptlrpcd+0xa82/0xe40 [ptlrpc] [13616.091411] ? do_wait_intr+0xf0/0xf0 [13616.091420] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13616.091601] kthread+0x1d1/0x200 [13616.091606] ? set_kthread_struct+0x70/0x70 [13616.091609] ret_from_fork+0x1f/0x30 [13616.290384] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13616.317530] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13616.327431] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13616.334520] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13616.337980] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13616.341403] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13616.347326] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13616.350370] RAX: 00000000000074cd RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13616.353564] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [13616.356191] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [13616.359261] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [13616.361833] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [13616.364642] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [13616.368285] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13616.370823] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13616.374171] Call Trace: [13616.375351] [13616.376351] ? show_regs.cold.9+0x22/0x2f [13616.378264] ? watchdog_timer_fn.cold.6+0x62/0x10a [13616.380447] ? __hrtimer_run_queues+0x1c2/0x4d0 [13616.382632] ? watchdog+0x40/0x40 [13616.384234] ? hrtimer_interrupt+0x128/0x2c0 [13616.385858] ? smp_apic_timer_interrupt+0xa6/0x270 [13616.388726] ? apic_timer_interrupt+0xf/0x20 [13616.390495] [13616.391515] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13616.393812] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13616.396267] do_raw_spin_lock+0x177/0x190 [13616.397661] _raw_spin_lock+0x21/0x30 [13616.399013] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13616.401399] process_one_work+0x2c8/0x700 [13616.403328] worker_thread+0x296/0x6e0 [13616.405279] ? rescuer_thread+0x570/0x570 [13616.406825] kthread+0x1d1/0x200 [13616.408461] ? set_kthread_struct+0x70/0x70 [13616.410509] ret_from_fork+0x1f/0x30 [13628.052267] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [13628.057676] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13628.085201] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13628.091549] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13628.096380] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13628.099667] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13628.113131] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13628.116134] RAX: 0000000000000196 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13628.118643] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [13628.121322] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [13628.124657] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [13628.128611] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [13628.132220] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [13628.136432] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13628.139548] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [13628.142709] Call Trace: [13628.143792] [13628.144937] ? show_regs.cold.9+0x22/0x2f [13628.146631] ? watchdog_timer_fn.cold.6+0x62/0x10a [13628.148587] ? __hrtimer_run_queues+0x1c2/0x4d0 [13628.151320] ? watchdog+0x40/0x40 [13628.152961] ? hrtimer_interrupt+0x128/0x2c0 [13628.154915] ? smp_apic_timer_interrupt+0xa6/0x270 [13628.156523] ? apic_timer_interrupt+0xf/0x20 [13628.157670] [13628.159124] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13628.161269] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13628.164738] do_raw_spin_lock+0x177/0x190 [13628.166174] _raw_spin_lock+0x21/0x30 [13628.167600] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13628.170175] ? do_raw_spin_unlock+0x75/0x190 [13628.171761] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13628.176653] ? do_wait_intr+0xf0/0xf0 [13628.181806] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13628.185546] mdc_reint+0x2d/0x110 [mdc] [13628.190061] mdc_unlink+0x658/0x870 [mdc] [13628.193127] lmv_unlink+0x34d/0xbd0 [lmv] [13628.194421] ll_unlink+0x24d/0x7f0 [lustre] [13628.197519] vfs_unlink+0x202/0x330 [13628.199795] do_unlinkat+0x2fa/0x460 [13628.202625] __x64_sys_unlink+0x37/0x50 [13628.205118] do_syscall_64+0xc1/0x440 [13628.207588] entry_SYSCALL_64_after_hwframe+0x49/0xae [13628.211719] RIP: 0033:0x7f151bdee10b [13628.217193] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [13628.231572] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [13628.235443] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [13628.238772] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [13628.242057] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13628.245555] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13628.248121] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13630.945178] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11700s! [13630.948432] Showing busy workqueues and worker pools: [13630.950622] workqueue events: flags=0x0 [13630.952553] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13630.955587] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [13630.959670] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13630.962582] pending: kfree_rcu_monitor [13630.965378] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13630.969150] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [13630.973604] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13630.976838] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [13630.980657] workqueue events_long: flags=0x0 [13630.985411] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13630.992477] pending: lu_objects_flush [obdclass] [13630.993842] workqueue events_unbound: flags=0x2 [13631.003612] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13631.008899] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13631.017370] workqueue events_power_efficient: flags=0x80 [13631.025722] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13631.030594] pending: do_cache_clean, neigh_periodic_work, check_lifetime [13631.037312] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13631.047320] pending: neigh_periodic_work [13631.051333] workqueue mm_percpu_wq: flags=0x8 [13631.054913] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13631.060672] pending: vmstat_update, lru_add_drain_per_cpu [13631.064428] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13631.071113] pending: vmstat_update [13631.073679] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13631.080648] pending: lru_add_drain_per_cpu [13631.085346] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [13631.096621] pending: lru_add_drain_per_cpu BAR(48) [13631.102072] workqueue writeback: flags=0x4a [13631.104318] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [13631.107362] pending: wb_workfn [13631.109394] workqueue xprtiod: flags=0xa [13631.111615] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13631.115663] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [13631.122562] workqueue ipv6_addrconf: flags=0x40008 [13631.124173] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13631.134517] pending: addrconf_verify_work [13631.136773] workqueue knbd0-recv: flags=0x1a [13631.140544] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13631.146585] in-flight: 658:recv_work [13631.149021] workqueue ptlrpc_pinger: flags=0x2 [13631.153164] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13631.163595] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13631.166721] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11700s workers=4 idle: 32984 37 8 [13631.171470] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13615s workers=2 idle: 661 [13632.057233] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [13632.065701] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13632.090493] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13632.098042] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13632.102098] RIP: 0010:native_safe_halt+0x2e/0x50 [13632.103634] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13632.118451] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13632.122186] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13632.126284] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [13632.131230] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [13632.135135] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [13632.137451] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [13632.140513] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [13632.145106] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13632.147651] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [13632.150797] Call Trace: [13632.151395] [13632.151877] ? show_regs.cold.9+0x22/0x2f [13632.152776] ? watchdog_timer_fn.cold.6+0x62/0x10a [13632.155412] ? __hrtimer_run_queues+0x1c2/0x4d0 [13632.157297] ? watchdog+0x40/0x40 [13632.158599] ? hrtimer_interrupt+0x128/0x2c0 [13632.160903] ? smp_apic_timer_interrupt+0xa6/0x270 [13632.163309] ? apic_timer_interrupt+0xf/0x20 [13632.165562] [13632.166665] ? native_safe_halt+0x2e/0x50 [13632.169302] kvm_wait+0xcd/0xe0 [13632.171778] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13632.175867] do_raw_spin_lock+0x177/0x190 [13632.177985] _raw_spin_lock+0x21/0x30 [13632.181678] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13632.184320] ? do_raw_spin_unlock+0x75/0x190 [13632.186517] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13632.189886] ? do_wait_intr+0xf0/0xf0 [13632.191466] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13632.193784] mdc_close+0x53b/0xd70 [mdc] [13632.194775] lmv_close+0x18d/0x490 [lmv] [13632.198134] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13632.200417] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13632.202866] ll_md_real_close+0xa5/0x2a0 [lustre] [13632.204520] ll_md_close+0x274/0x800 [lustre] [13632.205970] ll_file_release+0x128/0xc50 [lustre] [13632.209120] __fput+0x10d/0x440 [13632.210866] ____fput+0x12/0x20 [13632.212893] task_work_run+0xc6/0x110 [13632.215066] exit_to_usermode_loop+0x1dd/0x1f0 [13632.217469] do_syscall_64+0x3d6/0x440 [13632.219866] entry_SYSCALL_64_after_hwframe+0x49/0xae [13632.222538] RIP: 0033:0x7f151bee3cd8 [13632.224201] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13632.231472] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13632.234407] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [13632.241389] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [13632.246243] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13632.248995] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13632.254152] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13644.055107] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [13644.056045] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [13644.068329] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13644.068494] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13644.068502] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13644.068505] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13644.068518] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13644.068523] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13644.068534] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [13644.068537] RDX: 0000000000004e86 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [13644.068539] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [13644.068541] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [13644.068544] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13644.068547] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [13644.068550] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13644.068553] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13644.068562] Call Trace: [13644.068568] [13644.068574] ? show_regs.cold.9+0x22/0x2f [13644.068585] ? watchdog_timer_fn.cold.6+0x62/0x10a [13644.068592] ? __hrtimer_run_queues+0x1c2/0x4d0 [13644.068596] ? watchdog+0x40/0x40 [13644.068601] ? hrtimer_interrupt+0x128/0x2c0 [13644.068606] ? smp_apic_timer_interrupt+0xa6/0x270 [13644.068622] ? apic_timer_interrupt+0xf/0x20 [13644.068627] [13644.068631] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13644.068636] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13644.068641] do_raw_spin_lock+0x177/0x190 [13644.068651] _raw_spin_lock+0x21/0x30 [13644.068656] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13644.069206] obd_mod_free+0x46/0x140 [ptlrpc] [13644.069503] mdc_commit_open+0x91/0xb0 [mdc] [13644.069540] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13644.069753] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13644.070475] after_reply+0xe2c/0x1780 [ptlrpc] [13644.070779] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13644.076413] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13644.076494] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13644.076501] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13644.076508] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13644.077006] RIP: 0010:native_safe_halt+0x2e/0x50 [13644.077017] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13644.077021] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13644.077026] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13644.077034] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [13644.077036] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [13644.077039] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [13644.077041] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [13644.077044] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [13644.077047] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13644.077050] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13644.077058] Call Trace: [13644.077064] [13644.077141] ? show_regs.cold.9+0x22/0x2f [13644.077153] ? watchdog_timer_fn.cold.6+0x62/0x10a [13644.077159] ? __hrtimer_run_queues+0x1c2/0x4d0 [13644.077163] ? watchdog+0x40/0x40 [13644.077167] ? hrtimer_interrupt+0x128/0x2c0 [13644.077171] ? smp_apic_timer_interrupt+0xa6/0x270 [13644.077177] ? apic_timer_interrupt+0xf/0x20 [13644.077182] [13644.077186] ? native_safe_halt+0x2e/0x50 [13644.077190] kvm_wait+0xcd/0xe0 [13644.077204] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13644.077212] do_raw_spin_lock+0x177/0x190 [13644.077217] _raw_spin_lock+0x21/0x30 [13644.077221] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13644.077516] process_one_work+0x2c8/0x700 [13644.077523] worker_thread+0x296/0x6e0 [13644.077528] ? rescuer_thread+0x570/0x570 [13644.077531] kthread+0x1d1/0x200 [13644.077536] ? set_kthread_struct+0x70/0x70 [13644.077539] ret_from_fork+0x1f/0x30 [13644.604627] ? __next_timer_interrupt+0x160/0x160 [13644.609055] ptlrpcd+0xa82/0xe40 [ptlrpc] [13644.612663] ? do_wait_intr+0xf0/0xf0 [13644.615894] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13644.626830] kthread+0x1d1/0x200 [13644.638118] ? set_kthread_struct+0x70/0x70 [13644.639685] ret_from_fork+0x1f/0x30 [13656.052303] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [13656.056175] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13656.107377] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13656.114771] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13656.120402] RIP: 0010:native_safe_halt+0x2e/0x50 [13656.123365] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13656.147017] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13656.155257] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13656.163248] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [13656.173150] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [13656.177882] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [13656.189446] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [13656.194815] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [13656.206112] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13656.215819] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [13656.221265] Call Trace: [13656.223036] [13656.223884] ? show_regs.cold.9+0x22/0x2f [13656.227042] ? watchdog_timer_fn.cold.6+0x62/0x10a [13656.229306] ? __hrtimer_run_queues+0x1c2/0x4d0 [13656.230779] ? watchdog+0x40/0x40 [13656.231952] ? hrtimer_interrupt+0x128/0x2c0 [13656.239820] ? smp_apic_timer_interrupt+0xa6/0x270 [13656.242851] ? apic_timer_interrupt+0xf/0x20 [13656.245139] [13656.246353] ? native_safe_halt+0x2e/0x50 [13656.249314] kvm_wait+0xcd/0xe0 [13656.250936] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13656.253503] do_raw_spin_lock+0x177/0x190 [13656.255677] _raw_spin_lock+0x21/0x30 [13656.257197] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13656.262279] ? do_raw_spin_unlock+0x75/0x190 [13656.266955] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13656.273286] ? do_wait_intr+0xf0/0xf0 [13656.274773] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13656.280225] mdc_reint+0x2d/0x110 [mdc] [13656.282236] mdc_unlink+0x658/0x870 [mdc] [13656.283903] lmv_unlink+0x34d/0xbd0 [lmv] [13656.285297] ll_unlink+0x24d/0x7f0 [lustre] [13656.288374] vfs_unlink+0x202/0x330 [13656.290785] do_unlinkat+0x2fa/0x460 [13656.293474] __x64_sys_unlink+0x37/0x50 [13656.296468] do_syscall_64+0xc1/0x440 [13656.298848] entry_SYSCALL_64_after_hwframe+0x49/0xae [13656.301988] RIP: 0033:0x7f151bdee10b [13656.303980] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [13656.314218] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [13656.317444] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [13656.321894] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [13656.325953] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13656.333919] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13656.340970] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13660.057100] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [13660.062315] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13660.105349] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13660.114704] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13660.125314] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13660.137044] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13660.156709] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13660.164728] RAX: 000000000000631c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13660.174053] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [13660.179790] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [13660.187736] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [13660.193873] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [13660.198059] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [13660.204161] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13660.207662] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [13660.209487] Call Trace: [13660.210346] [13660.211124] ? show_regs.cold.9+0x22/0x2f [13660.212781] ? watchdog_timer_fn.cold.6+0x62/0x10a [13660.214761] ? __hrtimer_run_queues+0x1c2/0x4d0 [13660.216618] ? watchdog+0x40/0x40 [13660.223220] ? hrtimer_interrupt+0x128/0x2c0 [13660.227336] ? smp_apic_timer_interrupt+0xa6/0x270 [13660.233466] ? apic_timer_interrupt+0xf/0x20 [13660.240957] [13660.245309] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13660.251373] do_raw_spin_lock+0x177/0x190 [13660.254525] _raw_spin_lock+0x21/0x30 [13660.261045] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13660.267282] ? do_raw_spin_unlock+0x75/0x190 [13660.269419] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13660.276129] ? do_wait_intr+0xf0/0xf0 [13660.277455] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13660.280258] mdc_close+0x53b/0xd70 [mdc] [13660.281892] lmv_close+0x18d/0x490 [lmv] [13660.283452] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13660.289390] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13660.294220] ll_md_real_close+0xa5/0x2a0 [lustre] [13660.299047] ll_md_close+0x274/0x800 [lustre] [13660.302110] ll_file_release+0x128/0xc50 [lustre] [13660.306365] __fput+0x10d/0x440 [13660.307985] ____fput+0x12/0x20 [13660.311451] task_work_run+0xc6/0x110 [13660.315504] exit_to_usermode_loop+0x1dd/0x1f0 [13660.321609] do_syscall_64+0x3d6/0x440 [13660.327730] entry_SYSCALL_64_after_hwframe+0x49/0xae [13660.342136] RIP: 0033:0x7f151bee3cd8 [13660.346756] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13660.363160] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13660.370199] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [13660.380458] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [13660.394061] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13660.409033] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13660.419394] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13661.664373] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11731s! [13661.668100] Showing busy workqueues and worker pools: [13661.679938] workqueue events: flags=0x0 [13661.687039] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13661.696237] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [13661.708330] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13661.714251] pending: kfree_rcu_monitor [13661.719164] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13661.728293] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [13661.734226] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13661.739985] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [13661.745920] workqueue events_long: flags=0x0 [13661.748401] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13661.752645] pending: lu_objects_flush [obdclass] [13661.755026] workqueue events_unbound: flags=0x2 [13661.759289] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13661.763498] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13661.768974] workqueue events_power_efficient: flags=0x80 [13661.772707] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13661.775968] pending: do_cache_clean, neigh_periodic_work, check_lifetime [13661.779514] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13661.783343] pending: neigh_periodic_work [13661.785778] workqueue mm_percpu_wq: flags=0x8 [13661.789896] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13661.791887] pending: vmstat_update, lru_add_drain_per_cpu [13661.794948] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13661.798292] pending: vmstat_update [13661.799986] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13661.803346] pending: lru_add_drain_per_cpu [13661.812477] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [13661.817302] pending: lru_add_drain_per_cpu BAR(48) [13661.819458] workqueue writeback: flags=0x4a [13661.821628] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [13661.824861] pending: wb_workfn [13661.828994] workqueue xprtiod: flags=0xa [13661.832986] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13661.837905] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [13661.843036] workqueue ipv6_addrconf: flags=0x40008 [13661.846017] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13661.851464] pending: addrconf_verify_work [13661.855444] workqueue knbd0-recv: flags=0x1a [13661.859678] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13661.866482] in-flight: 658:recv_work [13661.869768] workqueue ptlrpc_pinger: flags=0x2 [13661.872424] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13661.876495] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13661.885132] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11731s workers=4 idle: 32984 37 8 [13661.892262] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13646s workers=2 idle: 661 [13672.056407] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [13672.058186] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [13672.060436] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13672.060510] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13672.060518] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13672.060525] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13672.060820] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13672.060833] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13672.060837] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13672.060841] RAX: 0000000000001c06 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13672.060843] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [13672.060845] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [13672.060847] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [13672.060849] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [13672.060852] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [13672.060854] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13672.060857] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13672.060865] Call Trace: [13672.060871] [13672.060877] ? show_regs.cold.9+0x22/0x2f [13672.060885] ? watchdog_timer_fn.cold.6+0x62/0x10a [13672.060891] ? __hrtimer_run_queues+0x1c2/0x4d0 [13672.060895] ? watchdog+0x40/0x40 [13672.060899] ? hrtimer_interrupt+0x128/0x2c0 [13672.060903] ? smp_apic_timer_interrupt+0xa6/0x270 [13672.060910] ? apic_timer_interrupt+0xf/0x20 [13672.060914] [13672.060916] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13672.060921] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13672.060926] do_raw_spin_lock+0x177/0x190 [13672.060932] _raw_spin_lock+0x21/0x30 [13672.060937] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13672.061436] process_one_work+0x2c8/0x700 [13672.061443] worker_thread+0x296/0x6e0 [13672.061448] ? rescuer_thread+0x570/0x570 [13672.061452] kthread+0x1d1/0x200 [13672.061456] ? set_kthread_struct+0x70/0x70 [13672.061459] ret_from_fork+0x1f/0x30 [13672.212890] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13672.237784] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13672.245800] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13672.250654] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13672.253365] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13672.263793] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13672.268113] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [13672.277893] RDX: 0000000000001d99 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [13672.281174] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [13672.284843] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [13672.288642] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13672.295184] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [13672.300009] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13672.303218] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13672.306567] Call Trace: [13672.307328] [13672.307866] ? show_regs.cold.9+0x22/0x2f [13672.314994] ? watchdog_timer_fn.cold.6+0x62/0x10a [13672.316830] ? __hrtimer_run_queues+0x1c2/0x4d0 [13672.325457] ? watchdog+0x40/0x40 [13672.327232] ? hrtimer_interrupt+0x128/0x2c0 [13672.332959] ? smp_apic_timer_interrupt+0xa6/0x270 [13672.334488] ? apic_timer_interrupt+0xf/0x20 [13672.335693] [13672.336343] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13672.346115] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13672.348981] do_raw_spin_lock+0x177/0x190 [13672.356341] _raw_spin_lock+0x21/0x30 [13672.359984] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13672.362061] obd_mod_free+0x46/0x140 [ptlrpc] [13672.372833] mdc_commit_open+0x91/0xb0 [mdc] [13672.381901] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13672.391627] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13672.396106] after_reply+0xe2c/0x1780 [ptlrpc] [13672.401729] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13672.411912] ? __next_timer_interrupt+0x160/0x160 [13672.420194] ptlrpcd+0xa82/0xe40 [ptlrpc] [13672.423685] ? do_wait_intr+0xf0/0xf0 [13672.426492] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13672.432933] kthread+0x1d1/0x200 [13672.438382] ? set_kthread_struct+0x70/0x70 [13672.444766] ret_from_fork+0x1f/0x30 [13684.900024] rcu: INFO: rcu_sched self-detected stall on CPU [13684.903722] rcu: 1-...!: (6882899 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [13684.909209] (t=11760253 jiffies g=249657 q=6014) [13684.912002] rcu: rcu_sched kthread starved for 11755205 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [13684.916304] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [13684.919244] rcu: RCU grace-period kthread stack dump: [13684.920572] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [13684.924216] Call Trace: [13684.925108] __schedule+0x351/0xcb0 [13684.926607] schedule+0xc0/0x180 [13684.927850] schedule_timeout+0xb4/0x190 [13684.929796] ? do_raw_spin_unlock+0x75/0x190 [13684.933804] ? __next_timer_interrupt+0x160/0x160 [13684.935940] ? prepare_to_swait_event+0xf3/0x140 [13684.937900] rcu_gp_kthread+0x8dd/0x15b0 [13684.939460] ? force_qs_rnp+0x2c0/0x2c0 [13684.941029] kthread+0x1d1/0x200 [13684.942694] ? set_kthread_struct+0x70/0x70 [13684.944576] ret_from_fork+0x1f/0x30 [13684.946022] rcu: Stack dump where RCU GP kthread last ran: [13684.948582] Sending NMI from CPU 1 to CPUs 0: [13684.950439] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [13684.951650] NMI backtrace for cpu 1 [13684.956690] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13684.966711] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13684.971109] Call Trace: [13684.971960] [13684.974062] ? dump_stack+0xbb/0x10e [13684.977183] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [13684.980150] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [13684.984777] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [13684.992753] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [13684.995077] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [13684.997753] ? rcu_dump_cpu_stacks+0x11b/0x171 [13684.999988] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [13685.002222] ? update_process_times+0x6d/0xc0 [13685.004988] ? tick_sched_handle.isra.18+0x27/0x90 [13685.008211] ? tick_sched_timer+0x53/0xb0 [13685.010181] ? __hrtimer_run_queues+0x1c2/0x4d0 [13685.012200] ? tick_sched_handle.isra.18+0x90/0x90 [13685.015762] ? hrtimer_interrupt+0x128/0x2c0 [13685.017234] ? smp_apic_timer_interrupt+0xa6/0x270 [13685.021113] ? apic_timer_interrupt+0xf/0x20 [13685.025798] [13685.028405] ? native_safe_halt+0x2e/0x50 [13685.032190] ? kvm_wait+0xcd/0xe0 [13685.038649] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13685.040306] ? do_raw_spin_lock+0x177/0x190 [13685.043813] ? _raw_spin_lock+0x21/0x30 [13685.045249] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13685.047815] ? obd_mod_free+0x46/0x140 [ptlrpc] [13685.050092] ? mdc_commit_open+0x91/0xb0 [mdc] [13685.052044] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13685.054639] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13685.057444] ? after_reply+0xe2c/0x1780 [ptlrpc] [13685.060885] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13685.063598] ? __next_timer_interrupt+0x160/0x160 [13685.065823] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [13685.070760] ? do_wait_intr+0xf0/0xf0 [13685.072643] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13685.075157] ? kthread+0x1d1/0x200 [13685.076568] ? set_kthread_struct+0x70/0x70 [13685.081397] ? ret_from_fork+0x1f/0x30 [13688.057068] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [13688.060528] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13688.113695] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13688.123151] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13688.129413] RIP: 0010:native_safe_halt+0x2e/0x50 [13688.135308] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13688.166193] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13688.173108] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13688.179232] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [13688.187140] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [13688.194135] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [13688.200141] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [13688.205072] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [13688.214620] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13688.218136] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [13688.226811] Call Trace: [13688.228097] [13688.229495] ? show_regs.cold.9+0x22/0x2f [13688.247175] ? watchdog_timer_fn.cold.6+0x62/0x10a [13688.258290] ? __hrtimer_run_queues+0x1c2/0x4d0 [13688.262916] ? watchdog+0x40/0x40 [13688.265857] ? hrtimer_interrupt+0x128/0x2c0 [13688.268358] ? smp_apic_timer_interrupt+0xa6/0x270 [13688.270345] ? apic_timer_interrupt+0xf/0x20 [13688.272146] [13688.274065] ? native_safe_halt+0x2e/0x50 [13688.277829] kvm_wait+0xcd/0xe0 [13688.280868] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13688.283920] do_raw_spin_lock+0x177/0x190 [13688.286436] _raw_spin_lock+0x21/0x30 [13688.290763] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13688.294039] ? do_raw_spin_unlock+0x75/0x190 [13688.296197] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13688.299171] ? do_wait_intr+0xf0/0xf0 [13688.300836] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13688.304699] mdc_close+0x53b/0xd70 [mdc] [13688.306717] lmv_close+0x18d/0x490 [lmv] [13688.309889] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13688.315899] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13688.322379] ll_md_real_close+0xa5/0x2a0 [lustre] [13688.324316] ll_md_close+0x274/0x800 [lustre] [13688.331303] ll_file_release+0x128/0xc50 [lustre] [13688.334152] __fput+0x10d/0x440 [13688.336263] ____fput+0x12/0x20 [13688.338086] task_work_run+0xc6/0x110 [13688.340014] exit_to_usermode_loop+0x1dd/0x1f0 [13688.345695] do_syscall_64+0x3d6/0x440 [13688.348472] entry_SYSCALL_64_after_hwframe+0x49/0xae [13688.358170] RIP: 0033:0x7f151bee3cd8 [13688.359404] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13688.379295] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13688.389064] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [13688.396408] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [13688.404737] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13688.409204] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13688.418817] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13692.391465] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11761s! [13692.406134] Showing busy workqueues and worker pools: [13692.411968] workqueue events: flags=0x0 [13692.415351] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13692.421162] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [13692.429534] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13692.434688] pending: kfree_rcu_monitor [13692.438520] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13692.444546] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [13692.451282] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13692.458366] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [13692.462665] workqueue events_long: flags=0x0 [13692.467790] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13692.470385] pending: lu_objects_flush [obdclass] [13692.472284] workqueue events_unbound: flags=0x2 [13692.477718] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13692.484017] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13692.501674] workqueue events_power_efficient: flags=0x80 [13692.506214] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13692.512429] pending: do_cache_clean, neigh_periodic_work, check_lifetime [13692.518009] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13692.523855] pending: neigh_periodic_work [13692.525707] workqueue mm_percpu_wq: flags=0x8 [13692.531866] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13692.539311] pending: vmstat_update, lru_add_drain_per_cpu [13692.544586] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13692.548232] pending: vmstat_update [13692.551515] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13692.554690] pending: lru_add_drain_per_cpu [13692.562674] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [13692.568444] pending: lru_add_drain_per_cpu BAR(48) [13692.570603] workqueue writeback: flags=0x4a [13692.575768] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [13692.579447] pending: wb_workfn [13692.581815] workqueue xprtiod: flags=0xa [13692.585186] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13692.590418] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [13692.596609] workqueue ipv6_addrconf: flags=0x40008 [13692.601369] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13692.606190] pending: addrconf_verify_work [13692.610053] workqueue knbd0-recv: flags=0x1a [13692.614258] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13692.621098] in-flight: 658:recv_work [13692.622883] workqueue ptlrpc_pinger: flags=0x2 [13692.625038] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13692.627550] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13692.631329] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11762s workers=4 idle: 32984 37 8 [13692.635678] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13677s workers=2 idle: 661 [13700.056041] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [13700.058952] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13700.075505] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13700.079572] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13700.082824] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13700.085497] RIP: 0010:native_safe_halt+0x2e/0x50 [13700.086969] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13700.093251] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13700.095813] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13700.098416] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [13700.101021] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [13700.103095] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [13700.105300] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [13700.107670] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [13700.110309] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13700.112499] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13700.114765] Call Trace: [13700.115857] [13700.116487] ? show_regs.cold.9+0x22/0x2f [13700.117848] ? watchdog_timer_fn.cold.6+0x62/0x10a [13700.119344] ? __hrtimer_run_queues+0x1c2/0x4d0 [13700.120984] ? watchdog+0x40/0x40 [13700.122123] ? hrtimer_interrupt+0x128/0x2c0 [13700.123816] ? smp_apic_timer_interrupt+0xa6/0x270 [13700.125223] ? apic_timer_interrupt+0xf/0x20 [13700.126914] [13700.127571] ? native_safe_halt+0x2e/0x50 [13700.128943] kvm_wait+0xcd/0xe0 [13700.129996] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13700.131843] do_raw_spin_lock+0x177/0x190 [13700.133095] _raw_spin_lock+0x21/0x30 [13700.134394] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13700.136566] process_one_work+0x2c8/0x700 [13700.137757] worker_thread+0x296/0x6e0 [13700.139036] ? rescuer_thread+0x570/0x570 [13700.140705] kthread+0x1d1/0x200 [13700.141988] ? set_kthread_struct+0x70/0x70 [13700.143506] ret_from_fork+0x1f/0x30 [13712.055053] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [13712.058721] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13712.077793] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13712.081438] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13712.085316] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13712.087985] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13712.096099] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13712.099618] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [13712.102629] RDX: 0000000000004c25 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [13712.105609] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [13712.108872] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [13712.112151] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13712.115304] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [13712.119120] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13712.122946] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13712.125638] Call Trace: [13712.128421] [13712.129274] ? show_regs.cold.9+0x22/0x2f [13712.131321] ? watchdog_timer_fn.cold.6+0x62/0x10a [13712.133311] ? __hrtimer_run_queues+0x1c2/0x4d0 [13712.135890] ? watchdog+0x40/0x40 [13712.137411] ? hrtimer_interrupt+0x128/0x2c0 [13712.139518] ? smp_apic_timer_interrupt+0xa6/0x270 [13712.141901] ? apic_timer_interrupt+0xf/0x20 [13712.143722] [13712.144780] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13712.147578] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13712.149967] do_raw_spin_lock+0x177/0x190 [13712.151793] _raw_spin_lock+0x21/0x30 [13712.153382] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13712.155876] obd_mod_free+0x46/0x140 [ptlrpc] [13712.158271] mdc_commit_open+0x91/0xb0 [mdc] [13712.160244] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13712.162515] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13712.165227] after_reply+0xe2c/0x1780 [ptlrpc] [13712.167532] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13712.172756] ? __next_timer_interrupt+0x160/0x160 [13712.175852] ptlrpcd+0xa82/0xe40 [ptlrpc] [13712.181891] ? do_wait_intr+0xf0/0xf0 [13712.185723] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13712.190979] kthread+0x1d1/0x200 [13712.194587] ? set_kthread_struct+0x70/0x70 [13712.196445] ret_from_fork+0x1f/0x30 [13716.059460] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [13716.073131] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13716.120108] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13716.132877] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13716.143610] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13716.149833] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13716.177187] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13716.189513] RAX: 0000000000002a83 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13716.195587] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [13716.199369] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [13716.203294] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [13716.206771] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [13716.210612] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [13716.215053] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13716.217709] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [13716.221940] Call Trace: [13716.223234] [13716.224255] ? show_regs.cold.9+0x22/0x2f [13716.228265] ? watchdog_timer_fn.cold.6+0x62/0x10a [13716.232663] ? __hrtimer_run_queues+0x1c2/0x4d0 [13716.239053] ? watchdog+0x40/0x40 [13716.242603] ? hrtimer_interrupt+0x128/0x2c0 [13716.247671] ? smp_apic_timer_interrupt+0xa6/0x270 [13716.250426] ? apic_timer_interrupt+0xf/0x20 [13716.253488] [13716.254810] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13716.258423] do_raw_spin_lock+0x177/0x190 [13716.260751] _raw_spin_lock+0x21/0x30 [13716.263649] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13716.268822] ? do_raw_spin_unlock+0x75/0x190 [13716.271722] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13716.275410] ? do_wait_intr+0xf0/0xf0 [13716.281670] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13716.289120] mdc_close+0x53b/0xd70 [mdc] [13716.291692] lmv_close+0x18d/0x490 [lmv] [13716.298389] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13716.303934] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13716.308187] ll_md_real_close+0xa5/0x2a0 [lustre] [13716.317242] ll_md_close+0x274/0x800 [lustre] [13716.320167] ll_file_release+0x128/0xc50 [lustre] [13716.327236] __fput+0x10d/0x440 [13716.329107] ____fput+0x12/0x20 [13716.331365] task_work_run+0xc6/0x110 [13716.333157] exit_to_usermode_loop+0x1dd/0x1f0 [13716.337375] do_syscall_64+0x3d6/0x440 [13716.339144] entry_SYSCALL_64_after_hwframe+0x49/0xae [13716.342744] RIP: 0033:0x7f151bee3cd8 [13716.347425] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13716.365297] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13716.369670] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [13716.377762] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [13716.386138] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13716.394953] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13716.406070] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13720.052037] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [13720.054893] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13720.075318] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13720.080201] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13720.084143] RIP: 0010:native_safe_halt+0x2e/0x50 [13720.086565] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13720.095073] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13720.098370] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13720.101825] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [13720.104656] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [13720.107813] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [13720.111287] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [13720.114093] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [13720.117299] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13720.119909] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [13720.123267] Call Trace: [13720.124428] [13720.125243] ? show_regs.cold.9+0x22/0x2f [13720.127190] ? watchdog_timer_fn.cold.6+0x62/0x10a [13720.128841] ? __hrtimer_run_queues+0x1c2/0x4d0 [13720.130395] ? watchdog+0x40/0x40 [13720.132150] ? hrtimer_interrupt+0x128/0x2c0 [13720.134211] ? smp_apic_timer_interrupt+0xa6/0x270 [13720.136374] ? apic_timer_interrupt+0xf/0x20 [13720.138466] [13720.139209] ? native_safe_halt+0x2e/0x50 [13720.140860] kvm_wait+0xcd/0xe0 [13720.142238] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13720.144901] do_raw_spin_lock+0x177/0x190 [13720.146346] _raw_spin_lock+0x21/0x30 [13720.148187] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13720.150660] ? do_raw_spin_unlock+0x75/0x190 [13720.152930] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13720.155661] ? do_wait_intr+0xf0/0xf0 [13720.157406] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13720.160263] mdc_reint+0x2d/0x110 [mdc] [13720.161651] mdc_unlink+0x658/0x870 [mdc] [13720.164096] lmv_unlink+0x34d/0xbd0 [lmv] [13720.167175] ll_unlink+0x24d/0x7f0 [lustre] [13720.169128] vfs_unlink+0x202/0x330 [13720.170494] do_unlinkat+0x2fa/0x460 [13720.172231] __x64_sys_unlink+0x37/0x50 [13720.174254] do_syscall_64+0xc1/0x440 [13720.175937] entry_SYSCALL_64_after_hwframe+0x49/0xae [13720.177948] RIP: 0033:0x7f151bdee10b [13720.179493] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [13720.187322] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [13720.190499] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [13720.193568] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [13720.196873] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13720.199652] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13720.202901] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13723.106189] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11792s! [13723.114064] Showing busy workqueues and worker pools: [13723.116307] workqueue events: flags=0x0 [13723.121450] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13723.125099] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [13723.129641] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13723.132645] pending: kfree_rcu_monitor [13723.135253] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13723.138682] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [13723.142569] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13723.145448] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [13723.153206] workqueue events_long: flags=0x0 [13723.155167] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13723.158387] pending: lu_objects_flush [obdclass] [13723.160785] workqueue events_unbound: flags=0x2 [13723.163104] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13723.168439] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13723.175337] workqueue events_power_efficient: flags=0x80 [13723.177603] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13723.180862] pending: do_cache_clean, neigh_periodic_work, check_lifetime [13723.185473] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13723.189208] pending: neigh_periodic_work [13723.193433] workqueue mm_percpu_wq: flags=0x8 [13723.196031] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13723.202466] pending: vmstat_update, lru_add_drain_per_cpu [13723.206367] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13723.208046] pending: vmstat_update [13723.209540] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13723.212284] pending: lru_add_drain_per_cpu [13723.215913] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [13723.217720] pending: lru_add_drain_per_cpu BAR(48) [13723.218945] workqueue writeback: flags=0x4a [13723.221071] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [13723.224883] pending: wb_workfn [13723.227032] workqueue xprtiod: flags=0xa [13723.229106] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13723.231025] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [13723.233365] workqueue ipv6_addrconf: flags=0x40008 [13723.237582] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13723.239769] pending: addrconf_verify_work [13723.241068] workqueue knbd0-recv: flags=0x1a [13723.246449] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13723.252480] in-flight: 658:recv_work [13723.259155] workqueue ptlrpc_pinger: flags=0x2 [13723.266563] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13723.272415] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13723.278548] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11792s workers=4 idle: 32984 37 8 [13723.285334] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13708s workers=2 idle: 661 [13728.056429] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [13728.061361] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13728.093837] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13728.099317] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13728.106695] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13728.110103] RIP: 0010:native_safe_halt+0x2e/0x50 [13728.112203] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13728.123353] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13728.126727] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13728.132298] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [13728.141708] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [13728.143730] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [13728.149252] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [13728.152384] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [13728.158219] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13728.162726] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13728.166242] Call Trace: [13728.167802] [13728.170067] ? show_regs.cold.9+0x22/0x2f [13728.178230] ? watchdog_timer_fn.cold.6+0x62/0x10a [13728.179621] ? __hrtimer_run_queues+0x1c2/0x4d0 [13728.183678] ? watchdog+0x40/0x40 [13728.185445] ? hrtimer_interrupt+0x128/0x2c0 [13728.189086] ? smp_apic_timer_interrupt+0xa6/0x270 [13728.192354] ? apic_timer_interrupt+0xf/0x20 [13728.194313] [13728.195289] ? native_safe_halt+0x2e/0x50 [13728.198449] kvm_wait+0xcd/0xe0 [13728.200416] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13728.204941] do_raw_spin_lock+0x177/0x190 [13728.206251] _raw_spin_lock+0x21/0x30 [13728.210359] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13728.218341] process_one_work+0x2c8/0x700 [13728.223782] worker_thread+0x296/0x6e0 [13728.225032] ? rescuer_thread+0x570/0x570 [13728.229637] kthread+0x1d1/0x200 [13728.232104] ? set_kthread_struct+0x70/0x70 [13728.235693] ret_from_fork+0x1f/0x30 [13740.055011] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [13740.060127] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13740.122549] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13740.141705] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13740.151466] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13740.156626] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13740.165593] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13740.169234] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [13740.171722] RDX: 000000000000154d RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [13740.174494] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [13740.177427] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [13740.180277] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13740.183284] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [13740.186673] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13740.188992] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13740.192197] Call Trace: [13740.193071] [13740.193764] ? show_regs.cold.9+0x22/0x2f [13740.195289] ? watchdog_timer_fn.cold.6+0x62/0x10a [13740.197123] ? __hrtimer_run_queues+0x1c2/0x4d0 [13740.200695] ? watchdog+0x40/0x40 [13740.202976] ? hrtimer_interrupt+0x128/0x2c0 [13740.205199] ? smp_apic_timer_interrupt+0xa6/0x270 [13740.207345] ? apic_timer_interrupt+0xf/0x20 [13740.208881] [13740.210869] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13740.213411] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13740.215599] do_raw_spin_lock+0x177/0x190 [13740.217456] _raw_spin_lock+0x21/0x30 [13740.219658] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13740.222844] obd_mod_free+0x46/0x140 [ptlrpc] [13740.225509] mdc_commit_open+0x91/0xb0 [mdc] [13740.227629] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13740.230151] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13740.233045] after_reply+0xe2c/0x1780 [ptlrpc] [13740.235136] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13740.237572] ? __next_timer_interrupt+0x160/0x160 [13740.240354] ptlrpcd+0xa82/0xe40 [ptlrpc] [13740.242737] ? do_wait_intr+0xf0/0xf0 [13740.244997] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13740.247995] kthread+0x1d1/0x200 [13740.249290] ? set_kthread_struct+0x70/0x70 [13740.251576] ret_from_fork+0x1f/0x30 [13744.057015] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [13744.059505] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13744.076410] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13744.081242] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13744.084811] RIP: 0010:native_safe_halt+0x2e/0x50 [13744.086776] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13744.101936] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13744.107684] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13744.123689] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [13744.135761] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [13744.145658] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [13744.157690] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [13744.167040] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [13744.180148] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13744.186346] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [13744.197400] Call Trace: [13744.199480] [13744.203198] ? show_regs.cold.9+0x22/0x2f [13744.211933] ? watchdog_timer_fn.cold.6+0x62/0x10a [13744.214417] ? __hrtimer_run_queues+0x1c2/0x4d0 [13744.222245] ? watchdog+0x40/0x40 [13744.228143] ? hrtimer_interrupt+0x128/0x2c0 [13744.230881] ? smp_apic_timer_interrupt+0xa6/0x270 [13744.233305] ? apic_timer_interrupt+0xf/0x20 [13744.238429] [13744.240052] ? native_safe_halt+0x2e/0x50 [13744.243690] kvm_wait+0xcd/0xe0 [13744.248219] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13744.255706] do_raw_spin_lock+0x177/0x190 [13744.264624] _raw_spin_lock+0x21/0x30 [13744.269120] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13744.281218] ? do_raw_spin_unlock+0x75/0x190 [13744.288809] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13744.295270] ? do_wait_intr+0xf0/0xf0 [13744.298160] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13744.306839] mdc_close+0x53b/0xd70 [mdc] [13744.313232] lmv_close+0x18d/0x490 [lmv] [13744.319871] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13744.329139] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13744.337161] ll_md_real_close+0xa5/0x2a0 [lustre] [13744.344173] ll_md_close+0x274/0x800 [lustre] [13744.349065] ll_file_release+0x128/0xc50 [lustre] [13744.354914] __fput+0x10d/0x440 [13744.357681] ____fput+0x12/0x20 [13744.360342] task_work_run+0xc6/0x110 [13744.367999] exit_to_usermode_loop+0x1dd/0x1f0 [13744.379371] do_syscall_64+0x3d6/0x440 [13744.380840] entry_SYSCALL_64_after_hwframe+0x49/0xae [13744.387344] RIP: 0033:0x7f151bee3cd8 [13744.390402] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13744.414653] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13744.425215] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [13744.431580] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [13744.438971] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13744.441518] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13744.446544] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13748.052891] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [13748.060880] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13748.102297] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13748.109329] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13748.117393] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13748.127396] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13748.138543] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13748.141567] RAX: 0000000000003c86 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13748.144500] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [13748.147352] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [13748.154208] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [13748.157063] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [13748.160114] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [13748.165293] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13748.168254] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [13748.171327] Call Trace: [13748.172441] [13748.173107] ? show_regs.cold.9+0x22/0x2f [13748.174739] ? watchdog_timer_fn.cold.6+0x62/0x10a [13748.177013] ? __hrtimer_run_queues+0x1c2/0x4d0 [13748.179842] ? watchdog+0x40/0x40 [13748.185239] ? hrtimer_interrupt+0x128/0x2c0 [13748.187279] ? smp_apic_timer_interrupt+0xa6/0x270 [13748.192172] ? apic_timer_interrupt+0xf/0x20 [13748.194636] [13748.198151] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13748.199659] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13748.201461] do_raw_spin_lock+0x177/0x190 [13748.202625] _raw_spin_lock+0x21/0x30 [13748.204206] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13748.206722] ? do_raw_spin_unlock+0x75/0x190 [13748.208291] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13748.216921] ? do_wait_intr+0xf0/0xf0 [13748.218710] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13748.220313] mdc_reint+0x2d/0x110 [mdc] [13748.221787] mdc_unlink+0x658/0x870 [mdc] [13748.223548] lmv_unlink+0x34d/0xbd0 [lmv] [13748.225305] ll_unlink+0x24d/0x7f0 [lustre] [13748.227296] vfs_unlink+0x202/0x330 [13748.228503] do_unlinkat+0x2fa/0x460 [13748.229923] __x64_sys_unlink+0x37/0x50 [13748.231054] do_syscall_64+0xc1/0x440 [13748.232594] entry_SYSCALL_64_after_hwframe+0x49/0xae [13748.235071] RIP: 0033:0x7f151bdee10b [13748.236225] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [13748.244546] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [13748.248536] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [13748.252294] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [13748.259328] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13748.262688] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13748.270983] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13753.313321] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11822s! [13753.316226] Showing busy workqueues and worker pools: [13753.318094] workqueue events: flags=0x0 [13753.322961] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13753.337778] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [13753.345567] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13753.351982] pending: kfree_rcu_monitor [13753.354224] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13753.359535] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [13753.363857] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13753.370073] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [13753.374170] workqueue events_long: flags=0x0 [13753.376034] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13753.380914] pending: lu_objects_flush [obdclass] [13753.384703] workqueue events_unbound: flags=0x2 [13753.386885] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13753.389610] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13753.395620] workqueue events_power_efficient: flags=0x80 [13753.398457] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13753.401736] pending: do_cache_clean, neigh_periodic_work, check_lifetime [13753.405159] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13753.408196] pending: neigh_periodic_work [13753.410249] workqueue mm_percpu_wq: flags=0x8 [13753.414155] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13753.417558] pending: vmstat_update, lru_add_drain_per_cpu [13753.421176] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13753.424637] pending: vmstat_update [13753.426607] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13753.430458] pending: lru_add_drain_per_cpu [13753.433402] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [13753.444174] pending: lru_add_drain_per_cpu BAR(48) [13753.450371] workqueue writeback: flags=0x4a [13753.452537] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [13753.455404] pending: wb_workfn [13753.457781] workqueue xprtiod: flags=0xa [13753.459871] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13753.465345] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [13753.472776] workqueue ipv6_addrconf: flags=0x40008 [13753.476520] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13753.482548] pending: addrconf_verify_work [13753.487350] workqueue knbd0-recv: flags=0x1a [13753.488886] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13753.501603] in-flight: 658:recv_work [13753.506740] workqueue ptlrpc_pinger: flags=0x2 [13753.509634] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13753.516105] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13753.522151] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11822s workers=4 idle: 32984 37 8 [13753.529509] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13738s workers=2 idle: 661 [13756.056024] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [13756.058730] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13756.081800] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13756.086394] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13756.090309] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13756.093370] RIP: 0010:native_safe_halt+0x2e/0x50 [13756.095259] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13756.102932] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13756.106390] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13756.109295] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [13756.112229] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [13756.115299] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [13756.119019] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [13756.122157] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [13756.126188] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13756.127908] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13756.130344] Call Trace: [13756.132972] [13756.134123] ? show_regs.cold.9+0x22/0x2f [13756.135243] ? watchdog_timer_fn.cold.6+0x62/0x10a [13756.138543] ? __hrtimer_run_queues+0x1c2/0x4d0 [13756.141202] ? watchdog+0x40/0x40 [13756.142569] ? hrtimer_interrupt+0x128/0x2c0 [13756.144414] ? smp_apic_timer_interrupt+0xa6/0x270 [13756.146387] ? apic_timer_interrupt+0xf/0x20 [13756.148208] [13756.149142] ? native_safe_halt+0x2e/0x50 [13756.150585] kvm_wait+0xcd/0xe0 [13756.151900] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13756.153844] do_raw_spin_lock+0x177/0x190 [13756.155831] _raw_spin_lock+0x21/0x30 [13756.157332] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13756.159343] process_one_work+0x2c8/0x700 [13756.161028] worker_thread+0x296/0x6e0 [13756.162463] ? rescuer_thread+0x570/0x570 [13756.164204] kthread+0x1d1/0x200 [13756.165376] ? set_kthread_struct+0x70/0x70 [13756.168427] ret_from_fork+0x1f/0x30 [13768.055025] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [13768.058702] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13768.078320] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13768.083396] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13768.087300] RIP: 0010:native_safe_halt+0x2e/0x50 [13768.090273] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13768.101458] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13768.105786] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [13768.109402] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [13768.113215] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [13768.116923] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [13768.119845] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13768.122566] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [13768.126918] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13768.129694] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13768.133388] Call Trace: [13768.134888] [13768.135931] ? show_regs.cold.9+0x22/0x2f [13768.137824] ? watchdog_timer_fn.cold.6+0x62/0x10a [13768.140224] ? __hrtimer_run_queues+0x1c2/0x4d0 [13768.142886] ? watchdog+0x40/0x40 [13768.144640] ? hrtimer_interrupt+0x128/0x2c0 [13768.147015] ? smp_apic_timer_interrupt+0xa6/0x270 [13768.149881] ? apic_timer_interrupt+0xf/0x20 [13768.152198] [13768.153300] ? native_safe_halt+0x2e/0x50 [13768.155652] kvm_wait+0xcd/0xe0 [13768.157050] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13768.159897] do_raw_spin_lock+0x177/0x190 [13768.162248] _raw_spin_lock+0x21/0x30 [13768.163871] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13768.167020] obd_mod_free+0x46/0x140 [ptlrpc] [13768.169604] mdc_commit_open+0x91/0xb0 [mdc] [13768.171535] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13768.174808] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13768.177609] after_reply+0xe2c/0x1780 [ptlrpc] [13768.180528] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13768.183345] ? __next_timer_interrupt+0x160/0x160 [13768.186192] ptlrpcd+0xa82/0xe40 [ptlrpc] [13768.188830] ? do_wait_intr+0xf0/0xf0 [13768.190849] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13768.194215] kthread+0x1d1/0x200 [13768.195394] ? set_kthread_struct+0x70/0x70 [13768.198914] ret_from_fork+0x1f/0x30 [13772.057080] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [13772.059656] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13772.108433] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13772.120016] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13772.127945] RIP: 0010:native_safe_halt+0x2e/0x50 [13772.131046] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13772.149222] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13772.153621] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13772.158041] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [13772.162603] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [13772.166261] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [13772.169845] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [13772.173243] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [13772.177904] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13772.180285] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [13772.186184] Call Trace: [13772.188226] [13772.189351] ? show_regs.cold.9+0x22/0x2f [13772.192213] ? watchdog_timer_fn.cold.6+0x62/0x10a [13772.194403] ? __hrtimer_run_queues+0x1c2/0x4d0 [13772.200215] ? watchdog+0x40/0x40 [13772.209992] ? hrtimer_interrupt+0x128/0x2c0 [13772.211430] ? smp_apic_timer_interrupt+0xa6/0x270 [13772.216007] ? apic_timer_interrupt+0xf/0x20 [13772.218993] [13772.219982] ? native_safe_halt+0x2e/0x50 [13772.222323] kvm_wait+0xcd/0xe0 [13772.225097] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13772.228080] do_raw_spin_lock+0x177/0x190 [13772.230828] _raw_spin_lock+0x21/0x30 [13772.233096] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13772.235391] ? do_raw_spin_unlock+0x75/0x190 [13772.241630] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13772.247266] ? do_wait_intr+0xf0/0xf0 [13772.251313] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13772.253160] mdc_close+0x53b/0xd70 [mdc] [13772.255408] lmv_close+0x18d/0x490 [lmv] [13772.259156] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13772.268197] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13772.279772] ll_md_real_close+0xa5/0x2a0 [lustre] [13772.290627] ll_md_close+0x274/0x800 [lustre] [13772.295014] ll_file_release+0x128/0xc50 [lustre] [13772.304287] __fput+0x10d/0x440 [13772.309224] ____fput+0x12/0x20 [13772.311072] task_work_run+0xc6/0x110 [13772.313238] exit_to_usermode_loop+0x1dd/0x1f0 [13772.318242] do_syscall_64+0x3d6/0x440 [13772.322355] entry_SYSCALL_64_after_hwframe+0x49/0xae [13772.330869] RIP: 0033:0x7f151bee3cd8 [13772.332169] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13772.343730] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13772.347409] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [13772.352681] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [13772.356391] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13772.362373] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13772.370108] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13780.053390] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [13780.067182] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13780.115960] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13780.131655] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13780.141393] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13780.147822] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13780.172747] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13780.189569] RAX: 000000000000127b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13780.194368] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [13780.211401] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [13780.219172] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [13780.229487] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [13780.232025] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [13780.240141] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13780.244830] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [13780.250331] Call Trace: [13780.251100] [13780.254798] ? show_regs.cold.9+0x22/0x2f [13780.257859] ? watchdog_timer_fn.cold.6+0x62/0x10a [13780.269252] ? __hrtimer_run_queues+0x1c2/0x4d0 [13780.271049] ? watchdog+0x40/0x40 [13780.275694] ? hrtimer_interrupt+0x128/0x2c0 [13780.279872] ? smp_apic_timer_interrupt+0xa6/0x270 [13780.282181] ? apic_timer_interrupt+0xf/0x20 [13780.285451] [13780.286634] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13780.293312] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13780.300724] do_raw_spin_lock+0x177/0x190 [13780.304452] _raw_spin_lock+0x21/0x30 [13780.305714] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13780.314047] ? do_raw_spin_unlock+0x75/0x190 [13780.316753] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13780.329012] ? do_wait_intr+0xf0/0xf0 [13780.330255] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13780.346352] mdc_reint+0x2d/0x110 [mdc] [13780.349308] mdc_unlink+0x658/0x870 [mdc] [13780.353111] lmv_unlink+0x34d/0xbd0 [lmv] [13780.358823] ll_unlink+0x24d/0x7f0 [lustre] [13780.364361] vfs_unlink+0x202/0x330 [13780.366835] do_unlinkat+0x2fa/0x460 [13780.372102] __x64_sys_unlink+0x37/0x50 [13780.373849] do_syscall_64+0xc1/0x440 [13780.376012] entry_SYSCALL_64_after_hwframe+0x49/0xae [13780.381714] RIP: 0033:0x7f151bdee10b [13780.385307] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [13780.407044] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [13780.411353] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [13780.419627] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [13780.425791] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13780.441153] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13780.450230] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13784.034413] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11853s! [13784.048544] Showing busy workqueues and worker pools: [13784.056016] workqueue events: flags=0x0 [13784.057822] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13784.060996] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13784.069972] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [13784.070529] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13784.070543] pending: kfree_rcu_monitor [13784.072910] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13784.072985] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13784.072992] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13784.072999] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13784.073250] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13784.073261] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13784.073265] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [13784.073269] RAX: 0000000000007692 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13784.073272] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [13784.073274] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [13784.073276] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [13784.073279] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [13784.073282] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [13784.073285] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13784.073288] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13784.073297] Call Trace: [13784.073303] [13784.073307] ? show_regs.cold.9+0x22/0x2f [13784.073316] ? watchdog_timer_fn.cold.6+0x62/0x10a [13784.073322] ? __hrtimer_run_queues+0x1c2/0x4d0 [13784.073325] ? watchdog+0x40/0x40 [13784.073329] ? hrtimer_interrupt+0x128/0x2c0 [13784.073333] ? smp_apic_timer_interrupt+0xa6/0x270 [13784.073341] ? apic_timer_interrupt+0xf/0x20 [13784.073345] [13784.073347] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13784.073351] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13784.073356] do_raw_spin_lock+0x177/0x190 [13784.073360] _raw_spin_lock+0x21/0x30 [13784.073364] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13784.073573] process_one_work+0x2c8/0x700 [13784.073579] worker_thread+0x296/0x6e0 [13784.073583] ? rescuer_thread+0x570/0x570 [13784.073586] kthread+0x1d1/0x200 [13784.073590] ? set_kthread_struct+0x70/0x70 [13784.073594] ret_from_fork+0x1f/0x30 [13784.242237] [13784.242939] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13784.245770] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [13784.250918] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13784.253719] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [13784.257117] workqueue events_long: flags=0x0 [13784.259295] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13784.262753] pending: lu_objects_flush [obdclass] [13784.267246] workqueue events_unbound: flags=0x2 [13784.270398] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13784.274136] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13784.278854] workqueue events_power_efficient: flags=0x80 [13784.281609] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13784.284620] pending: do_cache_clean, neigh_periodic_work, check_lifetime [13784.288220] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13784.291781] pending: neigh_periodic_work [13784.294148] workqueue mm_percpu_wq: flags=0x8 [13784.296410] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13784.299990] pending: vmstat_update, lru_add_drain_per_cpu [13784.303091] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13784.307437] pending: vmstat_update [13784.309816] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13784.314429] pending: lru_add_drain_per_cpu [13784.317317] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [13784.321795] pending: lru_add_drain_per_cpu BAR(48) [13784.324985] workqueue writeback: flags=0x4a [13784.327373] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [13784.332101] pending: wb_workfn [13784.334530] workqueue xprtiod: flags=0xa [13784.336771] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13784.343504] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [13784.347895] workqueue ipv6_addrconf: flags=0x40008 [13784.350772] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13784.354657] pending: addrconf_verify_work [13784.357495] workqueue knbd0-recv: flags=0x1a [13784.360017] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13784.365096] in-flight: 658:recv_work [13784.367015] workqueue ptlrpc_pinger: flags=0x2 [13784.369090] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13784.372302] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13784.375298] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11853s workers=4 idle: 32984 37 8 [13784.378110] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13769s workers=2 idle: 661 [13796.057313] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [13796.060192] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13796.103445] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13796.122662] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13796.128494] RIP: 0010:native_safe_halt+0x2e/0x50 [13796.130454] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13796.146809] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [13796.158768] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [13796.165044] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [13796.176536] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [13796.183491] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [13796.186314] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13796.192455] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [13796.199143] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13796.209202] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13796.222591] Call Trace: [13796.227793] [13796.229897] ? show_regs.cold.9+0x22/0x2f [13796.233069] ? watchdog_timer_fn.cold.6+0x62/0x10a [13796.235576] ? __hrtimer_run_queues+0x1c2/0x4d0 [13796.242119] ? watchdog+0x40/0x40 [13796.249226] ? hrtimer_interrupt+0x128/0x2c0 [13796.250296] ? smp_apic_timer_interrupt+0xa6/0x270 [13796.258911] ? apic_timer_interrupt+0xf/0x20 [13796.267061] [13796.272607] ? native_safe_halt+0x2e/0x50 [13796.280582] kvm_wait+0xcd/0xe0 [13796.282094] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13796.283958] do_raw_spin_lock+0x177/0x190 [13796.285553] _raw_spin_lock+0x21/0x30 [13796.286600] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13796.288733] obd_mod_free+0x46/0x140 [ptlrpc] [13796.292573] mdc_commit_open+0x91/0xb0 [mdc] [13796.296557] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13796.304975] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13796.307034] after_reply+0xe2c/0x1780 [ptlrpc] [13796.311140] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13796.315025] ? __next_timer_interrupt+0x160/0x160 [13796.317051] ptlrpcd+0xa82/0xe40 [ptlrpc] [13796.320733] ? do_wait_intr+0xf0/0xf0 [13796.322132] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13796.335669] kthread+0x1d1/0x200 [13796.339180] ? set_kthread_struct+0x70/0x70 [13796.349787] ret_from_fork+0x1f/0x30 [13800.057022] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [13800.062066] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13800.090324] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13800.098090] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13800.104894] RIP: 0010:native_safe_halt+0x2e/0x50 [13800.107656] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13800.119441] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13800.124028] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13800.127684] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [13800.131854] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [13800.136262] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [13800.140091] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [13800.146007] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [13800.150482] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13800.153477] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [13800.158071] Call Trace: [13800.163068] [13800.164055] ? show_regs.cold.9+0x22/0x2f [13800.166784] ? watchdog_timer_fn.cold.6+0x62/0x10a [13800.168637] ? __hrtimer_run_queues+0x1c2/0x4d0 [13800.174546] ? watchdog+0x40/0x40 [13800.176049] ? hrtimer_interrupt+0x128/0x2c0 [13800.180510] ? smp_apic_timer_interrupt+0xa6/0x270 [13800.182569] ? apic_timer_interrupt+0xf/0x20 [13800.184824] [13800.185614] ? native_safe_halt+0x2e/0x50 [13800.187681] kvm_wait+0xcd/0xe0 [13800.189031] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13800.191086] do_raw_spin_lock+0x177/0x190 [13800.197560] _raw_spin_lock+0x21/0x30 [13800.205717] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13800.218964] ? do_raw_spin_unlock+0x75/0x190 [13800.221131] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13800.225381] ? do_wait_intr+0xf0/0xf0 [13800.227830] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13800.231468] mdc_close+0x53b/0xd70 [mdc] [13800.233890] lmv_close+0x18d/0x490 [lmv] [13800.236600] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13800.240263] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13800.247050] ll_md_real_close+0xa5/0x2a0 [lustre] [13800.250132] ll_md_close+0x274/0x800 [lustre] [13800.261082] ll_file_release+0x128/0xc50 [lustre] [13800.270937] __fput+0x10d/0x440 [13800.273850] ____fput+0x12/0x20 [13800.278950] task_work_run+0xc6/0x110 [13800.281241] exit_to_usermode_loop+0x1dd/0x1f0 [13800.287236] do_syscall_64+0x3d6/0x440 [13800.292040] entry_SYSCALL_64_after_hwframe+0x49/0xae [13800.295825] RIP: 0033:0x7f151bee3cd8 [13800.302936] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13800.314077] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13800.318645] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [13800.322151] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [13800.326246] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13800.330068] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13800.333242] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13812.052097] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [13812.057524] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13812.057782] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13812.057852] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13812.057858] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13812.057860] RIP: 0010:native_safe_halt+0x2e/0x50 [13812.057873] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13812.057877] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13812.057882] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13812.057884] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [13812.057886] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [13812.057888] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [13812.057890] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [13812.057893] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [13812.057896] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13812.057898] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [13812.057910] Call Trace: [13812.057915] [13812.057921] ? show_regs.cold.9+0x22/0x2f [13812.057935] ? watchdog_timer_fn.cold.6+0x62/0x10a [13812.057946] ? __hrtimer_run_queues+0x1c2/0x4d0 [13812.057950] ? watchdog+0x40/0x40 [13812.057954] ? hrtimer_interrupt+0x128/0x2c0 [13812.057958] ? smp_apic_timer_interrupt+0xa6/0x270 [13812.057964] ? apic_timer_interrupt+0xf/0x20 [13812.057973] [13812.057976] ? native_safe_halt+0x2e/0x50 [13812.057980] kvm_wait+0xcd/0xe0 [13812.057991] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13812.058008] do_raw_spin_lock+0x177/0x190 [13812.058013] _raw_spin_lock+0x21/0x30 [13812.058139] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13812.058453] ? do_raw_spin_unlock+0x75/0x190 [13812.058459] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13812.058671] ? do_wait_intr+0xf0/0xf0 [13812.058679] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13812.058890] mdc_reint+0x2d/0x110 [mdc] [13812.058925] mdc_unlink+0x658/0x870 [mdc] [13812.058952] lmv_unlink+0x34d/0xbd0 [lmv] [13812.058981] ll_unlink+0x24d/0x7f0 [lustre] [13812.059368] vfs_unlink+0x202/0x330 [13812.059376] do_unlinkat+0x2fa/0x460 [13812.059380] __x64_sys_unlink+0x37/0x50 [13812.059383] do_syscall_64+0xc1/0x440 [13812.059452] entry_SYSCALL_64_after_hwframe+0x49/0xae [13812.059459] RIP: 0033:0x7f151bdee10b [13812.059466] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [13812.059469] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [13812.059473] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [13812.059476] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [13812.059478] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13812.059480] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13812.059482] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13812.373550] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13812.396629] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13812.404704] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13812.407501] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13812.410016] RIP: 0010:native_safe_halt+0x2e/0x50 [13812.411851] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13812.418010] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13812.420340] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13812.422407] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [13812.424538] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [13812.426875] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [13812.429787] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [13812.432613] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [13812.435674] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13812.437439] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13812.439672] Call Trace: [13812.440592] [13812.441333] ? show_regs.cold.9+0x22/0x2f [13812.442482] ? watchdog_timer_fn.cold.6+0x62/0x10a [13812.444297] ? __hrtimer_run_queues+0x1c2/0x4d0 [13812.445944] ? watchdog+0x40/0x40 [13812.447328] ? hrtimer_interrupt+0x128/0x2c0 [13812.448896] ? smp_apic_timer_interrupt+0xa6/0x270 [13812.450854] ? apic_timer_interrupt+0xf/0x20 [13812.452583] [13812.453451] ? native_safe_halt+0x2e/0x50 [13812.456287] kvm_wait+0xcd/0xe0 [13812.459718] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13812.461936] do_raw_spin_lock+0x177/0x190 [13812.465591] _raw_spin_lock+0x21/0x30 [13812.467354] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13812.475888] process_one_work+0x2c8/0x700 [13812.480231] worker_thread+0x296/0x6e0 [13812.482376] ? rescuer_thread+0x570/0x570 [13812.484420] kthread+0x1d1/0x200 [13812.487258] ? set_kthread_struct+0x70/0x70 [13812.489783] ret_from_fork+0x1f/0x30 [13814.753116] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11884s! [13814.757687] Showing busy workqueues and worker pools: [13814.760640] workqueue events: flags=0x0 [13814.766457] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13814.772422] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [13814.779887] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13814.784190] pending: kfree_rcu_monitor [13814.786244] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13814.790234] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [13814.793060] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13814.806968] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [13814.819632] workqueue events_long: flags=0x0 [13814.829421] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13814.834457] pending: lu_objects_flush [obdclass] [13814.844899] workqueue events_unbound: flags=0x2 [13814.851589] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13814.860790] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13814.874251] workqueue events_power_efficient: flags=0x80 [13814.883634] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13814.890973] pending: do_cache_clean, neigh_periodic_work, check_lifetime [13814.896632] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13814.903446] pending: neigh_periodic_work [13814.908024] workqueue mm_percpu_wq: flags=0x8 [13814.910339] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13814.917732] pending: vmstat_update, lru_add_drain_per_cpu [13814.928373] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13814.937831] pending: vmstat_update [13814.943467] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13814.952616] pending: lru_add_drain_per_cpu [13814.953728] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [13814.965363] pending: lru_add_drain_per_cpu BAR(48) [13814.971592] workqueue writeback: flags=0x4a [13814.973066] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [13814.986380] pending: wb_workfn [13814.992402] workqueue xprtiod: flags=0xa [13814.999020] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13815.011836] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [13815.026320] workqueue ipv6_addrconf: flags=0x40008 [13815.035123] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13815.038461] pending: addrconf_verify_work [13815.046379] workqueue knbd0-recv: flags=0x1a [13815.053446] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13815.063840] in-flight: 658:recv_work [13815.073752] workqueue ptlrpc_pinger: flags=0x2 [13815.079662] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13815.089445] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13815.099760] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11884s workers=4 idle: 32984 37 8 [13815.107893] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13799s workers=2 idle: 661 [13824.055466] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [13824.061710] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13824.089626] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13824.098617] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13824.103283] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13824.114725] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13824.130406] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13824.133565] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [13824.144388] RDX: 00000000000063ed RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [13824.150885] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [13824.160272] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [13824.171113] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13824.174651] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [13824.179304] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13824.186136] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13824.190349] Call Trace: [13824.191658] [13824.193167] ? show_regs.cold.9+0x22/0x2f [13824.195181] ? watchdog_timer_fn.cold.6+0x62/0x10a [13824.203390] ? __hrtimer_run_queues+0x1c2/0x4d0 [13824.205687] ? watchdog+0x40/0x40 [13824.207667] ? hrtimer_interrupt+0x128/0x2c0 [13824.213445] ? smp_apic_timer_interrupt+0xa6/0x270 [13824.215546] ? apic_timer_interrupt+0xf/0x20 [13824.219130] [13824.221208] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13824.225898] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13824.227655] do_raw_spin_lock+0x177/0x190 [13824.239638] _raw_spin_lock+0x21/0x30 [13824.241516] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13824.245670] obd_mod_free+0x46/0x140 [ptlrpc] [13824.252687] mdc_commit_open+0x91/0xb0 [mdc] [13824.258825] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13824.265281] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13824.269508] after_reply+0xe2c/0x1780 [ptlrpc] [13824.275674] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13824.279229] ? __next_timer_interrupt+0x160/0x160 [13824.283216] ptlrpcd+0xa82/0xe40 [ptlrpc] [13824.289273] ? do_wait_intr+0xf0/0xf0 [13824.297965] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13824.312685] kthread+0x1d1/0x200 [13824.317567] ? set_kthread_struct+0x70/0x70 [13824.323582] ret_from_fork+0x1f/0x30 [13828.057023] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [13828.060575] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13828.081380] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13828.086255] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13828.089988] RIP: 0010:native_safe_halt+0x2e/0x50 [13828.092569] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13828.098977] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [13828.101961] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13828.105186] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [13828.108253] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [13828.111226] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [13828.114236] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [13828.117291] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [13828.120459] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13828.123254] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [13828.125914] Call Trace: [13828.126961] [13828.127843] ? show_regs.cold.9+0x22/0x2f [13828.129621] ? watchdog_timer_fn.cold.6+0x62/0x10a [13828.131589] ? __hrtimer_run_queues+0x1c2/0x4d0 [13828.133616] ? watchdog+0x40/0x40 [13828.134767] ? hrtimer_interrupt+0x128/0x2c0 [13828.136420] ? smp_apic_timer_interrupt+0xa6/0x270 [13828.139003] ? apic_timer_interrupt+0xf/0x20 [13828.141208] [13828.142023] ? native_safe_halt+0x2e/0x50 [13828.143633] kvm_wait+0xcd/0xe0 [13828.145484] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13828.147681] do_raw_spin_lock+0x177/0x190 [13828.149404] _raw_spin_lock+0x21/0x30 [13828.150974] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13828.153550] ? do_raw_spin_unlock+0x75/0x190 [13828.155530] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13828.157670] ? do_wait_intr+0xf0/0xf0 [13828.159711] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13828.161787] mdc_close+0x53b/0xd70 [mdc] [13828.163756] lmv_close+0x18d/0x490 [lmv] [13828.165609] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13828.169507] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13828.171815] ll_md_real_close+0xa5/0x2a0 [lustre] [13828.173637] ll_md_close+0x274/0x800 [lustre] [13828.175878] ll_file_release+0x128/0xc50 [lustre] [13828.178749] __fput+0x10d/0x440 [13828.180830] ____fput+0x12/0x20 [13828.183009] task_work_run+0xc6/0x110 [13828.186447] exit_to_usermode_loop+0x1dd/0x1f0 [13828.189804] do_syscall_64+0x3d6/0x440 [13828.192128] entry_SYSCALL_64_after_hwframe+0x49/0xae [13828.195600] RIP: 0033:0x7f151bee3cd8 [13828.197187] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13828.206268] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13828.209716] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [13828.213556] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [13828.217843] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13828.226661] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13828.233872] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13840.052084] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [13840.054898] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13840.056079] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13840.075301] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13840.078973] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) [13840.083965] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13840.083975] RIP: 0010:native_safe_halt+0x2e/0x50 [13840.083988] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13840.083992] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13840.084043] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13840.084046] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [13840.084048] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [13840.084050] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [13840.084052] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [13840.084056] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [13840.084058] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13840.084061] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [13840.084067] Call Trace: [13840.084072] [13840.084077] ? show_regs.cold.9+0x22/0x2f [13840.084085] ? watchdog_timer_fn.cold.6+0x62/0x10a [13840.084091] ? __hrtimer_run_queues+0x1c2/0x4d0 [13840.084095] ? watchdog+0x40/0x40 [13840.084098] ? hrtimer_interrupt+0x128/0x2c0 [13840.084102] ? smp_apic_timer_interrupt+0xa6/0x270 [13840.084109] ? apic_timer_interrupt+0xf/0x20 [13840.084113] [13840.084116] ? native_safe_halt+0x2e/0x50 [13840.084120] kvm_wait+0xcd/0xe0 [13840.084127] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13840.084133] do_raw_spin_lock+0x177/0x190 [13840.084137] _raw_spin_lock+0x21/0x30 [13840.084143] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13840.084475] ? do_raw_spin_unlock+0x75/0x190 [13840.084481] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13840.084762] ? do_wait_intr+0xf0/0xf0 [13840.084770] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13840.089235] mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13840.089338] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13840.089344] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13840.089349] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13840.089617] RIP: 0010:native_safe_halt+0x2e/0x50 [13840.089684] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13840.089688] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13840.089696] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13840.089699] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [13840.089701] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [13840.089703] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [13840.089706] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [13840.089710] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [13840.089712] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13840.089715] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13840.089722] Call Trace: [13840.089727] [13840.089732] ? show_regs.cold.9+0x22/0x2f [13840.089742] ? watchdog_timer_fn.cold.6+0x62/0x10a [13840.089749] ? __hrtimer_run_queues+0x1c2/0x4d0 [13840.089753] ? watchdog+0x40/0x40 [13840.089758] ? hrtimer_interrupt+0x128/0x2c0 [13840.089762] ? smp_apic_timer_interrupt+0xa6/0x270 [13840.089769] ? apic_timer_interrupt+0xf/0x20 [13840.089773] [13840.089776] ? native_safe_halt+0x2e/0x50 [13840.089779] kvm_wait+0xcd/0xe0 [13840.093391] mdc_reint+0x2d/0x110 [mdc] [13840.093444] mdc_unlink+0x658/0x870 [mdc] [13840.093472] lmv_unlink+0x34d/0xbd0 [lmv] [13840.093502] ll_unlink+0x24d/0x7f0 [lustre] [13840.093984] vfs_unlink+0x202/0x330 [13840.093993] do_unlinkat+0x2fa/0x460 [13840.094047] __x64_sys_unlink+0x37/0x50 [13840.094054] do_syscall_64+0xc1/0x440 [13840.094060] entry_SYSCALL_64_after_hwframe+0x49/0xae [13840.094069] RIP: 0033:0x7f151bdee10b [13840.094076] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [13840.094079] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [13840.094084] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [13840.094087] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [13840.094089] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13840.094091] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13840.094093] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13840.294998] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13840.298428] do_raw_spin_lock+0x177/0x190 [13840.301074] _raw_spin_lock+0x21/0x30 [13840.304057] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13840.306606] process_one_work+0x2c8/0x700 [13840.308481] worker_thread+0x296/0x6e0 [13840.309668] ? rescuer_thread+0x570/0x570 [13840.311476] kthread+0x1d1/0x200 [13840.313299] ? set_kthread_struct+0x70/0x70 [13840.315389] ret_from_fork+0x1f/0x30 [13845.477166] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11914s! [13845.483598] Showing busy workqueues and worker pools: [13845.486919] workqueue events: flags=0x0 [13845.488645] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13845.492530] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [13845.498923] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13845.502378] pending: kfree_rcu_monitor [13845.504364] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13845.508058] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [13845.513398] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13845.517061] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [13845.521489] workqueue events_long: flags=0x0 [13845.523687] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13845.527334] pending: lu_objects_flush [obdclass] [13845.530351] workqueue events_unbound: flags=0x2 [13845.532501] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13845.535569] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13845.540978] workqueue events_power_efficient: flags=0x80 [13845.544309] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13845.547761] pending: do_cache_clean, neigh_periodic_work, check_lifetime [13845.551296] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13845.554904] pending: neigh_periodic_work [13845.557033] workqueue mm_percpu_wq: flags=0x8 [13845.559339] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13845.562577] pending: vmstat_update, lru_add_drain_per_cpu [13845.565586] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13845.569229] pending: vmstat_update [13845.571018] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13845.574241] pending: lru_add_drain_per_cpu [13845.577018] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [13845.580558] pending: lru_add_drain_per_cpu BAR(48) [13845.584266] workqueue writeback: flags=0x4a [13845.586710] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [13845.589997] pending: wb_workfn [13845.591570] workqueue xprtiod: flags=0xa [13845.594399] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13845.599381] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [13845.602970] workqueue ipv6_addrconf: flags=0x40008 [13845.605896] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13845.608979] pending: addrconf_verify_work [13845.614103] workqueue knbd0-recv: flags=0x1a [13845.616127] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13845.618389] in-flight: 658:recv_work [13845.620245] workqueue ptlrpc_pinger: flags=0x2 [13845.622261] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13845.625525] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13845.628625] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11915s workers=4 idle: 32984 37 8 [13845.632502] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13830s workers=2 idle: 661 [13852.055104] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [13852.059396] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13852.082145] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13852.087007] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13852.091629] RIP: 0010:native_safe_halt+0x2e/0x50 [13852.093887] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13852.102361] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13852.106173] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [13852.109521] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [13852.112490] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [13852.115573] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [13852.118792] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13852.121892] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [13852.125645] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13852.128968] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13852.132288] Call Trace: [13852.133351] [13852.134272] ? show_regs.cold.9+0x22/0x2f [13852.136213] ? watchdog_timer_fn.cold.6+0x62/0x10a [13852.138663] ? __hrtimer_run_queues+0x1c2/0x4d0 [13852.141158] ? watchdog+0x40/0x40 [13852.143021] ? hrtimer_interrupt+0x128/0x2c0 [13852.144847] ? smp_apic_timer_interrupt+0xa6/0x270 [13852.147106] ? apic_timer_interrupt+0xf/0x20 [13852.149062] [13852.150011] ? native_safe_halt+0x2e/0x50 [13852.152391] kvm_wait+0xcd/0xe0 [13852.153983] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13852.157020] do_raw_spin_lock+0x177/0x190 [13852.159094] _raw_spin_lock+0x21/0x30 [13852.161263] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13852.164247] obd_mod_free+0x46/0x140 [ptlrpc] [13852.167237] mdc_commit_open+0x91/0xb0 [mdc] [13852.169214] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13852.172172] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13852.177143] after_reply+0xe2c/0x1780 [ptlrpc] [13852.180658] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13852.183944] ? __next_timer_interrupt+0x160/0x160 [13852.186110] ptlrpcd+0xa82/0xe40 [ptlrpc] [13852.189139] ? do_wait_intr+0xf0/0xf0 [13852.191155] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13852.194194] kthread+0x1d1/0x200 [13852.196409] ? set_kthread_struct+0x70/0x70 [13852.198303] ret_from_fork+0x1f/0x30 [13856.057107] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [13856.060485] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13856.092312] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13856.101033] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13856.111781] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13856.117349] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13856.140989] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13856.149146] RAX: 0000000000005c84 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13856.152375] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [13856.160780] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [13856.167109] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [13856.171561] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [13856.178299] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [13856.181491] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13856.188157] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [13856.192605] Call Trace: [13856.198479] [13856.199380] ? show_regs.cold.9+0x22/0x2f [13856.205175] ? watchdog_timer_fn.cold.6+0x62/0x10a [13856.213963] ? __hrtimer_run_queues+0x1c2/0x4d0 [13856.225319] ? watchdog+0x40/0x40 [13856.227384] ? hrtimer_interrupt+0x128/0x2c0 [13856.236378] ? smp_apic_timer_interrupt+0xa6/0x270 [13856.238998] ? apic_timer_interrupt+0xf/0x20 [13856.243014] [13856.243855] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13856.252021] do_raw_spin_lock+0x177/0x190 [13856.258185] _raw_spin_lock+0x21/0x30 [13856.261651] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13856.263732] ? do_raw_spin_unlock+0x75/0x190 [13856.269164] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13856.276987] ? do_wait_intr+0xf0/0xf0 [13856.279894] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13856.285963] mdc_close+0x53b/0xd70 [mdc] [13856.291535] lmv_close+0x18d/0x490 [lmv] [13856.293973] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13856.298719] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13856.301789] ll_md_real_close+0xa5/0x2a0 [lustre] [13856.304226] ll_md_close+0x274/0x800 [lustre] [13856.313328] ll_file_release+0x128/0xc50 [lustre] [13856.315087] __fput+0x10d/0x440 [13856.320571] ____fput+0x12/0x20 [13856.322321] task_work_run+0xc6/0x110 [13856.325989] exit_to_usermode_loop+0x1dd/0x1f0 [13856.335156] do_syscall_64+0x3d6/0x440 [13856.339096] entry_SYSCALL_64_after_hwframe+0x49/0xae [13856.346108] RIP: 0033:0x7f151bee3cd8 [13856.349069] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13856.361352] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13856.367475] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [13856.372443] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [13856.377526] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13856.380134] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13856.383493] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13864.903031] rcu: INFO: rcu_sched self-detected stall on CPU [13864.905308] rcu: 1-...!: (7009917 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [13864.909869] (t=11940256 jiffies g=249657 q=6020) [13864.912079] rcu: rcu_sched kthread starved for 11935208 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [13864.916254] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [13864.920000] rcu: RCU grace-period kthread stack dump: [13864.922220] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [13864.926237] Call Trace: [13864.927246] __schedule+0x351/0xcb0 [13864.928801] schedule+0xc0/0x180 [13864.930069] schedule_timeout+0xb4/0x190 [13864.931897] ? do_raw_spin_unlock+0x75/0x190 [13864.933739] ? __next_timer_interrupt+0x160/0x160 [13864.935913] ? prepare_to_swait_event+0xf3/0x140 [13864.937916] rcu_gp_kthread+0x8dd/0x15b0 [13864.939562] ? force_qs_rnp+0x2c0/0x2c0 [13864.941796] kthread+0x1d1/0x200 [13864.943307] ? set_kthread_struct+0x70/0x70 [13864.945656] ret_from_fork+0x1f/0x30 [13864.947245] rcu: Stack dump where RCU GP kthread last ran: [13864.950562] Sending NMI from CPU 1 to CPUs 0: [13864.953402] NMI backtrace for cpu 0 [13864.953408] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13864.953410] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13864.953412] RIP: 0010:update_irq_load_avg+0x1b6/0x530 [13864.953414] Code: 8d be 7e b6 00 00 31 d2 48 83 05 24 f0 dc 02 01 48 f7 f7 31 d2 48 83 05 07 eb dc 02 01 48 89 83 60 0c 00 00 48 89 f0 48 f7 f7 <31> d2 48 89 83 68 0c 00 00 89 c8 f7 f7 48 89 83 70 0c 00 00 0f 1f [13864.953416] RSP: 0018:ffffb2cb80003e50 EFLAGS: 00000002 [13864.953418] RAX: 00000000000000c4 RBX: ffff8ee582033380 RCX: 00000000008cc3f7 [13864.953420] RDX: 0000000000007e9f RSI: 00000000008cc3f7 RDI: 000000000000b736 [13864.953421] RBP: ffffb2cb80003e70 R08: 00000000000000b8 R09: 0000000000000001 [13864.953422] R10: 00000000fa83b2da R11: 00000000000020ed R12: 0000000000000002 [13864.953423] R13: 0000000000000001 R14: 00000000000000b8 R15: ffff8ee582024fb8 [13864.953425] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [13864.953426] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13864.953427] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [13864.953428] Call Trace: [13864.953429] [13864.953430] ? show_regs.cold.9+0x22/0x2f [13864.953431] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [13864.953432] ? nmi_cpu_backtrace_handler+0x15/0x40 [13864.953433] ? nmi_handle+0xbd/0x210 [13864.953434] ? default_do_nmi+0x4b/0x1b0 [13864.953436] ? do_nmi+0x280/0x380 [13864.953437] ? end_repeat_nmi+0x16/0x1f [13864.953438] ? update_irq_load_avg+0x1b6/0x530 [13864.953439] ? update_irq_load_avg+0x1b6/0x530 [13864.953440] ? update_irq_load_avg+0x1b6/0x530 [13864.953441] [13864.953442] [13864.953442] update_rq_clock+0x17e/0x1c0 [13864.953443] scheduler_tick+0x95/0x190 [13864.953444] update_process_times+0x88/0xc0 [13864.953445] tick_sched_handle.isra.18+0x27/0x90 [13864.953446] tick_sched_timer+0x53/0xb0 [13864.953447] __hrtimer_run_queues+0x1c2/0x4d0 [13864.953448] ? tick_sched_handle.isra.18+0x90/0x90 [13864.953449] hrtimer_interrupt+0x128/0x2c0 [13864.953450] smp_apic_timer_interrupt+0xa6/0x270 [13864.953451] apic_timer_interrupt+0xf/0x20 [13864.953452] [13864.953453] RIP: 0010:native_safe_halt+0x2e/0x50 [13864.953454] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13864.953456] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13864.953458] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13864.953460] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [13864.953461] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [13864.953462] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [13864.953463] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [13864.953464] kvm_wait+0xcd/0xe0 [13864.953465] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13864.953466] do_raw_spin_lock+0x177/0x190 [13864.953467] _raw_spin_lock+0x21/0x30 [13864.953468] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13864.953469] ? do_raw_spin_unlock+0x75/0x190 [13864.953470] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13864.953471] ? do_wait_intr+0xf0/0xf0 [13864.953472] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13864.953473] mdc_reint+0x2d/0x110 [mdc] [13864.953474] mdc_unlink+0x658/0x870 [mdc] [13864.953475] lmv_unlink+0x34d/0xbd0 [lmv] [13864.953476] ll_unlink+0x24d/0x7f0 [lustre] [13864.953477] vfs_unlink+0x202/0x330 [13864.953478] do_unlinkat+0x2fa/0x460 [13864.953479] __x64_sys_unlink+0x37/0x50 [13864.953479] do_syscall_64+0xc1/0x440 [13864.953480] entry_SYSCALL_64_after_hwframe+0x49/0xae [13864.953482] RIP: 0033:0x7f151bdee10b [13864.953483] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [13864.953485] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [13864.953487] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [13864.953488] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [13864.953489] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13864.953490] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13864.953492] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13864.953597] NMI backtrace for cpu 1 [13865.133065] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13865.137243] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13865.140738] Call Trace: [13865.141755] [13865.143336] ? dump_stack+0xbb/0x10e [13865.144759] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [13865.146885] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [13865.148817] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [13865.151094] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [13865.153334] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [13865.155415] ? rcu_dump_cpu_stacks+0x11b/0x171 [13865.157166] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [13865.158614] ? update_process_times+0x6d/0xc0 [13865.160076] ? tick_sched_handle.isra.18+0x27/0x90 [13865.161648] ? tick_sched_timer+0x53/0xb0 [13865.162732] ? __hrtimer_run_queues+0x1c2/0x4d0 [13865.164061] ? tick_sched_handle.isra.18+0x90/0x90 [13865.165670] ? hrtimer_interrupt+0x128/0x2c0 [13865.167209] ? smp_apic_timer_interrupt+0xa6/0x270 [13865.168809] ? apic_timer_interrupt+0xf/0x20 [13865.170794] [13865.171422] ? native_safe_halt+0x2e/0x50 [13865.173009] ? kvm_wait+0xcd/0xe0 [13865.174057] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13865.175821] ? do_raw_spin_lock+0x177/0x190 [13865.177001] ? _raw_spin_lock+0x21/0x30 [13865.178870] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13865.180818] ? obd_mod_free+0x46/0x140 [ptlrpc] [13865.182589] ? mdc_commit_open+0x91/0xb0 [mdc] [13865.183820] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13865.186089] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13865.187907] ? after_reply+0xe2c/0x1780 [ptlrpc] [13865.189615] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13865.191085] ? __next_timer_interrupt+0x160/0x160 [13865.192619] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [13865.194117] ? do_wait_intr+0xf0/0xf0 [13865.195380] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13865.197386] ? kthread+0x1d1/0x200 [13865.198762] ? set_kthread_struct+0x70/0x70 [13865.200182] ? ret_from_fork+0x1f/0x30 [13868.056559] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13868.058893] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13868.078286] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13868.081155] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13868.083281] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13868.087433] RIP: 0010:native_safe_halt+0x2e/0x50 [13868.089542] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13868.095982] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13868.098210] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13868.100366] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [13868.102386] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [13868.104445] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [13868.106329] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [13868.109298] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [13868.112366] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13868.113620] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13868.121299] Call Trace: [13868.124912] [13868.125446] ? show_regs.cold.9+0x22/0x2f [13868.127600] ? watchdog_timer_fn.cold.6+0x62/0x10a [13868.129822] ? __hrtimer_run_queues+0x1c2/0x4d0 [13868.131934] ? watchdog+0x40/0x40 [13868.133073] ? hrtimer_interrupt+0x128/0x2c0 [13868.134383] ? smp_apic_timer_interrupt+0xa6/0x270 [13868.136270] ? apic_timer_interrupt+0xf/0x20 [13868.137462] [13868.138774] ? native_safe_halt+0x2e/0x50 [13868.141174] kvm_wait+0xcd/0xe0 [13868.141875] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13868.145472] do_raw_spin_lock+0x177/0x190 [13868.149087] _raw_spin_lock+0x21/0x30 [13868.150610] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13868.152847] process_one_work+0x2c8/0x700 [13868.154461] worker_thread+0x296/0x6e0 [13868.155938] ? rescuer_thread+0x570/0x570 [13868.157645] kthread+0x1d1/0x200 [13868.159033] ? set_kthread_struct+0x70/0x70 [13868.162126] ret_from_fork+0x1f/0x30 [13875.683931] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11945s! [13875.689939] Showing busy workqueues and worker pools: [13875.691523] workqueue events: flags=0x0 [13875.692317] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13875.701210] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [13875.707348] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13875.710425] pending: kfree_rcu_monitor [13875.712337] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13875.714504] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [13875.735595] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13875.739057] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [13875.750777] workqueue events_long: flags=0x0 [13875.755373] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13875.759257] pending: lu_objects_flush [obdclass] [13875.763584] workqueue events_unbound: flags=0x2 [13875.769334] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13875.785098] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13875.790509] workqueue events_power_efficient: flags=0x80 [13875.795435] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13875.801944] pending: do_cache_clean, neigh_periodic_work, check_lifetime [13875.821047] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13875.831402] pending: neigh_periodic_work [13875.837817] workqueue mm_percpu_wq: flags=0x8 [13875.840900] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13875.848400] pending: vmstat_update, lru_add_drain_per_cpu [13875.853974] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13875.873860] pending: vmstat_update [13875.881357] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13875.892408] pending: lru_add_drain_per_cpu [13875.901860] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [13875.904282] pending: lru_add_drain_per_cpu BAR(48) [13875.915405] workqueue writeback: flags=0x4a [13875.922567] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [13875.926020] pending: wb_workfn [13875.942638] workqueue xprtiod: flags=0xa [13875.944021] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13875.963797] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [13875.966193] workqueue ipv6_addrconf: flags=0x40008 [13875.981836] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13875.998929] pending: addrconf_verify_work [13876.001738] workqueue knbd0-recv: flags=0x1a [13876.002854] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13876.017266] in-flight: 658:recv_work [13876.019311] workqueue ptlrpc_pinger: flags=0x2 [13876.021387] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13876.024348] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13876.027010] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11945s workers=4 idle: 32984 37 8 [13876.030548] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13860s workers=2 idle: 661 [13884.057287] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [13884.061060] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13884.079380] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13884.091244] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13884.101305] RIP: 0010:native_safe_halt+0x2e/0x50 [13884.108046] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13884.121471] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13884.127650] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13884.133618] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [13884.140097] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [13884.151324] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [13884.160977] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [13884.167138] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [13884.171353] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13884.175266] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [13884.181184] Call Trace: [13884.183178] [13884.184410] ? show_regs.cold.9+0x22/0x2f [13884.189966] ? watchdog_timer_fn.cold.6+0x62/0x10a [13884.195830] ? __hrtimer_run_queues+0x1c2/0x4d0 [13884.204947] ? watchdog+0x40/0x40 [13884.210104] ? hrtimer_interrupt+0x128/0x2c0 [13884.212036] ? smp_apic_timer_interrupt+0xa6/0x270 [13884.215192] ? apic_timer_interrupt+0xf/0x20 [13884.218196] [13884.219458] ? native_safe_halt+0x2e/0x50 [13884.221891] kvm_wait+0xcd/0xe0 [13884.223693] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13884.227330] do_raw_spin_lock+0x177/0x190 [13884.230072] _raw_spin_lock+0x21/0x30 [13884.234955] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13884.241038] ? do_raw_spin_unlock+0x75/0x190 [13884.247112] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13884.249242] ? do_wait_intr+0xf0/0xf0 [13884.251868] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13884.262091] mdc_close+0x53b/0xd70 [mdc] [13884.274210] lmv_close+0x18d/0x490 [lmv] [13884.276817] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13884.278948] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13884.281958] ll_md_real_close+0xa5/0x2a0 [lustre] [13884.287202] ll_md_close+0x274/0x800 [lustre] [13884.290789] ll_file_release+0x128/0xc50 [lustre] [13884.294293] __fput+0x10d/0x440 [13884.296003] ____fput+0x12/0x20 [13884.298049] task_work_run+0xc6/0x110 [13884.299919] exit_to_usermode_loop+0x1dd/0x1f0 [13884.303226] do_syscall_64+0x3d6/0x440 [13884.308320] entry_SYSCALL_64_after_hwframe+0x49/0xae [13884.311455] RIP: 0033:0x7f151bee3cd8 [13884.312422] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13884.325172] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13884.329654] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [13884.343014] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [13884.346335] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13884.352983] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13884.360039] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13892.060162] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [13892.068269] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13892.120794] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13892.127950] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13892.136854] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13892.146827] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13892.159391] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13892.164956] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [13892.172946] RDX: 0000000000001516 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [13892.187255] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [13892.197451] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [13892.204189] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13892.218567] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [13892.230189] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13892.241424] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13892.247354] Call Trace: [13892.248958] [13892.250613] ? show_regs.cold.9+0x22/0x2f [13892.257156] ? watchdog_timer_fn.cold.6+0x62/0x10a [13892.261643] ? __hrtimer_run_queues+0x1c2/0x4d0 [13892.269217] ? watchdog+0x40/0x40 [13892.276541] ? hrtimer_interrupt+0x128/0x2c0 [13892.292791] ? smp_apic_timer_interrupt+0xa6/0x270 [13892.296993] ? apic_timer_interrupt+0xf/0x20 [13892.298143] [13892.304251] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13892.309521] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13892.315187] do_raw_spin_lock+0x177/0x190 [13892.316203] _raw_spin_lock+0x21/0x30 [13892.318198] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13892.321365] obd_mod_free+0x46/0x140 [ptlrpc] [13892.326972] mdc_commit_open+0x91/0xb0 [mdc] [13892.330077] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13892.334410] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13892.338065] after_reply+0xe2c/0x1780 [ptlrpc] [13892.345795] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13892.352790] ? __next_timer_interrupt+0x160/0x160 [13892.356973] ptlrpcd+0xa82/0xe40 [ptlrpc] [13892.362039] ? do_wait_intr+0xf0/0xf0 [13892.363180] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13892.375167] kthread+0x1d1/0x200 [13892.379737] ? set_kthread_struct+0x70/0x70 [13892.383379] ret_from_fork+0x1f/0x30 [13896.058074] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13896.063634] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13896.105756] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13896.112888] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13896.117018] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13896.121584] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13896.128719] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13896.150115] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13896.156618] RAX: 0000000000007d83 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13896.160909] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [13896.165640] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [13896.176208] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [13896.178870] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [13896.182316] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [13896.186444] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13896.189767] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13896.195172] Call Trace: [13896.196509] [13896.197430] ? show_regs.cold.9+0x22/0x2f [13896.200804] ? watchdog_timer_fn.cold.6+0x62/0x10a [13896.202781] ? __hrtimer_run_queues+0x1c2/0x4d0 [13896.207522] ? watchdog+0x40/0x40 [13896.208683] ? hrtimer_interrupt+0x128/0x2c0 [13896.214819] ? smp_apic_timer_interrupt+0xa6/0x270 [13896.219234] ? apic_timer_interrupt+0xf/0x20 [13896.222218] [13896.225513] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13896.230109] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13896.236112] do_raw_spin_lock+0x177/0x190 [13896.238708] _raw_spin_lock+0x21/0x30 [13896.248792] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13896.251500] process_one_work+0x2c8/0x700 [13896.253180] worker_thread+0x296/0x6e0 [13896.255212] ? rescuer_thread+0x570/0x570 [13896.257473] kthread+0x1d1/0x200 [13896.258816] ? set_kthread_struct+0x70/0x70 [13896.262902] ret_from_fork+0x1f/0x30 [13900.053413] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [13900.060088] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13900.113646] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13900.118455] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13900.127982] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13900.132473] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13900.157735] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [13900.161350] RAX: 000000000000506b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13900.165758] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [13900.169083] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [13900.173544] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [13900.178051] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [13900.181426] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [13900.188136] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13900.196452] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [13900.202672] Call Trace: [13900.205843] [13900.206657] ? show_regs.cold.9+0x22/0x2f [13900.209595] ? watchdog_timer_fn.cold.6+0x62/0x10a [13900.212548] ? __hrtimer_run_queues+0x1c2/0x4d0 [13900.215540] ? watchdog+0x40/0x40 [13900.219814] ? hrtimer_interrupt+0x128/0x2c0 [13900.222623] ? smp_apic_timer_interrupt+0xa6/0x270 [13900.225136] ? apic_timer_interrupt+0xf/0x20 [13900.227958] [13900.228848] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13900.231831] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13900.235094] do_raw_spin_lock+0x177/0x190 [13900.237774] _raw_spin_lock+0x21/0x30 [13900.239922] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13900.242995] ? do_raw_spin_unlock+0x75/0x190 [13900.245163] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13900.248150] ? do_wait_intr+0xf0/0xf0 [13900.250084] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13900.252649] mdc_reint+0x2d/0x110 [mdc] [13900.254610] mdc_unlink+0x658/0x870 [mdc] [13900.256767] lmv_unlink+0x34d/0xbd0 [lmv] [13900.258875] ll_unlink+0x24d/0x7f0 [lustre] [13900.260863] vfs_unlink+0x202/0x330 [13900.262052] do_unlinkat+0x2fa/0x460 [13900.263550] __x64_sys_unlink+0x37/0x50 [13900.265518] do_syscall_64+0xc1/0x440 [13900.267132] entry_SYSCALL_64_after_hwframe+0x49/0xae [13900.270523] RIP: 0033:0x7f151bdee10b [13900.272127] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [13900.282009] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [13900.287087] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [13900.291601] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [13900.297064] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13900.302048] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13900.306056] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13906.400129] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11975s! [13906.404020] Showing busy workqueues and worker pools: [13906.406480] workqueue events: flags=0x0 [13906.409447] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13906.412796] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [13906.417023] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13906.420372] pending: kfree_rcu_monitor [13906.422418] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13906.426577] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [13906.432717] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13906.435980] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [13906.447686] workqueue events_long: flags=0x0 [13906.451898] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13906.462315] pending: lu_objects_flush [obdclass] [13906.469940] workqueue events_unbound: flags=0x2 [13906.472591] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13906.477450] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13906.483697] workqueue events_power_efficient: flags=0x80 [13906.491249] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13906.498691] pending: do_cache_clean, neigh_periodic_work, check_lifetime [13906.506149] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13906.509602] pending: neigh_periodic_work [13906.518030] workqueue mm_percpu_wq: flags=0x8 [13906.522891] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13906.536313] pending: vmstat_update, lru_add_drain_per_cpu [13906.544647] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13906.553422] pending: vmstat_update [13906.560020] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13906.578945] pending: lru_add_drain_per_cpu [13906.583117] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [13906.589765] pending: lru_add_drain_per_cpu BAR(48) [13906.596939] workqueue writeback: flags=0x4a [13906.602511] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [13906.614548] pending: wb_workfn [13906.620677] workqueue xprtiod: flags=0xa [13906.626026] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13906.631831] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [13906.637539] workqueue ipv6_addrconf: flags=0x40008 [13906.644297] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13906.649731] pending: addrconf_verify_work [13906.657068] workqueue knbd0-recv: flags=0x1a [13906.665055] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13906.673589] in-flight: 658:recv_work [13906.678784] workqueue ptlrpc_pinger: flags=0x2 [13906.682422] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13906.687819] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13906.696458] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11976s workers=4 idle: 32984 37 8 [13906.705059] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13891s workers=2 idle: 661 [13912.057040] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [13912.059991] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13912.087860] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13912.102283] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13912.112535] RIP: 0010:native_safe_halt+0x2e/0x50 [13912.115027] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13912.135200] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13912.140515] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13912.148071] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [13912.154045] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [13912.171637] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [13912.180122] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [13912.184507] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [13912.188302] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13912.191356] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [13912.200369] Call Trace: [13912.202261] [13912.204377] ? show_regs.cold.9+0x22/0x2f [13912.206841] ? watchdog_timer_fn.cold.6+0x62/0x10a [13912.216485] ? __hrtimer_run_queues+0x1c2/0x4d0 [13912.227433] ? watchdog+0x40/0x40 [13912.229169] ? hrtimer_interrupt+0x128/0x2c0 [13912.232074] ? smp_apic_timer_interrupt+0xa6/0x270 [13912.236949] ? apic_timer_interrupt+0xf/0x20 [13912.241612] [13912.243299] ? native_safe_halt+0x2e/0x50 [13912.246349] kvm_wait+0xcd/0xe0 [13912.248200] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13912.250459] do_raw_spin_lock+0x177/0x190 [13912.253592] _raw_spin_lock+0x21/0x30 [13912.256406] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13912.263212] ? do_raw_spin_unlock+0x75/0x190 [13912.268761] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13912.270798] ? do_wait_intr+0xf0/0xf0 [13912.273621] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13912.275250] mdc_close+0x53b/0xd70 [mdc] [13912.281974] lmv_close+0x18d/0x490 [lmv] [13912.284247] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13912.287336] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13912.294914] ll_md_real_close+0xa5/0x2a0 [lustre] [13912.304665] ll_md_close+0x274/0x800 [lustre] [13912.307316] ll_file_release+0x128/0xc50 [lustre] [13912.315314] __fput+0x10d/0x440 [13912.318577] ____fput+0x12/0x20 [13912.320049] task_work_run+0xc6/0x110 [13912.323195] exit_to_usermode_loop+0x1dd/0x1f0 [13912.326282] do_syscall_64+0x3d6/0x440 [13912.336870] entry_SYSCALL_64_after_hwframe+0x49/0xae [13912.346594] RIP: 0033:0x7f151bee3cd8 [13912.350429] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13912.360685] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13912.364273] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [13912.369294] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [13912.376404] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13912.388817] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13912.399828] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13920.055083] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [13920.059040] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13920.075545] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13920.079861] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13920.082970] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13920.084990] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13920.091739] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13920.094234] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [13920.096553] RDX: 0000000000006e24 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [13920.099174] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [13920.101659] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [13920.104249] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13920.106688] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [13920.109242] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13920.111377] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13920.113790] Call Trace: [13920.114646] [13920.115447] ? show_regs.cold.9+0x22/0x2f [13920.116738] ? watchdog_timer_fn.cold.6+0x62/0x10a [13920.119204] ? __hrtimer_run_queues+0x1c2/0x4d0 [13920.121206] ? watchdog+0x40/0x40 [13920.122802] ? hrtimer_interrupt+0x128/0x2c0 [13920.124940] ? smp_apic_timer_interrupt+0xa6/0x270 [13920.126964] ? apic_timer_interrupt+0xf/0x20 [13920.129262] [13920.130715] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13920.133248] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13920.138044] do_raw_spin_lock+0x177/0x190 [13920.140225] _raw_spin_lock+0x21/0x30 [13920.142071] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13920.144997] obd_mod_free+0x46/0x140 [ptlrpc] [13920.148429] mdc_commit_open+0x91/0xb0 [mdc] [13920.150515] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13920.152823] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13920.156253] after_reply+0xe2c/0x1780 [ptlrpc] [13920.157944] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13920.162982] ? __next_timer_interrupt+0x160/0x160 [13920.165962] ptlrpcd+0xa82/0xe40 [ptlrpc] [13920.168543] ? do_wait_intr+0xf0/0xf0 [13920.171116] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13920.174744] kthread+0x1d1/0x200 [13920.176418] ? set_kthread_struct+0x70/0x70 [13920.178426] ret_from_fork+0x1f/0x30 [13924.056022] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13924.058862] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13924.075578] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13924.079137] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13924.081643] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13924.084735] RIP: 0010:native_safe_halt+0x2e/0x50 [13924.086360] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13924.092597] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13924.095359] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13924.097519] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [13924.099592] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [13924.102704] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [13924.105417] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [13924.108573] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [13924.111430] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13924.113864] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13924.117186] Call Trace: [13924.118252] [13924.119245] ? show_regs.cold.9+0x22/0x2f [13924.120871] ? watchdog_timer_fn.cold.6+0x62/0x10a [13924.122914] ? __hrtimer_run_queues+0x1c2/0x4d0 [13924.124920] ? watchdog+0x40/0x40 [13924.126521] ? hrtimer_interrupt+0x128/0x2c0 [13924.128433] ? smp_apic_timer_interrupt+0xa6/0x270 [13924.130498] ? apic_timer_interrupt+0xf/0x20 [13924.132340] [13924.133318] ? native_safe_halt+0x2e/0x50 [13924.135118] kvm_wait+0xcd/0xe0 [13924.136249] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13924.138507] do_raw_spin_lock+0x177/0x190 [13924.140237] _raw_spin_lock+0x21/0x30 [13924.141908] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13924.144553] process_one_work+0x2c8/0x700 [13924.146451] worker_thread+0x296/0x6e0 [13924.147943] ? rescuer_thread+0x570/0x570 [13924.149893] kthread+0x1d1/0x200 [13924.151172] ? set_kthread_struct+0x70/0x70 [13924.152807] ret_from_fork+0x1f/0x30 [13932.057306] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [13932.069404] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13932.114092] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13932.119525] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13932.123693] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13932.126702] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13932.135769] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13932.143456] RAX: 0000000000001299 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13932.155432] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [13932.163542] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [13932.170520] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [13932.174421] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [13932.178402] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [13932.186287] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13932.190405] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [13932.194877] Call Trace: [13932.196305] [13932.198514] ? show_regs.cold.9+0x22/0x2f [13932.202153] ? watchdog_timer_fn.cold.6+0x62/0x10a [13932.205675] ? __hrtimer_run_queues+0x1c2/0x4d0 [13932.208487] ? watchdog+0x40/0x40 [13932.211825] ? hrtimer_interrupt+0x128/0x2c0 [13932.215190] ? smp_apic_timer_interrupt+0xa6/0x270 [13932.219217] ? apic_timer_interrupt+0xf/0x20 [13932.221467] [13932.223454] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13932.231902] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13932.234314] do_raw_spin_lock+0x177/0x190 [13932.236768] _raw_spin_lock+0x21/0x30 [13932.239086] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13932.242519] ? do_raw_spin_unlock+0x75/0x190 [13932.250117] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13932.252707] ? do_wait_intr+0xf0/0xf0 [13932.255669] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13932.259472] mdc_reint+0x2d/0x110 [mdc] [13932.264824] mdc_unlink+0x658/0x870 [mdc] [13932.266351] lmv_unlink+0x34d/0xbd0 [lmv] [13932.274689] ll_unlink+0x24d/0x7f0 [lustre] [13932.278712] vfs_unlink+0x202/0x330 [13932.281265] do_unlinkat+0x2fa/0x460 [13932.283555] __x64_sys_unlink+0x37/0x50 [13932.291275] do_syscall_64+0xc1/0x440 [13932.293736] entry_SYSCALL_64_after_hwframe+0x49/0xae [13932.300176] RIP: 0033:0x7f151bdee10b [13932.303978] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [13932.315512] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [13932.322886] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [13932.331197] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [13932.338511] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13932.346703] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13932.356810] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13937.121295] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12006s! [13937.128087] Showing busy workqueues and worker pools: [13937.133400] workqueue events: flags=0x0 [13937.139476] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13937.148215] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [13937.157477] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13937.163628] pending: kfree_rcu_monitor [13937.167368] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13937.173446] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [13937.184367] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13937.189546] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [13937.198124] workqueue events_long: flags=0x0 [13937.202682] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13937.205223] pending: lu_objects_flush [obdclass] [13937.210806] workqueue events_unbound: flags=0x2 [13937.214120] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13937.219015] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13937.240649] workqueue events_power_efficient: flags=0x80 [13937.253962] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13937.270510] pending: do_cache_clean, neigh_periodic_work, check_lifetime [13937.278765] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13937.290782] pending: neigh_periodic_work [13937.296374] workqueue mm_percpu_wq: flags=0x8 [13937.299277] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13937.302446] pending: vmstat_update, lru_add_drain_per_cpu [13937.305413] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13937.310075] pending: vmstat_update [13937.312016] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13937.315611] pending: lru_add_drain_per_cpu [13937.319776] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [13937.324240] pending: lru_add_drain_per_cpu BAR(48) [13937.327507] workqueue writeback: flags=0x4a [13937.330572] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [13937.336457] pending: wb_workfn [13937.340597] workqueue xprtiod: flags=0xa [13937.343790] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13937.351625] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [13937.358065] workqueue ipv6_addrconf: flags=0x40008 [13937.362023] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13937.365558] pending: addrconf_verify_work [13937.367835] workqueue knbd0-recv: flags=0x1a [13937.370148] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13937.373318] in-flight: 658:recv_work [13937.376076] workqueue ptlrpc_pinger: flags=0x2 [13937.378115] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13937.380596] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13937.383675] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12006s workers=4 idle: 32984 37 8 [13937.387236] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13922s workers=2 idle: 661 [13940.057263] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [13940.066707] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13940.133399] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13940.146730] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13940.163040] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13940.173022] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13940.185304] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13940.197266] RAX: 0000000000002b84 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13940.203094] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [13940.216048] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [13940.225892] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [13940.235255] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [13940.243197] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [13940.253451] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13940.257476] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [13940.267093] Call Trace: [13940.270471] [13940.272459] ? show_regs.cold.9+0x22/0x2f [13940.275735] ? watchdog_timer_fn.cold.6+0x62/0x10a [13940.280452] ? __hrtimer_run_queues+0x1c2/0x4d0 [13940.289269] ? watchdog+0x40/0x40 [13940.299438] ? hrtimer_interrupt+0x128/0x2c0 [13940.301045] ? smp_apic_timer_interrupt+0xa6/0x270 [13940.309748] ? apic_timer_interrupt+0xf/0x20 [13940.319146] [13940.327372] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13940.333771] do_raw_spin_lock+0x177/0x190 [13940.348960] _raw_spin_lock+0x21/0x30 [13940.350057] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13940.361838] ? do_raw_spin_unlock+0x75/0x190 [13940.363262] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13940.372487] ? do_wait_intr+0xf0/0xf0 [13940.377245] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13940.385304] mdc_close+0x53b/0xd70 [mdc] [13940.388005] lmv_close+0x18d/0x490 [lmv] [13940.390571] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13940.395572] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13940.398289] ll_md_real_close+0xa5/0x2a0 [lustre] [13940.401579] ll_md_close+0x274/0x800 [lustre] [13940.403664] ll_file_release+0x128/0xc50 [lustre] [13940.412041] __fput+0x10d/0x440 [13940.413240] ____fput+0x12/0x20 [13940.414386] task_work_run+0xc6/0x110 [13940.422968] exit_to_usermode_loop+0x1dd/0x1f0 [13940.424543] do_syscall_64+0x3d6/0x440 [13940.433572] entry_SYSCALL_64_after_hwframe+0x49/0xae [13940.437752] RIP: 0033:0x7f151bee3cd8 [13940.440650] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13940.457015] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13940.463218] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [13940.475205] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [13940.482171] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13940.495932] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13940.503010] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13948.055106] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [13948.059597] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13948.076357] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13948.086593] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13948.092550] RIP: 0010:native_safe_halt+0x2e/0x50 [13948.094721] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13948.102382] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13948.105703] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [13948.109139] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [13948.111632] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [13948.114538] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [13948.117702] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13948.120944] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [13948.124553] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13948.126498] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13948.129379] Call Trace: [13948.130701] [13948.131405] ? show_regs.cold.9+0x22/0x2f [13948.132487] ? watchdog_timer_fn.cold.6+0x62/0x10a [13948.134020] ? __hrtimer_run_queues+0x1c2/0x4d0 [13948.135510] ? watchdog+0x40/0x40 [13948.137027] ? hrtimer_interrupt+0x128/0x2c0 [13948.138541] ? smp_apic_timer_interrupt+0xa6/0x270 [13948.140018] ? apic_timer_interrupt+0xf/0x20 [13948.141339] [13948.141994] ? native_safe_halt+0x2e/0x50 [13948.143712] kvm_wait+0xcd/0xe0 [13948.144543] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13948.146437] do_raw_spin_lock+0x177/0x190 [13948.147642] _raw_spin_lock+0x21/0x30 [13948.148713] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13948.154314] obd_mod_free+0x46/0x140 [ptlrpc] [13948.157949] mdc_commit_open+0x91/0xb0 [mdc] [13948.160621] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13948.164524] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13948.167487] after_reply+0xe2c/0x1780 [ptlrpc] [13948.171334] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13948.176497] ? __next_timer_interrupt+0x160/0x160 [13948.180824] ptlrpcd+0xa82/0xe40 [ptlrpc] [13948.188319] ? do_wait_intr+0xf0/0xf0 [13948.190407] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13948.193613] kthread+0x1d1/0x200 [13948.195290] ? set_kthread_struct+0x70/0x70 [13948.197518] ret_from_fork+0x1f/0x30 [13952.056878] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13952.069374] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13952.122451] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13952.137096] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13952.148110] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13952.156501] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13952.168484] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13952.197194] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [13952.209514] RAX: 000000000000100f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13952.219438] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [13952.225348] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [13952.230281] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [13952.237050] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [13952.245364] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [13952.252300] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13952.261043] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13952.265447] Call Trace: [13952.268533] [13952.271667] ? show_regs.cold.9+0x22/0x2f [13952.275770] ? watchdog_timer_fn.cold.6+0x62/0x10a [13952.281509] ? __hrtimer_run_queues+0x1c2/0x4d0 [13952.284449] ? watchdog+0x40/0x40 [13952.286570] ? hrtimer_interrupt+0x128/0x2c0 [13952.288952] ? smp_apic_timer_interrupt+0xa6/0x270 [13952.294493] ? apic_timer_interrupt+0xf/0x20 [13952.301001] [13952.302468] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13952.306574] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13952.310686] do_raw_spin_lock+0x177/0x190 [13952.319280] _raw_spin_lock+0x21/0x30 [13952.321953] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13952.328429] process_one_work+0x2c8/0x700 [13952.332489] worker_thread+0x296/0x6e0 [13952.336760] ? rescuer_thread+0x570/0x570 [13952.348275] kthread+0x1d1/0x200 [13952.356430] ? set_kthread_struct+0x70/0x70 [13952.360596] ret_from_fork+0x1f/0x30 [13964.052522] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [13964.065980] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13964.138420] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13964.157360] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13964.165559] RIP: 0010:native_safe_halt+0x2e/0x50 [13964.168828] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13964.191373] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13964.199503] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13964.205730] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [13964.215990] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [13964.225727] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [13964.237875] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [13964.247795] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [13964.256361] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13964.260921] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [13964.273611] Call Trace: [13964.276675] [13964.277558] ? show_regs.cold.9+0x22/0x2f [13964.284275] ? watchdog_timer_fn.cold.6+0x62/0x10a [13964.290319] ? __hrtimer_run_queues+0x1c2/0x4d0 [13964.294751] ? watchdog+0x40/0x40 [13964.297140] ? hrtimer_interrupt+0x128/0x2c0 [13964.299742] ? smp_apic_timer_interrupt+0xa6/0x270 [13964.307284] ? apic_timer_interrupt+0xf/0x20 [13964.313092] [13964.314499] ? native_safe_halt+0x2e/0x50 [13964.316794] kvm_wait+0xcd/0xe0 [13964.321464] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13964.329058] do_raw_spin_lock+0x177/0x190 [13964.335918] _raw_spin_lock+0x21/0x30 [13964.340559] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13964.344649] ? do_raw_spin_unlock+0x75/0x190 [13964.345841] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13964.350134] ? do_wait_intr+0xf0/0xf0 [13964.351944] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13964.355545] mdc_reint+0x2d/0x110 [mdc] [13964.358966] mdc_unlink+0x658/0x870 [mdc] [13964.360145] lmv_unlink+0x34d/0xbd0 [lmv] [13964.363584] ll_unlink+0x24d/0x7f0 [lustre] [13964.366369] vfs_unlink+0x202/0x330 [13964.368309] do_unlinkat+0x2fa/0x460 [13964.370033] __x64_sys_unlink+0x37/0x50 [13964.372020] do_syscall_64+0xc1/0x440 [13964.374204] entry_SYSCALL_64_after_hwframe+0x49/0xae [13964.376826] RIP: 0033:0x7f151bdee10b [13964.379002] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [13964.388326] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [13964.396385] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [13964.401047] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [13964.405486] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13964.416292] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13964.420361] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13967.840757] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12037s! [13967.844208] Showing busy workqueues and worker pools: [13967.847300] workqueue events: flags=0x0 [13967.849864] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13967.854827] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [13967.863793] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13967.869106] pending: kfree_rcu_monitor [13967.873120] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13967.879112] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [13967.885459] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13967.887436] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [13967.896436] workqueue events_long: flags=0x0 [13967.898646] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13967.901935] pending: lu_objects_flush [obdclass] [13967.904678] workqueue events_unbound: flags=0x2 [13967.907404] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13967.911094] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13967.915189] workqueue events_power_efficient: flags=0x80 [13967.916548] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13967.918525] pending: do_cache_clean, neigh_periodic_work, check_lifetime [13967.921103] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13967.922982] pending: neigh_periodic_work [13967.923977] workqueue mm_percpu_wq: flags=0x8 [13967.925427] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13967.928062] pending: vmstat_update, lru_add_drain_per_cpu [13967.931108] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13967.934421] pending: vmstat_update [13967.936256] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13967.939309] pending: lru_add_drain_per_cpu [13967.941318] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [13967.944480] pending: lru_add_drain_per_cpu BAR(48) [13967.948976] workqueue writeback: flags=0x4a [13967.950868] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [13967.955300] pending: wb_workfn [13967.959904] workqueue xprtiod: flags=0xa [13967.961649] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13967.971503] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [13967.979367] workqueue ipv6_addrconf: flags=0x40008 [13967.986152] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13967.989335] pending: addrconf_verify_work [13967.991209] workqueue knbd0-recv: flags=0x1a [13967.994024] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13967.997459] in-flight: 658:recv_work [13967.999483] workqueue ptlrpc_pinger: flags=0x2 [13968.001008] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13968.003066] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13968.005252] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12037s workers=4 idle: 32984 37 8 [13968.008336] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13952s workers=2 idle: 661 [13968.057953] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [13968.066767] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13968.119217] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13968.132891] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13968.143552] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13968.145533] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13968.169417] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13968.174612] RAX: 0000000000006405 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13968.177602] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [13968.182737] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [13968.187535] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [13968.190697] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [13968.194219] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [13968.201989] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13968.208910] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [13968.217169] Call Trace: [13968.218416] [13968.219376] ? show_regs.cold.9+0x22/0x2f [13968.221234] ? watchdog_timer_fn.cold.6+0x62/0x10a [13968.223953] ? __hrtimer_run_queues+0x1c2/0x4d0 [13968.230165] ? watchdog+0x40/0x40 [13968.231889] ? hrtimer_interrupt+0x128/0x2c0 [13968.233878] ? smp_apic_timer_interrupt+0xa6/0x270 [13968.236193] ? apic_timer_interrupt+0xf/0x20 [13968.238824] [13968.240001] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13968.243934] do_raw_spin_lock+0x177/0x190 [13968.245341] _raw_spin_lock+0x21/0x30 [13968.248320] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13968.250624] ? do_raw_spin_unlock+0x75/0x190 [13968.254439] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13968.258448] ? do_wait_intr+0xf0/0xf0 [13968.262434] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13968.265443] mdc_close+0x53b/0xd70 [mdc] [13968.267738] lmv_close+0x18d/0x490 [lmv] [13968.271774] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13968.273967] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13968.276326] ll_md_real_close+0xa5/0x2a0 [lustre] [13968.278400] ll_md_close+0x274/0x800 [lustre] [13968.280383] ll_file_release+0x128/0xc50 [lustre] [13968.283048] __fput+0x10d/0x440 [13968.286013] ____fput+0x12/0x20 [13968.289067] task_work_run+0xc6/0x110 [13968.290594] exit_to_usermode_loop+0x1dd/0x1f0 [13968.294283] do_syscall_64+0x3d6/0x440 [13968.296194] entry_SYSCALL_64_after_hwframe+0x49/0xae [13968.302511] RIP: 0033:0x7f151bee3cd8 [13968.304395] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13968.315136] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13968.318898] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [13968.321559] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [13968.327061] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13968.332535] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13968.334939] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13976.055050] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [13976.062391] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13976.109159] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13976.115155] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13976.122013] RIP: 0010:native_safe_halt+0x2e/0x50 [13976.125563] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13976.133686] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [13976.143457] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [13976.155042] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [13976.160227] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [13976.169353] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [13976.178687] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13976.189659] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [13976.197699] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13976.204865] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13976.212330] Call Trace: [13976.213212] [13976.214089] ? show_regs.cold.9+0x22/0x2f [13976.215798] ? watchdog_timer_fn.cold.6+0x62/0x10a [13976.218720] ? __hrtimer_run_queues+0x1c2/0x4d0 [13976.221415] ? watchdog+0x40/0x40 [13976.227575] ? hrtimer_interrupt+0x128/0x2c0 [13976.229503] ? smp_apic_timer_interrupt+0xa6/0x270 [13976.235685] ? apic_timer_interrupt+0xf/0x20 [13976.237272] [13976.238780] ? native_safe_halt+0x2e/0x50 [13976.240306] kvm_wait+0xcd/0xe0 [13976.241452] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13976.243912] do_raw_spin_lock+0x177/0x190 [13976.245338] _raw_spin_lock+0x21/0x30 [13976.246941] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13976.249524] obd_mod_free+0x46/0x140 [ptlrpc] [13976.255011] mdc_commit_open+0x91/0xb0 [mdc] [13976.256443] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13976.260152] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13976.265060] after_reply+0xe2c/0x1780 [ptlrpc] [13976.268425] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13976.271447] ? __next_timer_interrupt+0x160/0x160 [13976.274138] ptlrpcd+0xa82/0xe40 [ptlrpc] [13976.280691] ? do_wait_intr+0xf0/0xf0 [13976.286123] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13976.288373] kthread+0x1d1/0x200 [13976.289840] ? set_kthread_struct+0x70/0x70 [13976.291588] ret_from_fork+0x1f/0x30 [13980.056053] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [13980.058262] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13980.136686] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13980.165959] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13980.180924] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13980.190421] RIP: 0010:native_safe_halt+0x2e/0x50 [13980.198546] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13980.226518] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13980.234956] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13980.251365] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [13980.266218] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [13980.273024] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [13980.283332] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [13980.290427] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [13980.304056] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13980.312387] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [13980.322513] Call Trace: [13980.327413] [13980.328308] ? show_regs.cold.9+0x22/0x2f [13980.332383] ? watchdog_timer_fn.cold.6+0x62/0x10a [13980.339038] ? __hrtimer_run_queues+0x1c2/0x4d0 [13980.345424] ? watchdog+0x40/0x40 [13980.350470] ? hrtimer_interrupt+0x128/0x2c0 [13980.353668] ? smp_apic_timer_interrupt+0xa6/0x270 [13980.359435] ? apic_timer_interrupt+0xf/0x20 [13980.364919] [13980.367814] ? native_safe_halt+0x2e/0x50 [13980.371709] kvm_wait+0xcd/0xe0 [13980.372953] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13980.382352] do_raw_spin_lock+0x177/0x190 [13980.387279] _raw_spin_lock+0x21/0x30 [13980.393350] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13980.396973] process_one_work+0x2c8/0x700 [13980.404571] worker_thread+0x296/0x6e0 [13980.411923] ? rescuer_thread+0x570/0x570 [13980.413481] kthread+0x1d1/0x200 [13980.417201] ? set_kthread_struct+0x70/0x70 [13980.426514] ret_from_fork+0x1f/0x30 [13992.052409] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [13992.056670] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13992.085881] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13992.091676] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13992.095879] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13992.099238] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13992.107920] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13992.112396] RAX: 000000000000067e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13992.115442] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [13992.119103] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [13992.122430] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [13992.124633] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [13992.127556] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [13992.134270] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13992.137734] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [13992.141128] Call Trace: [13992.142287] [13992.143606] ? show_regs.cold.9+0x22/0x2f [13992.145628] ? watchdog_timer_fn.cold.6+0x62/0x10a [13992.147897] ? __hrtimer_run_queues+0x1c2/0x4d0 [13992.154398] ? watchdog+0x40/0x40 [13992.156274] ? hrtimer_interrupt+0x128/0x2c0 [13992.158708] ? smp_apic_timer_interrupt+0xa6/0x270 [13992.161065] ? apic_timer_interrupt+0xf/0x20 [13992.163226] [13992.164020] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13992.166569] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13992.169026] do_raw_spin_lock+0x177/0x190 [13992.170924] _raw_spin_lock+0x21/0x30 [13992.172257] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13992.174711] ? do_raw_spin_unlock+0x75/0x190 [13992.176502] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13992.179121] ? do_wait_intr+0xf0/0xf0 [13992.181384] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13992.183469] mdc_reint+0x2d/0x110 [mdc] [13992.186146] mdc_unlink+0x658/0x870 [mdc] [13992.187951] lmv_unlink+0x34d/0xbd0 [lmv] [13992.190009] ll_unlink+0x24d/0x7f0 [lustre] [13992.191903] vfs_unlink+0x202/0x330 [13992.193691] do_unlinkat+0x2fa/0x460 [13992.196411] __x64_sys_unlink+0x37/0x50 [13992.200057] do_syscall_64+0xc1/0x440 [13992.202150] entry_SYSCALL_64_after_hwframe+0x49/0xae [13992.207360] RIP: 0033:0x7f151bdee10b [13992.208949] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [13992.224263] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [13992.229174] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [13992.232754] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [13992.235917] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13992.239266] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13992.242280] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13996.057080] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [13996.061807] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13996.088120] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13996.096779] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13996.103146] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13996.110288] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13996.121361] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13996.125982] RAX: 0000000000005d76 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [13996.129298] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [13996.133726] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [13996.137933] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [13996.141918] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [13996.145927] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [13996.150568] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13996.156342] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [13996.160765] Call Trace: [13996.162076] [13996.162945] ? show_regs.cold.9+0x22/0x2f [13996.166171] ? watchdog_timer_fn.cold.6+0x62/0x10a [13996.169532] ? __hrtimer_run_queues+0x1c2/0x4d0 [13996.171699] ? watchdog+0x40/0x40 [13996.173571] ? hrtimer_interrupt+0x128/0x2c0 [13996.175849] ? smp_apic_timer_interrupt+0xa6/0x270 [13996.180761] ? apic_timer_interrupt+0xf/0x20 [13996.186073] [13996.187850] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13996.191040] do_raw_spin_lock+0x177/0x190 [13996.193997] _raw_spin_lock+0x21/0x30 [13996.197846] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13996.203668] ? do_raw_spin_unlock+0x75/0x190 [13996.205926] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13996.212042] ? do_wait_intr+0xf0/0xf0 [13996.214472] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13996.224219] mdc_close+0x53b/0xd70 [mdc] [13996.232305] lmv_close+0x18d/0x490 [lmv] [13996.234754] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13996.243231] ll_close_inode_openhandle+0x493/0xea0 [lustre] [13996.249649] ll_md_real_close+0xa5/0x2a0 [lustre] [13996.255624] ll_md_close+0x274/0x800 [lustre] [13996.258921] ll_file_release+0x128/0xc50 [lustre] [13996.261736] __fput+0x10d/0x440 [13996.263681] ____fput+0x12/0x20 [13996.265677] task_work_run+0xc6/0x110 [13996.270597] exit_to_usermode_loop+0x1dd/0x1f0 [13996.273050] do_syscall_64+0x3d6/0x440 [13996.275574] entry_SYSCALL_64_after_hwframe+0x49/0xae [13996.278503] RIP: 0033:0x7f151bee3cd8 [13996.280834] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [13996.298966] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [13996.307051] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [13996.310294] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [13996.316387] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [13996.321881] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [13996.325942] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [13998.051043] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12067s! [13998.054841] Showing busy workqueues and worker pools: [13998.057225] workqueue events: flags=0x0 [13998.058847] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13998.061872] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [13998.065667] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13998.068465] pending: kfree_rcu_monitor [13998.069841] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13998.072596] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [13998.076688] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13998.079491] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [13998.082116] workqueue events_long: flags=0x0 [13998.083839] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13998.086541] pending: lu_objects_flush [obdclass] [13998.088708] workqueue events_unbound: flags=0x2 [13998.090109] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [13998.092672] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [13998.099812] workqueue events_power_efficient: flags=0x80 [13998.102643] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13998.107831] pending: do_cache_clean, neigh_periodic_work, check_lifetime [13998.114581] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13998.119051] pending: neigh_periodic_work [13998.124751] workqueue mm_percpu_wq: flags=0x8 [13998.126666] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13998.133425] pending: vmstat_update, lru_add_drain_per_cpu [13998.136031] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13998.140242] pending: vmstat_update [13998.146633] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13998.152648] pending: lru_add_drain_per_cpu [13998.154201] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [13998.166861] pending: lru_add_drain_per_cpu BAR(48) [13998.175790] workqueue writeback: flags=0x4a [13998.181772] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [13998.189944] pending: wb_workfn [13998.201818] workqueue xprtiod: flags=0xa [13998.204919] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [13998.213990] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [13998.221743] workqueue ipv6_addrconf: flags=0x40008 [13998.226545] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13998.229678] pending: addrconf_verify_work [13998.234081] workqueue knbd0-recv: flags=0x1a [13998.236240] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13998.240498] in-flight: 658:recv_work [13998.242739] workqueue ptlrpc_pinger: flags=0x2 [13998.245017] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13998.249211] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [13998.252791] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12067s workers=4 idle: 32984 37 8 [13998.257808] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13983s workers=2 idle: 661 [14004.055105] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [14004.066297] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14004.121580] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14004.132682] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14004.139472] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14004.147579] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14004.170370] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14004.184864] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [14004.190946] RDX: 0000000000001ecc RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [14004.196750] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [14004.203765] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [14004.207748] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14004.221352] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [14004.232747] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14004.237177] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14004.247290] Call Trace: [14004.251471] [14004.258181] ? show_regs.cold.9+0x22/0x2f [14004.259703] ? watchdog_timer_fn.cold.6+0x62/0x10a [14004.280324] ? __hrtimer_run_queues+0x1c2/0x4d0 [14004.297797] ? watchdog+0x40/0x40 [14004.303774] ? hrtimer_interrupt+0x128/0x2c0 [14004.308560] ? smp_apic_timer_interrupt+0xa6/0x270 [14004.314886] ? apic_timer_interrupt+0xf/0x20 [14004.319289] [14004.324949] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14004.337299] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14004.349027] do_raw_spin_lock+0x177/0x190 [14004.360998] _raw_spin_lock+0x21/0x30 [14004.376628] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14004.381889] obd_mod_free+0x46/0x140 [ptlrpc] [14004.393449] mdc_commit_open+0x91/0xb0 [mdc] [14004.399800] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14004.406536] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14004.416940] after_reply+0xe2c/0x1780 [ptlrpc] [14004.428601] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14004.436736] ? __next_timer_interrupt+0x160/0x160 [14004.440896] ptlrpcd+0xa82/0xe40 [ptlrpc] [14004.444745] ? do_wait_intr+0xf0/0xf0 [14004.446929] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14004.455146] kthread+0x1d1/0x200 [14004.458993] ? set_kthread_struct+0x70/0x70 [14004.460845] ret_from_fork+0x1f/0x30 [14008.056034] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [14008.058923] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14008.083269] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14008.092429] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14008.099964] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14008.107747] RIP: 0010:native_safe_halt+0x2e/0x50 [14008.111017] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14008.128595] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14008.130802] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14008.133492] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [14008.140461] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [14008.143410] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [14008.146735] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [14008.151569] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [14008.155130] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14008.159553] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14008.162851] Call Trace: [14008.164739] [14008.165485] ? show_regs.cold.9+0x22/0x2f [14008.167669] ? watchdog_timer_fn.cold.6+0x62/0x10a [14008.169363] ? __hrtimer_run_queues+0x1c2/0x4d0 [14008.173042] ? watchdog+0x40/0x40 [14008.175374] ? hrtimer_interrupt+0x128/0x2c0 [14008.179848] ? smp_apic_timer_interrupt+0xa6/0x270 [14008.182142] ? apic_timer_interrupt+0xf/0x20 [14008.183911] [14008.184738] ? native_safe_halt+0x2e/0x50 [14008.192530] kvm_wait+0xcd/0xe0 [14008.196366] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14008.200681] do_raw_spin_lock+0x177/0x190 [14008.203386] _raw_spin_lock+0x21/0x30 [14008.206922] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14008.214717] process_one_work+0x2c8/0x700 [14008.216367] worker_thread+0x296/0x6e0 [14008.221516] ? rescuer_thread+0x570/0x570 [14008.225978] kthread+0x1d1/0x200 [14008.227102] ? set_kthread_struct+0x70/0x70 [14008.233119] ret_from_fork+0x1f/0x30 [14024.053840] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [14024.057577] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [14024.060882] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14024.060963] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14024.060970] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14024.060972] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14024.060985] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14024.060989] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14024.060993] RAX: 0000000000007f09 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14024.060996] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [14024.062361] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [14024.062366] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [14024.062368] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [14024.062371] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [14024.062374] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14024.062376] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [14024.062385] Call Trace: [14024.062390] [14024.062395] ? show_regs.cold.9+0x22/0x2f [14024.062406] ? watchdog_timer_fn.cold.6+0x62/0x10a [14024.062412] ? __hrtimer_run_queues+0x1c2/0x4d0 [14024.062416] ? watchdog+0x40/0x40 [14024.062420] ? hrtimer_interrupt+0x128/0x2c0 [14024.062424] ? smp_apic_timer_interrupt+0xa6/0x270 [14024.062433] ? apic_timer_interrupt+0xf/0x20 [14024.062437] [14024.062440] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14024.062445] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14024.062450] do_raw_spin_lock+0x177/0x190 [14024.062455] _raw_spin_lock+0x21/0x30 [14024.062461] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14024.062737] ? do_raw_spin_unlock+0x75/0x190 [14024.062743] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14024.062944] ? do_wait_intr+0xf0/0xf0 [14024.062954] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14024.063500] mdc_reint+0x2d/0x110 [mdc] [14024.063540] mdc_unlink+0x658/0x870 [mdc] [14024.063567] lmv_unlink+0x34d/0xbd0 [lmv] [14024.063596] ll_unlink+0x24d/0x7f0 [lustre] [14024.063707] vfs_unlink+0x202/0x330 [14024.063716] do_unlinkat+0x2fa/0x460 [14024.063721] __x64_sys_unlink+0x37/0x50 [14024.063724] do_syscall_64+0xc1/0x440 [14024.063731] entry_SYSCALL_64_after_hwframe+0x49/0xae [14024.063738] RIP: 0033:0x7f151bdee10b [14024.063745] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [14024.063749] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [14024.063754] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [14024.063757] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [14024.063759] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14024.063761] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14024.063763] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14024.410689] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14024.435741] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14024.440879] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14024.446490] RIP: 0010:native_safe_halt+0x2e/0x50 [14024.449514] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14024.458383] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14024.462124] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14024.465993] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [14024.469835] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [14024.473772] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [14024.477770] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [14024.482500] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [14024.486581] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14024.489859] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [14024.495204] Call Trace: [14024.496727] [14024.497481] ? show_regs.cold.9+0x22/0x2f [14024.499234] ? watchdog_timer_fn.cold.6+0x62/0x10a [14024.502108] ? __hrtimer_run_queues+0x1c2/0x4d0 [14024.504198] ? watchdog+0x40/0x40 [14024.505811] ? hrtimer_interrupt+0x128/0x2c0 [14024.507685] ? smp_apic_timer_interrupt+0xa6/0x270 [14024.510172] ? apic_timer_interrupt+0xf/0x20 [14024.512027] [14024.513168] ? native_safe_halt+0x2e/0x50 [14024.514927] kvm_wait+0xcd/0xe0 [14024.516607] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14024.519424] do_raw_spin_lock+0x177/0x190 [14024.521529] _raw_spin_lock+0x21/0x30 [14024.523293] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14024.526400] ? do_raw_spin_unlock+0x75/0x190 [14024.527860] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14024.530423] ? do_wait_intr+0xf0/0xf0 [14024.532549] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14024.534933] mdc_close+0x53b/0xd70 [mdc] [14024.536717] lmv_close+0x18d/0x490 [lmv] [14024.538959] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14024.543250] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14024.546220] ll_md_real_close+0xa5/0x2a0 [lustre] [14024.549323] ll_md_close+0x274/0x800 [lustre] [14024.551778] ll_file_release+0x128/0xc50 [lustre] [14024.554406] __fput+0x10d/0x440 [14024.555880] ____fput+0x12/0x20 [14024.557454] task_work_run+0xc6/0x110 [14024.559651] exit_to_usermode_loop+0x1dd/0x1f0 [14024.562790] do_syscall_64+0x3d6/0x440 [14024.564729] entry_SYSCALL_64_after_hwframe+0x49/0xae [14024.567296] RIP: 0033:0x7f151bee3cd8 [14024.569216] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14024.579887] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14024.584512] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [14024.588211] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [14024.591913] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14024.595450] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14024.599589] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14028.769052] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12098s! [14028.772621] Showing busy workqueues and worker pools: [14028.775594] workqueue events: flags=0x0 [14028.777639] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14028.779766] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [14028.783658] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14028.786323] pending: kfree_rcu_monitor [14028.787844] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14028.789952] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [14028.794187] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14028.797357] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [14028.800702] workqueue events_long: flags=0x0 [14028.803178] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14028.806344] pending: lu_objects_flush [obdclass] [14028.812273] workqueue events_unbound: flags=0x2 [14028.816351] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14028.823578] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14028.835781] workqueue events_power_efficient: flags=0x80 [14028.839878] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14028.846632] pending: do_cache_clean, neigh_periodic_work, check_lifetime [14028.853287] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14028.860482] pending: neigh_periodic_work [14028.865821] workqueue mm_percpu_wq: flags=0x8 [14028.872891] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14028.878880] pending: vmstat_update, lru_add_drain_per_cpu [14028.883517] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14028.887854] pending: vmstat_update [14028.889485] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14028.897792] pending: lru_add_drain_per_cpu [14028.902465] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [14028.905937] pending: lru_add_drain_per_cpu BAR(48) [14028.912280] workqueue writeback: flags=0x4a [14028.917144] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [14028.920084] pending: wb_workfn [14028.925407] workqueue xprtiod: flags=0xa [14028.928460] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14028.937437] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [14028.947135] workqueue ipv6_addrconf: flags=0x40008 [14028.957299] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14028.966521] pending: addrconf_verify_work [14028.971165] workqueue knbd0-recv: flags=0x1a [14028.978266] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14028.985883] in-flight: 658:recv_work [14028.989110] workqueue ptlrpc_pinger: flags=0x2 [14028.992520] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14028.996291] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14028.999717] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12098s workers=4 idle: 32984 37 8 [14029.003945] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14013s workers=2 idle: 661 [14032.055024] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [14032.058976] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14032.090434] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14032.102639] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14032.108446] RIP: 0010:native_safe_halt+0x2e/0x50 [14032.112913] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14032.129067] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14032.131772] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [14032.139971] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [14032.144403] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [14032.149612] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [14032.157070] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14032.160607] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [14032.166325] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14032.170461] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14032.174706] Call Trace: [14032.176130] [14032.176969] ? show_regs.cold.9+0x22/0x2f [14032.179133] ? watchdog_timer_fn.cold.6+0x62/0x10a [14032.184283] ? __hrtimer_run_queues+0x1c2/0x4d0 [14032.186346] ? watchdog+0x40/0x40 [14032.188393] ? hrtimer_interrupt+0x128/0x2c0 [14032.190411] ? smp_apic_timer_interrupt+0xa6/0x270 [14032.193194] ? apic_timer_interrupt+0xf/0x20 [14032.194673] [14032.195478] ? native_safe_halt+0x2e/0x50 [14032.199308] kvm_wait+0xcd/0xe0 [14032.200355] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14032.205755] do_raw_spin_lock+0x177/0x190 [14032.207317] _raw_spin_lock+0x21/0x30 [14032.209818] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14032.214138] obd_mod_free+0x46/0x140 [ptlrpc] [14032.216391] mdc_commit_open+0x91/0xb0 [mdc] [14032.222465] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14032.225344] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14032.227972] after_reply+0xe2c/0x1780 [ptlrpc] [14032.232404] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14032.238483] ? __next_timer_interrupt+0x160/0x160 [14032.245159] ptlrpcd+0xa82/0xe40 [ptlrpc] [14032.249196] ? do_wait_intr+0xf0/0xf0 [14032.255816] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14032.260605] kthread+0x1d1/0x200 [14032.261710] ? set_kthread_struct+0x70/0x70 [14032.263410] ret_from_fork+0x1f/0x30 [14036.059639] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [14036.068764] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14036.111837] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14036.136360] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14036.157469] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14036.169695] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14036.177721] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14036.184419] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14036.186641] RAX: 00000000000076b4 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14036.189855] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [14036.193615] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [14036.196069] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [14036.202729] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [14036.209896] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [14036.215257] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14036.218208] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14036.229000] Call Trace: [14036.231594] [14036.234097] ? show_regs.cold.9+0x22/0x2f [14036.236835] ? watchdog_timer_fn.cold.6+0x62/0x10a [14036.245927] ? __hrtimer_run_queues+0x1c2/0x4d0 [14036.248791] ? watchdog+0x40/0x40 [14036.250950] ? hrtimer_interrupt+0x128/0x2c0 [14036.252766] ? smp_apic_timer_interrupt+0xa6/0x270 [14036.255522] ? apic_timer_interrupt+0xf/0x20 [14036.258786] [14036.261163] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14036.263159] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14036.267837] do_raw_spin_lock+0x177/0x190 [14036.270047] _raw_spin_lock+0x21/0x30 [14036.271689] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14036.275760] process_one_work+0x2c8/0x700 [14036.280136] worker_thread+0x296/0x6e0 [14036.282995] ? rescuer_thread+0x570/0x570 [14036.286716] kthread+0x1d1/0x200 [14036.292398] ? set_kthread_struct+0x70/0x70 [14036.301253] ret_from_fork+0x1f/0x30 [14044.908246] rcu: INFO: rcu_sched self-detected stall on CPU [14044.911917] rcu: 1-...!: (7124523 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [14044.917299] (t=12120261 jiffies g=249657 q=6083) [14044.918984] rcu: rcu_sched kthread starved for 12115213 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [14044.924435] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [14044.930099] rcu: RCU grace-period kthread stack dump: [14044.932330] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [14044.937037] Call Trace: [14044.938149] __schedule+0x351/0xcb0 [14044.940015] schedule+0xc0/0x180 [14044.941423] schedule_timeout+0xb4/0x190 [14044.943499] ? do_raw_spin_unlock+0x75/0x190 [14044.945499] ? __next_timer_interrupt+0x160/0x160 [14044.948308] ? prepare_to_swait_event+0xf3/0x140 [14044.950920] rcu_gp_kthread+0x8dd/0x15b0 [14044.952623] ? force_qs_rnp+0x2c0/0x2c0 [14044.954743] kthread+0x1d1/0x200 [14044.956410] ? set_kthread_struct+0x70/0x70 [14044.958369] ret_from_fork+0x1f/0x30 [14044.961199] rcu: Stack dump where RCU GP kthread last ran: [14044.963188] Sending NMI from CPU 1 to CPUs 0: [14044.965621] NMI backtrace for cpu 0 [14044.965627] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14044.965629] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14044.965631] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14044.965635] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14044.965637] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 [14044.965639] RAX: 0000000000003e23 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14044.965641] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [14044.965642] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [14044.965644] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [14044.965645] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [14044.965647] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [14044.965648] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14044.965649] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [14044.965651] Call Trace: [14044.965651] [14044.965652] ? show_regs.cold.9+0x22/0x2f [14044.965653] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [14044.965654] ? nmi_cpu_backtrace_handler+0x15/0x40 [14044.965656] ? nmi_handle+0xbd/0x210 [14044.965657] ? default_do_nmi+0x4b/0x1b0 [14044.965658] ? do_nmi+0x280/0x380 [14044.965659] ? end_repeat_nmi+0x16/0x1f [14044.965660] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14044.965661] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14044.965662] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14044.965664] [14044.965664] do_raw_spin_lock+0x177/0x190 [14044.965665] _raw_spin_lock+0x21/0x30 [14044.965667] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14044.965668] ? do_raw_spin_unlock+0x75/0x190 [14044.965669] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14044.965670] ? do_wait_intr+0xf0/0xf0 [14044.965671] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14044.965672] mdc_reint+0x2d/0x110 [mdc] [14044.965673] mdc_unlink+0x658/0x870 [mdc] [14044.965674] lmv_unlink+0x34d/0xbd0 [lmv] [14044.965675] ll_unlink+0x24d/0x7f0 [lustre] [14044.965676] vfs_unlink+0x202/0x330 [14044.965677] do_unlinkat+0x2fa/0x460 [14044.965678] __x64_sys_unlink+0x37/0x50 [14044.965679] do_syscall_64+0xc1/0x440 [14044.965680] entry_SYSCALL_64_after_hwframe+0x49/0xae [14044.965681] RIP: 0033:0x7f151bdee10b [14044.965682] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [14044.965684] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [14044.965686] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [14044.965687] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [14044.965689] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14044.965690] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14044.965692] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14044.968295] NMI backtrace for cpu 1 [14045.156196] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14045.162250] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14045.166122] Call Trace: [14045.166650] [14045.171460] ? dump_stack+0xbb/0x10e [14045.173170] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [14045.176265] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [14045.178456] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [14045.179714] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [14045.184660] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [14045.186577] ? rcu_dump_cpu_stacks+0x11b/0x171 [14045.188376] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [14045.189686] ? update_process_times+0x6d/0xc0 [14045.192159] ? tick_sched_handle.isra.18+0x27/0x90 [14045.195248] ? tick_sched_timer+0x53/0xb0 [14045.197171] ? __hrtimer_run_queues+0x1c2/0x4d0 [14045.198725] ? tick_sched_handle.isra.18+0x90/0x90 [14045.201345] ? hrtimer_interrupt+0x128/0x2c0 [14045.203169] ? smp_apic_timer_interrupt+0xa6/0x270 [14045.204486] ? apic_timer_interrupt+0xf/0x20 [14045.205996] [14045.206584] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14045.208298] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14045.210448] ? do_raw_spin_lock+0x177/0x190 [14045.212048] ? _raw_spin_lock+0x21/0x30 [14045.213455] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14045.217458] ? obd_mod_free+0x46/0x140 [ptlrpc] [14045.219494] ? mdc_commit_open+0x91/0xb0 [mdc] [14045.221605] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14045.224403] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14045.226396] ? after_reply+0xe2c/0x1780 [ptlrpc] [14045.228732] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14045.231368] ? __next_timer_interrupt+0x160/0x160 [14045.234675] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [14045.242719] ? do_wait_intr+0xf0/0xf0 [14045.244076] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14045.247123] ? kthread+0x1d1/0x200 [14045.249082] ? set_kthread_struct+0x70/0x70 [14045.250645] ? ret_from_fork+0x1f/0x30 [14052.058563] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [14052.063454] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14052.090676] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14052.098700] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14052.107518] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14052.109723] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14052.124566] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14052.132094] RAX: 0000000000007cdb RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14052.135350] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [14052.139205] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [14052.142722] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [14052.145955] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [14052.151450] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [14052.155338] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14052.160240] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [14052.162857] Call Trace: [14052.163469] [14052.164201] ? show_regs.cold.9+0x22/0x2f [14052.165392] ? watchdog_timer_fn.cold.6+0x62/0x10a [14052.167001] ? __hrtimer_run_queues+0x1c2/0x4d0 [14052.168454] ? watchdog+0x40/0x40 [14052.169642] ? hrtimer_interrupt+0x128/0x2c0 [14052.171278] ? smp_apic_timer_interrupt+0xa6/0x270 [14052.172710] ? apic_timer_interrupt+0xf/0x20 [14052.175558] [14052.176173] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14052.177650] do_raw_spin_lock+0x177/0x190 [14052.178660] _raw_spin_lock+0x21/0x30 [14052.179664] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14052.181404] ? do_raw_spin_unlock+0x75/0x190 [14052.182573] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14052.184261] ? do_wait_intr+0xf0/0xf0 [14052.185252] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14052.187066] mdc_close+0x53b/0xd70 [mdc] [14052.188243] lmv_close+0x18d/0x490 [lmv] [14052.189556] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14052.191502] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14052.193282] ll_md_real_close+0xa5/0x2a0 [lustre] [14052.194651] ll_md_close+0x274/0x800 [lustre] [14052.196845] ll_file_release+0x128/0xc50 [lustre] [14052.198472] __fput+0x10d/0x440 [14052.199405] ____fput+0x12/0x20 [14052.200425] task_work_run+0xc6/0x110 [14052.201492] exit_to_usermode_loop+0x1dd/0x1f0 [14052.203466] do_syscall_64+0x3d6/0x440 [14052.204537] entry_SYSCALL_64_after_hwframe+0x49/0xae [14052.206435] RIP: 0033:0x7f151bee3cd8 [14052.208123] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14052.214443] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14052.217073] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [14052.219607] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [14052.223217] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14052.225086] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14052.227221] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14059.490059] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12128s! [14059.495146] Showing busy workqueues and worker pools: [14059.498932] workqueue events: flags=0x0 [14059.499994] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14059.505455] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [14059.511409] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14059.516358] pending: kfree_rcu_monitor [14059.519881] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14059.525045] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [14059.533088] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14059.536113] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [14059.542292] workqueue events_long: flags=0x0 [14059.544715] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14059.548114] pending: lu_objects_flush [obdclass] [14059.552941] workqueue events_unbound: flags=0x2 [14059.555590] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14059.562226] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14059.567364] workqueue events_power_efficient: flags=0x80 [14059.568801] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14059.572008] pending: do_cache_clean, neigh_periodic_work, check_lifetime [14059.574420] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14059.580370] pending: neigh_periodic_work [14059.583340] workqueue mm_percpu_wq: flags=0x8 [14059.585555] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14059.591324] pending: vmstat_update, lru_add_drain_per_cpu [14059.595627] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14059.602146] pending: vmstat_update [14059.603575] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14059.607299] pending: lru_add_drain_per_cpu [14059.612049] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [14059.616421] pending: lru_add_drain_per_cpu BAR(48) [14059.621032] workqueue writeback: flags=0x4a [14059.622064] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [14059.627673] pending: wb_workfn [14059.629441] workqueue xprtiod: flags=0xa [14059.630493] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14059.632144] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [14059.634384] workqueue ipv6_addrconf: flags=0x40008 [14059.635553] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14059.637326] pending: addrconf_verify_work [14059.638473] workqueue knbd0-recv: flags=0x1a [14059.639526] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14059.641730] in-flight: 658:recv_work [14059.642902] workqueue ptlrpc_pinger: flags=0x2 [14059.644137] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14059.645960] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14059.647743] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12129s workers=4 idle: 32984 37 8 [14059.649849] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14044s workers=2 idle: 661 [14064.056029] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [14064.058674] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14064.093415] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14064.105314] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14064.109915] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14064.114029] RIP: 0010:native_safe_halt+0x2e/0x50 [14064.116327] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14064.124405] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14064.128081] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14064.130936] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [14064.135440] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [14064.140925] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [14064.147207] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [14064.150678] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [14064.157514] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14064.160107] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14064.163778] Call Trace: [14064.165201] [14064.166304] ? show_regs.cold.9+0x22/0x2f [14064.168281] ? watchdog_timer_fn.cold.6+0x62/0x10a [14064.170457] ? __hrtimer_run_queues+0x1c2/0x4d0 [14064.174141] ? watchdog+0x40/0x40 [14064.175891] ? hrtimer_interrupt+0x128/0x2c0 [14064.178125] ? smp_apic_timer_interrupt+0xa6/0x270 [14064.180835] ? apic_timer_interrupt+0xf/0x20 [14064.182857] [14064.183604] ? native_safe_halt+0x2e/0x50 [14064.189762] kvm_wait+0xcd/0xe0 [14064.190883] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14064.194290] do_raw_spin_lock+0x177/0x190 [14064.196392] _raw_spin_lock+0x21/0x30 [14064.197928] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14064.200984] process_one_work+0x2c8/0x700 [14064.202418] worker_thread+0x296/0x6e0 [14064.205737] ? rescuer_thread+0x570/0x570 [14064.207095] kthread+0x1d1/0x200 [14064.208180] ? set_kthread_struct+0x70/0x70 [14064.211399] ret_from_fork+0x1f/0x30 [14072.055019] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [14072.057985] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14072.079242] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14072.084722] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14072.088152] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14072.090726] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14072.099787] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14072.103215] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [14072.106138] RDX: 0000000000002092 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [14072.109011] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [14072.111798] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [14072.114876] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14072.117683] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [14072.120451] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14072.122092] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14072.123995] Call Trace: [14072.124653] [14072.125162] ? show_regs.cold.9+0x22/0x2f [14072.126392] ? watchdog_timer_fn.cold.6+0x62/0x10a [14072.127851] ? __hrtimer_run_queues+0x1c2/0x4d0 [14072.129011] ? watchdog+0x40/0x40 [14072.130066] ? hrtimer_interrupt+0x128/0x2c0 [14072.131359] ? smp_apic_timer_interrupt+0xa6/0x270 [14072.132859] ? apic_timer_interrupt+0xf/0x20 [14072.134131] [14072.134859] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14072.136843] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14072.138357] do_raw_spin_lock+0x177/0x190 [14072.139747] _raw_spin_lock+0x21/0x30 [14072.140793] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14072.142367] obd_mod_free+0x46/0x140 [ptlrpc] [14072.144118] mdc_commit_open+0x91/0xb0 [mdc] [14072.145607] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14072.147461] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14072.149160] after_reply+0xe2c/0x1780 [ptlrpc] [14072.150632] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14072.152174] ? __next_timer_interrupt+0x160/0x160 [14072.153491] ptlrpcd+0xa82/0xe40 [ptlrpc] [14072.154870] ? do_wait_intr+0xf0/0xf0 [14072.155935] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14072.157520] kthread+0x1d1/0x200 [14072.158506] ? set_kthread_struct+0x70/0x70 [14072.160181] ret_from_fork+0x1f/0x30 [14080.059903] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [14080.069902] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14080.106724] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14080.114244] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14080.120363] RIP: 0010:native_safe_halt+0x2e/0x50 [14080.123347] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14080.137835] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14080.143111] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14080.148063] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [14080.153406] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [14080.159113] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [14080.165171] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [14080.170531] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [14080.175313] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14080.178798] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [14080.183249] Call Trace: [14080.184642] [14080.185776] ? show_regs.cold.9+0x22/0x2f [14080.188313] ? watchdog_timer_fn.cold.6+0x62/0x10a [14080.191808] ? __hrtimer_run_queues+0x1c2/0x4d0 [14080.195208] ? watchdog+0x40/0x40 [14080.197673] ? hrtimer_interrupt+0x128/0x2c0 [14080.200136] ? smp_apic_timer_interrupt+0xa6/0x270 [14080.202825] ? apic_timer_interrupt+0xf/0x20 [14080.206788] [14080.207918] ? native_safe_halt+0x2e/0x50 [14080.210260] kvm_wait+0xcd/0xe0 [14080.212697] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14080.214727] do_raw_spin_lock+0x177/0x190 [14080.216259] _raw_spin_lock+0x21/0x30 [14080.217711] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14080.220686] ? do_raw_spin_unlock+0x75/0x190 [14080.222211] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14080.224380] ? do_wait_intr+0xf0/0xf0 [14080.225564] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14080.227201] mdc_close+0x53b/0xd70 [mdc] [14080.228229] lmv_close+0x18d/0x490 [lmv] [14080.230068] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14080.232828] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14080.234860] ll_md_real_close+0xa5/0x2a0 [lustre] [14080.236632] ll_md_close+0x274/0x800 [lustre] [14080.238110] ll_file_release+0x128/0xc50 [lustre] [14080.239961] __fput+0x10d/0x440 [14080.241129] ____fput+0x12/0x20 [14080.242143] task_work_run+0xc6/0x110 [14080.243785] exit_to_usermode_loop+0x1dd/0x1f0 [14080.245427] do_syscall_64+0x3d6/0x440 [14080.247211] entry_SYSCALL_64_after_hwframe+0x49/0xae [14080.249113] RIP: 0033:0x7f151bee3cd8 [14080.250338] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14080.256970] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14080.259609] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [14080.262522] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [14080.265202] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14080.267333] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14080.270449] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14084.053775] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [14084.060540] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14084.087004] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14084.097272] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14084.103839] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14084.109571] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14084.129345] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14084.144144] RAX: 0000000000003856 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14084.151870] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [14084.156431] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [14084.162395] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [14084.167818] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [14084.177019] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [14084.184639] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14084.190883] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [14084.197919] Call Trace: [14084.199958] [14084.201334] ? show_regs.cold.9+0x22/0x2f [14084.203429] ? watchdog_timer_fn.cold.6+0x62/0x10a [14084.209645] ? __hrtimer_run_queues+0x1c2/0x4d0 [14084.213138] ? watchdog+0x40/0x40 [14084.218251] ? hrtimer_interrupt+0x128/0x2c0 [14084.221331] ? smp_apic_timer_interrupt+0xa6/0x270 [14084.224451] ? apic_timer_interrupt+0xf/0x20 [14084.225657] [14084.226277] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14084.228830] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14084.234901] do_raw_spin_lock+0x177/0x190 [14084.239577] _raw_spin_lock+0x21/0x30 [14084.241478] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14084.243746] ? do_raw_spin_unlock+0x75/0x190 [14084.246451] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14084.249961] ? do_wait_intr+0xf0/0xf0 [14084.251913] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14084.254303] mdc_reint+0x2d/0x110 [mdc] [14084.256785] mdc_unlink+0x658/0x870 [mdc] [14084.258633] lmv_unlink+0x34d/0xbd0 [lmv] [14084.260011] ll_unlink+0x24d/0x7f0 [lustre] [14084.262203] vfs_unlink+0x202/0x330 [14084.263925] do_unlinkat+0x2fa/0x460 [14084.265779] __x64_sys_unlink+0x37/0x50 [14084.267303] do_syscall_64+0xc1/0x440 [14084.268825] entry_SYSCALL_64_after_hwframe+0x49/0xae [14084.270982] RIP: 0033:0x7f151bdee10b [14084.272765] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [14084.281603] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [14084.285133] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [14084.287789] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [14084.292030] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14084.295087] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14084.298106] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14089.699348] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12159s! [14089.703577] Showing busy workqueues and worker pools: [14089.706702] workqueue events: flags=0x0 [14089.709062] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14089.712936] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [14089.718843] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14089.724922] pending: kfree_rcu_monitor [14089.728815] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14089.732026] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [14089.739814] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14089.744479] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [14089.759260] workqueue events_long: flags=0x0 [14089.763840] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14089.770334] pending: lu_objects_flush [obdclass] [14089.779093] workqueue events_unbound: flags=0x2 [14089.784107] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14089.790145] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14089.806878] workqueue events_power_efficient: flags=0x80 [14089.814067] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14089.821940] pending: do_cache_clean, neigh_periodic_work, check_lifetime [14089.831624] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14089.837877] pending: neigh_periodic_work [14089.839246] workqueue mm_percpu_wq: flags=0x8 [14089.852126] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14089.853929] pending: vmstat_update, lru_add_drain_per_cpu [14089.862700] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14089.876360] pending: vmstat_update [14089.877930] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14089.889210] pending: lru_add_drain_per_cpu [14089.894775] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [14089.898991] pending: lru_add_drain_per_cpu BAR(48) [14089.905011] workqueue writeback: flags=0x4a [14089.911034] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [14089.916132] pending: wb_workfn [14089.918279] workqueue xprtiod: flags=0xa [14089.920968] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14089.929716] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [14089.943826] workqueue ipv6_addrconf: flags=0x40008 [14089.946992] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14089.956799] pending: addrconf_verify_work [14089.964780] workqueue knbd0-recv: flags=0x1a [14089.972175] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14089.977069] in-flight: 658:recv_work [14089.979651] workqueue ptlrpc_pinger: flags=0x2 [14089.988748] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14089.990944] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14090.001469] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12159s workers=4 idle: 32984 37 8 [14090.014732] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14074s workers=2 idle: 661 [14092.056685] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [14092.061871] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14092.100252] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14092.106538] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14092.111525] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14092.121803] RIP: 0010:native_safe_halt+0x2e/0x50 [14092.124314] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14092.138289] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14092.143756] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14092.148062] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [14092.152309] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [14092.158884] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [14092.166038] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [14092.171072] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [14092.178252] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14092.184842] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14092.189316] Call Trace: [14092.193102] [14092.195833] ? show_regs.cold.9+0x22/0x2f [14092.197920] ? watchdog_timer_fn.cold.6+0x62/0x10a [14092.202303] ? __hrtimer_run_queues+0x1c2/0x4d0 [14092.205872] ? watchdog+0x40/0x40 [14092.209164] ? hrtimer_interrupt+0x128/0x2c0 [14092.212941] ? smp_apic_timer_interrupt+0xa6/0x270 [14092.215702] ? apic_timer_interrupt+0xf/0x20 [14092.218712] [14092.220081] ? native_safe_halt+0x2e/0x50 [14092.221948] kvm_wait+0xcd/0xe0 [14092.225732] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14092.227662] do_raw_spin_lock+0x177/0x190 [14092.231516] _raw_spin_lock+0x21/0x30 [14092.234082] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14092.239044] process_one_work+0x2c8/0x700 [14092.241609] worker_thread+0x296/0x6e0 [14092.244087] ? rescuer_thread+0x570/0x570 [14092.245936] kthread+0x1d1/0x200 [14092.247029] ? set_kthread_struct+0x70/0x70 [14092.248903] ret_from_fork+0x1f/0x30 [14100.055087] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [14100.062619] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14100.094021] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14100.106700] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14100.111265] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14100.118411] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14100.146335] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14100.161866] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [14100.167132] RDX: 00000000000038e6 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [14100.175059] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [14100.183258] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [14100.194272] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14100.205119] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [14100.215043] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14100.219805] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14100.225113] Call Trace: [14100.227000] [14100.228074] ? show_regs.cold.9+0x22/0x2f [14100.230481] ? watchdog_timer_fn.cold.6+0x62/0x10a [14100.233755] ? __hrtimer_run_queues+0x1c2/0x4d0 [14100.236982] ? watchdog+0x40/0x40 [14100.239051] ? hrtimer_interrupt+0x128/0x2c0 [14100.241409] ? smp_apic_timer_interrupt+0xa6/0x270 [14100.243608] ? apic_timer_interrupt+0xf/0x20 [14100.245637] [14100.247041] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14100.249488] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14100.251552] do_raw_spin_lock+0x177/0x190 [14100.252955] _raw_spin_lock+0x21/0x30 [14100.254662] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14100.257370] obd_mod_free+0x46/0x140 [ptlrpc] [14100.260118] mdc_commit_open+0x91/0xb0 [mdc] [14100.262357] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14100.264814] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14100.267557] after_reply+0xe2c/0x1780 [ptlrpc] [14100.269946] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14100.272851] ? __next_timer_interrupt+0x160/0x160 [14100.275141] ptlrpcd+0xa82/0xe40 [ptlrpc] [14100.277540] ? do_wait_intr+0xf0/0xf0 [14100.279426] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14100.282251] kthread+0x1d1/0x200 [14100.283877] ? set_kthread_struct+0x70/0x70 [14100.286137] ret_from_fork+0x1f/0x30 [14108.057915] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [14108.066262] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14108.109123] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14108.115363] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14108.122082] RIP: 0010:native_safe_halt+0x2e/0x50 [14108.127258] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14108.133801] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14108.137559] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14108.143154] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [14108.149581] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [14108.154706] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [14108.161238] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [14108.166627] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [14108.171819] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14108.175417] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [14108.179910] Call Trace: [14108.181482] [14108.185646] ? show_regs.cold.9+0x22/0x2f [14108.188242] ? watchdog_timer_fn.cold.6+0x62/0x10a [14108.195213] ? __hrtimer_run_queues+0x1c2/0x4d0 [14108.197284] ? watchdog+0x40/0x40 [14108.199105] ? hrtimer_interrupt+0x128/0x2c0 [14108.204776] ? smp_apic_timer_interrupt+0xa6/0x270 [14108.211149] ? apic_timer_interrupt+0xf/0x20 [14108.212575] [14108.213032] ? native_safe_halt+0x2e/0x50 [14108.214144] kvm_wait+0xcd/0xe0 [14108.214907] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14108.216304] do_raw_spin_lock+0x177/0x190 [14108.219714] _raw_spin_lock+0x21/0x30 [14108.220957] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14108.224641] ? do_raw_spin_unlock+0x75/0x190 [14108.226143] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14108.229269] ? do_wait_intr+0xf0/0xf0 [14108.231692] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14108.236028] mdc_close+0x53b/0xd70 [mdc] [14108.240132] lmv_close+0x18d/0x490 [lmv] [14108.241952] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14108.245490] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14108.252740] ll_md_real_close+0xa5/0x2a0 [lustre] [14108.255290] ll_md_close+0x274/0x800 [lustre] [14108.259731] ll_file_release+0x128/0xc50 [lustre] [14108.261958] __fput+0x10d/0x440 [14108.263341] ____fput+0x12/0x20 [14108.264764] task_work_run+0xc6/0x110 [14108.266260] exit_to_usermode_loop+0x1dd/0x1f0 [14108.268094] do_syscall_64+0x3d6/0x440 [14108.269544] entry_SYSCALL_64_after_hwframe+0x49/0xae [14108.271428] RIP: 0033:0x7f151bee3cd8 [14108.272966] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14108.281938] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14108.289044] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [14108.292432] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [14108.297660] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14108.300689] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14108.308030] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14116.052421] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [14116.056885] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14116.109672] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14116.115416] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14116.118988] RIP: 0010:native_safe_halt+0x2e/0x50 [14116.121257] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14116.144374] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14116.155695] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14116.162431] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [14116.168865] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [14116.176867] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [14116.181235] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [14116.207881] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [14116.210517] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14116.220051] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [14116.231231] Call Trace: [14116.232485] [14116.236077] ? show_regs.cold.9+0x22/0x2f [14116.244267] ? watchdog_timer_fn.cold.6+0x62/0x10a [14116.254384] ? __hrtimer_run_queues+0x1c2/0x4d0 [14116.255911] ? watchdog+0x40/0x40 [14116.258552] ? hrtimer_interrupt+0x128/0x2c0 [14116.259717] ? smp_apic_timer_interrupt+0xa6/0x270 [14116.264869] ? apic_timer_interrupt+0xf/0x20 [14116.266645] [14116.267558] ? native_safe_halt+0x2e/0x50 [14116.271801] kvm_wait+0xcd/0xe0 [14116.273838] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14116.276493] do_raw_spin_lock+0x177/0x190 [14116.280853] _raw_spin_lock+0x21/0x30 [14116.282817] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14116.287688] ? do_raw_spin_unlock+0x75/0x190 [14116.289424] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14116.292979] ? do_wait_intr+0xf0/0xf0 [14116.294114] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14116.308424] mdc_reint+0x2d/0x110 [mdc] [14116.309623] mdc_unlink+0x658/0x870 [mdc] [14116.310820] lmv_unlink+0x34d/0xbd0 [lmv] [14116.317224] ll_unlink+0x24d/0x7f0 [lustre] [14116.320090] vfs_unlink+0x202/0x330 [14116.322361] do_unlinkat+0x2fa/0x460 [14116.324528] __x64_sys_unlink+0x37/0x50 [14116.326150] do_syscall_64+0xc1/0x440 [14116.327874] entry_SYSCALL_64_after_hwframe+0x49/0xae [14116.330762] RIP: 0033:0x7f151bdee10b [14116.333217] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [14116.342322] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [14116.345984] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [14116.349428] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [14116.352559] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14116.355188] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14116.358453] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14120.056355] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [14120.066185] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14120.110452] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14120.120029] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14120.126665] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14120.133690] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14120.136743] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14120.149961] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14120.156490] RAX: 0000000000005a84 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14120.166954] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [14120.181644] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [14120.192713] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [14120.195890] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [14120.202972] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [14120.213995] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14120.224063] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14120.233175] Call Trace: [14120.234092] [14120.234832] ? show_regs.cold.9+0x22/0x2f [14120.241905] ? watchdog_timer_fn.cold.6+0x62/0x10a [14120.259764] ? __hrtimer_run_queues+0x1c2/0x4d0 [14120.268049] ? watchdog+0x40/0x40 [14120.270184] ? hrtimer_interrupt+0x128/0x2c0 [14120.273826] ? smp_apic_timer_interrupt+0xa6/0x270 [14120.286038] ? apic_timer_interrupt+0xf/0x20 [14120.292626] [14120.296459] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14120.302849] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14120.318286] do_raw_spin_lock+0x177/0x190 [14120.330682] _raw_spin_lock+0x21/0x30 [14120.333915] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14120.340750] process_one_work+0x2c8/0x700 [14120.355446] worker_thread+0x296/0x6e0 [14120.359674] ? rescuer_thread+0x570/0x570 [14120.367708] kthread+0x1d1/0x200 [14120.371307] ? set_kthread_struct+0x70/0x70 [14120.378846] ret_from_fork+0x1f/0x30 [14120.417036] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12189s! [14120.426482] Showing busy workqueues and worker pools: [14120.432636] workqueue events: flags=0x0 [14120.438400] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14120.444311] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [14120.450747] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14120.456487] pending: kfree_rcu_monitor [14120.458425] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14120.464509] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [14120.469741] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14120.477867] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [14120.484758] workqueue events_long: flags=0x0 [14120.488697] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14120.494128] pending: lu_objects_flush [obdclass] [14120.499850] workqueue events_unbound: flags=0x2 [14120.506714] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14120.510036] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14120.516196] workqueue events_power_efficient: flags=0x80 [14120.518184] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14120.520818] pending: do_cache_clean, neigh_periodic_work, check_lifetime [14120.523699] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14120.526465] pending: neigh_periodic_work [14120.528307] workqueue mm_percpu_wq: flags=0x8 [14120.529992] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14120.532768] pending: vmstat_update, lru_add_drain_per_cpu [14120.535151] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14120.537807] pending: vmstat_update [14120.539353] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14120.542005] pending: lru_add_drain_per_cpu [14120.543891] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [14120.546665] pending: lru_add_drain_per_cpu BAR(48) [14120.548751] workqueue writeback: flags=0x4a [14120.550465] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [14120.552796] pending: wb_workfn [14120.554313] workqueue xprtiod: flags=0xa [14120.555828] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14120.558329] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [14120.561415] workqueue ipv6_addrconf: flags=0x40008 [14120.563276] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14120.565736] pending: addrconf_verify_work [14120.567440] workqueue knbd0-recv: flags=0x1a [14120.569033] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14120.571530] in-flight: 658:recv_work [14120.572978] workqueue ptlrpc_pinger: flags=0x2 [14120.574659] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14120.576744] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14120.579044] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12189s workers=4 idle: 32984 37 8 [14120.582179] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14105s workers=2 idle: 661 [14128.055025] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [14128.058321] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14128.085562] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14128.091179] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14128.095552] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14128.098209] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14128.105363] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14128.108453] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [14128.111190] RDX: 0000000000006034 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [14128.113554] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [14128.115320] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [14128.117154] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14128.118699] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [14128.120683] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14128.121960] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14128.123969] Call Trace: [14128.124570] [14128.125228] ? show_regs.cold.9+0x22/0x2f [14128.126561] ? watchdog_timer_fn.cold.6+0x62/0x10a [14128.128237] ? __hrtimer_run_queues+0x1c2/0x4d0 [14128.129528] ? watchdog+0x40/0x40 [14128.130406] ? hrtimer_interrupt+0x128/0x2c0 [14128.131417] ? smp_apic_timer_interrupt+0xa6/0x270 [14128.132632] ? apic_timer_interrupt+0xf/0x20 [14128.133830] [14128.134557] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14128.135868] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14128.137316] do_raw_spin_lock+0x177/0x190 [14128.138170] _raw_spin_lock+0x21/0x30 [14128.139136] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14128.140475] obd_mod_free+0x46/0x140 [ptlrpc] [14128.141554] mdc_commit_open+0x91/0xb0 [mdc] [14128.143158] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14128.144783] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14128.147088] after_reply+0xe2c/0x1780 [ptlrpc] [14128.148941] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14128.151216] ? __next_timer_interrupt+0x160/0x160 [14128.152761] ptlrpcd+0xa82/0xe40 [ptlrpc] [14128.154621] ? do_wait_intr+0xf0/0xf0 [14128.156123] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14128.158186] kthread+0x1d1/0x200 [14128.159475] ? set_kthread_struct+0x70/0x70 [14128.160884] ret_from_fork+0x1f/0x30 [14136.064394] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [14136.075656] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14136.121182] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14136.134040] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14136.139026] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14136.141848] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14136.149193] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14136.152615] RAX: 0000000000007d12 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14136.157240] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [14136.160727] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [14136.167420] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [14136.174499] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [14136.181601] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [14136.191798] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14136.197004] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [14136.203266] Call Trace: [14136.204552] [14136.205506] ? show_regs.cold.9+0x22/0x2f [14136.208642] ? watchdog_timer_fn.cold.6+0x62/0x10a [14136.211312] ? __hrtimer_run_queues+0x1c2/0x4d0 [14136.213253] ? watchdog+0x40/0x40 [14136.214898] ? hrtimer_interrupt+0x128/0x2c0 [14136.216678] ? smp_apic_timer_interrupt+0xa6/0x270 [14136.219132] ? apic_timer_interrupt+0xf/0x20 [14136.220867] [14136.221877] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14136.224712] do_raw_spin_lock+0x177/0x190 [14136.226564] _raw_spin_lock+0x21/0x30 [14136.228382] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14136.231396] ? do_raw_spin_unlock+0x75/0x190 [14136.233572] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14136.236013] ? do_wait_intr+0xf0/0xf0 [14136.237585] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14136.239744] mdc_close+0x53b/0xd70 [mdc] [14136.241531] lmv_close+0x18d/0x490 [lmv] [14136.243179] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14136.245540] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14136.248590] ll_md_real_close+0xa5/0x2a0 [lustre] [14136.250543] ll_md_close+0x274/0x800 [lustre] [14136.252403] ll_file_release+0x128/0xc50 [lustre] [14136.255123] __fput+0x10d/0x440 [14136.256440] ____fput+0x12/0x20 [14136.257734] task_work_run+0xc6/0x110 [14136.261418] exit_to_usermode_loop+0x1dd/0x1f0 [14136.263853] do_syscall_64+0x3d6/0x440 [14136.265608] entry_SYSCALL_64_after_hwframe+0x49/0xae [14136.267792] RIP: 0033:0x7f151bee3cd8 [14136.269450] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14136.277210] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14136.280705] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [14136.282921] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [14136.285262] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14136.288694] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14136.295518] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14148.052026] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [14148.055288] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14148.056026] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14148.074342] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14148.074358] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14148.074362] RIP: 0010:native_safe_halt+0x2e/0x50 [14148.074375] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14148.074380] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14148.074386] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14148.074389] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [14148.074391] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [14148.074394] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [14148.074396] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [14148.074400] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [14148.074403] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14148.074405] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [14148.074414] Call Trace: [14148.074419] [14148.074424] ? show_regs.cold.9+0x22/0x2f [14148.074433] ? watchdog_timer_fn.cold.6+0x62/0x10a [14148.074440] ? __hrtimer_run_queues+0x1c2/0x4d0 [14148.074445] ? watchdog+0x40/0x40 [14148.074449] ? hrtimer_interrupt+0x128/0x2c0 [14148.074454] ? smp_apic_timer_interrupt+0xa6/0x270 [14148.074461] ? apic_timer_interrupt+0xf/0x20 [14148.074466] [14148.074469] ? native_safe_halt+0x2e/0x50 [14148.074473] kvm_wait+0xcd/0xe0 [14148.074480] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14148.074489] do_raw_spin_lock+0x177/0x190 [14148.074493] _raw_spin_lock+0x21/0x30 [14148.074569] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14148.074869] ? do_raw_spin_unlock+0x75/0x190 [14148.074876] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14148.075093] ? do_wait_intr+0xf0/0xf0 [14148.075101] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14148.075309] mdc_reint+0x2d/0x110 [mdc] [14148.075345] mdc_unlink+0x658/0x870 [mdc] [14148.075379] lmv_unlink+0x34d/0xbd0 [lmv] [14148.075418] ll_unlink+0x24d/0x7f0 [lustre] [14148.075664] vfs_unlink+0x202/0x330 [14148.075672] do_unlinkat+0x2fa/0x460 [14148.075678] __x64_sys_unlink+0x37/0x50 [14148.075681] do_syscall_64+0xc1/0x440 [14148.075688] entry_SYSCALL_64_after_hwframe+0x49/0xae [14148.075695] RIP: 0033:0x7f151bdee10b [14148.075702] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [14148.075706] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [14148.075711] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [14148.075714] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [14148.075716] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14148.075718] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14148.075721] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14148.199639] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14148.213696] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14148.216890] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14148.219369] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14148.221453] RIP: 0010:native_safe_halt+0x2e/0x50 [14148.222780] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14148.228341] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14148.230618] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14148.232448] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [14148.234547] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [14148.236394] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [14148.238273] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [14148.240303] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [14148.242767] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14148.244230] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14148.246312] Call Trace: [14148.247071] [14148.247611] ? show_regs.cold.9+0x22/0x2f [14148.248751] ? watchdog_timer_fn.cold.6+0x62/0x10a [14148.250093] ? __hrtimer_run_queues+0x1c2/0x4d0 [14148.251363] ? watchdog+0x40/0x40 [14148.252299] ? hrtimer_interrupt+0x128/0x2c0 [14148.253661] ? smp_apic_timer_interrupt+0xa6/0x270 [14148.255224] ? apic_timer_interrupt+0xf/0x20 [14148.256693] [14148.257377] ? native_safe_halt+0x2e/0x50 [14148.258742] kvm_wait+0xcd/0xe0 [14148.259741] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14148.261256] do_raw_spin_lock+0x177/0x190 [14148.262332] _raw_spin_lock+0x21/0x30 [14148.263446] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14148.265333] process_one_work+0x2c8/0x700 [14148.266600] worker_thread+0x296/0x6e0 [14148.267661] ? rescuer_thread+0x570/0x570 [14148.268982] kthread+0x1d1/0x200 [14148.269972] ? set_kthread_struct+0x70/0x70 [14148.271151] ret_from_fork+0x1f/0x30 [14150.624033] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12220s! [14150.626366] Showing busy workqueues and worker pools: [14150.627734] workqueue events: flags=0x0 [14150.628970] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14150.631144] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [14150.634363] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14150.636367] pending: kfree_rcu_monitor [14150.637609] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14150.639431] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [14150.642592] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14150.644664] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [14150.647271] workqueue events_long: flags=0x0 [14150.648744] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14150.650905] pending: lu_objects_flush [obdclass] [14150.652540] workqueue events_unbound: flags=0x2 [14150.653954] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14150.655681] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14150.658830] workqueue events_power_efficient: flags=0x80 [14150.660286] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14150.662427] pending: do_cache_clean, neigh_periodic_work, check_lifetime [14150.664946] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14150.666876] pending: neigh_periodic_work [14150.668142] workqueue mm_percpu_wq: flags=0x8 [14150.669397] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14150.671465] pending: vmstat_update, lru_add_drain_per_cpu [14150.674044] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14150.676289] pending: vmstat_update [14150.677507] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14150.680496] pending: lru_add_drain_per_cpu [14150.681954] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [14150.684101] pending: lru_add_drain_per_cpu BAR(48) [14150.685525] workqueue writeback: flags=0x4a [14150.686571] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [14150.688391] pending: wb_workfn [14150.689676] workqueue xprtiod: flags=0xa [14150.690921] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14150.692698] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [14150.695016] workqueue ipv6_addrconf: flags=0x40008 [14150.696381] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14150.698134] pending: addrconf_verify_work [14150.699560] workqueue knbd0-recv: flags=0x1a [14150.700710] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14150.702643] in-flight: 658:recv_work [14150.703709] workqueue ptlrpc_pinger: flags=0x2 [14150.705073] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14150.707738] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14150.709625] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12220s workers=4 idle: 32984 37 8 [14150.711889] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14135s workers=2 idle: 661 [14156.055025] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [14156.058510] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14156.081320] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14156.085834] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14156.090198] RIP: 0010:native_safe_halt+0x2e/0x50 [14156.094636] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14156.113194] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14156.117928] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [14156.119875] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [14156.122195] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [14156.130287] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [14156.134400] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14156.138949] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [14156.143498] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14156.145375] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14156.147919] Call Trace: [14156.148591] [14156.149344] ? show_regs.cold.9+0x22/0x2f [14156.150608] ? watchdog_timer_fn.cold.6+0x62/0x10a [14156.152264] ? __hrtimer_run_queues+0x1c2/0x4d0 [14156.154030] ? watchdog+0x40/0x40 [14156.155045] ? hrtimer_interrupt+0x128/0x2c0 [14156.156514] ? smp_apic_timer_interrupt+0xa6/0x270 [14156.158147] ? apic_timer_interrupt+0xf/0x20 [14156.159536] [14156.160182] ? native_safe_halt+0x2e/0x50 [14156.161376] kvm_wait+0xcd/0xe0 [14156.162391] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14156.164068] do_raw_spin_lock+0x177/0x190 [14156.165340] _raw_spin_lock+0x21/0x30 [14156.166639] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14156.168429] obd_mod_free+0x46/0x140 [ptlrpc] [14156.173166] mdc_commit_open+0x91/0xb0 [mdc] [14156.175599] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14156.180050] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14156.184906] after_reply+0xe2c/0x1780 [ptlrpc] [14156.188315] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14156.190334] ? __next_timer_interrupt+0x160/0x160 [14156.192247] ptlrpcd+0xa82/0xe40 [ptlrpc] [14156.194215] ? do_wait_intr+0xf0/0xf0 [14156.195631] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14156.197982] kthread+0x1d1/0x200 [14156.199215] ? set_kthread_struct+0x70/0x70 [14156.201110] ret_from_fork+0x1f/0x30 [14164.057012] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [14164.061005] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14164.095878] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14164.105629] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14164.113482] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [14164.116560] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [14164.125454] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14164.132277] RAX: 000000000000266a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14164.136358] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [14164.141085] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [14164.145017] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [14164.148459] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [14164.152406] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [14164.157394] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14164.159517] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [14164.164231] Call Trace: [14164.165843] [14164.167104] ? show_regs.cold.9+0x22/0x2f [14164.169147] ? watchdog_timer_fn.cold.6+0x62/0x10a [14164.171547] ? __hrtimer_run_queues+0x1c2/0x4d0 [14164.173960] ? watchdog+0x40/0x40 [14164.176174] ? hrtimer_interrupt+0x128/0x2c0 [14164.177912] ? smp_apic_timer_interrupt+0xa6/0x270 [14164.179855] ? apic_timer_interrupt+0xf/0x20 [14164.181451] [14164.182107] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [14164.184622] do_raw_spin_lock+0x177/0x190 [14164.186209] _raw_spin_lock+0x21/0x30 [14164.188456] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14164.190363] ? do_raw_spin_unlock+0x75/0x190 [14164.192195] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14164.193836] ? do_wait_intr+0xf0/0xf0 [14164.195058] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14164.196563] mdc_close+0x53b/0xd70 [mdc] [14164.198244] lmv_close+0x18d/0x490 [lmv] [14164.199422] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14164.201184] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14164.203152] ll_md_real_close+0xa5/0x2a0 [lustre] [14164.204631] ll_md_close+0x274/0x800 [lustre] [14164.206309] ll_file_release+0x128/0xc50 [lustre] [14164.207610] __fput+0x10d/0x440 [14164.208413] ____fput+0x12/0x20 [14164.209267] task_work_run+0xc6/0x110 [14164.210121] exit_to_usermode_loop+0x1dd/0x1f0 [14164.211295] do_syscall_64+0x3d6/0x440 [14164.212252] entry_SYSCALL_64_after_hwframe+0x49/0xae [14164.213529] RIP: 0033:0x7f151bee3cd8 [14164.214510] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14164.219567] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14164.221812] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [14164.223855] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [14164.225598] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14164.227427] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14164.229501] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14176.052062] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [14176.056029] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14176.062294] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14176.062375] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14176.062383] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14176.062386] RIP: 0010:native_safe_halt+0x2e/0x50 [14176.062400] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14176.062404] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14176.062409] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14176.062411] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [14176.062414] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [14176.062416] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [14176.062418] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [14176.062422] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [14176.062425] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14176.062428] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [14176.062436] Call Trace: [14176.062441] [14176.062447] ? show_regs.cold.9+0x22/0x2f [14176.062456] ? watchdog_timer_fn.cold.6+0x62/0x10a [14176.062463] ? __hrtimer_run_queues+0x1c2/0x4d0 [14176.062467] ? watchdog+0x40/0x40 [14176.062471] ? hrtimer_interrupt+0x128/0x2c0 [14176.062476] ? smp_apic_timer_interrupt+0xa6/0x270 [14176.062484] ? apic_timer_interrupt+0xf/0x20 [14176.062489] [14176.062492] ? native_safe_halt+0x2e/0x50 [14176.062497] kvm_wait+0xcd/0xe0 [14176.062504] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14176.062511] do_raw_spin_lock+0x177/0x190 [14176.062515] _raw_spin_lock+0x21/0x30 [14176.062520] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14176.063459] ? do_raw_spin_unlock+0x75/0x190 [14176.063467] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14176.063659] ? do_wait_intr+0xf0/0xf0 [14176.063678] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14176.063933] mdc_reint+0x2d/0x110 [mdc] [14176.063973] mdc_unlink+0x658/0x870 [mdc] [14176.063997] lmv_unlink+0x34d/0xbd0 [lmv] [14176.064032] ll_unlink+0x24d/0x7f0 [lustre] [14176.064152] vfs_unlink+0x202/0x330 [14176.064160] do_unlinkat+0x2fa/0x460 [14176.064164] __x64_sys_unlink+0x37/0x50 [14176.064168] do_syscall_64+0xc1/0x440 [14176.064174] entry_SYSCALL_64_after_hwframe+0x49/0xae [14176.064181] RIP: 0033:0x7f151bdee10b [14176.064189] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [14176.064193] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [14176.064198] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [14176.064201] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [14176.064203] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14176.064205] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14176.064207] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14176.274946] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14176.299709] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14176.309056] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14176.314097] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14176.317239] RIP: 0010:native_safe_halt+0x2e/0x50 [14176.319344] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14176.329070] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14176.335020] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14176.337885] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [14176.343935] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [14176.349474] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [14176.352725] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [14176.356695] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [14176.364143] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14176.367450] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14176.374370] Call Trace: [14176.376222] [14176.376878] ? show_regs.cold.9+0x22/0x2f [14176.381621] ? watchdog_timer_fn.cold.6+0x62/0x10a [14176.383723] ? __hrtimer_run_queues+0x1c2/0x4d0 [14176.385962] ? watchdog+0x40/0x40 [14176.387327] ? hrtimer_interrupt+0x128/0x2c0 [14176.389322] ? smp_apic_timer_interrupt+0xa6/0x270 [14176.391902] ? apic_timer_interrupt+0xf/0x20 [14176.395785] [14176.396928] ? native_safe_halt+0x2e/0x50 [14176.399506] kvm_wait+0xcd/0xe0 [14176.400860] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14176.410091] do_raw_spin_lock+0x177/0x190 [14176.413631] _raw_spin_lock+0x21/0x30 [14176.415875] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14176.418861] process_one_work+0x2c8/0x700 [14176.421855] worker_thread+0x296/0x6e0 [14176.423824] ? rescuer_thread+0x570/0x570 [14176.427876] kthread+0x1d1/0x200 [14176.430902] ? set_kthread_struct+0x70/0x70 [14176.433028] ret_from_fork+0x1f/0x30 [14180.832584] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12250s! [14180.839323] Showing busy workqueues and worker pools: [14180.841337] workqueue events: flags=0x0 [14180.848184] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14180.855982] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [14180.860490] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14180.862679] pending: kfree_rcu_monitor [14180.865172] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14180.870449] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [14180.877466] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14180.885722] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [14180.889375] workqueue events_long: flags=0x0 [14180.894478] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14180.896916] pending: lu_objects_flush [obdclass] [14180.898975] workqueue events_unbound: flags=0x2 [14180.900907] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14180.903892] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14180.910712] workqueue events_power_efficient: flags=0x80 [14180.912453] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14180.918807] pending: do_cache_clean, neigh_periodic_work, check_lifetime [14180.921665] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14180.923676] pending: neigh_periodic_work [14180.924853] workqueue mm_percpu_wq: flags=0x8 [14180.926344] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14180.929003] pending: vmstat_update, lru_add_drain_per_cpu [14180.931329] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14180.934021] pending: vmstat_update [14180.935448] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14180.938165] pending: lru_add_drain_per_cpu [14180.939661] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [14180.941601] pending: lru_add_drain_per_cpu BAR(48) [14180.943010] workqueue writeback: flags=0x4a [14180.944249] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [14180.945932] pending: wb_workfn [14180.946818] workqueue xprtiod: flags=0xa [14180.948071] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14180.949787] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [14180.951967] workqueue ipv6_addrconf: flags=0x40008 [14180.953258] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14180.955156] pending: addrconf_verify_work [14180.956943] workqueue knbd0-recv: flags=0x1a [14180.958109] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14180.959962] in-flight: 658:recv_work [14180.961058] workqueue ptlrpc_pinger: flags=0x2 [14180.962327] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14180.964423] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14180.967019] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12250s workers=4 idle: 32984 37 8 [14180.970985] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14165s workers=2 idle: 661 [14184.056176] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [14184.071194] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14184.126640] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14184.138655] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14184.146080] RIP: 0010:native_safe_halt+0x2e/0x50 [14184.148799] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14184.160057] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14184.168600] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [14184.172818] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [14184.176815] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [14184.180101] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [14184.183547] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14184.186671] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [14184.194320] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14184.204238] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14184.212785] Call Trace: [14184.213828] [14184.214545] ? show_regs.cold.9+0x22/0x2f [14184.233102] ? watchdog_timer_fn.cold.6+0x62/0x10a [14184.238886] ? __hrtimer_run_queues+0x1c2/0x4d0 [14184.243615] ? watchdog+0x40/0x40 [14184.248292] ? hrtimer_interrupt+0x128/0x2c0 [14184.251814] ? smp_apic_timer_interrupt+0xa6/0x270 [14184.256068] ? apic_timer_interrupt+0xf/0x20 [14184.260121] [14184.263917] ? native_safe_halt+0x2e/0x50 [14184.265752] kvm_wait+0xcd/0xe0 [14184.267567] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14184.272169] do_raw_spin_lock+0x177/0x190 [14184.275175] _raw_spin_lock+0x21/0x30 [14184.277758] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14184.281563] obd_mod_free+0x46/0x140 [ptlrpc] [14184.284945] mdc_commit_open+0x91/0xb0 [mdc] [14184.290431] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14184.296596] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14184.300142] after_reply+0xe2c/0x1780 [ptlrpc] [14184.303651] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14184.307788] ? __next_timer_interrupt+0x160/0x160 [14184.309901] ptlrpcd+0xa82/0xe40 [ptlrpc] [14184.315722] ? do_wait_intr+0xf0/0xf0 [14184.319544] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14184.321943] kthread+0x1d1/0x200 [14184.323172] ? set_kthread_struct+0x70/0x70 [14184.334294] ret_from_fork+0x1f/0x30 [14192.057029] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [14192.069090] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14192.125325] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14192.140970] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14192.150217] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14192.155493] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14192.164236] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14192.171167] RAX: 0000000000002e0c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14192.176571] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [14192.188200] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [14192.195238] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [14192.203086] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [14192.205467] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [14192.213356] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14192.215102] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [14192.230742] Call Trace: [14192.235154] [14192.236817] ? show_regs.cold.9+0x22/0x2f [14192.238610] ? watchdog_timer_fn.cold.6+0x62/0x10a [14192.241123] ? __hrtimer_run_queues+0x1c2/0x4d0 [14192.242919] ? watchdog+0x40/0x40 [14192.246087] ? hrtimer_interrupt+0x128/0x2c0 [14192.249578] ? smp_apic_timer_interrupt+0xa6/0x270 [14192.252314] ? apic_timer_interrupt+0xf/0x20 [14192.261005] [14192.263483] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14192.266521] do_raw_spin_lock+0x177/0x190 [14192.275248] _raw_spin_lock+0x21/0x30 [14192.280258] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14192.282373] ? do_raw_spin_unlock+0x75/0x190 [14192.291073] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14192.295716] ? do_wait_intr+0xf0/0xf0 [14192.298585] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14192.301117] mdc_close+0x53b/0xd70 [mdc] [14192.304636] lmv_close+0x18d/0x490 [lmv] [14192.307469] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14192.314881] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14192.323537] ll_md_real_close+0xa5/0x2a0 [lustre] [14192.327859] ll_md_close+0x274/0x800 [lustre] [14192.332363] ll_file_release+0x128/0xc50 [lustre] [14192.339000] __fput+0x10d/0x440 [14192.340121] ____fput+0x12/0x20 [14192.346576] task_work_run+0xc6/0x110 [14192.350988] exit_to_usermode_loop+0x1dd/0x1f0 [14192.358188] do_syscall_64+0x3d6/0x440 [14192.361313] entry_SYSCALL_64_after_hwframe+0x49/0xae [14192.366736] RIP: 0033:0x7f151bee3cd8 [14192.372778] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14192.393236] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14192.405988] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [14192.408952] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [14192.421972] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14192.428619] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14192.433206] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14204.056040] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14204.061611] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14204.084129] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14204.088884] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14204.092472] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14204.095584] RIP: 0010:native_safe_halt+0x2e/0x50 [14204.097543] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14204.105189] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14204.108567] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14204.111272] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [14204.114690] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [14204.117683] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [14204.121024] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [14204.123964] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [14204.127788] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14204.130001] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14204.133076] Call Trace: [14204.134150] [14204.135075] ? show_regs.cold.9+0x22/0x2f [14204.136647] ? watchdog_timer_fn.cold.6+0x62/0x10a [14204.139221] ? __hrtimer_run_queues+0x1c2/0x4d0 [14204.141294] ? watchdog+0x40/0x40 [14204.142480] ? hrtimer_interrupt+0x128/0x2c0 [14204.144606] ? smp_apic_timer_interrupt+0xa6/0x270 [14204.147346] ? apic_timer_interrupt+0xf/0x20 [14204.149355] [14204.150114] ? native_safe_halt+0x2e/0x50 [14204.151917] kvm_wait+0xcd/0xe0 [14204.153536] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14204.155990] do_raw_spin_lock+0x177/0x190 [14204.157595] _raw_spin_lock+0x21/0x30 [14204.159605] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14204.162117] process_one_work+0x2c8/0x700 [14204.163792] worker_thread+0x296/0x6e0 [14204.168127] ? rescuer_thread+0x570/0x570 [14204.169875] kthread+0x1d1/0x200 [14204.171783] ? set_kthread_struct+0x70/0x70 [14204.177159] ret_from_fork+0x1f/0x30 [14208.052151] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [14208.054494] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14208.083775] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14208.090127] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14208.095835] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14208.099789] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14208.109067] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14208.115151] RAX: 0000000000005855 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14208.117736] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [14208.120486] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [14208.123646] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [14208.126625] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [14208.129432] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [14208.132767] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14208.135032] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [14208.137815] Call Trace: [14208.138818] [14208.139549] ? show_regs.cold.9+0x22/0x2f [14208.140818] ? watchdog_timer_fn.cold.6+0x62/0x10a [14208.142192] ? __hrtimer_run_queues+0x1c2/0x4d0 [14208.143815] ? watchdog+0x40/0x40 [14208.144944] ? hrtimer_interrupt+0x128/0x2c0 [14208.146204] ? smp_apic_timer_interrupt+0xa6/0x270 [14208.147880] ? apic_timer_interrupt+0xf/0x20 [14208.151103] [14208.151728] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14208.154455] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14208.156190] do_raw_spin_lock+0x177/0x190 [14208.159991] _raw_spin_lock+0x21/0x30 [14208.168978] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14208.176884] ? do_raw_spin_unlock+0x75/0x190 [14208.182925] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14208.189045] ? do_wait_intr+0xf0/0xf0 [14208.192911] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14208.197626] mdc_reint+0x2d/0x110 [mdc] [14208.199165] mdc_unlink+0x658/0x870 [mdc] [14208.203977] lmv_unlink+0x34d/0xbd0 [lmv] [14208.207726] ll_unlink+0x24d/0x7f0 [lustre] [14208.210447] vfs_unlink+0x202/0x330 [14208.211930] do_unlinkat+0x2fa/0x460 [14208.214792] __x64_sys_unlink+0x37/0x50 [14208.215699] do_syscall_64+0xc1/0x440 [14208.216611] entry_SYSCALL_64_after_hwframe+0x49/0xae [14208.217883] RIP: 0033:0x7f151bdee10b [14208.220533] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [14208.230298] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [14208.233400] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [14208.236113] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [14208.238615] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14208.241984] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14208.245733] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14211.040303] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12280s! [14211.045242] Showing busy workqueues and worker pools: [14211.049677] workqueue events: flags=0x0 [14211.051891] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14211.057132] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [14211.061118] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14211.063977] pending: kfree_rcu_monitor [14211.066167] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14211.068532] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [14211.072436] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14211.075548] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [14211.078664] workqueue events_long: flags=0x0 [14211.080143] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14211.082946] pending: lu_objects_flush [obdclass] [14211.084960] workqueue events_unbound: flags=0x2 [14211.086608] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14211.088799] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14211.092496] workqueue events_power_efficient: flags=0x80 [14211.094128] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14211.096421] pending: do_cache_clean, neigh_periodic_work, check_lifetime [14211.098762] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14211.101395] pending: neigh_periodic_work [14211.103012] workqueue mm_percpu_wq: flags=0x8 [14211.105093] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14211.107767] pending: vmstat_update, lru_add_drain_per_cpu [14211.113066] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14211.115956] pending: vmstat_update [14211.117380] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14211.119542] pending: lru_add_drain_per_cpu [14211.120794] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [14211.123531] pending: lru_add_drain_per_cpu BAR(48) [14211.134760] workqueue writeback: flags=0x4a [14211.138014] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [14211.142792] pending: wb_workfn [14211.145509] workqueue xprtiod: flags=0xa [14211.147242] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14211.151579] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [14211.156368] workqueue ipv6_addrconf: flags=0x40008 [14211.158478] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14211.161582] pending: addrconf_verify_work [14211.164195] workqueue knbd0-recv: flags=0x1a [14211.166533] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14211.169503] in-flight: 658:recv_work [14211.171226] workqueue ptlrpc_pinger: flags=0x2 [14211.173244] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14211.176389] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14211.179176] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12280s workers=4 idle: 32984 37 8 [14211.182650] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14195s workers=2 idle: 661 [14212.057252] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [14212.061660] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14212.088455] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14212.094932] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14212.099678] RIP: 0010:native_safe_halt+0x2e/0x50 [14212.102831] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14212.113662] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14212.118051] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [14212.122758] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [14212.126940] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [14212.131154] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [14212.135942] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14212.139942] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [14212.144728] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14212.148410] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14212.153138] Call Trace: [14212.154447] [14212.155887] ? show_regs.cold.9+0x22/0x2f [14212.158484] ? watchdog_timer_fn.cold.6+0x62/0x10a [14212.161993] ? __hrtimer_run_queues+0x1c2/0x4d0 [14212.165109] ? watchdog+0x40/0x40 [14212.167115] ? hrtimer_interrupt+0x128/0x2c0 [14212.169827] ? smp_apic_timer_interrupt+0xa6/0x270 [14212.173471] ? apic_timer_interrupt+0xf/0x20 [14212.175938] [14212.177109] ? native_safe_halt+0x2e/0x50 [14212.179694] kvm_wait+0xcd/0xe0 [14212.182033] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14212.185063] do_raw_spin_lock+0x177/0x190 [14212.186824] _raw_spin_lock+0x21/0x30 [14212.188839] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14212.192170] obd_mod_free+0x46/0x140 [ptlrpc] [14212.194848] mdc_commit_open+0x91/0xb0 [mdc] [14212.197671] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14212.200599] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14212.204060] after_reply+0xe2c/0x1780 [ptlrpc] [14212.207630] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14212.212083] ? __next_timer_interrupt+0x160/0x160 [14212.215034] ptlrpcd+0xa82/0xe40 [ptlrpc] [14212.218913] ? do_wait_intr+0xf0/0xf0 [14212.221826] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14212.226913] kthread+0x1d1/0x200 [14212.228986] ? set_kthread_struct+0x70/0x70 [14212.231617] ret_from_fork+0x1f/0x30 [14220.057014] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [14220.061661] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14220.088697] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14220.097658] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14220.106728] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14220.111965] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14220.123339] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14220.127064] RAX: 0000000000004597 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14220.130737] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [14220.133820] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [14220.137173] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [14220.140817] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [14220.144498] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [14220.148017] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14220.150868] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [14220.154119] Call Trace: [14220.155366] [14220.156156] ? show_regs.cold.9+0x22/0x2f [14220.158029] ? watchdog_timer_fn.cold.6+0x62/0x10a [14220.160056] ? __hrtimer_run_queues+0x1c2/0x4d0 [14220.161992] ? watchdog+0x40/0x40 [14220.163406] ? hrtimer_interrupt+0x128/0x2c0 [14220.165072] ? smp_apic_timer_interrupt+0xa6/0x270 [14220.167407] ? apic_timer_interrupt+0xf/0x20 [14220.169844] [14220.170522] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14220.172676] do_raw_spin_lock+0x177/0x190 [14220.174441] _raw_spin_lock+0x21/0x30 [14220.176102] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14220.178509] ? do_raw_spin_unlock+0x75/0x190 [14220.180674] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14220.183349] ? do_wait_intr+0xf0/0xf0 [14220.184708] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14220.187303] mdc_close+0x53b/0xd70 [mdc] [14220.189072] lmv_close+0x18d/0x490 [lmv] [14220.190887] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14220.193296] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14220.197954] ll_md_real_close+0xa5/0x2a0 [lustre] [14220.201695] ll_md_close+0x274/0x800 [lustre] [14220.204033] ll_file_release+0x128/0xc50 [lustre] [14220.207912] __fput+0x10d/0x440 [14220.209052] ____fput+0x12/0x20 [14220.210585] task_work_run+0xc6/0x110 [14220.212345] exit_to_usermode_loop+0x1dd/0x1f0 [14220.214451] do_syscall_64+0x3d6/0x440 [14220.216634] entry_SYSCALL_64_after_hwframe+0x49/0xae [14220.219067] RIP: 0033:0x7f151bee3cd8 [14220.220739] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14220.229506] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14220.233327] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [14220.236356] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [14220.239756] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14220.243113] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14220.246032] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14224.911704] rcu: INFO: rcu_sched self-detected stall on CPU [14224.918844] rcu: 1-...!: (7251565 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [14224.931646] (t=12300264 jiffies g=249657 q=6114) [14224.935042] rcu: rcu_sched kthread starved for 12295216 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [14224.944771] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [14224.951322] rcu: RCU grace-period kthread stack dump: [14224.954985] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [14224.961989] Call Trace: [14224.965939] __schedule+0x351/0xcb0 [14224.969643] schedule+0xc0/0x180 [14224.974033] schedule_timeout+0xb4/0x190 [14224.976712] ? do_raw_spin_unlock+0x75/0x190 [14224.986307] ? __next_timer_interrupt+0x160/0x160 [14224.991432] ? prepare_to_swait_event+0xf3/0x140 [14224.998074] rcu_gp_kthread+0x8dd/0x15b0 [14224.999596] ? force_qs_rnp+0x2c0/0x2c0 [14225.001000] kthread+0x1d1/0x200 [14225.006631] ? set_kthread_struct+0x70/0x70 [14225.014031] ret_from_fork+0x1f/0x30 [14225.016529] rcu: Stack dump where RCU GP kthread last ran: [14225.022619] Sending NMI from CPU 1 to CPUs 2: [14225.026206] NMI backtrace for cpu 2 [14225.026213] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14225.026216] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14225.026217] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14225.026220] RIP: 0010:__pv_queued_spin_lock_slowpath+0x320/0x6d0 [14225.026222] Code: 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 <48> 83 05 18 d4 dc 02 01 eb e0 48 83 05 06 d4 dc 02 01 48 83 05 be [14225.026224] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 [14225.026227] RAX: 0000000000003f6e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14225.026228] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [14225.026230] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [14225.026232] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [14225.026234] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [14225.026236] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [14225.026237] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14225.026239] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14225.026240] Call Trace: [14225.026241] [14225.026242] ? show_regs.cold.9+0x22/0x2f [14225.026244] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [14225.026245] ? nmi_cpu_backtrace_handler+0x15/0x40 [14225.026246] ? nmi_handle+0xbd/0x210 [14225.026247] ? default_do_nmi+0x4b/0x1b0 [14225.026248] ? do_nmi+0x280/0x380 [14225.026250] ? end_repeat_nmi+0x16/0x1f [14225.026251] ? __pv_queued_spin_lock_slowpath+0x320/0x6d0 [14225.026252] ? __pv_queued_spin_lock_slowpath+0x320/0x6d0 [14225.026254] ? __pv_queued_spin_lock_slowpath+0x320/0x6d0 [14225.026255] [14225.026256] do_raw_spin_lock+0x177/0x190 [14225.026257] _raw_spin_lock+0x21/0x30 [14225.026258] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14225.026260] process_one_work+0x2c8/0x700 [14225.026261] worker_thread+0x296/0x6e0 [14225.026262] ? rescuer_thread+0x570/0x570 [14225.026263] kthread+0x1d1/0x200 [14225.026264] ? set_kthread_struct+0x70/0x70 [14225.026265] ret_from_fork+0x1f/0x30 [14225.026336] NMI backtrace for cpu 1 [14225.281520] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14225.285528] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14225.288925] Call Trace: [14225.289817] [14225.290569] ? dump_stack+0xbb/0x10e [14225.291847] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [14225.293736] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [14225.295910] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [14225.298176] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [14225.300772] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [14225.307982] ? rcu_dump_cpu_stacks+0x11b/0x171 [14225.313936] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [14225.318956] ? update_process_times+0x6d/0xc0 [14225.324195] ? tick_sched_handle.isra.18+0x27/0x90 [14225.326821] ? tick_sched_timer+0x53/0xb0 [14225.332905] ? __hrtimer_run_queues+0x1c2/0x4d0 [14225.337641] ? tick_sched_handle.isra.18+0x90/0x90 [14225.347540] ? hrtimer_interrupt+0x128/0x2c0 [14225.349023] ? smp_apic_timer_interrupt+0xa6/0x270 [14225.350738] ? apic_timer_interrupt+0xf/0x20 [14225.356022] [14225.357003] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14225.362544] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14225.370801] ? do_raw_spin_lock+0x177/0x190 [14225.372568] ? _raw_spin_lock+0x21/0x30 [14225.381643] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14225.385915] ? obd_mod_free+0x46/0x140 [ptlrpc] [14225.389547] ? mdc_commit_open+0x91/0xb0 [mdc] [14225.398349] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14225.402891] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14225.410387] ? after_reply+0xe2c/0x1780 [ptlrpc] [14225.416164] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14225.420802] ? __next_timer_interrupt+0x160/0x160 [14225.425698] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [14225.429336] ? do_wait_intr+0xf0/0xf0 [14225.431771] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14225.436591] ? kthread+0x1d1/0x200 [14225.438667] ? set_kthread_struct+0x70/0x70 [14225.441491] ? ret_from_fork+0x1f/0x30 [14236.052104] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [14236.055829] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14236.087854] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14236.093168] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14236.097141] RIP: 0010:native_safe_halt+0x2e/0x50 [14236.099617] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14236.107201] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14236.111396] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14236.114987] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [14236.118629] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [14236.122493] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [14236.126435] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [14236.129818] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [14236.134110] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14236.137219] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [14236.141226] Call Trace: [14236.142521] [14236.143758] ? show_regs.cold.9+0x22/0x2f [14236.145967] ? watchdog_timer_fn.cold.6+0x62/0x10a [14236.148511] ? __hrtimer_run_queues+0x1c2/0x4d0 [14236.150668] ? watchdog+0x40/0x40 [14236.152483] ? hrtimer_interrupt+0x128/0x2c0 [14236.154579] ? smp_apic_timer_interrupt+0xa6/0x270 [14236.157200] ? apic_timer_interrupt+0xf/0x20 [14236.159274] [14236.160255] ? native_safe_halt+0x2e/0x50 [14236.162750] kvm_wait+0xcd/0xe0 [14236.165294] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14236.168927] do_raw_spin_lock+0x177/0x190 [14236.170831] _raw_spin_lock+0x21/0x30 [14236.172511] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14236.175401] ? do_raw_spin_unlock+0x75/0x190 [14236.177449] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14236.180522] ? do_wait_intr+0xf0/0xf0 [14236.182291] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14236.184666] mdc_reint+0x2d/0x110 [mdc] [14236.186345] mdc_unlink+0x658/0x870 [mdc] [14236.187901] lmv_unlink+0x34d/0xbd0 [lmv] [14236.189441] ll_unlink+0x24d/0x7f0 [lustre] [14236.191262] vfs_unlink+0x202/0x330 [14236.192643] do_unlinkat+0x2fa/0x460 [14236.194312] __x64_sys_unlink+0x37/0x50 [14236.195801] do_syscall_64+0xc1/0x440 [14236.197681] entry_SYSCALL_64_after_hwframe+0x49/0xae [14236.199686] RIP: 0033:0x7f151bdee10b [14236.201340] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [14236.209709] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [14236.213264] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [14236.217365] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [14236.220495] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14236.223861] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14236.227185] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14241.253063] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12310s! [14241.257129] Showing busy workqueues and worker pools: [14241.259439] workqueue events: flags=0x0 [14241.261609] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14241.265110] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [14241.269988] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14241.276781] pending: kfree_rcu_monitor [14241.281813] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14241.285456] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [14241.290135] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14241.293359] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [14241.297556] workqueue events_long: flags=0x0 [14241.299582] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14241.303162] pending: lu_objects_flush [obdclass] [14241.305648] workqueue events_unbound: flags=0x2 [14241.307607] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14241.310246] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14241.314645] workqueue events_power_efficient: flags=0x80 [14241.316403] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14241.318611] pending: do_cache_clean, neigh_periodic_work, check_lifetime [14241.321324] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14241.324780] pending: neigh_periodic_work [14241.326656] workqueue mm_percpu_wq: flags=0x8 [14241.328473] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14241.331446] pending: vmstat_update, lru_add_drain_per_cpu [14241.334139] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14241.337055] pending: vmstat_update [14241.338582] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14241.341409] pending: lru_add_drain_per_cpu [14241.343414] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [14241.346013] pending: lru_add_drain_per_cpu BAR(48) [14241.348226] workqueue writeback: flags=0x4a [14241.349846] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [14241.352160] pending: wb_workfn [14241.353532] workqueue xprtiod: flags=0xa [14241.354666] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14241.357007] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [14241.360225] workqueue ipv6_addrconf: flags=0x40008 [14241.361869] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14241.365110] pending: addrconf_verify_work [14241.367120] workqueue knbd0-recv: flags=0x1a [14241.368411] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14241.371171] in-flight: 658:recv_work [14241.372715] workqueue ptlrpc_pinger: flags=0x2 [14241.374648] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14241.377149] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14241.380019] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12310s workers=4 idle: 32984 37 8 [14241.383282] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14226s workers=2 idle: 661 [14248.057041] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [14248.060607] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14248.080593] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14248.085676] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14248.090033] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14248.093544] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14248.102144] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14248.105615] RAX: 0000000000006c8d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14248.109310] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [14248.112548] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [14248.116110] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [14248.118757] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [14248.121676] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [14248.125529] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14248.128193] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [14248.131161] Call Trace: [14248.132548] [14248.133468] ? show_regs.cold.9+0x22/0x2f [14248.135224] ? watchdog_timer_fn.cold.6+0x62/0x10a [14248.137439] ? __hrtimer_run_queues+0x1c2/0x4d0 [14248.139290] ? watchdog+0x40/0x40 [14248.140861] ? hrtimer_interrupt+0x128/0x2c0 [14248.142872] ? smp_apic_timer_interrupt+0xa6/0x270 [14248.144796] ? apic_timer_interrupt+0xf/0x20 [14248.146979] [14248.147972] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14248.150372] do_raw_spin_lock+0x177/0x190 [14248.152302] _raw_spin_lock+0x21/0x30 [14248.153784] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14248.156361] ? do_raw_spin_unlock+0x75/0x190 [14248.158234] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14248.161163] ? do_wait_intr+0xf0/0xf0 [14248.162758] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14248.165486] mdc_close+0x53b/0xd70 [mdc] [14248.167584] lmv_close+0x18d/0x490 [lmv] [14248.169188] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14248.171951] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14248.174722] ll_md_real_close+0xa5/0x2a0 [lustre] [14248.177794] ll_md_close+0x274/0x800 [lustre] [14248.181091] ll_file_release+0x128/0xc50 [lustre] [14248.183368] __fput+0x10d/0x440 [14248.184787] ____fput+0x12/0x20 [14248.186068] task_work_run+0xc6/0x110 [14248.187749] exit_to_usermode_loop+0x1dd/0x1f0 [14248.189696] do_syscall_64+0x3d6/0x440 [14248.191559] entry_SYSCALL_64_after_hwframe+0x49/0xae [14248.193820] RIP: 0033:0x7f151bee3cd8 [14248.196614] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14248.204331] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14248.207649] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [14248.210658] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [14248.213725] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14248.216814] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14248.220043] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14252.055015] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [14252.060636] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [14252.061849] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14252.061939] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14252.061945] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14252.061948] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14252.061962] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14252.061966] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14252.061971] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [14252.061973] RDX: 0000000000001712 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [14252.061976] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [14252.061979] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [14252.061981] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14252.061985] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [14252.061988] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14252.061990] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14252.061999] Call Trace: [14252.062004] [14252.062010] ? show_regs.cold.9+0x22/0x2f [14252.062020] ? watchdog_timer_fn.cold.6+0x62/0x10a [14252.062027] ? __hrtimer_run_queues+0x1c2/0x4d0 [14252.062038] ? watchdog+0x40/0x40 [14252.062043] ? hrtimer_interrupt+0x128/0x2c0 [14252.062047] ? smp_apic_timer_interrupt+0xa6/0x270 [14252.062055] ? apic_timer_interrupt+0xf/0x20 [14252.062060] [14252.062063] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14252.062068] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14252.062073] do_raw_spin_lock+0x177/0x190 [14252.062078] _raw_spin_lock+0x21/0x30 [14252.062084] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14252.062488] obd_mod_free+0x46/0x140 [ptlrpc] [14252.062695] mdc_commit_open+0x91/0xb0 [mdc] [14252.062733] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14252.063066] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14252.067548] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14252.067631] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14252.067638] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14252.067644] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14252.068205] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14252.069015] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14252.069019] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14252.069025] RAX: 00000000000072d8 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14252.069027] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [14252.069030] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [14252.069032] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [14252.069034] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [14252.069037] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [14252.069041] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14252.069043] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14252.069051] Call Trace: [14252.069057] [14252.069095] ? show_regs.cold.9+0x22/0x2f [14252.069105] ? watchdog_timer_fn.cold.6+0x62/0x10a [14252.069112] ? __hrtimer_run_queues+0x1c2/0x4d0 [14252.069117] ? watchdog+0x40/0x40 [14252.069122] ? hrtimer_interrupt+0x128/0x2c0 [14252.069126] ? smp_apic_timer_interrupt+0xa6/0x270 [14252.069134] ? apic_timer_interrupt+0xf/0x20 [14252.069139] [14252.069178] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14252.100691] after_reply+0xe2c/0x1780 [ptlrpc] [14252.101037] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14252.101428] ? __next_timer_interrupt+0x160/0x160 [14252.101440] ptlrpcd+0xa82/0xe40 [ptlrpc] [14252.101704] ? do_wait_intr+0xf0/0xf0 [14252.101713] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14252.102007] kthread+0x1d1/0x200 [14252.102013] ? set_kthread_struct+0x70/0x70 [14252.102018] ret_from_fork+0x1f/0x30 [14252.306165] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14252.308219] do_raw_spin_lock+0x177/0x190 [14252.310145] _raw_spin_lock+0x21/0x30 [14252.312026] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14252.314517] process_one_work+0x2c8/0x700 [14252.316448] worker_thread+0x296/0x6e0 [14252.318124] ? rescuer_thread+0x570/0x570 [14252.320152] kthread+0x1d1/0x200 [14252.321157] ? set_kthread_struct+0x70/0x70 [14252.322660] ret_from_fork+0x1f/0x30 [14268.052076] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [14268.057577] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14268.088120] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14268.095982] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14268.101823] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14268.106395] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14268.118189] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14268.122574] RAX: 0000000000005d03 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14268.126550] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [14268.130962] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [14268.135213] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [14268.138994] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [14268.142949] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [14268.148191] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14268.152111] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [14268.157241] Call Trace: [14268.158763] [14268.159984] ? show_regs.cold.9+0x22/0x2f [14268.162195] ? watchdog_timer_fn.cold.6+0x62/0x10a [14268.165338] ? __hrtimer_run_queues+0x1c2/0x4d0 [14268.168464] ? watchdog+0x40/0x40 [14268.169884] ? hrtimer_interrupt+0x128/0x2c0 [14268.172273] ? smp_apic_timer_interrupt+0xa6/0x270 [14268.175212] ? apic_timer_interrupt+0xf/0x20 [14268.177867] [14268.178781] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14268.181659] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14268.184534] do_raw_spin_lock+0x177/0x190 [14268.186514] _raw_spin_lock+0x21/0x30 [14268.188265] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14268.191132] ? do_raw_spin_unlock+0x75/0x190 [14268.193379] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14268.196123] ? do_wait_intr+0xf0/0xf0 [14268.198005] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14268.201349] mdc_reint+0x2d/0x110 [mdc] [14268.203734] mdc_unlink+0x658/0x870 [mdc] [14268.205850] lmv_unlink+0x34d/0xbd0 [lmv] [14268.207951] ll_unlink+0x24d/0x7f0 [lustre] [14268.210938] vfs_unlink+0x202/0x330 [14268.212381] do_unlinkat+0x2fa/0x460 [14268.213992] __x64_sys_unlink+0x37/0x50 [14268.216395] do_syscall_64+0xc1/0x440 [14268.218450] entry_SYSCALL_64_after_hwframe+0x49/0xae [14268.220833] RIP: 0033:0x7f151bdee10b [14268.222823] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [14268.231980] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [14268.235820] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [14268.239121] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [14268.242059] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14268.245203] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14268.249174] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14271.462802] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12340s! [14271.468302] Showing busy workqueues and worker pools: [14271.470173] workqueue events: flags=0x0 [14271.471559] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14271.473890] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [14271.476805] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14271.478699] pending: kfree_rcu_monitor [14271.480284] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14271.482239] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [14271.485480] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14271.487999] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [14271.490787] workqueue events_long: flags=0x0 [14271.492353] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14271.494941] pending: lu_objects_flush [obdclass] [14271.496665] workqueue events_unbound: flags=0x2 [14271.498217] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14271.500279] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14271.504528] workqueue events_power_efficient: flags=0x80 [14271.506879] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14271.509107] pending: do_cache_clean, neigh_periodic_work, check_lifetime [14271.511408] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14271.514026] pending: neigh_periodic_work [14271.515831] workqueue mm_percpu_wq: flags=0x8 [14271.517300] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14271.520072] pending: vmstat_update, lru_add_drain_per_cpu [14271.521993] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14271.523927] pending: vmstat_update [14271.525273] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14271.528329] pending: lru_add_drain_per_cpu [14271.530228] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [14271.534459] pending: lru_add_drain_per_cpu BAR(48) [14271.536322] workqueue writeback: flags=0x4a [14271.538028] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [14271.539940] pending: wb_workfn [14271.541082] workqueue xprtiod: flags=0xa [14271.542238] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14271.544263] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [14271.546532] workqueue ipv6_addrconf: flags=0x40008 [14271.548154] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14271.550037] pending: addrconf_verify_work [14271.551498] workqueue knbd0-recv: flags=0x1a [14271.552977] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14271.555014] in-flight: 658:recv_work [14271.556189] workqueue ptlrpc_pinger: flags=0x2 [14271.557803] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14271.560146] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14271.562535] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12340s workers=4 idle: 32984 37 8 [14271.565528] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14256s workers=2 idle: 661 [14276.057064] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [14276.060829] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14276.118255] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14276.132288] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14276.140777] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14276.142852] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14276.168102] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14276.177293] RAX: 0000000000000ff8 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14276.188887] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [14276.198233] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [14276.204176] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [14276.214968] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [14276.221410] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [14276.234121] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14276.236681] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [14276.243793] Call Trace: [14276.244748] [14276.245483] ? show_regs.cold.9+0x22/0x2f [14276.249041] ? watchdog_timer_fn.cold.6+0x62/0x10a [14276.253948] ? __hrtimer_run_queues+0x1c2/0x4d0 [14276.260188] ? watchdog+0x40/0x40 [14276.265201] ? hrtimer_interrupt+0x128/0x2c0 [14276.272707] ? smp_apic_timer_interrupt+0xa6/0x270 [14276.280024] ? apic_timer_interrupt+0xf/0x20 [14276.283679] [14276.286755] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14276.293313] do_raw_spin_lock+0x177/0x190 [14276.299597] _raw_spin_lock+0x21/0x30 [14276.301233] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14276.305724] ? do_raw_spin_unlock+0x75/0x190 [14276.307826] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14276.310230] ? do_wait_intr+0xf0/0xf0 [14276.312196] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14276.315094] mdc_close+0x53b/0xd70 [mdc] [14276.316579] lmv_close+0x18d/0x490 [lmv] [14276.318187] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14276.320625] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14276.323130] ll_md_real_close+0xa5/0x2a0 [lustre] [14276.325127] ll_md_close+0x274/0x800 [lustre] [14276.327063] ll_file_release+0x128/0xc50 [lustre] [14276.329598] __fput+0x10d/0x440 [14276.331009] ____fput+0x12/0x20 [14276.332279] task_work_run+0xc6/0x110 [14276.333998] exit_to_usermode_loop+0x1dd/0x1f0 [14276.335871] do_syscall_64+0x3d6/0x440 [14276.337731] entry_SYSCALL_64_after_hwframe+0x49/0xae [14276.340009] RIP: 0033:0x7f151bee3cd8 [14276.341682] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14276.348970] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14276.352230] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [14276.355723] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [14276.359085] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14276.364906] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14276.370740] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14280.055188] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [14280.056008] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [14280.058912] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14280.059311] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14280.059318] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14280.059321] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14280.059334] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14280.059338] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14280.059343] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [14280.059345] RDX: 0000000000000d48 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [14280.059348] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [14280.059350] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [14280.059353] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14280.059356] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [14280.059359] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14280.059362] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14280.059830] Call Trace: [14280.065801] Modules linked in: lustre(O) osp(O) [14280.092428] [14280.092442] ? show_regs.cold.9+0x22/0x2f [14280.092458] ? watchdog_timer_fn.cold.6+0x62/0x10a [14280.092464] ? __hrtimer_run_queues+0x1c2/0x4d0 [14280.092469] ? watchdog+0x40/0x40 [14280.092473] ? hrtimer_interrupt+0x128/0x2c0 [14280.092478] ? smp_apic_timer_interrupt+0xa6/0x270 [14280.092487] ? apic_timer_interrupt+0xf/0x20 [14280.092492] [14280.092495] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14280.092503] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14280.092508] do_raw_spin_lock+0x177/0x190 [14280.092512] _raw_spin_lock+0x21/0x30 [14280.102671] ofd(O) [14280.109174] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14280.109597] obd_mod_free+0x46/0x140 [ptlrpc] [14280.109854] mdc_commit_open+0x91/0xb0 [mdc] [14280.109892] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14280.113111] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14280.113197] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14280.113204] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14280.113211] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14280.122809] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14280.123286] after_reply+0xe2c/0x1780 [ptlrpc] [14280.129261] [14280.133412] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14280.133883] ? __next_timer_interrupt+0x160/0x160 [14280.133904] ptlrpcd+0xa82/0xe40 [ptlrpc] [14280.134197] ? do_wait_intr+0xf0/0xf0 [14280.134206] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14280.134377] kthread+0x1d1/0x200 [14280.134385] ? set_kthread_struct+0x70/0x70 [14280.134388] ret_from_fork+0x1f/0x30 [14280.266304] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14280.271149] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14280.282093] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14280.288917] RAX: 0000000000002086 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14280.293523] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [14280.296804] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [14280.301308] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [14280.304397] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [14280.306676] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [14280.317234] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14280.320257] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14280.323676] Call Trace: [14280.324749] [14280.325605] ? show_regs.cold.9+0x22/0x2f [14280.327308] ? watchdog_timer_fn.cold.6+0x62/0x10a [14280.329891] ? __hrtimer_run_queues+0x1c2/0x4d0 [14280.332118] ? watchdog+0x40/0x40 [14280.333761] ? hrtimer_interrupt+0x128/0x2c0 [14280.335737] ? smp_apic_timer_interrupt+0xa6/0x270 [14280.338190] ? apic_timer_interrupt+0xf/0x20 [14280.340468] [14280.341480] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14280.344239] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14280.346983] do_raw_spin_lock+0x177/0x190 [14280.349720] _raw_spin_lock+0x21/0x30 [14280.351256] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14280.355104] process_one_work+0x2c8/0x700 [14280.357293] worker_thread+0x296/0x6e0 [14280.359181] ? rescuer_thread+0x570/0x570 [14280.360987] kthread+0x1d1/0x200 [14280.362469] ? set_kthread_struct+0x70/0x70 [14280.364643] ret_from_fork+0x1f/0x30 [14296.052101] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [14296.055637] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14296.075554] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14296.079463] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14296.082595] RIP: 0010:native_safe_halt+0x2e/0x50 [14296.084019] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14296.091457] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14296.094671] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14296.098887] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [14296.102007] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [14296.104919] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [14296.108030] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [14296.110306] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [14296.113840] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14296.116732] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [14296.119815] Call Trace: [14296.120770] [14296.121532] ? show_regs.cold.9+0x22/0x2f [14296.124199] ? watchdog_timer_fn.cold.6+0x62/0x10a [14296.125937] ? __hrtimer_run_queues+0x1c2/0x4d0 [14296.127827] ? watchdog+0x40/0x40 [14296.128918] ? hrtimer_interrupt+0x128/0x2c0 [14296.130651] ? smp_apic_timer_interrupt+0xa6/0x270 [14296.132287] ? apic_timer_interrupt+0xf/0x20 [14296.133797] [14296.135350] ? native_safe_halt+0x2e/0x50 [14296.139749] kvm_wait+0xcd/0xe0 [14296.141867] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14296.145632] do_raw_spin_lock+0x177/0x190 [14296.148023] _raw_spin_lock+0x21/0x30 [14296.150899] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14296.153709] ? do_raw_spin_unlock+0x75/0x190 [14296.155821] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14296.157871] ? do_wait_intr+0xf0/0xf0 [14296.159020] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14296.161014] mdc_reint+0x2d/0x110 [mdc] [14296.162549] mdc_unlink+0x658/0x870 [mdc] [14296.163963] lmv_unlink+0x34d/0xbd0 [lmv] [14296.165650] ll_unlink+0x24d/0x7f0 [lustre] [14296.167288] vfs_unlink+0x202/0x330 [14296.168699] do_unlinkat+0x2fa/0x460 [14296.171636] __x64_sys_unlink+0x37/0x50 [14296.173887] do_syscall_64+0xc1/0x440 [14296.175951] entry_SYSCALL_64_after_hwframe+0x49/0xae [14296.179227] RIP: 0033:0x7f151bdee10b [14296.181509] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [14296.190991] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [14296.193519] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [14296.195860] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [14296.198275] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14296.200752] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14296.203143] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14301.665058] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12371s! [14301.669585] Showing busy workqueues and worker pools: [14301.672927] workqueue events: flags=0x0 [14301.675066] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14301.679014] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [14301.684433] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14301.688113] pending: kfree_rcu_monitor [14301.690653] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14301.694699] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [14301.699983] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14301.703291] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [14301.707259] workqueue events_long: flags=0x0 [14301.709964] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14301.713726] pending: lu_objects_flush [obdclass] [14301.717228] workqueue events_unbound: flags=0x2 [14301.719913] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14301.723242] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14301.727070] workqueue events_power_efficient: flags=0x80 [14301.729640] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14301.734144] pending: do_cache_clean, neigh_periodic_work, check_lifetime [14301.736272] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14301.741205] pending: neigh_periodic_work [14301.746225] workqueue mm_percpu_wq: flags=0x8 [14301.750570] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14301.755629] pending: vmstat_update, lru_add_drain_per_cpu [14301.759056] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14301.764060] pending: vmstat_update [14301.765919] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14301.769462] pending: lru_add_drain_per_cpu [14301.771767] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [14301.775625] pending: lru_add_drain_per_cpu BAR(48) [14301.777883] workqueue writeback: flags=0x4a [14301.779874] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [14301.791481] pending: wb_workfn [14301.797641] workqueue xprtiod: flags=0xa [14301.799913] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14301.804959] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [14301.807970] workqueue ipv6_addrconf: flags=0x40008 [14301.816667] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14301.823697] pending: addrconf_verify_work [14301.825276] workqueue knbd0-recv: flags=0x1a [14301.827558] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14301.830157] in-flight: 658:recv_work [14301.835110] workqueue ptlrpc_pinger: flags=0x2 [14301.838790] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14301.842937] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14301.846202] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12371s workers=4 idle: 32984 37 8 [14301.855188] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14286s workers=2 idle: 661 [14304.062461] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [14304.067517] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14304.124098] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14304.134067] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14304.146703] RIP: 0010:native_safe_halt+0x2e/0x50 [14304.149922] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14304.170170] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14304.182519] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14304.189754] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [14304.200365] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [14304.202605] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [14304.208018] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [14304.213845] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [14304.221263] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14304.223960] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [14304.227135] Call Trace: [14304.228682] [14304.229595] ? show_regs.cold.9+0x22/0x2f [14304.231362] ? watchdog_timer_fn.cold.6+0x62/0x10a [14304.234732] ? __hrtimer_run_queues+0x1c2/0x4d0 [14304.238600] ? watchdog+0x40/0x40 [14304.241762] ? hrtimer_interrupt+0x128/0x2c0 [14304.246755] ? smp_apic_timer_interrupt+0xa6/0x270 [14304.251207] ? apic_timer_interrupt+0xf/0x20 [14304.252708] [14304.253740] ? native_safe_halt+0x2e/0x50 [14304.255651] kvm_wait+0xcd/0xe0 [14304.256858] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14304.262635] do_raw_spin_lock+0x177/0x190 [14304.264703] _raw_spin_lock+0x21/0x30 [14304.266176] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14304.268745] ? do_raw_spin_unlock+0x75/0x190 [14304.270887] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14304.273498] ? do_wait_intr+0xf0/0xf0 [14304.274564] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14304.280807] mdc_close+0x53b/0xd70 [mdc] [14304.284189] lmv_close+0x18d/0x490 [lmv] [14304.288884] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14304.291282] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14304.294830] ll_md_real_close+0xa5/0x2a0 [lustre] [14304.297227] ll_md_close+0x274/0x800 [lustre] [14304.301826] ll_file_release+0x128/0xc50 [lustre] [14304.304071] __fput+0x10d/0x440 [14304.305241] ____fput+0x12/0x20 [14304.306244] task_work_run+0xc6/0x110 [14304.307528] exit_to_usermode_loop+0x1dd/0x1f0 [14304.309832] do_syscall_64+0x3d6/0x440 [14304.311588] entry_SYSCALL_64_after_hwframe+0x49/0xae [14304.314179] RIP: 0033:0x7f151bee3cd8 [14304.315815] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14304.332255] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14304.344315] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [14304.356073] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [14304.365060] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14304.372368] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14304.377376] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14308.055054] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [14308.056274] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14308.058472] Modules linked in: [14308.062700] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14308.062781] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14308.062790] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14308.062796] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14308.066034] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul [14308.099345] [14308.099354] RIP: 0010:native_safe_halt+0x2e/0x50 [14308.099370] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14308.099375] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14308.099391] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14308.099394] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [14308.099396] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [14308.099399] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [14308.099402] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [14308.099406] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [14308.099409] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14308.099411] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14308.099419] Call Trace: [14308.099425] [14308.099430] ? show_regs.cold.9+0x22/0x2f [14308.099440] ? watchdog_timer_fn.cold.6+0x62/0x10a [14308.099446] ? __hrtimer_run_queues+0x1c2/0x4d0 [14308.099451] ? watchdog+0x40/0x40 [14308.099456] ? hrtimer_interrupt+0x128/0x2c0 [14308.099460] ? smp_apic_timer_interrupt+0xa6/0x270 [14308.099467] ? apic_timer_interrupt+0xf/0x20 [14308.099472] [14308.099475] ? native_safe_halt+0x2e/0x50 [14308.099480] kvm_wait+0xcd/0xe0 [14308.099487] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14308.099498] do_raw_spin_lock+0x177/0x190 [14308.099503] _raw_spin_lock+0x21/0x30 [14308.099508] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14308.099941] process_one_work+0x2c8/0x700 [14308.099949] worker_thread+0x296/0x6e0 [14308.099954] ? rescuer_thread+0x570/0x570 [14308.099958] kthread+0x1d1/0x200 [14308.099963] ? set_kthread_struct+0x70/0x70 [14308.099967] ret_from_fork+0x1f/0x30 [14308.210179] crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14308.215943] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14308.221716] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14308.225126] RIP: 0010:native_safe_halt+0x2e/0x50 [14308.227713] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14308.236202] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14308.240017] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [14308.243138] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [14308.246751] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [14308.249889] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [14308.252154] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14308.256069] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [14308.259936] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14308.262757] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14308.266017] Call Trace: [14308.267156] [14308.268171] ? show_regs.cold.9+0x22/0x2f [14308.270182] ? watchdog_timer_fn.cold.6+0x62/0x10a [14308.272693] ? __hrtimer_run_queues+0x1c2/0x4d0 [14308.274799] ? watchdog+0x40/0x40 [14308.276261] ? hrtimer_interrupt+0x128/0x2c0 [14308.278657] ? smp_apic_timer_interrupt+0xa6/0x270 [14308.280848] ? apic_timer_interrupt+0xf/0x20 [14308.282891] [14308.283929] ? native_safe_halt+0x2e/0x50 [14308.285915] kvm_wait+0xcd/0xe0 [14308.287748] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14308.290095] do_raw_spin_lock+0x177/0x190 [14308.291990] _raw_spin_lock+0x21/0x30 [14308.293733] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14308.296064] obd_mod_free+0x46/0x140 [ptlrpc] [14308.298141] mdc_commit_open+0x91/0xb0 [mdc] [14308.300166] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14308.302881] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14308.305942] after_reply+0xe2c/0x1780 [ptlrpc] [14308.308451] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14308.311045] ? __next_timer_interrupt+0x160/0x160 [14308.313135] ptlrpcd+0xa82/0xe40 [ptlrpc] [14308.315624] ? do_wait_intr+0xf0/0xf0 [14308.317061] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14308.319846] kthread+0x1d1/0x200 [14308.321177] ? set_kthread_struct+0x70/0x70 [14308.323106] ret_from_fork+0x1f/0x30 [14328.053115] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [14328.056551] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14328.087626] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14328.092378] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14328.097213] RIP: 0010:native_safe_halt+0x2e/0x50 [14328.098948] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14328.108226] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14328.110999] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14328.114060] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [14328.118349] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [14328.123267] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [14328.127543] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [14328.131270] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [14328.135153] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14328.137319] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [14328.141477] Call Trace: [14328.142416] [14328.143429] ? show_regs.cold.9+0x22/0x2f [14328.145331] ? watchdog_timer_fn.cold.6+0x62/0x10a [14328.147467] ? __hrtimer_run_queues+0x1c2/0x4d0 [14328.149732] ? watchdog+0x40/0x40 [14328.151329] ? hrtimer_interrupt+0x128/0x2c0 [14328.153950] ? smp_apic_timer_interrupt+0xa6/0x270 [14328.155974] ? apic_timer_interrupt+0xf/0x20 [14328.158327] [14328.159347] ? native_safe_halt+0x2e/0x50 [14328.162458] kvm_wait+0xcd/0xe0 [14328.164526] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14328.166849] do_raw_spin_lock+0x177/0x190 [14328.169374] _raw_spin_lock+0x21/0x30 [14328.171271] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14328.174502] ? do_raw_spin_unlock+0x75/0x190 [14328.176705] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14328.179555] ? do_wait_intr+0xf0/0xf0 [14328.181256] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14328.184694] mdc_reint+0x2d/0x110 [mdc] [14328.186714] mdc_unlink+0x658/0x870 [mdc] [14328.189696] lmv_unlink+0x34d/0xbd0 [lmv] [14328.193541] ll_unlink+0x24d/0x7f0 [lustre] [14328.195501] vfs_unlink+0x202/0x330 [14328.197130] do_unlinkat+0x2fa/0x460 [14328.200959] __x64_sys_unlink+0x37/0x50 [14328.204849] do_syscall_64+0xc1/0x440 [14328.207387] entry_SYSCALL_64_after_hwframe+0x49/0xae [14328.209174] RIP: 0033:0x7f151bdee10b [14328.212533] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [14328.222959] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [14328.230265] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [14328.234031] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [14328.240018] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14328.243470] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14328.246897] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14331.873068] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12401s! [14331.878413] Showing busy workqueues and worker pools: [14331.880912] workqueue events: flags=0x0 [14331.883252] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14331.885695] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [14331.890982] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14331.893863] pending: kfree_rcu_monitor [14331.898248] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14331.900597] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [14331.905876] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14331.909267] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [14331.912468] workqueue events_long: flags=0x0 [14331.914273] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14331.917264] pending: lu_objects_flush [obdclass] [14331.919805] workqueue events_unbound: flags=0x2 [14331.921612] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14331.924391] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14331.928702] workqueue events_power_efficient: flags=0x80 [14331.930793] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14331.933493] pending: do_cache_clean, neigh_periodic_work, check_lifetime [14331.936344] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14331.938965] pending: neigh_periodic_work [14331.940661] workqueue mm_percpu_wq: flags=0x8 [14331.942517] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14331.945363] pending: vmstat_update, lru_add_drain_per_cpu [14331.947700] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14331.950286] pending: vmstat_update [14331.951782] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14331.954602] pending: lru_add_drain_per_cpu [14331.956503] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [14331.959345] pending: lru_add_drain_per_cpu BAR(48) [14331.961511] workqueue writeback: flags=0x4a [14331.966461] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [14331.968121] pending: wb_workfn [14331.971310] workqueue xprtiod: flags=0xa [14331.973159] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14331.975943] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [14331.980532] workqueue ipv6_addrconf: flags=0x40008 [14331.983457] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14331.986839] pending: addrconf_verify_work [14331.989667] workqueue knbd0-recv: flags=0x1a [14331.991525] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14331.994209] in-flight: 658:recv_work [14331.995846] workqueue ptlrpc_pinger: flags=0x2 [14331.997727] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14332.000427] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14332.002836] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12401s workers=4 idle: 32984 37 8 [14332.006320] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14316s workers=2 idle: 661 [14332.057185] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [14332.061681] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14332.079901] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14332.083580] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14332.087377] RIP: 0010:native_safe_halt+0x2e/0x50 [14332.089298] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14332.096309] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14332.099550] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14332.102625] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [14332.106040] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [14332.113709] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [14332.118967] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [14332.125128] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [14332.136990] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14332.140876] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [14332.145807] Call Trace: [14332.147536] [14332.148686] ? show_regs.cold.9+0x22/0x2f [14332.150833] ? watchdog_timer_fn.cold.6+0x62/0x10a [14332.153339] ? __hrtimer_run_queues+0x1c2/0x4d0 [14332.155429] ? watchdog+0x40/0x40 [14332.156889] ? hrtimer_interrupt+0x128/0x2c0 [14332.158206] ? smp_apic_timer_interrupt+0xa6/0x270 [14332.164185] ? apic_timer_interrupt+0xf/0x20 [14332.166563] [14332.169953] ? native_safe_halt+0x2e/0x50 [14332.172274] kvm_wait+0xcd/0xe0 [14332.177542] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14332.180866] do_raw_spin_lock+0x177/0x190 [14332.185340] _raw_spin_lock+0x21/0x30 [14332.186526] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14332.188888] ? do_raw_spin_unlock+0x75/0x190 [14332.190873] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14332.193417] ? do_wait_intr+0xf0/0xf0 [14332.194770] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14332.197673] mdc_close+0x53b/0xd70 [mdc] [14332.199769] lmv_close+0x18d/0x490 [lmv] [14332.202816] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14332.205988] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14332.209988] ll_md_real_close+0xa5/0x2a0 [lustre] [14332.212435] ll_md_close+0x274/0x800 [lustre] [14332.214469] ll_file_release+0x128/0xc50 [lustre] [14332.215763] __fput+0x10d/0x440 [14332.216811] ____fput+0x12/0x20 [14332.217771] task_work_run+0xc6/0x110 [14332.218724] exit_to_usermode_loop+0x1dd/0x1f0 [14332.220355] do_syscall_64+0x3d6/0x440 [14332.221639] entry_SYSCALL_64_after_hwframe+0x49/0xae [14332.223321] RIP: 0033:0x7f151bee3cd8 [14332.224576] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14332.231728] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14332.233768] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [14332.237174] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [14332.240204] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14332.243624] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14332.246760] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14336.056023] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14336.058255] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [14336.060276] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14336.060346] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14336.060353] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14336.060359] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14336.060790] RIP: 0010:native_safe_halt+0x2e/0x50 [14336.060801] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14336.060805] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14336.060809] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14336.060812] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [14336.060814] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [14336.060817] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [14336.060819] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [14336.060823] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [14336.060826] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14336.060828] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14336.060836] Call Trace: [14336.060841] [14336.060890] ? show_regs.cold.9+0x22/0x2f [14336.060957] ? watchdog_timer_fn.cold.6+0x62/0x10a [14336.060963] ? __hrtimer_run_queues+0x1c2/0x4d0 [14336.060967] ? watchdog+0x40/0x40 [14336.060972] ? hrtimer_interrupt+0x128/0x2c0 [14336.061038] ? smp_apic_timer_interrupt+0xa6/0x270 [14336.061045] ? apic_timer_interrupt+0xf/0x20 [14336.061049] [14336.061052] ? native_safe_halt+0x2e/0x50 [14336.061055] kvm_wait+0xcd/0xe0 [14336.061063] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14336.061071] do_raw_spin_lock+0x177/0x190 [14336.061075] _raw_spin_lock+0x21/0x30 [14336.061079] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14336.067999] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14336.068073] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14336.068081] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14336.068084] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14336.068099] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14336.068103] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14336.068118] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [14336.068121] RDX: 0000000000000190 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [14336.068123] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [14336.068126] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [14336.068128] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14336.068131] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [14336.068134] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14336.068137] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14336.068144] Call Trace: [14336.068150] [14336.068155] ? show_regs.cold.9+0x22/0x2f [14336.068164] ? watchdog_timer_fn.cold.6+0x62/0x10a [14336.068171] ? __hrtimer_run_queues+0x1c2/0x4d0 [14336.068174] ? watchdog+0x40/0x40 [14336.068178] ? hrtimer_interrupt+0x128/0x2c0 [14336.068182] ? smp_apic_timer_interrupt+0xa6/0x270 [14336.068200] ? apic_timer_interrupt+0xf/0x20 [14336.068204] [14336.068207] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14336.068211] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14336.068215] do_raw_spin_lock+0x177/0x190 [14336.068219] _raw_spin_lock+0x21/0x30 [14336.068224] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14336.068475] obd_mod_free+0x46/0x140 [ptlrpc] [14336.104258] process_one_work+0x2c8/0x700 [14336.104279] worker_thread+0x296/0x6e0 [14336.104284] ? rescuer_thread+0x570/0x570 [14336.104288] kthread+0x1d1/0x200 [14336.104292] ? set_kthread_struct+0x70/0x70 [14336.104296] ret_from_fork+0x1f/0x30 [14336.275187] mdc_commit_open+0x91/0xb0 [mdc] [14336.276539] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14336.278307] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14336.280001] after_reply+0xe2c/0x1780 [ptlrpc] [14336.281578] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14336.283338] ? __next_timer_interrupt+0x160/0x160 [14336.284784] ptlrpcd+0xa82/0xe40 [ptlrpc] [14336.286335] ? do_wait_intr+0xf0/0xf0 [14336.287550] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14336.289915] kthread+0x1d1/0x200 [14336.290896] ? set_kthread_struct+0x70/0x70 [14336.292195] ret_from_fork+0x1f/0x30 [14356.052024] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [14356.054480] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14356.094656] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14356.105466] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14356.112516] RIP: 0010:native_safe_halt+0x2e/0x50 [14356.114462] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14356.123924] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14356.127244] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14356.130267] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [14356.133775] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [14356.137031] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [14356.140598] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [14356.143639] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [14356.147552] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14356.150946] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [14356.153930] Call Trace: [14356.154685] [14356.155528] ? show_regs.cold.9+0x22/0x2f [14356.157723] ? watchdog_timer_fn.cold.6+0x62/0x10a [14356.159978] ? __hrtimer_run_queues+0x1c2/0x4d0 [14356.161703] ? watchdog+0x40/0x40 [14356.163280] ? hrtimer_interrupt+0x128/0x2c0 [14356.165436] ? smp_apic_timer_interrupt+0xa6/0x270 [14356.167487] ? apic_timer_interrupt+0xf/0x20 [14356.169355] [14356.170260] ? native_safe_halt+0x2e/0x50 [14356.172233] kvm_wait+0xcd/0xe0 [14356.173463] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14356.175344] do_raw_spin_lock+0x177/0x190 [14356.176963] _raw_spin_lock+0x21/0x30 [14356.178403] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14356.181180] ? do_raw_spin_unlock+0x75/0x190 [14356.183438] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14356.185648] ? do_wait_intr+0xf0/0xf0 [14356.187511] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14356.189886] mdc_reint+0x2d/0x110 [mdc] [14356.191409] mdc_unlink+0x658/0x870 [mdc] [14356.193221] lmv_unlink+0x34d/0xbd0 [lmv] [14356.195069] ll_unlink+0x24d/0x7f0 [lustre] [14356.197118] vfs_unlink+0x202/0x330 [14356.198478] do_unlinkat+0x2fa/0x460 [14356.200285] __x64_sys_unlink+0x37/0x50 [14356.202238] do_syscall_64+0xc1/0x440 [14356.203694] entry_SYSCALL_64_after_hwframe+0x49/0xae [14356.205976] RIP: 0033:0x7f151bdee10b [14356.207856] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [14356.215750] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [14356.219702] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [14356.222727] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [14356.225748] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14356.228345] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14356.230324] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14360.057027] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [14360.061002] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14360.100400] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14360.107996] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14360.117327] RIP: 0010:native_safe_halt+0x2e/0x50 [14360.120516] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14360.134396] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14360.138083] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14360.143886] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [14360.152188] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [14360.157589] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [14360.161267] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [14360.164690] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [14360.170433] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14360.174122] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [14360.179019] Call Trace: [14360.180483] [14360.182198] ? show_regs.cold.9+0x22/0x2f [14360.184531] ? watchdog_timer_fn.cold.6+0x62/0x10a [14360.186700] ? __hrtimer_run_queues+0x1c2/0x4d0 [14360.188410] ? watchdog+0x40/0x40 [14360.189640] ? hrtimer_interrupt+0x128/0x2c0 [14360.192146] ? smp_apic_timer_interrupt+0xa6/0x270 [14360.194268] ? apic_timer_interrupt+0xf/0x20 [14360.196919] [14360.197476] ? native_safe_halt+0x2e/0x50 [14360.199557] kvm_wait+0xcd/0xe0 [14360.201510] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14360.206354] do_raw_spin_lock+0x177/0x190 [14360.209797] _raw_spin_lock+0x21/0x30 [14360.211596] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14360.215230] ? do_raw_spin_unlock+0x75/0x190 [14360.217456] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14360.222667] ? do_wait_intr+0xf0/0xf0 [14360.224404] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14360.227308] mdc_close+0x53b/0xd70 [mdc] [14360.229175] lmv_close+0x18d/0x490 [lmv] [14360.235004] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14360.237653] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14360.249658] ll_md_real_close+0xa5/0x2a0 [lustre] [14360.252013] ll_md_close+0x274/0x800 [lustre] [14360.254186] ll_file_release+0x128/0xc50 [lustre] [14360.256222] __fput+0x10d/0x440 [14360.257424] ____fput+0x12/0x20 [14360.258614] task_work_run+0xc6/0x110 [14360.260181] exit_to_usermode_loop+0x1dd/0x1f0 [14360.262091] do_syscall_64+0x3d6/0x440 [14360.263516] entry_SYSCALL_64_after_hwframe+0x49/0xae [14360.265841] RIP: 0033:0x7f151bee3cd8 [14360.267451] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14360.275225] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14360.277603] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [14360.280455] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [14360.282526] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14360.284407] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14360.287354] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14362.081848] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12431s! [14362.086350] Showing busy workqueues and worker pools: [14362.090297] workqueue events: flags=0x0 [14362.093482] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14362.095397] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [14362.101651] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14362.106916] pending: kfree_rcu_monitor [14362.110991] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14362.115563] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [14362.127336] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14362.131569] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [14362.139838] workqueue events_long: flags=0x0 [14362.146713] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14362.150458] pending: lu_objects_flush [obdclass] [14362.153802] workqueue events_unbound: flags=0x2 [14362.155293] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14362.159227] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14362.170387] workqueue events_power_efficient: flags=0x80 [14362.172711] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14362.182510] pending: do_cache_clean, neigh_periodic_work, check_lifetime [14362.186652] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14362.190449] pending: neigh_periodic_work [14362.192408] workqueue mm_percpu_wq: flags=0x8 [14362.194297] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14362.198285] pending: vmstat_update, lru_add_drain_per_cpu [14362.200544] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14362.206121] pending: vmstat_update [14362.207266] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14362.212814] pending: lru_add_drain_per_cpu [14362.218534] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [14362.225557] pending: lru_add_drain_per_cpu BAR(48) [14362.229183] workqueue writeback: flags=0x4a [14362.230584] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [14362.233664] pending: wb_workfn [14362.235437] workqueue xprtiod: flags=0xa [14362.236831] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14362.239121] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [14362.243677] workqueue ipv6_addrconf: flags=0x40008 [14362.246769] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14362.249973] pending: addrconf_verify_work [14362.252817] workqueue knbd0-recv: flags=0x1a [14362.254241] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14362.258919] in-flight: 658:recv_work [14362.260309] workqueue ptlrpc_pinger: flags=0x2 [14362.262126] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14362.264457] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14362.267429] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12431s workers=4 idle: 32984 37 8 [14362.269883] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14347s workers=2 idle: 661 [14364.056160] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [14364.057900] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14364.059116] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14364.059190] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14364.059196] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14364.059199] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14364.059212] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14364.059217] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14364.059221] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [14364.059224] RDX: 0000000000002c11 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [14364.059226] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [14364.059229] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [14364.059231] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14364.059234] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [14364.059236] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14364.059239] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14364.059247] Call Trace: [14364.059253] [14364.059259] ? show_regs.cold.9+0x22/0x2f [14364.059268] ? watchdog_timer_fn.cold.6+0x62/0x10a [14364.059275] ? __hrtimer_run_queues+0x1c2/0x4d0 [14364.059279] ? watchdog+0x40/0x40 [14364.059283] ? hrtimer_interrupt+0x128/0x2c0 [14364.059287] ? smp_apic_timer_interrupt+0xa6/0x270 [14364.059298] ? apic_timer_interrupt+0xf/0x20 [14364.059303] [14364.059306] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14364.059316] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14364.059321] do_raw_spin_lock+0x177/0x190 [14364.059326] _raw_spin_lock+0x21/0x30 [14364.059332] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14364.059584] obd_mod_free+0x46/0x140 [ptlrpc] [14364.059847] mdc_commit_open+0x91/0xb0 [mdc] [14364.059942] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14364.060247] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14364.060415] after_reply+0xe2c/0x1780 [ptlrpc] [14364.060545] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14364.060686] ? __next_timer_interrupt+0x160/0x160 [14364.060693] ptlrpcd+0xa82/0xe40 [ptlrpc] [14364.061138] ? do_wait_intr+0xf0/0xf0 [14364.061147] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14364.066297] Modules linked in: [14364.097657] kthread+0x1d1/0x200 [14364.097677] ? set_kthread_struct+0x70/0x70 [14364.097682] ret_from_fork+0x1f/0x30 [14364.214931] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14364.231089] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14364.235469] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14364.238936] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14364.241389] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14364.243463] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14364.250310] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14364.252993] RAX: 0000000000005ec1 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14364.255478] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [14364.258043] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [14364.260374] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [14364.262450] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [14364.264826] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [14364.267051] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14364.268764] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14364.270655] Call Trace: [14364.271359] [14364.272138] ? show_regs.cold.9+0x22/0x2f [14364.273326] ? watchdog_timer_fn.cold.6+0x62/0x10a [14364.274669] ? __hrtimer_run_queues+0x1c2/0x4d0 [14364.276009] ? watchdog+0x40/0x40 [14364.276933] ? hrtimer_interrupt+0x128/0x2c0 [14364.278130] ? smp_apic_timer_interrupt+0xa6/0x270 [14364.279408] ? apic_timer_interrupt+0xf/0x20 [14364.280848] [14364.281602] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14364.283462] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14364.285334] do_raw_spin_lock+0x177/0x190 [14364.286500] _raw_spin_lock+0x21/0x30 [14364.287571] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14364.289422] process_one_work+0x2c8/0x700 [14364.290829] worker_thread+0x296/0x6e0 [14364.292161] ? rescuer_thread+0x570/0x570 [14364.293289] kthread+0x1d1/0x200 [14364.294360] ? set_kthread_struct+0x70/0x70 [14364.295648] ret_from_fork+0x1f/0x30 [14388.052019] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [14388.056120] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14388.057636] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [14388.093749] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14388.093771] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14388.093773] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14388.093787] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14388.093791] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [14388.093796] RAX: 0000000000007297 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14388.093799] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [14388.093801] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [14388.093803] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [14388.093806] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [14388.093809] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [14388.093812] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14388.093814] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [14388.093822] Call Trace: [14388.093826] [14388.093831] ? show_regs.cold.9+0x22/0x2f [14388.093839] ? watchdog_timer_fn.cold.6+0x62/0x10a [14388.093845] ? __hrtimer_run_queues+0x1c2/0x4d0 [14388.093848] ? watchdog+0x40/0x40 [14388.093852] ? hrtimer_interrupt+0x128/0x2c0 [14388.093856] ? smp_apic_timer_interrupt+0xa6/0x270 [14388.093864] ? apic_timer_interrupt+0xf/0x20 [14388.093868] [14388.093870] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14388.093875] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14388.093879] do_raw_spin_lock+0x177/0x190 [14388.093883] _raw_spin_lock+0x21/0x30 [14388.093890] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14388.094151] ? do_raw_spin_unlock+0x75/0x190 [14388.094157] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14388.106653] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14388.106731] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14388.106738] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14388.106742] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [14388.106756] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [14388.106760] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14388.106764] RAX: 00000000000068af RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14388.106766] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [14388.106768] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [14388.106770] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [14388.106773] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [14388.106776] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [14388.106779] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14388.106781] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [14388.107539] Call Trace: [14388.107548] [14388.107553] ? show_regs.cold.9+0x22/0x2f [14388.107566] ? watchdog_timer_fn.cold.6+0x62/0x10a [14388.107652] ? __hrtimer_run_queues+0x1c2/0x4d0 [14388.107656] ? watchdog+0x40/0x40 [14388.107660] ? hrtimer_interrupt+0x128/0x2c0 [14388.107664] ? smp_apic_timer_interrupt+0xa6/0x270 [14388.107672] ? apic_timer_interrupt+0xf/0x20 [14388.107678] [14388.107681] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [14388.107688] do_raw_spin_lock+0x177/0x190 [14388.107693] _raw_spin_lock+0x21/0x30 [14388.107699] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14388.108176] ? do_raw_spin_unlock+0x75/0x190 [14388.108182] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14388.109152] ? do_wait_intr+0xf0/0xf0 [14388.109161] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14388.109381] mdc_close+0x53b/0xd70 [mdc] [14388.109428] lmv_close+0x18d/0x490 [lmv] [14388.109454] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14388.117753] ? do_wait_intr+0xf0/0xf0 [14388.117822] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14388.118171] mdc_reint+0x2d/0x110 [mdc] [14388.118935] mdc_unlink+0x658/0x870 [mdc] [14388.118964] lmv_unlink+0x34d/0xbd0 [lmv] [14388.118999] ll_unlink+0x24d/0x7f0 [lustre] [14388.119121] vfs_unlink+0x202/0x330 [14388.119130] do_unlinkat+0x2fa/0x460 [14388.119135] __x64_sys_unlink+0x37/0x50 [14388.119138] do_syscall_64+0xc1/0x440 [14388.119145] entry_SYSCALL_64_after_hwframe+0x49/0xae [14388.119154] RIP: 0033:0x7f151bdee10b [14388.119162] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [14388.119166] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [14388.119171] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [14388.119174] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [14388.119176] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14388.119178] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14388.119181] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14388.773719] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14388.784670] ll_md_real_close+0xa5/0x2a0 [lustre] [14388.789249] ll_md_close+0x274/0x800 [lustre] [14388.792741] ll_file_release+0x128/0xc50 [lustre] [14388.797949] __fput+0x10d/0x440 [14388.799114] ____fput+0x12/0x20 [14388.802025] task_work_run+0xc6/0x110 [14388.803903] exit_to_usermode_loop+0x1dd/0x1f0 [14388.808365] do_syscall_64+0x3d6/0x440 [14388.812223] entry_SYSCALL_64_after_hwframe+0x49/0xae [14388.825137] RIP: 0033:0x7f151bee3cd8 [14388.829062] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14388.846225] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14388.859111] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [14388.863060] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [14388.870614] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14388.876746] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14388.884070] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14392.060666] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [14392.061440] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14392.075699] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14392.076096] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14392.076104] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14392.076107] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14392.076122] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14392.076127] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14392.076132] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [14392.076134] RDX: 000000000000104c RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [14392.076137] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [14392.076140] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [14392.076142] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14392.076145] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [14392.076149] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14392.076151] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14392.076161] Call Trace: [14392.076166] [14392.076173] ? show_regs.cold.9+0x22/0x2f [14392.076183] ? watchdog_timer_fn.cold.6+0x62/0x10a [14392.076191] ? __hrtimer_run_queues+0x1c2/0x4d0 [14392.076195] ? watchdog+0x40/0x40 [14392.076200] ? hrtimer_interrupt+0x128/0x2c0 [14392.076204] ? smp_apic_timer_interrupt+0xa6/0x270 [14392.076212] ? apic_timer_interrupt+0xf/0x20 [14392.076217] [14392.076220] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14392.076224] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14392.076229] do_raw_spin_lock+0x177/0x190 [14392.076234] _raw_spin_lock+0x21/0x30 [14392.076240] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14392.076641] obd_mod_free+0x46/0x140 [ptlrpc] [14392.077230] mdc_commit_open+0x91/0xb0 [mdc] [14392.077266] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14392.078323] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14392.091851] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14392.091929] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14392.091937] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14392.091943] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14392.092616] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14392.092636] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14392.092640] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14392.092644] RAX: 00000000000020d4 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14392.092647] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [14392.092650] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [14392.092652] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [14392.092654] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [14392.092657] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [14392.092660] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14392.092662] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14392.092670] Call Trace: [14392.092674] [14392.092680] ? show_regs.cold.9+0x22/0x2f [14392.092688] ? watchdog_timer_fn.cold.6+0x62/0x10a [14392.092695] ? __hrtimer_run_queues+0x1c2/0x4d0 [14392.092699] ? watchdog+0x40/0x40 [14392.092703] ? hrtimer_interrupt+0x128/0x2c0 [14392.092707] ? smp_apic_timer_interrupt+0xa6/0x270 [14392.092720] ? apic_timer_interrupt+0xf/0x20 [14392.092724] [14392.092727] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14392.092732] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14392.092737] do_raw_spin_lock+0x177/0x190 [14392.092741] _raw_spin_lock+0x21/0x30 [14392.092746] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14392.092972] process_one_work+0x2c8/0x700 [14392.092985] worker_thread+0x296/0x6e0 [14392.092990] ? rescuer_thread+0x570/0x570 [14392.092994] kthread+0x1d1/0x200 [14392.092998] ? set_kthread_struct+0x70/0x70 [14392.093001] ret_from_fork+0x1f/0x30 [14392.657696] after_reply+0xe2c/0x1780 [ptlrpc] [14392.665059] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14392.676970] ? __next_timer_interrupt+0x160/0x160 [14392.683781] ptlrpcd+0xa82/0xe40 [ptlrpc] [14392.685539] ? do_wait_intr+0xf0/0xf0 [14392.690173] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14392.694366] kthread+0x1d1/0x200 [14392.697140] ? set_kthread_struct+0x70/0x70 [14392.701739] ret_from_fork+0x1f/0x30 [14392.703788] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12462s! [14392.710908] Showing busy workqueues and worker pools: [14392.713461] workqueue events: flags=0x0 [14392.718046] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14392.724375] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [14392.736270] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14392.743309] pending: kfree_rcu_monitor [14392.746917] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14392.754085] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [14392.768341] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14392.778516] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [14392.789420] workqueue events_long: flags=0x0 [14392.795734] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14392.808277] pending: lu_objects_flush [obdclass] [14392.810287] workqueue events_unbound: flags=0x2 [14392.821082] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14392.828883] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14392.845284] workqueue events_power_efficient: flags=0x80 [14392.846464] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14392.868286] pending: do_cache_clean, neigh_periodic_work, check_lifetime [14392.880162] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14392.892986] pending: neigh_periodic_work [14392.899111] workqueue mm_percpu_wq: flags=0x8 [14392.913522] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14392.924277] pending: vmstat_update, lru_add_drain_per_cpu [14392.953091] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14392.970350] pending: vmstat_update [14392.976214] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14393.005209] pending: lru_add_drain_per_cpu [14393.008286] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [14393.028886] pending: lru_add_drain_per_cpu BAR(48) [14393.034317] workqueue writeback: flags=0x4a [14393.038404] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [14393.045895] pending: wb_workfn [14393.053123] workqueue xprtiod: flags=0xa [14393.059961] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14393.071364] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [14393.083672] workqueue ipv6_addrconf: flags=0x40008 [14393.087395] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14393.100106] pending: addrconf_verify_work [14393.101804] workqueue knbd0-recv: flags=0x1a [14393.106254] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14393.108960] in-flight: 658:recv_work [14393.112687] workqueue ptlrpc_pinger: flags=0x2 [14393.116179] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14393.121896] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14393.126789] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12462s workers=4 idle: 32984 37 8 [14393.133077] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14377s workers=2 idle: 661 [14404.914026] rcu: INFO: rcu_sched self-detected stall on CPU [14404.917206] rcu: 1-...!: (7382043 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [14404.922852] (t=12480267 jiffies g=249657 q=6142) [14404.924598] rcu: rcu_sched kthread starved for 12475219 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [14404.929459] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [14404.932962] rcu: RCU grace-period kthread stack dump: [14404.935621] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [14404.940257] Call Trace: [14404.941500] __schedule+0x351/0xcb0 [14404.943196] schedule+0xc0/0x180 [14404.944761] schedule_timeout+0xb4/0x190 [14404.946236] ? do_raw_spin_unlock+0x75/0x190 [14404.948697] ? __next_timer_interrupt+0x160/0x160 [14404.950117] ? prepare_to_swait_event+0xf3/0x140 [14404.952718] rcu_gp_kthread+0x8dd/0x15b0 [14404.954631] ? force_qs_rnp+0x2c0/0x2c0 [14404.956945] kthread+0x1d1/0x200 [14404.958256] ? set_kthread_struct+0x70/0x70 [14404.964959] ret_from_fork+0x1f/0x30 [14404.967799] rcu: Stack dump where RCU GP kthread last ran: [14404.972544] Sending NMI from CPU 1 to CPUs 3: [14404.974693] NMI backtrace for cpu 3 skipped: idling at native_safe_halt+0x2e/0x50 [14404.977507] NMI backtrace for cpu 1 [14404.980325] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14404.988735] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14404.998813] Call Trace: [14405.000181] [14405.001788] ? dump_stack+0xbb/0x10e [14405.005933] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [14405.009913] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [14405.013391] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [14405.016253] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [14405.020190] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [14405.024556] ? rcu_dump_cpu_stacks+0x11b/0x171 [14405.027124] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [14405.030362] ? update_process_times+0x6d/0xc0 [14405.033905] ? tick_sched_handle.isra.18+0x27/0x90 [14405.035990] ? tick_sched_timer+0x53/0xb0 [14405.039143] ? __hrtimer_run_queues+0x1c2/0x4d0 [14405.041945] ? tick_sched_handle.isra.18+0x90/0x90 [14405.044946] ? hrtimer_interrupt+0x128/0x2c0 [14405.047833] ? smp_apic_timer_interrupt+0xa6/0x270 [14405.051770] ? apic_timer_interrupt+0xf/0x20 [14405.054008] [14405.055152] ? native_safe_halt+0x2e/0x50 [14405.058782] ? kvm_wait+0xcd/0xe0 [14405.060752] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14405.063831] ? do_raw_spin_lock+0x177/0x190 [14405.066122] ? _raw_spin_lock+0x21/0x30 [14405.068622] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14405.071870] ? obd_mod_free+0x46/0x140 [ptlrpc] [14405.074945] ? mdc_commit_open+0x91/0xb0 [mdc] [14405.077334] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14405.080852] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14405.085380] ? after_reply+0xe2c/0x1780 [ptlrpc] [14405.089805] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14405.093614] ? __next_timer_interrupt+0x160/0x160 [14405.096165] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [14405.106189] ? do_wait_intr+0xf0/0xf0 [14405.110153] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14405.116107] ? kthread+0x1d1/0x200 [14405.118755] ? set_kthread_struct+0x70/0x70 [14405.121893] ? ret_from_fork+0x1f/0x30 [14420.052390] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [14420.055114] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14420.056154] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14420.083974] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14420.083988] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14420.083991] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14420.084004] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14420.084008] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14420.084015] RAX: 000000000000310a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14420.084017] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [14420.084020] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [14420.084022] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [14420.084024] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [14420.084028] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [14420.084031] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14420.084033] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [14420.084041] Call Trace: [14420.084047] [14420.084052] ? show_regs.cold.9+0x22/0x2f [14420.084067] ? watchdog_timer_fn.cold.6+0x62/0x10a [14420.084073] ? __hrtimer_run_queues+0x1c2/0x4d0 [14420.084077] ? watchdog+0x40/0x40 [14420.084081] ? hrtimer_interrupt+0x128/0x2c0 [14420.084086] ? smp_apic_timer_interrupt+0xa6/0x270 [14420.084093] ? apic_timer_interrupt+0xf/0x20 [14420.084098] [14420.084101] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14420.084106] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14420.086784] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14420.086869] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14420.086877] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14420.086884] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14420.087375] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14420.087385] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14420.087389] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14420.087394] RAX: 0000000000005613 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14420.087397] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [14420.087400] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [14420.087402] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [14420.087405] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [14420.087408] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [14420.087411] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14420.087414] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14420.087423] Call Trace: [14420.087427] [14420.087433] ? show_regs.cold.9+0x22/0x2f [14420.087442] ? watchdog_timer_fn.cold.6+0x62/0x10a [14420.087449] ? __hrtimer_run_queues+0x1c2/0x4d0 [14420.087453] ? watchdog+0x40/0x40 [14420.087458] ? hrtimer_interrupt+0x128/0x2c0 [14420.087463] ? smp_apic_timer_interrupt+0xa6/0x270 [14420.087471] ? apic_timer_interrupt+0xf/0x20 [14420.087475] [14420.087478] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14420.087483] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14420.087489] do_raw_spin_lock+0x177/0x190 [14420.087494] _raw_spin_lock+0x21/0x30 [14420.087499] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14420.088010] process_one_work+0x2c8/0x700 [14420.088017] worker_thread+0x296/0x6e0 [14420.088022] ? rescuer_thread+0x570/0x570 [14420.088025] kthread+0x1d1/0x200 [14420.088030] ? set_kthread_struct+0x70/0x70 [14420.088033] ret_from_fork+0x1f/0x30 [14420.398829] do_raw_spin_lock+0x177/0x190 [14420.400602] _raw_spin_lock+0x21/0x30 [14420.406371] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14420.409633] ? do_raw_spin_unlock+0x75/0x190 [14420.413120] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14420.417483] ? do_wait_intr+0xf0/0xf0 [14420.422771] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14420.426482] mdc_reint+0x2d/0x110 [mdc] [14420.428010] mdc_unlink+0x658/0x870 [mdc] [14420.429536] lmv_unlink+0x34d/0xbd0 [lmv] [14420.431204] ll_unlink+0x24d/0x7f0 [lustre] [14420.434861] vfs_unlink+0x202/0x330 [14420.437936] do_unlinkat+0x2fa/0x460 [14420.439146] __x64_sys_unlink+0x37/0x50 [14420.440185] do_syscall_64+0xc1/0x440 [14420.444129] entry_SYSCALL_64_after_hwframe+0x49/0xae [14420.447616] RIP: 0033:0x7f151bdee10b [14420.449275] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [14420.456803] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [14420.461318] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [14420.467790] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [14420.474565] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14420.479957] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14420.486360] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14423.520186] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12492s! [14423.524662] Showing busy workqueues and worker pools: [14423.529564] workqueue events: flags=0x0 [14423.531614] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14423.535289] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [14423.543307] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14423.550556] pending: kfree_rcu_monitor [14423.551811] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14423.553270] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [14423.563428] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14423.566901] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [14423.571590] workqueue events_long: flags=0x0 [14423.574515] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14423.579614] pending: lu_objects_flush [obdclass] [14423.580825] workqueue events_unbound: flags=0x2 [14423.581690] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14423.592171] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14423.602253] workqueue events_power_efficient: flags=0x80 [14423.604307] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14423.607155] pending: do_cache_clean, neigh_periodic_work, check_lifetime [14423.610080] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14423.613239] pending: neigh_periodic_work [14423.615951] workqueue mm_percpu_wq: flags=0x8 [14423.618607] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14423.622351] pending: vmstat_update, lru_add_drain_per_cpu [14423.625351] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14423.628657] pending: vmstat_update [14423.630473] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14423.638230] pending: lru_add_drain_per_cpu [14423.639704] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [14423.645862] pending: lru_add_drain_per_cpu BAR(48) [14423.649919] workqueue writeback: flags=0x4a [14423.652047] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [14423.655376] pending: wb_workfn [14423.657303] workqueue xprtiod: flags=0xa [14423.659307] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14423.662507] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [14423.667062] workqueue ipv6_addrconf: flags=0x40008 [14423.669616] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14423.673666] pending: addrconf_verify_work [14423.676376] workqueue knbd0-recv: flags=0x1a [14423.678685] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14423.682582] in-flight: 658:recv_work [14423.684883] workqueue ptlrpc_pinger: flags=0x2 [14423.687784] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14423.691622] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14423.695403] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12493s workers=4 idle: 32984 37 8 [14423.698783] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14408s workers=2 idle: 661 [14432.059887] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [14432.060360] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [14432.063560] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14432.063642] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14432.069320] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14432.069638] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14432.069774] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14432.069777] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14432.070446] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14432.070454] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [14432.070459] RAX: 0000000000001b99 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14432.070462] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [14432.070464] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [14432.070466] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [14432.070468] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [14432.070472] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [14432.070476] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14432.070479] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [14432.070486] Call Trace: [14432.070492] [14432.070498] ? show_regs.cold.9+0x22/0x2f [14432.070510] ? watchdog_timer_fn.cold.6+0x62/0x10a [14432.070602] ? __hrtimer_run_queues+0x1c2/0x4d0 [14432.070610] ? watchdog+0x40/0x40 [14432.070615] ? hrtimer_interrupt+0x128/0x2c0 [14432.070619] ? smp_apic_timer_interrupt+0xa6/0x270 [14432.070626] ? apic_timer_interrupt+0xf/0x20 [14432.070631] [14432.070635] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14432.070651] do_raw_spin_lock+0x177/0x190 [14432.070657] _raw_spin_lock+0x21/0x30 [14432.070665] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14432.071197] ? do_raw_spin_unlock+0x75/0x190 [14432.071204] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14432.071588] ? do_wait_intr+0xf0/0xf0 [14432.071597] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14432.093509] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14432.093524] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14432.093541] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14432.093545] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14432.093551] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [14432.093553] RDX: 0000000000005c32 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [14432.093556] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [14432.093558] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [14432.093560] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14432.093563] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [14432.093566] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14432.093568] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14432.093651] Call Trace: [14432.093656] [14432.093666] ? show_regs.cold.9+0x22/0x2f [14432.093676] ? watchdog_timer_fn.cold.6+0x62/0x10a [14432.093683] ? __hrtimer_run_queues+0x1c2/0x4d0 [14432.093687] ? watchdog+0x40/0x40 [14432.093695] ? hrtimer_interrupt+0x128/0x2c0 [14432.093699] ? smp_apic_timer_interrupt+0xa6/0x270 [14432.093707] ? apic_timer_interrupt+0xf/0x20 [14432.093713] [14432.093716] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14432.093721] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14432.093729] do_raw_spin_lock+0x177/0x190 [14432.093734] _raw_spin_lock+0x21/0x30 [14432.093744] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14432.094187] obd_mod_free+0x46/0x140 [ptlrpc] [14432.099001] mdc_close+0x53b/0xd70 [mdc] [14432.099066] lmv_close+0x18d/0x490 [lmv] [14432.099095] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14432.099270] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14432.099467] ll_md_real_close+0xa5/0x2a0 [lustre] [14432.099564] ll_md_close+0x274/0x800 [lustre] [14432.099655] ll_file_release+0x128/0xc50 [lustre] [14432.099800] __fput+0x10d/0x440 [14432.099809] ____fput+0x12/0x20 [14432.099814] task_work_run+0xc6/0x110 [14432.099821] exit_to_usermode_loop+0x1dd/0x1f0 [14432.099829] do_syscall_64+0x3d6/0x440 [14432.099834] entry_SYSCALL_64_after_hwframe+0x49/0xae [14432.099842] RIP: 0033:0x7f151bee3cd8 [14432.099854] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14432.099858] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14432.099863] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [14432.099960] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [14432.099963] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14432.099965] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14432.099967] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14432.328287] mdc_commit_open+0x91/0xb0 [mdc] [14432.330261] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14432.332590] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14432.335296] after_reply+0xe2c/0x1780 [ptlrpc] [14432.337875] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14432.340171] ? __next_timer_interrupt+0x160/0x160 [14432.342491] ptlrpcd+0xa82/0xe40 [ptlrpc] [14432.344594] ? do_wait_intr+0xf0/0xf0 [14432.345977] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14432.348436] kthread+0x1d1/0x200 [14432.349744] ? set_kthread_struct+0x70/0x70 [14432.352750] ret_from_fork+0x1f/0x30 [14448.053396] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [14448.058497] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14448.063248] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14448.063329] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14448.063336] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14448.063339] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14448.063353] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14448.063358] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [14448.063363] RAX: 000000000000740b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14448.063366] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [14448.063368] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [14448.063371] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [14448.063373] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [14448.063377] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [14448.063380] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14448.063383] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [14448.063391] Call Trace: [14448.063396] [14448.063401] ? show_regs.cold.9+0x22/0x2f [14448.063411] ? watchdog_timer_fn.cold.6+0x62/0x10a [14448.063417] ? __hrtimer_run_queues+0x1c2/0x4d0 [14448.063422] ? watchdog+0x40/0x40 [14448.063426] ? hrtimer_interrupt+0x128/0x2c0 [14448.063430] ? smp_apic_timer_interrupt+0xa6/0x270 [14448.063438] ? apic_timer_interrupt+0xf/0x20 [14448.063442] [14448.063445] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14448.063450] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14448.063455] do_raw_spin_lock+0x177/0x190 [14448.063460] _raw_spin_lock+0x21/0x30 [14448.063466] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14448.063792] ? do_raw_spin_unlock+0x75/0x190 [14448.063798] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14448.064317] ? do_wait_intr+0xf0/0xf0 [14448.064325] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14448.064695] mdc_reint+0x2d/0x110 [mdc] [14448.064736] mdc_unlink+0x658/0x870 [mdc] [14448.064985] lmv_unlink+0x34d/0xbd0 [lmv] [14448.065018] ll_unlink+0x24d/0x7f0 [lustre] [14448.065151] vfs_unlink+0x202/0x330 [14448.065160] do_unlinkat+0x2fa/0x460 [14448.065166] __x64_sys_unlink+0x37/0x50 [14448.065170] do_syscall_64+0xc1/0x440 [14448.065176] entry_SYSCALL_64_after_hwframe+0x49/0xae [14448.065183] RIP: 0033:0x7f151bdee10b [14448.065189] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [14448.065193] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [14448.065197] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [14448.065200] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [14448.065202] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14448.065205] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14448.065207] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14448.489723] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14448.532345] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14448.544968] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14448.558350] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14448.571454] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14448.579287] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14448.594144] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14448.604342] RAX: 0000000000004642 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14448.613255] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [14448.620693] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [14448.630653] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [14448.639948] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [14448.646712] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [14448.651705] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14448.660717] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14448.669406] Call Trace: [14448.671679] [14448.677155] ? show_regs.cold.9+0x22/0x2f [14448.679111] ? watchdog_timer_fn.cold.6+0x62/0x10a [14448.681442] ? __hrtimer_run_queues+0x1c2/0x4d0 [14448.684620] ? watchdog+0x40/0x40 [14448.691187] ? hrtimer_interrupt+0x128/0x2c0 [14448.696039] ? smp_apic_timer_interrupt+0xa6/0x270 [14448.701057] ? apic_timer_interrupt+0xf/0x20 [14448.703800] [14448.704799] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14448.707682] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14448.711491] do_raw_spin_lock+0x177/0x190 [14448.714989] _raw_spin_lock+0x21/0x30 [14448.717322] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14448.720334] process_one_work+0x2c8/0x700 [14448.722315] worker_thread+0x296/0x6e0 [14448.728428] ? rescuer_thread+0x570/0x570 [14448.734723] kthread+0x1d1/0x200 [14448.739896] ? set_kthread_struct+0x70/0x70 [14448.744687] ret_from_fork+0x1f/0x30 [14453.729029] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12523s! [14453.738524] Showing busy workqueues and worker pools: [14453.747755] workqueue events: flags=0x0 [14453.753406] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14453.761912] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [14453.773383] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14453.784095] pending: kfree_rcu_monitor [14453.793904] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14453.799505] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [14453.813113] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14453.826478] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [14453.839347] workqueue events_long: flags=0x0 [14453.847463] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14453.858578] pending: lu_objects_flush [obdclass] [14453.869670] workqueue events_unbound: flags=0x2 [14453.875718] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14453.883524] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14453.897732] workqueue events_power_efficient: flags=0x80 [14453.901156] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14453.905893] pending: do_cache_clean, neigh_periodic_work, check_lifetime [14453.918138] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14453.924562] pending: neigh_periodic_work [14453.925927] workqueue mm_percpu_wq: flags=0x8 [14453.927373] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14453.935801] pending: vmstat_update, lru_add_drain_per_cpu [14453.940788] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14453.948480] pending: vmstat_update [14453.953188] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14453.958590] pending: lru_add_drain_per_cpu [14453.964797] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [14453.975672] pending: lru_add_drain_per_cpu BAR(48) [14453.985185] workqueue writeback: flags=0x4a [14453.994630] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [14454.007048] pending: wb_workfn [14454.011516] workqueue xprtiod: flags=0xa [14454.020023] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14454.030906] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [14454.040673] workqueue ipv6_addrconf: flags=0x40008 [14454.044695] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14454.049214] pending: addrconf_verify_work [14454.052526] workqueue knbd0-recv: flags=0x1a [14454.057330] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14454.066158] in-flight: 658:recv_work [14454.067618] workqueue ptlrpc_pinger: flags=0x2 [14454.070777] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14454.074787] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14454.084766] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12523s workers=4 idle: 32984 37 8 [14454.092975] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14438s workers=2 idle: 661 [14460.055126] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [14460.057028] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [14460.058104] Modules linked in: [14460.061106] Modules linked in: [14460.061107] lustre(O) osp(O) ofd(O) [14460.062482] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14460.062548] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14460.070856] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) [14460.072327] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14460.092016] lov(O) [14460.097330] RIP: 0010:native_safe_halt+0x2e/0x50 [14460.099460] osc(O) [14460.102998] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14460.103010] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14460.103017] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14460.103019] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [14460.103021] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [14460.103023] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [14460.103026] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [14460.103029] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [14460.103033] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14460.103035] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [14460.103043] Call Trace: [14460.103049] [14460.103055] ? show_regs.cold.9+0x22/0x2f [14460.103068] ? watchdog_timer_fn.cold.6+0x62/0x10a [14460.103074] ? __hrtimer_run_queues+0x1c2/0x4d0 [14460.103079] ? watchdog+0x40/0x40 [14460.103083] ? hrtimer_interrupt+0x128/0x2c0 [14460.103087] ? smp_apic_timer_interrupt+0xa6/0x270 [14460.103096] ? apic_timer_interrupt+0xf/0x20 [14460.103100] [14460.103103] ? native_safe_halt+0x2e/0x50 [14460.103107] kvm_wait+0xcd/0xe0 [14460.103120] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14460.103130] do_raw_spin_lock+0x177/0x190 [14460.103135] _raw_spin_lock+0x21/0x30 [14460.103139] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14460.103421] ? do_raw_spin_unlock+0x75/0x190 [14460.103427] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14460.103638] ? do_wait_intr+0xf0/0xf0 [14460.103645] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14460.103959] mdc_close+0x53b/0xd70 [mdc] [14460.104004] lmv_close+0x18d/0x490 [lmv] [14460.104031] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14460.104199] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14460.104299] ll_md_real_close+0xa5/0x2a0 [lustre] [14460.104380] ll_md_close+0x274/0x800 [lustre] [14460.104458] ll_file_release+0x128/0xc50 [lustre] [14460.104538] __fput+0x10d/0x440 [14460.104547] ____fput+0x12/0x20 [14460.104552] task_work_run+0xc6/0x110 [14460.104614] exit_to_usermode_loop+0x1dd/0x1f0 [14460.104622] do_syscall_64+0x3d6/0x440 [14460.104627] entry_SYSCALL_64_after_hwframe+0x49/0xae [14460.104635] RIP: 0033:0x7f151bee3cd8 [14460.104643] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14460.104646] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14460.104653] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [14460.104655] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [14460.104658] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14460.104660] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14460.104662] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14460.249891] lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14460.269715] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14460.274808] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14460.278867] RIP: 0010:native_safe_halt+0x2e/0x50 [14460.281074] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14460.288959] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14460.292627] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [14460.296276] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [14460.299957] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [14460.303855] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [14460.306752] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14460.310082] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [14460.314276] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14460.316982] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14460.319973] Call Trace: [14460.321547] [14460.322456] ? show_regs.cold.9+0x22/0x2f [14460.324079] ? watchdog_timer_fn.cold.6+0x62/0x10a [14460.326459] ? __hrtimer_run_queues+0x1c2/0x4d0 [14460.328682] ? watchdog+0x40/0x40 [14460.330467] ? hrtimer_interrupt+0x128/0x2c0 [14460.332204] ? smp_apic_timer_interrupt+0xa6/0x270 [14460.334525] ? apic_timer_interrupt+0xf/0x20 [14460.336407] [14460.337447] ? native_safe_halt+0x2e/0x50 [14460.339411] kvm_wait+0xcd/0xe0 [14460.341038] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14460.343465] do_raw_spin_lock+0x177/0x190 [14460.345136] _raw_spin_lock+0x21/0x30 [14460.347404] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14460.350101] obd_mod_free+0x46/0x140 [ptlrpc] [14460.353404] mdc_commit_open+0x91/0xb0 [mdc] [14460.355705] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14460.358507] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14460.361855] after_reply+0xe2c/0x1780 [ptlrpc] [14460.364233] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14460.366922] ? __next_timer_interrupt+0x160/0x160 [14460.369534] ptlrpcd+0xa82/0xe40 [ptlrpc] [14460.371640] ? do_wait_intr+0xf0/0xf0 [14460.373553] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14460.376691] kthread+0x1d1/0x200 [14460.378427] ? set_kthread_struct+0x70/0x70 [14460.380475] ret_from_fork+0x1f/0x30 [14476.056368] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14476.063536] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14476.105401] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14476.112371] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14476.122254] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14476.127529] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14476.132076] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14476.145226] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14476.149761] RAX: 00000000000078b7 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14476.152041] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [14476.156247] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [14476.161330] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [14476.165629] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [14476.169507] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [14476.176520] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14476.182484] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14476.187223] Call Trace: [14476.188491] [14476.190185] ? show_regs.cold.9+0x22/0x2f [14476.193620] ? watchdog_timer_fn.cold.6+0x62/0x10a [14476.196439] ? __hrtimer_run_queues+0x1c2/0x4d0 [14476.199014] ? watchdog+0x40/0x40 [14476.200889] ? hrtimer_interrupt+0x128/0x2c0 [14476.203333] ? smp_apic_timer_interrupt+0xa6/0x270 [14476.206021] ? apic_timer_interrupt+0xf/0x20 [14476.208898] [14476.210260] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14476.213653] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14476.216508] do_raw_spin_lock+0x177/0x190 [14476.218725] _raw_spin_lock+0x21/0x30 [14476.221075] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14476.225021] process_one_work+0x2c8/0x700 [14476.227130] worker_thread+0x296/0x6e0 [14476.229246] ? rescuer_thread+0x570/0x570 [14476.231339] kthread+0x1d1/0x200 [14476.233261] ? set_kthread_struct+0x70/0x70 [14476.235332] ret_from_fork+0x1f/0x30 [14480.060139] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [14480.077407] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14480.139857] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14480.157871] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14480.166901] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14480.172947] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14480.192897] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14480.205178] RAX: 000000000000027f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14480.215256] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [14480.223596] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [14480.231776] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [14480.236687] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [14480.244843] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [14480.260425] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14480.276972] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [14480.289092] Call Trace: [14480.291273] [14480.298281] ? show_regs.cold.9+0x22/0x2f [14480.308298] ? watchdog_timer_fn.cold.6+0x62/0x10a [14480.320505] ? __hrtimer_run_queues+0x1c2/0x4d0 [14480.322068] ? watchdog+0x40/0x40 [14480.336281] ? hrtimer_interrupt+0x128/0x2c0 [14480.344383] ? smp_apic_timer_interrupt+0xa6/0x270 [14480.354075] ? apic_timer_interrupt+0xf/0x20 [14480.368128] [14480.368955] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14480.377202] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14480.381114] do_raw_spin_lock+0x177/0x190 [14480.386163] _raw_spin_lock+0x21/0x30 [14480.391217] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14480.399853] ? do_raw_spin_unlock+0x75/0x190 [14480.402411] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14480.411226] ? do_wait_intr+0xf0/0xf0 [14480.413515] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14480.426392] mdc_reint+0x2d/0x110 [mdc] [14480.439345] mdc_unlink+0x658/0x870 [mdc] [14480.445411] lmv_unlink+0x34d/0xbd0 [lmv] [14480.453969] ll_unlink+0x24d/0x7f0 [lustre] [14480.462390] vfs_unlink+0x202/0x330 [14480.472481] do_unlinkat+0x2fa/0x460 [14480.476517] __x64_sys_unlink+0x37/0x50 [14480.479524] do_syscall_64+0xc1/0x440 [14480.480946] entry_SYSCALL_64_after_hwframe+0x49/0xae [14480.495068] RIP: 0033:0x7f151bdee10b [14480.502085] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [14480.530467] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [14480.541096] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [14480.563904] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [14480.577158] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14480.598477] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14480.610814] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14484.449626] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12553s! [14484.457012] Showing busy workqueues and worker pools: [14484.467832] workqueue events: flags=0x0 [14484.474581] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14484.486719] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [14484.504683] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14484.511861] pending: kfree_rcu_monitor [14484.515589] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14484.520796] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [14484.530282] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14484.534337] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [14484.540690] workqueue events_long: flags=0x0 [14484.551471] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14484.561783] pending: lu_objects_flush [obdclass] [14484.571820] workqueue events_unbound: flags=0x2 [14484.575296] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14484.586035] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14484.598910] workqueue events_power_efficient: flags=0x80 [14484.603903] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14484.614801] pending: do_cache_clean, neigh_periodic_work, check_lifetime [14484.626370] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14484.633793] pending: neigh_periodic_work [14484.638783] workqueue mm_percpu_wq: flags=0x8 [14484.648434] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14484.654388] pending: vmstat_update, lru_add_drain_per_cpu [14484.663091] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14484.675119] pending: vmstat_update [14484.687198] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14484.694075] pending: lru_add_drain_per_cpu [14484.706508] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [14484.718662] pending: lru_add_drain_per_cpu BAR(48) [14484.720470] workqueue writeback: flags=0x4a [14484.730323] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [14484.741157] pending: wb_workfn [14484.748426] workqueue xprtiod: flags=0xa [14484.754618] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14484.760422] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [14484.778040] workqueue ipv6_addrconf: flags=0x40008 [14484.785511] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14484.794136] pending: addrconf_verify_work [14484.801496] workqueue knbd0-recv: flags=0x1a [14484.805414] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14484.817309] in-flight: 658:recv_work [14484.826601] workqueue ptlrpc_pinger: flags=0x2 [14484.830228] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14484.836902] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14484.849797] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12554s workers=4 idle: 32984 37 8 [14484.862982] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14469s workers=2 idle: 661 [14488.060425] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [14488.065642] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [14488.067231] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14488.067314] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14488.067321] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14488.067324] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14488.067402] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14488.067407] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14488.067412] RAX: 0000000000007f8f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14488.067415] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [14488.067417] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [14488.067419] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [14488.067421] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [14488.067425] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [14488.067428] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14488.067430] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [14488.067440] Call Trace: [14488.067445] [14488.067452] ? show_regs.cold.9+0x22/0x2f [14488.067462] ? watchdog_timer_fn.cold.6+0x62/0x10a [14488.067469] ? __hrtimer_run_queues+0x1c2/0x4d0 [14488.067474] ? watchdog+0x40/0x40 [14488.067479] ? hrtimer_interrupt+0x128/0x2c0 [14488.067483] ? smp_apic_timer_interrupt+0xa6/0x270 [14488.067491] ? apic_timer_interrupt+0xf/0x20 [14488.068235] [14488.068241] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14488.068252] do_raw_spin_lock+0x177/0x190 [14488.068258] _raw_spin_lock+0x21/0x30 [14488.068263] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14488.071223] ? do_raw_spin_unlock+0x75/0x190 [14488.071235] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14488.072901] ? do_wait_intr+0xf0/0xf0 [14488.072910] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14488.073230] mdc_close+0x53b/0xd70 [mdc] [14488.073272] lmv_close+0x18d/0x490 [lmv] [14488.073302] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14488.073483] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14488.074708] ll_md_real_close+0xa5/0x2a0 [lustre] [14488.074793] ll_md_close+0x274/0x800 [lustre] [14488.074881] ll_file_release+0x128/0xc50 [lustre] [14488.074970] __fput+0x10d/0x440 [14488.074979] ____fput+0x12/0x20 [14488.074984] task_work_run+0xc6/0x110 [14488.074994] exit_to_usermode_loop+0x1dd/0x1f0 [14488.075002] do_syscall_64+0x3d6/0x440 [14488.075007] entry_SYSCALL_64_after_hwframe+0x49/0xae [14488.075016] RIP: 0033:0x7f151bee3cd8 [14488.075025] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14488.075030] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14488.075036] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [14488.075039] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [14488.075041] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14488.075043] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14488.075045] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14488.560552] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14488.601437] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14488.614056] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14488.635304] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14488.647626] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14488.671377] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14488.676932] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [14488.682617] RDX: 0000000000002d69 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [14488.690239] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [14488.693673] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [14488.696880] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14488.703024] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [14488.709910] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14488.718915] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14488.727592] Call Trace: [14488.730442] [14488.731747] ? show_regs.cold.9+0x22/0x2f [14488.735353] ? watchdog_timer_fn.cold.6+0x62/0x10a [14488.737107] ? __hrtimer_run_queues+0x1c2/0x4d0 [14488.742947] ? watchdog+0x40/0x40 [14488.744820] ? hrtimer_interrupt+0x128/0x2c0 [14488.747055] ? smp_apic_timer_interrupt+0xa6/0x270 [14488.749282] ? apic_timer_interrupt+0xf/0x20 [14488.752024] [14488.753220] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14488.756146] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14488.762146] do_raw_spin_lock+0x177/0x190 [14488.765554] _raw_spin_lock+0x21/0x30 [14488.767794] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14488.769856] obd_mod_free+0x46/0x140 [ptlrpc] [14488.777808] mdc_commit_open+0x91/0xb0 [mdc] [14488.785477] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14488.793575] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14488.799807] after_reply+0xe2c/0x1780 [ptlrpc] [14488.803748] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14488.810167] ? __next_timer_interrupt+0x160/0x160 [14488.819733] ptlrpcd+0xa82/0xe40 [ptlrpc] [14488.825662] ? do_wait_intr+0xf0/0xf0 [14488.827026] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14488.843834] kthread+0x1d1/0x200 [14488.846248] ? set_kthread_struct+0x70/0x70 [14488.849637] ret_from_fork+0x1f/0x30 [14504.056079] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14504.060075] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14504.092218] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14504.097671] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14504.102720] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14504.107030] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14504.110138] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14504.119682] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14504.123741] RAX: 0000000000003485 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14504.127554] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [14504.132208] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [14504.135558] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [14504.139112] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [14504.143111] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [14504.147658] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14504.150648] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14504.154139] Call Trace: [14504.155668] [14504.156800] ? show_regs.cold.9+0x22/0x2f [14504.158861] ? watchdog_timer_fn.cold.6+0x62/0x10a [14504.161550] ? __hrtimer_run_queues+0x1c2/0x4d0 [14504.163851] ? watchdog+0x40/0x40 [14504.165877] ? hrtimer_interrupt+0x128/0x2c0 [14504.167980] ? smp_apic_timer_interrupt+0xa6/0x270 [14504.170717] ? apic_timer_interrupt+0xf/0x20 [14504.172249] [14504.173156] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14504.175509] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14504.178012] do_raw_spin_lock+0x177/0x190 [14504.180256] _raw_spin_lock+0x21/0x30 [14504.182759] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14504.185165] process_one_work+0x2c8/0x700 [14504.187240] worker_thread+0x296/0x6e0 [14504.189012] ? rescuer_thread+0x570/0x570 [14504.191072] kthread+0x1d1/0x200 [14504.192665] ? set_kthread_struct+0x70/0x70 [14504.194802] ret_from_fork+0x1f/0x30 [14512.052509] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [14512.058061] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14512.112565] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14512.119158] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14512.127897] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [14512.133801] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [14512.144234] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14512.152879] RAX: 0000000000001a10 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14512.155383] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [14512.164607] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [14512.171573] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [14512.184965] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [14512.194735] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [14512.202435] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14512.204806] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [14512.211243] Call Trace: [14512.214997] [14512.217324] ? show_regs.cold.9+0x22/0x2f [14512.219853] ? watchdog_timer_fn.cold.6+0x62/0x10a [14512.231254] ? __hrtimer_run_queues+0x1c2/0x4d0 [14512.238442] ? watchdog+0x40/0x40 [14512.239947] ? hrtimer_interrupt+0x128/0x2c0 [14512.247042] ? smp_apic_timer_interrupt+0xa6/0x270 [14512.252159] ? apic_timer_interrupt+0xf/0x20 [14512.258855] [14512.259641] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [14512.266448] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14512.276363] do_raw_spin_lock+0x177/0x190 [14512.277760] _raw_spin_lock+0x21/0x30 [14512.284790] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14512.294212] ? do_raw_spin_unlock+0x75/0x190 [14512.297033] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14512.306756] ? do_wait_intr+0xf0/0xf0 [14512.309044] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14512.312918] mdc_reint+0x2d/0x110 [mdc] [14512.314915] mdc_unlink+0x658/0x870 [mdc] [14512.320190] lmv_unlink+0x34d/0xbd0 [lmv] [14512.322811] ll_unlink+0x24d/0x7f0 [lustre] [14512.326149] vfs_unlink+0x202/0x330 [14512.328230] do_unlinkat+0x2fa/0x460 [14512.332693] __x64_sys_unlink+0x37/0x50 [14512.334654] do_syscall_64+0xc1/0x440 [14512.338932] entry_SYSCALL_64_after_hwframe+0x49/0xae [14512.346168] RIP: 0033:0x7f151bdee10b [14512.352448] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [14512.361022] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [14512.368472] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [14512.371984] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [14512.376802] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14512.380196] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14512.383711] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14515.170094] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12584s! [14515.173577] Showing busy workqueues and worker pools: [14515.177090] workqueue events: flags=0x0 [14515.180109] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14515.186788] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [14515.195167] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14515.199583] pending: kfree_rcu_monitor [14515.205889] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14515.212171] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [14515.220343] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14515.227045] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [14515.231180] workqueue events_long: flags=0x0 [14515.233823] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14515.239601] pending: lu_objects_flush [obdclass] [14515.242417] workqueue events_unbound: flags=0x2 [14515.244997] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14515.248093] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14515.253121] workqueue events_power_efficient: flags=0x80 [14515.255971] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14515.262282] pending: do_cache_clean, neigh_periodic_work, check_lifetime [14515.271437] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14515.278607] pending: neigh_periodic_work [14515.284168] workqueue mm_percpu_wq: flags=0x8 [14515.286937] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14515.294075] pending: vmstat_update, lru_add_drain_per_cpu [14515.298819] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14515.303035] pending: vmstat_update [14515.304470] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14515.307444] pending: lru_add_drain_per_cpu [14515.309759] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [14515.312339] pending: lru_add_drain_per_cpu BAR(48) [14515.314405] workqueue writeback: flags=0x4a [14515.316167] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [14515.319206] pending: wb_workfn [14515.320608] workqueue xprtiod: flags=0xa [14515.322430] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14515.325755] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [14515.339048] workqueue ipv6_addrconf: flags=0x40008 [14515.344204] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14515.352135] pending: addrconf_verify_work [14515.361484] workqueue knbd0-recv: flags=0x1a [14515.369023] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14515.377098] in-flight: 658:recv_work [14515.389462] workqueue ptlrpc_pinger: flags=0x2 [14515.396351] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14515.403490] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14515.408257] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12584s workers=4 idle: 32984 37 8 [14515.417705] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14500s workers=2 idle: 661 [14516.057695] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [14516.059889] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [14516.065891] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14516.065970] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14516.065976] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14516.065980] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14516.065993] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14516.065997] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14516.066001] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [14516.066004] RDX: 00000000000011cc RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [14516.066006] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [14516.066008] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [14516.066010] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14516.066014] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [14516.066016] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14516.066018] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14516.066027] Call Trace: [14516.066034] [14516.066040] ? show_regs.cold.9+0x22/0x2f [14516.066049] ? watchdog_timer_fn.cold.6+0x62/0x10a [14516.066054] ? __hrtimer_run_queues+0x1c2/0x4d0 [14516.066058] ? watchdog+0x40/0x40 [14516.066062] ? hrtimer_interrupt+0x128/0x2c0 [14516.066066] ? smp_apic_timer_interrupt+0xa6/0x270 [14516.066073] ? apic_timer_interrupt+0xf/0x20 [14516.066078] [14516.066080] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14516.066084] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14516.066088] do_raw_spin_lock+0x177/0x190 [14516.066093] _raw_spin_lock+0x21/0x30 [14516.066098] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14516.066380] obd_mod_free+0x46/0x140 [ptlrpc] [14516.066578] mdc_commit_open+0x91/0xb0 [mdc] [14516.066613] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14516.066807] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14516.067003] after_reply+0xe2c/0x1780 [ptlrpc] [14516.067199] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14516.067402] ? __next_timer_interrupt+0x160/0x160 [14516.067411] ptlrpcd+0xa82/0xe40 [ptlrpc] [14516.068079] ? do_wait_intr+0xf0/0xf0 [14516.068089] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14516.068359] kthread+0x1d1/0x200 [14516.068377] ? set_kthread_struct+0x70/0x70 [14516.068381] ret_from_fork+0x1f/0x30 [14516.295871] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14516.344569] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14516.354525] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14516.364818] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14516.371833] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14516.388922] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14516.403466] RAX: 0000000000005509 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14516.407254] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [14516.415144] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [14516.423732] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [14516.433124] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [14516.444344] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [14516.453770] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14516.460082] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [14516.465958] Call Trace: [14516.468894] [14516.470959] ? show_regs.cold.9+0x22/0x2f [14516.472616] ? watchdog_timer_fn.cold.6+0x62/0x10a [14516.476971] ? __hrtimer_run_queues+0x1c2/0x4d0 [14516.484808] ? watchdog+0x40/0x40 [14516.485940] ? hrtimer_interrupt+0x128/0x2c0 [14516.492122] ? smp_apic_timer_interrupt+0xa6/0x270 [14516.496550] ? apic_timer_interrupt+0xf/0x20 [14516.501863] [14516.505481] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14516.510962] do_raw_spin_lock+0x177/0x190 [14516.515444] _raw_spin_lock+0x21/0x30 [14516.517163] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14516.526872] ? do_raw_spin_unlock+0x75/0x190 [14516.533484] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14516.539561] ? do_wait_intr+0xf0/0xf0 [14516.545458] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14516.547139] mdc_close+0x53b/0xd70 [mdc] [14516.549912] lmv_close+0x18d/0x490 [lmv] [14516.552105] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14516.559411] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14516.564744] ll_md_real_close+0xa5/0x2a0 [lustre] [14516.568049] ll_md_close+0x274/0x800 [lustre] [14516.577163] ll_file_release+0x128/0xc50 [lustre] [14516.579314] __fput+0x10d/0x440 [14516.581822] ____fput+0x12/0x20 [14516.587934] task_work_run+0xc6/0x110 [14516.591073] exit_to_usermode_loop+0x1dd/0x1f0 [14516.592617] do_syscall_64+0x3d6/0x440 [14516.597133] entry_SYSCALL_64_after_hwframe+0x49/0xae [14516.598904] RIP: 0033:0x7f151bee3cd8 [14516.602932] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14516.615431] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14516.620164] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [14516.625628] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [14516.628725] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14516.633306] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14516.635566] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14532.056011] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [14532.060402] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14532.082560] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14532.086242] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14532.093435] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14532.108515] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14532.116775] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14532.144060] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14532.155861] RAX: 000000000000101a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14532.166255] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [14532.174905] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [14532.181409] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [14532.188009] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [14532.195383] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [14532.205204] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14532.213121] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14532.222336] Call Trace: [14532.223528] [14532.225004] ? show_regs.cold.9+0x22/0x2f [14532.228031] ? watchdog_timer_fn.cold.6+0x62/0x10a [14532.234571] ? __hrtimer_run_queues+0x1c2/0x4d0 [14532.238642] ? watchdog+0x40/0x40 [14532.242382] ? hrtimer_interrupt+0x128/0x2c0 [14532.247433] ? smp_apic_timer_interrupt+0xa6/0x270 [14532.251033] ? apic_timer_interrupt+0xf/0x20 [14532.258512] [14532.259619] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14532.266698] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14532.270544] do_raw_spin_lock+0x177/0x190 [14532.275150] _raw_spin_lock+0x21/0x30 [14532.277321] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14532.281464] process_one_work+0x2c8/0x700 [14532.291433] worker_thread+0x296/0x6e0 [14532.296675] ? rescuer_thread+0x570/0x570 [14532.300734] kthread+0x1d1/0x200 [14532.304434] ? set_kthread_struct+0x70/0x70 [14532.310628] ret_from_fork+0x1f/0x30 [14540.052493] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [14540.069756] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14540.116744] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14540.123734] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14540.128741] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14540.134876] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14540.147280] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14540.152917] RAX: 000000000000635d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14540.158954] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [14540.163882] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [14540.166079] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [14540.170675] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [14540.174804] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [14540.177936] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14540.180234] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [14540.184229] Call Trace: [14540.185304] [14540.186784] ? show_regs.cold.9+0x22/0x2f [14540.188995] ? watchdog_timer_fn.cold.6+0x62/0x10a [14540.190561] ? __hrtimer_run_queues+0x1c2/0x4d0 [14540.193160] ? watchdog+0x40/0x40 [14540.194747] ? hrtimer_interrupt+0x128/0x2c0 [14540.196669] ? smp_apic_timer_interrupt+0xa6/0x270 [14540.198762] ? apic_timer_interrupt+0xf/0x20 [14540.200735] [14540.201713] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14540.204117] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14540.209558] do_raw_spin_lock+0x177/0x190 [14540.210939] _raw_spin_lock+0x21/0x30 [14540.213270] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14540.216717] ? do_raw_spin_unlock+0x75/0x190 [14540.220104] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14540.224117] ? do_wait_intr+0xf0/0xf0 [14540.225902] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14540.233688] mdc_reint+0x2d/0x110 [mdc] [14540.237334] mdc_unlink+0x658/0x870 [mdc] [14540.243178] lmv_unlink+0x34d/0xbd0 [lmv] [14540.244625] ll_unlink+0x24d/0x7f0 [lustre] [14540.249605] vfs_unlink+0x202/0x330 [14540.252066] do_unlinkat+0x2fa/0x460 [14540.255263] __x64_sys_unlink+0x37/0x50 [14540.257854] do_syscall_64+0xc1/0x440 [14540.260610] entry_SYSCALL_64_after_hwframe+0x49/0xae [14540.262506] RIP: 0033:0x7f151bdee10b [14540.264662] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [14540.275619] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [14540.280834] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [14540.285800] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [14540.288110] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14540.292977] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14540.300788] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14544.055068] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [14544.058871] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [14544.062224] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14544.062303] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14544.062311] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14544.062314] RIP: 0010:native_safe_halt+0x2e/0x50 [14544.062327] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14544.062331] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14544.062335] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [14544.062337] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [14544.062340] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [14544.062342] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [14544.062345] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14544.062348] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [14544.062351] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14544.062353] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14544.062362] Call Trace: [14544.062368] [14544.062374] ? show_regs.cold.9+0x22/0x2f [14544.062384] ? watchdog_timer_fn.cold.6+0x62/0x10a [14544.062390] ? __hrtimer_run_queues+0x1c2/0x4d0 [14544.062395] ? watchdog+0x40/0x40 [14544.062400] ? hrtimer_interrupt+0x128/0x2c0 [14544.062404] ? smp_apic_timer_interrupt+0xa6/0x270 [14544.062411] ? apic_timer_interrupt+0xf/0x20 [14544.062416] [14544.062419] ? native_safe_halt+0x2e/0x50 [14544.062422] kvm_wait+0xcd/0xe0 [14544.062430] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14544.062436] do_raw_spin_lock+0x177/0x190 [14544.062440] _raw_spin_lock+0x21/0x30 [14544.062446] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14544.062786] obd_mod_free+0x46/0x140 [ptlrpc] [14544.063549] mdc_commit_open+0x91/0xb0 [mdc] [14544.063587] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14544.068042] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14544.068131] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14544.068137] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14544.068140] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14544.068153] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14544.068157] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14544.068161] RAX: 0000000000007e23 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14544.068164] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [14544.068166] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [14544.068169] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [14544.068171] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [14544.068175] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [14544.068178] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14544.068180] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [14544.068188] Call Trace: [14544.068193] [14544.068198] ? show_regs.cold.9+0x22/0x2f [14544.068207] ? watchdog_timer_fn.cold.6+0x62/0x10a [14544.068213] ? __hrtimer_run_queues+0x1c2/0x4d0 [14544.068217] ? watchdog+0x40/0x40 [14544.068222] ? hrtimer_interrupt+0x128/0x2c0 [14544.068226] ? smp_apic_timer_interrupt+0xa6/0x270 [14544.068233] ? apic_timer_interrupt+0xf/0x20 [14544.068238] [14544.068241] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14544.068247] do_raw_spin_lock+0x177/0x190 [14544.068251] _raw_spin_lock+0x21/0x30 [14544.068255] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14544.068581] ? do_raw_spin_unlock+0x75/0x190 [14544.068587] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14544.068786] ? do_wait_intr+0xf0/0xf0 [14544.068793] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14544.068993] mdc_close+0x53b/0xd70 [mdc] [14544.069035] lmv_close+0x18d/0x490 [lmv] [14544.069058] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14544.069240] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14544.069354] ll_md_real_close+0xa5/0x2a0 [lustre] [14544.069461] ll_md_close+0x274/0x800 [lustre] [14544.069550] ll_file_release+0x128/0xc50 [lustre] [14544.069640] __fput+0x10d/0x440 [14544.069649] ____fput+0x12/0x20 [14544.069653] task_work_run+0xc6/0x110 [14544.069662] exit_to_usermode_loop+0x1dd/0x1f0 [14544.069668] do_syscall_64+0x3d6/0x440 [14544.069672] entry_SYSCALL_64_after_hwframe+0x49/0xae [14544.069679] RIP: 0033:0x7f151bee3cd8 [14544.069685] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14544.069689] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14544.069693] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [14544.069695] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [14544.069697] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14544.069700] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14544.069702] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14544.699169] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14544.704664] after_reply+0xe2c/0x1780 [ptlrpc] [14544.710663] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14544.713693] ? __next_timer_interrupt+0x160/0x160 [14544.716132] ptlrpcd+0xa82/0xe40 [ptlrpc] [14544.717967] ? do_wait_intr+0xf0/0xf0 [14544.728730] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14544.741565] kthread+0x1d1/0x200 [14544.742828] ? set_kthread_struct+0x70/0x70 [14544.749766] ret_from_fork+0x1f/0x30 [14545.900566] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12615s! [14545.919013] Showing busy workqueues and worker pools: [14545.923392] workqueue events: flags=0x0 [14545.927126] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14545.937006] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [14545.945697] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14545.953065] pending: kfree_rcu_monitor [14545.961553] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14545.966711] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [14545.981930] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14545.986202] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [14545.989896] workqueue events_long: flags=0x0 [14545.994614] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14545.999907] pending: lu_objects_flush [obdclass] [14546.006106] workqueue events_unbound: flags=0x2 [14546.010539] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14546.014619] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14546.025932] workqueue events_power_efficient: flags=0x80 [14546.028569] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14546.035342] pending: do_cache_clean, neigh_periodic_work, check_lifetime [14546.042218] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14546.050576] pending: neigh_periodic_work [14546.055435] workqueue mm_percpu_wq: flags=0x8 [14546.059618] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14546.072382] pending: vmstat_update, lru_add_drain_per_cpu [14546.076850] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14546.079220] pending: vmstat_update [14546.090402] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14546.096764] pending: lru_add_drain_per_cpu [14546.098479] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [14546.108181] pending: lru_add_drain_per_cpu BAR(48) [14546.112639] workqueue writeback: flags=0x4a [14546.115575] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [14546.122692] pending: wb_workfn [14546.126163] workqueue xprtiod: flags=0xa [14546.127568] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14546.134517] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [14546.151360] workqueue ipv6_addrconf: flags=0x40008 [14546.157310] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14546.166586] pending: addrconf_verify_work [14546.170396] workqueue knbd0-recv: flags=0x1a [14546.179462] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14546.188829] in-flight: 658:recv_work [14546.192621] workqueue ptlrpc_pinger: flags=0x2 [14546.200295] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14546.203751] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14546.213622] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12615s workers=4 idle: 32984 37 8 [14546.224544] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14530s workers=2 idle: 661 [14560.063005] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [14560.068960] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14560.135359] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14560.146325] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14560.153371] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14560.161004] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14560.167319] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14560.193086] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14560.217385] RAX: 0000000000003394 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14560.237727] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [14560.252077] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [14560.263302] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [14560.271193] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [14560.276012] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [14560.281540] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14560.285763] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14560.296897] Call Trace: [14560.301807] [14560.307346] ? show_regs.cold.9+0x22/0x2f [14560.315397] ? watchdog_timer_fn.cold.6+0x62/0x10a [14560.327317] ? __hrtimer_run_queues+0x1c2/0x4d0 [14560.341097] ? watchdog+0x40/0x40 [14560.348216] ? hrtimer_interrupt+0x128/0x2c0 [14560.353238] ? smp_apic_timer_interrupt+0xa6/0x270 [14560.365586] ? apic_timer_interrupt+0xf/0x20 [14560.369668] [14560.376418] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14560.381758] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14560.390606] do_raw_spin_lock+0x177/0x190 [14560.397330] _raw_spin_lock+0x21/0x30 [14560.398561] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14560.402353] process_one_work+0x2c8/0x700 [14560.405106] worker_thread+0x296/0x6e0 [14560.411648] ? rescuer_thread+0x570/0x570 [14560.415258] kthread+0x1d1/0x200 [14560.421128] ? set_kthread_struct+0x70/0x70 [14560.432625] ret_from_fork+0x1f/0x30 [14572.054699] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [14572.061284] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14572.062683] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [14572.067058] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [14572.067082] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14572.067152] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14572.067156] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14572.067159] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14572.067175] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14572.067180] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14572.067184] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [14572.067186] RDX: 0000000000005da1 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [14572.067189] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [14572.067191] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [14572.067193] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14572.067197] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [14572.067200] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14572.067203] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14572.067212] Call Trace: [14572.067218] [14572.067234] ? show_regs.cold.9+0x22/0x2f [14572.067244] ? watchdog_timer_fn.cold.6+0x62/0x10a [14572.067251] ? __hrtimer_run_queues+0x1c2/0x4d0 [14572.067256] ? watchdog+0x40/0x40 [14572.067260] ? hrtimer_interrupt+0x128/0x2c0 [14572.067264] ? smp_apic_timer_interrupt+0xa6/0x270 [14572.067272] ? apic_timer_interrupt+0xf/0x20 [14572.067276] [14572.067278] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14572.067284] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14572.067288] do_raw_spin_lock+0x177/0x190 [14572.067293] _raw_spin_lock+0x21/0x30 [14572.067299] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14572.067789] obd_mod_free+0x46/0x140 [ptlrpc] [14572.068184] mdc_commit_open+0x91/0xb0 [mdc] [14572.068284] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14572.068671] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14572.069342] after_reply+0xe2c/0x1780 [ptlrpc] [14572.069665] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14572.070438] ? __next_timer_interrupt+0x160/0x160 [14572.070449] ptlrpcd+0xa82/0xe40 [ptlrpc] [14572.070731] ? do_wait_intr+0xf0/0xf0 [14572.070739] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14572.072751] kthread+0x1d1/0x200 [14572.072759] ? set_kthread_struct+0x70/0x70 [14572.072763] ret_from_fork+0x1f/0x30 [14572.112773] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14572.112788] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14572.112791] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14572.112813] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14572.112827] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14572.112833] RAX: 0000000000002a88 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14572.112836] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [14572.112838] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [14572.112840] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [14572.112842] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [14572.112851] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [14572.112855] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14572.112858] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [14572.112867] Call Trace: [14572.112872] [14572.112880] ? show_regs.cold.9+0x22/0x2f [14572.112889] ? watchdog_timer_fn.cold.6+0x62/0x10a [14572.112896] ? __hrtimer_run_queues+0x1c2/0x4d0 [14572.112901] ? watchdog+0x40/0x40 [14572.112905] ? hrtimer_interrupt+0x128/0x2c0 [14572.112908] ? smp_apic_timer_interrupt+0xa6/0x270 [14572.112924] ? apic_timer_interrupt+0xf/0x20 [14572.112929] [14572.112932] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14572.112938] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14572.112946] do_raw_spin_lock+0x177/0x190 [14572.112951] _raw_spin_lock+0x21/0x30 [14572.112959] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14572.113646] ? do_raw_spin_unlock+0x75/0x190 [14572.113653] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14572.114026] ? do_wait_intr+0xf0/0xf0 [14572.114034] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14572.114471] mdc_reint+0x2d/0x110 [mdc] [14572.114523] mdc_unlink+0x658/0x870 [mdc] [14572.114551] lmv_unlink+0x34d/0xbd0 [lmv] [14572.114583] ll_unlink+0x24d/0x7f0 [lustre] [14572.114767] vfs_unlink+0x202/0x330 [14572.114776] do_unlinkat+0x2fa/0x460 [14572.114781] __x64_sys_unlink+0x37/0x50 [14572.114785] do_syscall_64+0xc1/0x440 [14572.114791] entry_SYSCALL_64_after_hwframe+0x49/0xae [14572.114797] RIP: 0033:0x7f151bdee10b [14572.114805] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [14572.114808] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [14572.114813] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [14572.114816] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [14572.114829] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14572.114831] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14572.114833] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14572.948602] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14573.013582] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14573.023585] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14573.033829] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14573.041216] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14573.069339] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14573.073564] RAX: 000000000000660a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14573.079609] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [14573.086318] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [14573.095322] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [14573.109209] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [14573.118456] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [14573.123186] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14573.130206] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [14573.133984] Call Trace: [14573.135096] [14573.136106] ? show_regs.cold.9+0x22/0x2f [14573.140999] ? watchdog_timer_fn.cold.6+0x62/0x10a [14573.144962] ? __hrtimer_run_queues+0x1c2/0x4d0 [14573.149867] ? watchdog+0x40/0x40 [14573.155523] ? hrtimer_interrupt+0x128/0x2c0 [14573.157148] ? smp_apic_timer_interrupt+0xa6/0x270 [14573.162381] ? apic_timer_interrupt+0xf/0x20 [14573.168171] [14573.171686] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14573.173727] do_raw_spin_lock+0x177/0x190 [14573.177237] _raw_spin_lock+0x21/0x30 [14573.179186] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14573.182162] ? do_raw_spin_unlock+0x75/0x190 [14573.186066] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14573.198340] ? do_wait_intr+0xf0/0xf0 [14573.206551] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14573.215851] mdc_close+0x53b/0xd70 [mdc] [14573.225203] lmv_close+0x18d/0x490 [lmv] [14573.226616] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14573.234399] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14573.244041] ll_md_real_close+0xa5/0x2a0 [lustre] [14573.255745] ll_md_close+0x274/0x800 [lustre] [14573.257426] ll_file_release+0x128/0xc50 [lustre] [14573.269945] __fput+0x10d/0x440 [14573.277043] ____fput+0x12/0x20 [14573.278560] task_work_run+0xc6/0x110 [14573.279956] exit_to_usermode_loop+0x1dd/0x1f0 [14573.283615] do_syscall_64+0x3d6/0x440 [14573.288200] entry_SYSCALL_64_after_hwframe+0x49/0xae [14573.297697] RIP: 0033:0x7f151bee3cd8 [14573.298988] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14573.332802] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14573.335522] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [14573.353646] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [14573.361642] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14573.369992] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14573.378984] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14576.614928] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12646s! [14576.632914] Showing busy workqueues and worker pools: [14576.639908] workqueue events: flags=0x0 [14576.644583] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14576.654078] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [14576.666248] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14576.671783] pending: kfree_rcu_monitor [14576.673835] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14576.682238] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [14576.686791] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14576.690923] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [14576.695662] workqueue events_long: flags=0x0 [14576.697867] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14576.706848] pending: lu_objects_flush [obdclass] [14576.708617] workqueue events_unbound: flags=0x2 [14576.719798] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14576.721315] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14576.735420] workqueue events_power_efficient: flags=0x80 [14576.742095] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14576.754042] pending: do_cache_clean, neigh_periodic_work, check_lifetime [14576.759879] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14576.767856] pending: neigh_periodic_work [14576.780620] workqueue mm_percpu_wq: flags=0x8 [14576.781838] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14576.783711] pending: vmstat_update, lru_add_drain_per_cpu [14576.794872] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14576.804201] pending: vmstat_update [14576.806664] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14576.814129] pending: lru_add_drain_per_cpu [14576.819130] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [14576.826577] pending: lru_add_drain_per_cpu BAR(48) [14576.830136] workqueue writeback: flags=0x4a [14576.833704] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [14576.841099] pending: wb_workfn [14576.843217] workqueue xprtiod: flags=0xa [14576.845097] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14576.847836] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [14576.856919] workqueue ipv6_addrconf: flags=0x40008 [14576.858588] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14576.863566] pending: addrconf_verify_work [14576.867243] workqueue knbd0-recv: flags=0x1a [14576.876872] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14576.887069] in-flight: 658:recv_work [14576.891325] workqueue ptlrpc_pinger: flags=0x2 [14576.893059] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14576.900069] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14576.903979] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12646s workers=4 idle: 32984 37 8 [14576.912154] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14561s workers=2 idle: 661 [14584.917014] rcu: INFO: rcu_sched self-detected stall on CPU [14584.920717] rcu: 1-...!: (7488251 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [14584.927407] (t=12660270 jiffies g=249657 q=6284) [14584.930153] rcu: rcu_sched kthread starved for 12655222 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [14584.935790] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [14584.942102] rcu: RCU grace-period kthread stack dump: [14584.946322] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [14584.952361] Call Trace: [14584.953900] __schedule+0x351/0xcb0 [14584.955891] schedule+0xc0/0x180 [14584.957213] schedule_timeout+0xb4/0x190 [14584.958981] ? do_raw_spin_unlock+0x75/0x190 [14584.960435] ? __next_timer_interrupt+0x160/0x160 [14584.962351] ? prepare_to_swait_event+0xf3/0x140 [14584.964263] rcu_gp_kthread+0x8dd/0x15b0 [14584.965928] ? force_qs_rnp+0x2c0/0x2c0 [14584.967608] kthread+0x1d1/0x200 [14584.969257] ? set_kthread_struct+0x70/0x70 [14584.971217] ret_from_fork+0x1f/0x30 [14584.972826] rcu: Stack dump where RCU GP kthread last ran: [14584.975266] Sending NMI from CPU 1 to CPUs 2: [14584.977775] NMI backtrace for cpu 2 [14584.977781] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14584.977783] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14584.977785] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14584.977788] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14584.977789] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14584.977791] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 [14584.977793] RAX: 0000000000004d0d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14584.977795] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [14584.977796] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [14584.977797] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [14584.977798] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [14584.977800] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [14584.977801] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14584.977802] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14584.977804] Call Trace: [14584.977804] [14584.977805] ? show_regs.cold.9+0x22/0x2f [14584.977807] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [14584.977808] ? nmi_cpu_backtrace_handler+0x15/0x40 [14584.977809] ? nmi_handle+0xbd/0x210 [14584.977810] ? default_do_nmi+0x4b/0x1b0 [14584.977811] ? do_nmi+0x280/0x380 [14584.977812] ? end_repeat_nmi+0x16/0x1f [14584.977813] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14584.977814] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14584.977815] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14584.977816] [14584.977817] do_raw_spin_lock+0x177/0x190 [14584.977818] _raw_spin_lock+0x21/0x30 [14584.977818] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14584.977820] process_one_work+0x2c8/0x700 [14584.977820] worker_thread+0x296/0x6e0 [14584.977826] ? rescuer_thread+0x570/0x570 [14584.977827] kthread+0x1d1/0x200 [14584.977828] ? set_kthread_struct+0x70/0x70 [14584.977829] ret_from_fork+0x1f/0x30 [14584.977921] NMI backtrace for cpu 1 [14585.077637] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14585.083154] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14585.087384] Call Trace: [14585.088668] [14585.089364] ? dump_stack+0xbb/0x10e [14585.091050] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [14585.093685] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [14585.097042] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [14585.099432] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [14585.102213] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [14585.104821] ? rcu_dump_cpu_stacks+0x11b/0x171 [14585.106965] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [14585.109743] ? update_process_times+0x6d/0xc0 [14585.111897] ? tick_sched_handle.isra.18+0x27/0x90 [14585.113882] ? tick_sched_timer+0x53/0xb0 [14585.115837] ? __hrtimer_run_queues+0x1c2/0x4d0 [14585.117952] ? tick_sched_handle.isra.18+0x90/0x90 [14585.120078] ? hrtimer_interrupt+0x128/0x2c0 [14585.122401] ? smp_apic_timer_interrupt+0xa6/0x270 [14585.124412] ? apic_timer_interrupt+0xf/0x20 [14585.126781] [14585.127872] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14585.130957] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14585.133814] ? do_raw_spin_lock+0x177/0x190 [14585.135782] ? _raw_spin_lock+0x21/0x30 [14585.137469] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14585.140014] ? obd_mod_free+0x46/0x140 [ptlrpc] [14585.142665] ? mdc_commit_open+0x91/0xb0 [mdc] [14585.144753] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14585.147847] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14585.150756] ? after_reply+0xe2c/0x1780 [ptlrpc] [14585.153325] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14585.156926] ? __next_timer_interrupt+0x160/0x160 [14585.159486] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [14585.161410] ? do_wait_intr+0xf0/0xf0 [14585.163310] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14585.168115] ? kthread+0x1d1/0x200 [14585.170689] ? set_kthread_struct+0x70/0x70 [14585.174130] ? ret_from_fork+0x1f/0x30 [14600.057129] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [14600.070396] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14600.100905] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14600.106849] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14600.112790] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14600.115827] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14600.125689] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14600.130314] RAX: 0000000000001e04 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14600.133559] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [14600.137887] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [14600.141271] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [14600.145525] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [14600.150062] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [14600.154961] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14600.158979] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [14600.163226] Call Trace: [14600.164750] [14600.165916] ? show_regs.cold.9+0x22/0x2f [14600.170865] ? watchdog_timer_fn.cold.6+0x62/0x10a [14600.173173] ? __hrtimer_run_queues+0x1c2/0x4d0 [14600.175297] ? watchdog+0x40/0x40 [14600.177712] ? hrtimer_interrupt+0x128/0x2c0 [14600.179805] ? smp_apic_timer_interrupt+0xa6/0x270 [14600.182098] ? apic_timer_interrupt+0xf/0x20 [14600.184046] [14600.185043] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14600.188130] do_raw_spin_lock+0x177/0x190 [14600.190121] _raw_spin_lock+0x21/0x30 [14600.191947] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14600.194087] ? do_raw_spin_unlock+0x75/0x190 [14600.196229] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14600.199461] ? do_wait_intr+0xf0/0xf0 [14600.201485] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14600.203901] mdc_close+0x53b/0xd70 [mdc] [14600.206094] lmv_close+0x18d/0x490 [lmv] [14600.208060] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14600.210789] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14600.213933] ll_md_real_close+0xa5/0x2a0 [lustre] [14600.215879] ll_md_close+0x274/0x800 [lustre] [14600.217876] ll_file_release+0x128/0xc50 [lustre] [14600.220045] __fput+0x10d/0x440 [14600.222422] ____fput+0x12/0x20 [14600.224175] task_work_run+0xc6/0x110 [14600.226454] exit_to_usermode_loop+0x1dd/0x1f0 [14600.228862] do_syscall_64+0x3d6/0x440 [14600.230871] entry_SYSCALL_64_after_hwframe+0x49/0xae [14600.233028] RIP: 0033:0x7f151bee3cd8 [14600.234612] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14600.243076] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14600.246052] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [14600.248641] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [14600.251296] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14600.253494] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14600.255697] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14604.052013] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [14604.058941] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14604.132952] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14604.152488] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14604.169569] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14604.177124] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14604.203351] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14604.220738] RAX: 000000000000704e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14604.225992] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [14604.233981] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [14604.244181] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [14604.255573] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [14604.271947] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [14604.274018] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14604.281469] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [14604.283647] Call Trace: [14604.284311] [14604.287092] ? show_regs.cold.9+0x22/0x2f [14604.289162] ? watchdog_timer_fn.cold.6+0x62/0x10a [14604.303265] ? __hrtimer_run_queues+0x1c2/0x4d0 [14604.307847] ? watchdog+0x40/0x40 [14604.309716] ? hrtimer_interrupt+0x128/0x2c0 [14604.310903] ? smp_apic_timer_interrupt+0xa6/0x270 [14604.315179] ? apic_timer_interrupt+0xf/0x20 [14604.319093] [14604.319791] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14604.321992] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14604.324947] do_raw_spin_lock+0x177/0x190 [14604.327064] _raw_spin_lock+0x21/0x30 [14604.328944] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14604.331740] ? do_raw_spin_unlock+0x75/0x190 [14604.333117] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14604.335804] ? do_wait_intr+0xf0/0xf0 [14604.338354] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14604.343108] mdc_reint+0x2d/0x110 [mdc] [14604.345299] mdc_unlink+0x658/0x870 [mdc] [14604.347365] lmv_unlink+0x34d/0xbd0 [lmv] [14604.349587] ll_unlink+0x24d/0x7f0 [lustre] [14604.351343] vfs_unlink+0x202/0x330 [14604.352318] do_unlinkat+0x2fa/0x460 [14604.353727] __x64_sys_unlink+0x37/0x50 [14604.355515] do_syscall_64+0xc1/0x440 [14604.357622] entry_SYSCALL_64_after_hwframe+0x49/0xae [14604.359120] RIP: 0033:0x7f151bdee10b [14604.360124] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [14604.368355] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [14604.371470] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [14604.375059] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [14604.377457] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14604.380268] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14604.382682] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14607.334802] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12676s! [14607.343255] Showing busy workqueues and worker pools: [14607.346251] workqueue events: flags=0x0 [14607.348764] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14607.351544] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [14607.356447] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14607.359447] pending: kfree_rcu_monitor [14607.361427] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14607.364546] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [14607.368972] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14607.372052] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [14607.375328] workqueue events_long: flags=0x0 [14607.377464] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14607.380567] pending: lu_objects_flush [obdclass] [14607.382959] workqueue events_unbound: flags=0x2 [14607.384953] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14607.387577] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14607.392154] workqueue events_power_efficient: flags=0x80 [14607.394391] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14607.398127] pending: do_cache_clean, neigh_periodic_work, check_lifetime [14607.401673] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14607.406339] pending: neigh_periodic_work [14607.408258] workqueue mm_percpu_wq: flags=0x8 [14607.410226] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14607.413222] pending: vmstat_update, lru_add_drain_per_cpu [14607.415807] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14607.418558] pending: vmstat_update [14607.420274] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14607.423415] pending: lru_add_drain_per_cpu [14607.425474] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [14607.428752] pending: lru_add_drain_per_cpu BAR(48) [14607.430967] workqueue writeback: flags=0x4a [14607.432900] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [14607.435911] pending: wb_workfn [14607.437217] workqueue xprtiod: flags=0xa [14607.439977] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14607.443897] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [14607.447164] workqueue ipv6_addrconf: flags=0x40008 [14607.449253] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14607.452244] pending: addrconf_verify_work [14607.454199] workqueue knbd0-recv: flags=0x1a [14607.456014] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14607.458786] in-flight: 658:recv_work [14607.460563] workqueue ptlrpc_pinger: flags=0x2 [14607.462611] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14607.465563] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14607.468819] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12676s workers=4 idle: 32984 37 8 [14607.472164] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14592s workers=2 idle: 661 [14612.055651] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [14612.056238] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14612.067144] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14612.067221] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14612.073768] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14612.073860] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14612.073867] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14612.073916] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14612.074238] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14612.074247] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14612.074251] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14612.074256] RAX: 0000000000006458 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14612.074259] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [14612.074261] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [14612.074264] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [14612.074266] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [14612.074270] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [14612.074273] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14612.074275] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14612.074283] Call Trace: [14612.074287] [14612.074640] ? show_regs.cold.9+0x22/0x2f [14612.074651] ? watchdog_timer_fn.cold.6+0x62/0x10a [14612.074657] ? __hrtimer_run_queues+0x1c2/0x4d0 [14612.074661] ? watchdog+0x40/0x40 [14612.074665] ? hrtimer_interrupt+0x128/0x2c0 [14612.074669] ? smp_apic_timer_interrupt+0xa6/0x270 [14612.074685] ? apic_timer_interrupt+0xf/0x20 [14612.074691] [14612.074694] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14612.074698] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14612.074703] do_raw_spin_lock+0x177/0x190 [14612.074707] _raw_spin_lock+0x21/0x30 [14612.074711] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14612.114874] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14612.114885] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14612.114900] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14612.114905] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14612.114912] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [14612.114914] RDX: 00000000000043c4 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [14612.114917] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [14612.114919] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [14612.114922] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14612.114925] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [14612.114928] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14612.114930] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14612.121943] process_one_work+0x2c8/0x700 [14612.121963] worker_thread+0x296/0x6e0 [14612.121969] ? rescuer_thread+0x570/0x570 [14612.121973] kthread+0x1d1/0x200 [14612.121978] ? set_kthread_struct+0x70/0x70 [14612.121982] ret_from_fork+0x1f/0x30 [14612.333939] Call Trace: [14612.334774] [14612.335743] ? show_regs.cold.9+0x22/0x2f [14612.338097] ? watchdog_timer_fn.cold.6+0x62/0x10a [14612.340282] ? __hrtimer_run_queues+0x1c2/0x4d0 [14612.342602] ? watchdog+0x40/0x40 [14612.344236] ? hrtimer_interrupt+0x128/0x2c0 [14612.346618] ? smp_apic_timer_interrupt+0xa6/0x270 [14612.349113] ? apic_timer_interrupt+0xf/0x20 [14612.352545] [14612.353385] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14612.356381] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14612.359216] do_raw_spin_lock+0x177/0x190 [14612.361380] _raw_spin_lock+0x21/0x30 [14612.363643] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14612.365248] obd_mod_free+0x46/0x140 [ptlrpc] [14612.367803] mdc_commit_open+0x91/0xb0 [mdc] [14612.371493] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14612.373233] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14612.375669] after_reply+0xe2c/0x1780 [ptlrpc] [14612.378020] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14612.380246] ? __next_timer_interrupt+0x160/0x160 [14612.383069] ptlrpcd+0xa82/0xe40 [ptlrpc] [14612.385179] ? do_wait_intr+0xf0/0xf0 [14612.390684] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14612.393079] kthread+0x1d1/0x200 [14612.394943] ? set_kthread_struct+0x70/0x70 [14612.396570] ret_from_fork+0x1f/0x30 [14628.057033] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [14628.059701] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14628.081514] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14628.086070] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14628.089817] RIP: 0010:native_safe_halt+0x2e/0x50 [14628.092162] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14628.100105] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14628.102999] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14628.107762] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [14628.110974] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [14628.114515] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [14628.118104] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [14628.122027] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [14628.127111] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14628.130286] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [14628.133809] Call Trace: [14628.135294] [14628.136180] ? show_regs.cold.9+0x22/0x2f [14628.138851] ? watchdog_timer_fn.cold.6+0x62/0x10a [14628.141373] ? __hrtimer_run_queues+0x1c2/0x4d0 [14628.143638] ? watchdog+0x40/0x40 [14628.146293] ? hrtimer_interrupt+0x128/0x2c0 [14628.149645] ? smp_apic_timer_interrupt+0xa6/0x270 [14628.153151] ? apic_timer_interrupt+0xf/0x20 [14628.156481] [14628.158045] ? native_safe_halt+0x2e/0x50 [14628.160862] kvm_wait+0xcd/0xe0 [14628.163152] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14628.165956] do_raw_spin_lock+0x177/0x190 [14628.168635] _raw_spin_lock+0x21/0x30 [14628.171420] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14628.175381] ? do_raw_spin_unlock+0x75/0x190 [14628.178050] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14628.182351] ? do_wait_intr+0xf0/0xf0 [14628.184406] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14628.188140] mdc_close+0x53b/0xd70 [mdc] [14628.189968] lmv_close+0x18d/0x490 [lmv] [14628.192089] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14628.194566] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14628.197416] ll_md_real_close+0xa5/0x2a0 [lustre] [14628.200072] ll_md_close+0x274/0x800 [lustre] [14628.202547] ll_file_release+0x128/0xc50 [lustre] [14628.205429] __fput+0x10d/0x440 [14628.206963] ____fput+0x12/0x20 [14628.210417] task_work_run+0xc6/0x110 [14628.212418] exit_to_usermode_loop+0x1dd/0x1f0 [14628.215919] do_syscall_64+0x3d6/0x440 [14628.217540] entry_SYSCALL_64_after_hwframe+0x49/0xae [14628.219921] RIP: 0033:0x7f151bee3cd8 [14628.221740] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14628.230109] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14628.233932] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [14628.237649] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [14628.241101] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14628.244550] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14628.247583] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14632.057698] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [14632.080609] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14632.149329] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14632.164065] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14632.175787] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14632.178481] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14632.197000] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14632.200469] RAX: 0000000000007924 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14632.202877] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [14632.205908] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [14632.208525] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [14632.211036] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [14632.213492] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [14632.219168] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14632.223004] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [14632.228433] Call Trace: [14632.229090] [14632.233304] ? show_regs.cold.9+0x22/0x2f [14632.238553] ? watchdog_timer_fn.cold.6+0x62/0x10a [14632.245875] ? __hrtimer_run_queues+0x1c2/0x4d0 [14632.252458] ? watchdog+0x40/0x40 [14632.260303] ? hrtimer_interrupt+0x128/0x2c0 [14632.264478] ? smp_apic_timer_interrupt+0xa6/0x270 [14632.269948] ? apic_timer_interrupt+0xf/0x20 [14632.274549] [14632.275312] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14632.286248] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14632.297388] do_raw_spin_lock+0x177/0x190 [14632.298856] _raw_spin_lock+0x21/0x30 [14632.299968] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14632.305870] ? do_raw_spin_unlock+0x75/0x190 [14632.316810] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14632.320676] ? do_wait_intr+0xf0/0xf0 [14632.326092] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14632.330644] mdc_reint+0x2d/0x110 [mdc] [14632.332067] mdc_unlink+0x658/0x870 [mdc] [14632.334843] lmv_unlink+0x34d/0xbd0 [lmv] [14632.336697] ll_unlink+0x24d/0x7f0 [lustre] [14632.338387] vfs_unlink+0x202/0x330 [14632.341709] do_unlinkat+0x2fa/0x460 [14632.352172] __x64_sys_unlink+0x37/0x50 [14632.363841] do_syscall_64+0xc1/0x440 [14632.365697] entry_SYSCALL_64_after_hwframe+0x49/0xae [14632.373641] RIP: 0033:0x7f151bdee10b [14632.383462] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [14632.420820] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [14632.436569] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [14632.454915] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [14632.471980] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14632.487530] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14632.501035] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14637.536675] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12706s! [14637.538668] Showing busy workqueues and worker pools: [14637.539848] workqueue events: flags=0x0 [14637.546599] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14637.548152] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [14637.553853] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14637.556933] pending: kfree_rcu_monitor [14637.559832] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14637.562778] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [14637.566946] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14637.570687] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [14637.575943] workqueue events_long: flags=0x0 [14637.579091] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14637.583041] pending: lu_objects_flush [obdclass] [14637.592860] workqueue events_unbound: flags=0x2 [14637.594939] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14637.596890] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14637.601336] workqueue events_power_efficient: flags=0x80 [14637.604109] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14637.608512] pending: do_cache_clean, neigh_periodic_work, check_lifetime [14637.611773] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14637.613523] pending: neigh_periodic_work [14637.616255] workqueue mm_percpu_wq: flags=0x8 [14637.617953] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14637.621370] pending: vmstat_update, lru_add_drain_per_cpu [14637.623923] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14637.627513] pending: vmstat_update [14637.629220] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14637.636516] pending: lru_add_drain_per_cpu [14637.637527] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [14637.638974] pending: lru_add_drain_per_cpu BAR(48) [14637.641582] workqueue writeback: flags=0x4a [14637.644358] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [14637.647430] pending: wb_workfn [14637.648920] workqueue xprtiod: flags=0xa [14637.651019] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14637.657463] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [14637.662631] workqueue ipv6_addrconf: flags=0x40008 [14637.664678] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14637.667045] pending: addrconf_verify_work [14637.669851] workqueue knbd0-recv: flags=0x1a [14637.671144] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14637.674938] in-flight: 658:recv_work [14637.676016] workqueue ptlrpc_pinger: flags=0x2 [14637.679393] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14637.682038] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14637.687031] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12707s workers=4 idle: 32984 37 8 [14637.690949] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14622s workers=2 idle: 661 [14640.055254] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [14640.056010] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14640.059071] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) [14640.062230] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14640.062317] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14640.062324] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14640.062330] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14640.062928] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14640.062940] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14640.062944] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14640.062948] RAX: 0000000000007587 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14640.062951] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [14640.062953] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [14640.062959] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [14640.062961] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [14640.062964] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [14640.062967] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14640.062970] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14640.062978] Call Trace: [14640.062983] [14640.062992] ? show_regs.cold.9+0x22/0x2f [14640.063001] ? watchdog_timer_fn.cold.6+0x62/0x10a [14640.063007] ? __hrtimer_run_queues+0x1c2/0x4d0 [14640.063012] ? watchdog+0x40/0x40 [14640.063016] ? hrtimer_interrupt+0x128/0x2c0 [14640.063031] ? smp_apic_timer_interrupt+0xa6/0x270 [14640.063040] ? apic_timer_interrupt+0xf/0x20 [14640.063052] [14640.063055] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14640.063060] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14640.063065] do_raw_spin_lock+0x177/0x190 [14640.063070] _raw_spin_lock+0x21/0x30 [14640.063075] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14640.063414] process_one_work+0x2c8/0x700 [14640.063421] worker_thread+0x296/0x6e0 [14640.063425] ? rescuer_thread+0x570/0x570 [14640.063434] kthread+0x1d1/0x200 [14640.063438] ? set_kthread_struct+0x70/0x70 [14640.063442] ret_from_fork+0x1f/0x30 [14640.175202] lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14640.192442] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14640.196982] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14640.200378] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14640.202754] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14640.210153] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14640.212983] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [14640.215668] RDX: 0000000000001025 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [14640.218174] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [14640.220830] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [14640.223536] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14640.226471] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [14640.229719] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14640.231654] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14640.234365] Call Trace: [14640.235346] [14640.236143] ? show_regs.cold.9+0x22/0x2f [14640.237715] ? watchdog_timer_fn.cold.6+0x62/0x10a [14640.239880] ? __hrtimer_run_queues+0x1c2/0x4d0 [14640.242133] ? watchdog+0x40/0x40 [14640.244730] ? hrtimer_interrupt+0x128/0x2c0 [14640.246389] ? smp_apic_timer_interrupt+0xa6/0x270 [14640.250396] ? apic_timer_interrupt+0xf/0x20 [14640.252865] [14640.254942] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14640.257478] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14640.259487] do_raw_spin_lock+0x177/0x190 [14640.260751] _raw_spin_lock+0x21/0x30 [14640.262304] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14640.264540] obd_mod_free+0x46/0x140 [ptlrpc] [14640.270410] mdc_commit_open+0x91/0xb0 [mdc] [14640.272840] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14640.278297] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14640.282722] after_reply+0xe2c/0x1780 [ptlrpc] [14640.284906] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14640.287079] ? __next_timer_interrupt+0x160/0x160 [14640.288676] ptlrpcd+0xa82/0xe40 [ptlrpc] [14640.291439] ? do_wait_intr+0xf0/0xf0 [14640.293303] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14640.295518] kthread+0x1d1/0x200 [14640.297189] ? set_kthread_struct+0x70/0x70 [14640.298712] ret_from_fork+0x1f/0x30 [14656.057050] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [14656.063600] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14656.097480] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14656.109731] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14656.114891] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14656.120469] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14656.134835] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14656.139935] RAX: 0000000000004bf4 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14656.145938] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [14656.152673] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [14656.158694] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [14656.164325] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [14656.169819] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [14656.174756] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14656.177933] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [14656.182532] Call Trace: [14656.185268] [14656.186309] ? show_regs.cold.9+0x22/0x2f [14656.189108] ? watchdog_timer_fn.cold.6+0x62/0x10a [14656.191248] ? __hrtimer_run_queues+0x1c2/0x4d0 [14656.194466] ? watchdog+0x40/0x40 [14656.197148] ? hrtimer_interrupt+0x128/0x2c0 [14656.199541] ? smp_apic_timer_interrupt+0xa6/0x270 [14656.201557] ? apic_timer_interrupt+0xf/0x20 [14656.203189] [14656.204390] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14656.207549] do_raw_spin_lock+0x177/0x190 [14656.209676] _raw_spin_lock+0x21/0x30 [14656.211579] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14656.215174] ? do_raw_spin_unlock+0x75/0x190 [14656.217536] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14656.221290] ? do_wait_intr+0xf0/0xf0 [14656.223599] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14656.227910] mdc_close+0x53b/0xd70 [mdc] [14656.230474] lmv_close+0x18d/0x490 [lmv] [14656.232368] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14656.235676] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14656.238165] ll_md_real_close+0xa5/0x2a0 [lustre] [14656.241086] ll_md_close+0x274/0x800 [lustre] [14656.243482] ll_file_release+0x128/0xc50 [lustre] [14656.246479] __fput+0x10d/0x440 [14656.249393] ____fput+0x12/0x20 [14656.251144] task_work_run+0xc6/0x110 [14656.252448] exit_to_usermode_loop+0x1dd/0x1f0 [14656.255126] do_syscall_64+0x3d6/0x440 [14656.257920] entry_SYSCALL_64_after_hwframe+0x49/0xae [14656.260585] RIP: 0033:0x7f151bee3cd8 [14656.262567] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14656.273316] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14656.277368] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [14656.281265] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [14656.284701] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14656.291746] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14656.297290] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14664.054194] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [14664.057695] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14664.083576] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14664.089290] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14664.093769] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14664.097099] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14664.106216] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14664.109404] RAX: 0000000000003611 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14664.112958] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [14664.116314] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [14664.120051] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [14664.123387] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [14664.127154] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [14664.131303] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14664.134024] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [14664.137296] Call Trace: [14664.138756] [14664.139840] ? show_regs.cold.9+0x22/0x2f [14664.141768] ? watchdog_timer_fn.cold.6+0x62/0x10a [14664.144028] ? __hrtimer_run_queues+0x1c2/0x4d0 [14664.146211] ? watchdog+0x40/0x40 [14664.148075] ? hrtimer_interrupt+0x128/0x2c0 [14664.150144] ? smp_apic_timer_interrupt+0xa6/0x270 [14664.152298] ? apic_timer_interrupt+0xf/0x20 [14664.154624] [14664.155690] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14664.158239] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14664.160897] do_raw_spin_lock+0x177/0x190 [14664.162458] _raw_spin_lock+0x21/0x30 [14664.164366] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14664.167355] ? do_raw_spin_unlock+0x75/0x190 [14664.169300] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14664.172104] ? do_wait_intr+0xf0/0xf0 [14664.174043] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14664.176787] mdc_reint+0x2d/0x110 [mdc] [14664.178638] mdc_unlink+0x658/0x870 [mdc] [14664.180002] lmv_unlink+0x34d/0xbd0 [lmv] [14664.181749] ll_unlink+0x24d/0x7f0 [lustre] [14664.184022] vfs_unlink+0x202/0x330 [14664.185720] do_unlinkat+0x2fa/0x460 [14664.187377] __x64_sys_unlink+0x37/0x50 [14664.189701] do_syscall_64+0xc1/0x440 [14664.191365] entry_SYSCALL_64_after_hwframe+0x49/0xae [14664.194176] RIP: 0033:0x7f151bdee10b [14664.195912] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [14664.205069] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [14664.208974] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [14664.212119] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [14664.215669] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14664.219038] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14664.222290] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14667.747718] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12737s! [14667.759728] Showing busy workqueues and worker pools: [14667.766761] workqueue events: flags=0x0 [14667.770352] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14667.777521] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [14667.785509] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14667.790825] pending: kfree_rcu_monitor [14667.793281] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14667.798656] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [14667.805013] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14667.809250] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [14667.814437] workqueue events_long: flags=0x0 [14667.817074] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14667.822094] pending: lu_objects_flush [obdclass] [14667.830096] workqueue events_unbound: flags=0x2 [14667.833524] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14667.837572] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14667.845025] workqueue events_power_efficient: flags=0x80 [14667.848733] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14667.852691] pending: do_cache_clean, neigh_periodic_work, check_lifetime [14667.860800] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14667.865056] pending: neigh_periodic_work [14667.868321] workqueue mm_percpu_wq: flags=0x8 [14667.876344] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14667.883498] pending: vmstat_update, lru_add_drain_per_cpu [14667.889603] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14667.899501] pending: vmstat_update [14667.901912] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14667.905983] pending: lru_add_drain_per_cpu [14667.909020] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [14667.913771] pending: lru_add_drain_per_cpu BAR(48) [14667.916269] workqueue writeback: flags=0x4a [14667.918223] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [14667.922957] pending: wb_workfn [14667.924741] workqueue xprtiod: flags=0xa [14667.927679] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14667.931432] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [14667.936144] workqueue ipv6_addrconf: flags=0x40008 [14667.940170] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14667.944349] pending: addrconf_verify_work [14667.947288] workqueue knbd0-recv: flags=0x1a [14667.951225] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14667.958575] in-flight: 658:recv_work [14667.960873] workqueue ptlrpc_pinger: flags=0x2 [14667.967297] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14667.978257] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14667.990216] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12737s workers=4 idle: 32984 37 8 [14667.997714] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14652s workers=2 idle: 661 [14668.055011] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [14668.062431] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14668.073138] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14668.073282] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14668.073355] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14668.073358] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14668.083146] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14668.085879] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14668.085898] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14668.085905] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14668.087722] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14668.087736] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14668.087740] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14668.087745] RAX: 0000000000005f15 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14668.087747] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [14668.087750] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [14668.088380] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [14668.088383] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [14668.088387] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [14668.088390] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14668.088392] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14668.088401] Call Trace: [14668.088406] [14668.088411] ? show_regs.cold.9+0x22/0x2f [14668.088421] ? watchdog_timer_fn.cold.6+0x62/0x10a [14668.088427] ? __hrtimer_run_queues+0x1c2/0x4d0 [14668.088431] ? watchdog+0x40/0x40 [14668.088436] ? hrtimer_interrupt+0x128/0x2c0 [14668.088440] ? smp_apic_timer_interrupt+0xa6/0x270 [14668.088448] ? apic_timer_interrupt+0xf/0x20 [14668.088452] [14668.088454] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14668.088459] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14668.088465] do_raw_spin_lock+0x177/0x190 [14668.088470] _raw_spin_lock+0x21/0x30 [14668.088475] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14668.156692] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14668.156707] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14668.156714] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [14668.156716] RDX: 00000000000060aa RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [14668.156719] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [14668.156721] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [14668.156723] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14668.156727] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [14668.156730] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14668.156732] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14668.156740] Call Trace: [14668.156915] [14668.156922] ? show_regs.cold.9+0x22/0x2f [14668.156936] ? watchdog_timer_fn.cold.6+0x62/0x10a [14668.156943] ? __hrtimer_run_queues+0x1c2/0x4d0 [14668.156948] ? watchdog+0x40/0x40 [14668.156952] ? hrtimer_interrupt+0x128/0x2c0 [14668.156956] ? smp_apic_timer_interrupt+0xa6/0x270 [14668.156964] ? apic_timer_interrupt+0xf/0x20 [14668.156968] [14668.156971] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14668.156978] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14668.156982] do_raw_spin_lock+0x177/0x190 [14668.156987] _raw_spin_lock+0x21/0x30 [14668.156995] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14668.173426] process_one_work+0x2c8/0x700 [14668.173447] worker_thread+0x296/0x6e0 [14668.173453] ? rescuer_thread+0x570/0x570 [14668.173457] kthread+0x1d1/0x200 [14668.173461] ? set_kthread_struct+0x70/0x70 [14668.173465] ret_from_fork+0x1f/0x30 [14668.723196] obd_mod_free+0x46/0x140 [ptlrpc] [14668.729814] mdc_commit_open+0x91/0xb0 [mdc] [14668.734934] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14668.739949] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14668.744025] after_reply+0xe2c/0x1780 [ptlrpc] [14668.747490] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14668.752254] ? __next_timer_interrupt+0x160/0x160 [14668.755570] ptlrpcd+0xa82/0xe40 [ptlrpc] [14668.759439] ? do_wait_intr+0xf0/0xf0 [14668.762948] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14668.767995] kthread+0x1d1/0x200 [14668.768795] ? set_kthread_struct+0x70/0x70 [14668.779006] ret_from_fork+0x1f/0x30 [14684.057046] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [14684.060478] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14684.079826] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14684.084112] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14684.087832] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14684.090466] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14684.098232] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14684.101167] RAX: 0000000000003b85 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14684.103876] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [14684.109000] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [14684.112507] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [14684.115886] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [14684.118323] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [14684.121666] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14684.124014] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [14684.126990] Call Trace: [14684.128087] [14684.129187] ? show_regs.cold.9+0x22/0x2f [14684.130641] ? watchdog_timer_fn.cold.6+0x62/0x10a [14684.132902] ? __hrtimer_run_queues+0x1c2/0x4d0 [14684.134989] ? watchdog+0x40/0x40 [14684.136394] ? hrtimer_interrupt+0x128/0x2c0 [14684.138594] ? smp_apic_timer_interrupt+0xa6/0x270 [14684.140292] ? apic_timer_interrupt+0xf/0x20 [14684.142330] [14684.143423] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14684.145767] do_raw_spin_lock+0x177/0x190 [14684.147305] _raw_spin_lock+0x21/0x30 [14684.149201] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14684.151360] ? do_raw_spin_unlock+0x75/0x190 [14684.153278] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14684.155389] ? do_wait_intr+0xf0/0xf0 [14684.157223] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14684.159656] mdc_close+0x53b/0xd70 [mdc] [14684.161295] lmv_close+0x18d/0x490 [lmv] [14684.162906] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14684.165400] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14684.167714] ll_md_real_close+0xa5/0x2a0 [lustre] [14684.169651] ll_md_close+0x274/0x800 [lustre] [14684.171492] ll_file_release+0x128/0xc50 [lustre] [14684.173759] __fput+0x10d/0x440 [14684.175133] ____fput+0x12/0x20 [14684.176520] task_work_run+0xc6/0x110 [14684.178098] exit_to_usermode_loop+0x1dd/0x1f0 [14684.180277] do_syscall_64+0x3d6/0x440 [14684.181831] entry_SYSCALL_64_after_hwframe+0x49/0xae [14684.184048] RIP: 0033:0x7f151bee3cd8 [14684.185397] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14684.197763] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14684.201435] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [14684.204278] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [14684.206692] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14684.209214] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14684.211718] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14692.053894] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [14692.061278] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14692.114019] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14692.124057] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14692.136248] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14692.142844] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14692.154948] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14692.160110] RAX: 00000000000069f6 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14692.169147] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [14692.185948] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [14692.191715] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [14692.201989] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [14692.210946] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [14692.218855] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14692.222735] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [14692.227338] Call Trace: [14692.229073] [14692.230885] ? show_regs.cold.9+0x22/0x2f [14692.233682] ? watchdog_timer_fn.cold.6+0x62/0x10a [14692.237301] ? __hrtimer_run_queues+0x1c2/0x4d0 [14692.240299] ? watchdog+0x40/0x40 [14692.244169] ? hrtimer_interrupt+0x128/0x2c0 [14692.246821] ? smp_apic_timer_interrupt+0xa6/0x270 [14692.248854] ? apic_timer_interrupt+0xf/0x20 [14692.250372] [14692.254166] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14692.257024] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14692.265303] do_raw_spin_lock+0x177/0x190 [14692.271264] _raw_spin_lock+0x21/0x30 [14692.272649] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14692.276034] ? do_raw_spin_unlock+0x75/0x190 [14692.278421] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14692.284951] ? do_wait_intr+0xf0/0xf0 [14692.286304] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14692.293040] mdc_reint+0x2d/0x110 [mdc] [14692.295794] mdc_unlink+0x658/0x870 [mdc] [14692.297255] lmv_unlink+0x34d/0xbd0 [lmv] [14692.301792] ll_unlink+0x24d/0x7f0 [lustre] [14692.304684] vfs_unlink+0x202/0x330 [14692.308906] do_unlinkat+0x2fa/0x460 [14692.312556] __x64_sys_unlink+0x37/0x50 [14692.315010] do_syscall_64+0xc1/0x440 [14692.317923] entry_SYSCALL_64_after_hwframe+0x49/0xae [14692.324033] RIP: 0033:0x7f151bdee10b [14692.326628] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [14692.342240] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [14692.348024] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [14692.354969] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [14692.359173] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14692.363423] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14692.370603] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14696.055050] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [14696.056033] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14696.056891] Modules linked in: [14696.060623] Modules linked in: [14696.060623] lustre(O) osp(O) [14696.061660] lustre(O) [14696.062455] ofd(O) [14696.065733] osp(O) [14696.066421] lod(O) [14696.067793] ofd(O) [14696.068324] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14696.068369] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14696.068373] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14696.068376] RIP: 0010:native_safe_halt+0x2e/0x50 [14696.069324] lod(O) [14696.069886] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14696.069892] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14696.069897] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [14696.069900] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [14696.069902] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [14696.069904] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [14696.069906] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14696.069910] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [14696.069912] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14696.070049] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14696.070056] Call Trace: [14696.070061] [14696.070065] ? show_regs.cold.9+0x22/0x2f [14696.070074] ? watchdog_timer_fn.cold.6+0x62/0x10a [14696.070079] ? __hrtimer_run_queues+0x1c2/0x4d0 [14696.070082] ? watchdog+0x40/0x40 [14696.070085] ? hrtimer_interrupt+0x128/0x2c0 [14696.070088] ? smp_apic_timer_interrupt+0xa6/0x270 [14696.070094] ? apic_timer_interrupt+0xf/0x20 [14696.070097] [14696.070099] ? native_safe_halt+0x2e/0x50 [14696.070102] kvm_wait+0xcd/0xe0 [14696.070107] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14696.070112] do_raw_spin_lock+0x177/0x190 [14696.070115] _raw_spin_lock+0x21/0x30 [14696.070120] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14696.070322] obd_mod_free+0x46/0x140 [ptlrpc] [14696.070445] mdc_commit_open+0x91/0xb0 [mdc] [14696.070481] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14696.070686] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14696.099844] mdt(O) [14696.108156] after_reply+0xe2c/0x1780 [ptlrpc] [14696.112234] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic [14696.113553] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14696.114838] crct10dif_pclmul [14696.122269] ? __next_timer_interrupt+0x160/0x160 [14696.126083] crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14696.126113] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14696.126122] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14696.126128] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14696.126610] RIP: 0010:native_safe_halt+0x2e/0x50 [14696.126634] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14696.126638] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14696.126643] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14696.126645] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [14696.126647] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [14696.126649] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [14696.126651] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [14696.126654] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [14696.126657] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14696.126659] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14696.126667] Call Trace: [14696.126672] [14696.126676] ? show_regs.cold.9+0x22/0x2f [14696.126686] ? watchdog_timer_fn.cold.6+0x62/0x10a [14696.126692] ? __hrtimer_run_queues+0x1c2/0x4d0 [14696.126696] ? watchdog+0x40/0x40 [14696.126701] ? hrtimer_interrupt+0x128/0x2c0 [14696.126705] ? smp_apic_timer_interrupt+0xa6/0x270 [14696.126712] ? apic_timer_interrupt+0xf/0x20 [14696.126717] [14696.126719] ? native_safe_halt+0x2e/0x50 [14696.126724] kvm_wait+0xcd/0xe0 [14696.126732] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14696.126739] do_raw_spin_lock+0x177/0x190 [14696.126743] _raw_spin_lock+0x21/0x30 [14696.126747] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14696.126948] process_one_work+0x2c8/0x700 [14696.126955] worker_thread+0x296/0x6e0 [14696.126960] ? rescuer_thread+0x570/0x570 [14696.126964] kthread+0x1d1/0x200 [14696.126968] ? set_kthread_struct+0x70/0x70 [14696.126972] ret_from_fork+0x1f/0x30 [14696.258626] ptlrpcd+0xa82/0xe40 [ptlrpc] [14696.263157] ? do_wait_intr+0xf0/0xf0 [14696.267726] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14696.271422] kthread+0x1d1/0x200 [14696.272451] ? set_kthread_struct+0x70/0x70 [14696.275379] ret_from_fork+0x1f/0x30 [14698.464045] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12767s! [14698.468339] Showing busy workqueues and worker pools: [14698.470871] workqueue events: flags=0x0 [14698.472628] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14698.475586] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [14698.481210] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14698.485158] pending: kfree_rcu_monitor [14698.487497] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14698.492860] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [14698.505748] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14698.515230] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [14698.523254] workqueue events_long: flags=0x0 [14698.529030] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14698.531516] pending: lu_objects_flush [obdclass] [14698.539092] workqueue events_unbound: flags=0x2 [14698.542071] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14698.545377] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14698.553197] workqueue events_power_efficient: flags=0x80 [14698.556056] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14698.565369] pending: do_cache_clean, neigh_periodic_work, check_lifetime [14698.582077] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14698.594209] pending: neigh_periodic_work [14698.601908] workqueue mm_percpu_wq: flags=0x8 [14698.604820] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14698.606966] pending: vmstat_update, lru_add_drain_per_cpu [14698.609828] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14698.613813] pending: vmstat_update [14698.615900] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14698.619076] pending: lru_add_drain_per_cpu [14698.621380] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [14698.625000] pending: lru_add_drain_per_cpu BAR(48) [14698.627922] workqueue writeback: flags=0x4a [14698.630097] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [14698.632880] pending: wb_workfn [14698.634657] workqueue xprtiod: flags=0xa [14698.636166] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14698.639017] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [14698.642029] workqueue ipv6_addrconf: flags=0x40008 [14698.644020] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14698.646681] pending: addrconf_verify_work [14698.648375] workqueue knbd0-recv: flags=0x1a [14698.650288] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14698.652939] in-flight: 658:recv_work [14698.655665] workqueue ptlrpc_pinger: flags=0x2 [14698.657786] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14698.660931] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14698.663826] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12768s workers=4 idle: 32984 37 8 [14698.667184] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14683s workers=2 idle: 661 [14712.057036] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [14712.061042] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14712.080474] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14712.083797] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14712.086029] RIP: 0010:native_safe_halt+0x2e/0x50 [14712.087145] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14712.093184] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14712.095961] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14712.098545] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [14712.101091] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [14712.103640] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [14712.106094] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [14712.109097] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [14712.112080] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14712.113593] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [14712.115650] Call Trace: [14712.116510] [14712.117143] ? show_regs.cold.9+0x22/0x2f [14712.118446] ? watchdog_timer_fn.cold.6+0x62/0x10a [14712.119683] ? __hrtimer_run_queues+0x1c2/0x4d0 [14712.120860] ? watchdog+0x40/0x40 [14712.121907] ? hrtimer_interrupt+0x128/0x2c0 [14712.123541] ? smp_apic_timer_interrupt+0xa6/0x270 [14712.125314] ? apic_timer_interrupt+0xf/0x20 [14712.126921] [14712.127758] ? native_safe_halt+0x2e/0x50 [14712.129244] kvm_wait+0xcd/0xe0 [14712.130882] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14712.132873] do_raw_spin_lock+0x177/0x190 [14712.134471] _raw_spin_lock+0x21/0x30 [14712.135860] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14712.137975] ? do_raw_spin_unlock+0x75/0x190 [14712.139665] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14712.141769] ? do_wait_intr+0xf0/0xf0 [14712.143060] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14712.145195] mdc_close+0x53b/0xd70 [mdc] [14712.147078] lmv_close+0x18d/0x490 [lmv] [14712.148851] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14712.151062] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14712.153501] ll_md_real_close+0xa5/0x2a0 [lustre] [14712.155230] ll_md_close+0x274/0x800 [lustre] [14712.157228] ll_file_release+0x128/0xc50 [lustre] [14712.159468] __fput+0x10d/0x440 [14712.160795] ____fput+0x12/0x20 [14712.162009] task_work_run+0xc6/0x110 [14712.163501] exit_to_usermode_loop+0x1dd/0x1f0 [14712.165400] do_syscall_64+0x3d6/0x440 [14712.166828] entry_SYSCALL_64_after_hwframe+0x49/0xae [14712.168903] RIP: 0033:0x7f151bee3cd8 [14712.170263] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14712.179960] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14712.185511] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [14712.189555] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [14712.191996] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14712.194610] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14712.196923] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14724.052025] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [14724.055029] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [14724.055973] Modules linked in: [14724.056023] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14724.056030] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14724.056111] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14724.056117] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14724.056123] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14724.056451] RIP: 0010:native_safe_halt+0x2e/0x50 [14724.056459] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14724.056463] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14724.056468] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14724.056470] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [14724.056472] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [14724.056474] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [14724.056477] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [14724.056480] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [14724.056483] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14724.056485] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14724.056494] Call Trace: [14724.056499] [14724.056503] ? show_regs.cold.9+0x22/0x2f [14724.056511] ? watchdog_timer_fn.cold.6+0x62/0x10a [14724.056517] ? __hrtimer_run_queues+0x1c2/0x4d0 [14724.056521] ? watchdog+0x40/0x40 [14724.056525] ? hrtimer_interrupt+0x128/0x2c0 [14724.056529] ? smp_apic_timer_interrupt+0xa6/0x270 [14724.056535] ? apic_timer_interrupt+0xf/0x20 [14724.056540] [14724.056542] ? native_safe_halt+0x2e/0x50 [14724.056546] kvm_wait+0xcd/0xe0 [14724.056553] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14724.056560] do_raw_spin_lock+0x177/0x190 [14724.056564] _raw_spin_lock+0x21/0x30 [14724.056573] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14724.056777] process_one_work+0x2c8/0x700 [14724.056783] worker_thread+0x296/0x6e0 [14724.056788] ? rescuer_thread+0x570/0x570 [14724.056791] kthread+0x1d1/0x200 [14724.056795] ? set_kthread_struct+0x70/0x70 [14724.056799] ret_from_fork+0x1f/0x30 [14724.061140] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14724.061217] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14724.061223] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14724.061225] RIP: 0010:native_safe_halt+0x2e/0x50 [14724.061239] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14724.061244] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14724.061248] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [14724.061251] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [14724.061252] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [14724.061255] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [14724.061257] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14724.061260] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [14724.061263] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14724.062742] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14724.062895] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14724.062902] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14724.062905] RIP: 0010:native_safe_halt+0x2e/0x50 [14724.062915] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14724.062919] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14724.062923] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14724.062925] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [14724.062927] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [14724.062929] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [14724.062931] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [14724.062933] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [14724.062936] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14724.062939] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [14724.062948] Call Trace: [14724.062953] [14724.062958] ? show_regs.cold.9+0x22/0x2f [14724.062966] ? watchdog_timer_fn.cold.6+0x62/0x10a [14724.062972] ? __hrtimer_run_queues+0x1c2/0x4d0 [14724.062976] ? watchdog+0x40/0x40 [14724.062979] ? hrtimer_interrupt+0x128/0x2c0 [14724.062982] ? smp_apic_timer_interrupt+0xa6/0x270 [14724.062988] ? apic_timer_interrupt+0xf/0x20 [14724.062992] [14724.062995] ? native_safe_halt+0x2e/0x50 [14724.062998] kvm_wait+0xcd/0xe0 [14724.063004] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14724.063010] do_raw_spin_lock+0x177/0x190 [14724.063014] _raw_spin_lock+0x21/0x30 [14724.063019] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14724.063464] ? do_raw_spin_unlock+0x75/0x190 [14724.063471] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14724.073709] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14724.073726] Call Trace: [14724.073733] [14724.073738] ? show_regs.cold.9+0x22/0x2f [14724.090474] ? do_wait_intr+0xf0/0xf0 [14724.090496] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14724.090759] mdc_reint+0x2d/0x110 [mdc] [14724.090798] mdc_unlink+0x658/0x870 [mdc] [14724.090825] lmv_unlink+0x34d/0xbd0 [lmv] [14724.090856] ll_unlink+0x24d/0x7f0 [lustre] [14724.093918] ? watchdog_timer_fn.cold.6+0x62/0x10a [14724.096124] vfs_unlink+0x202/0x330 [14724.096140] do_unlinkat+0x2fa/0x460 [14724.096145] __x64_sys_unlink+0x37/0x50 [14724.096148] do_syscall_64+0xc1/0x440 [14724.096155] entry_SYSCALL_64_after_hwframe+0x49/0xae [14724.096163] RIP: 0033:0x7f151bdee10b [14724.096171] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [14724.096175] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [14724.096180] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [14724.096182] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [14724.096184] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14724.096186] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14724.096188] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14724.483521] ? __hrtimer_run_queues+0x1c2/0x4d0 [14724.484627] ? watchdog+0x40/0x40 [14724.485767] ? hrtimer_interrupt+0x128/0x2c0 [14724.486713] ? smp_apic_timer_interrupt+0xa6/0x270 [14724.490974] ? apic_timer_interrupt+0xf/0x20 [14724.492022] [14724.492582] ? native_safe_halt+0x2e/0x50 [14724.494460] kvm_wait+0xcd/0xe0 [14724.496613] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14724.498477] do_raw_spin_lock+0x177/0x190 [14724.499513] _raw_spin_lock+0x21/0x30 [14724.500331] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14724.502983] obd_mod_free+0x46/0x140 [ptlrpc] [14724.505164] mdc_commit_open+0x91/0xb0 [mdc] [14724.508034] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14724.513601] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14724.517107] after_reply+0xe2c/0x1780 [ptlrpc] [14724.520029] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14724.523745] ? __next_timer_interrupt+0x160/0x160 [14724.525865] ptlrpcd+0xa82/0xe40 [ptlrpc] [14724.528089] ? do_wait_intr+0xf0/0xf0 [14724.529272] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14724.531728] kthread+0x1d1/0x200 [14724.533121] ? set_kthread_struct+0x70/0x70 [14724.538018] ret_from_fork+0x1f/0x30 [14728.675760] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12798s! [14728.682150] Showing busy workqueues and worker pools: [14728.692130] workqueue events: flags=0x0 [14728.698410] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14728.708005] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [14728.718955] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14728.722872] pending: kfree_rcu_monitor [14728.726014] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14728.730551] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [14728.734959] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14728.738040] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [14728.740441] workqueue events_long: flags=0x0 [14728.741721] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14728.746350] pending: lu_objects_flush [obdclass] [14728.749473] workqueue events_unbound: flags=0x2 [14728.751390] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14728.753899] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14728.757949] workqueue events_power_efficient: flags=0x80 [14728.760084] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14728.762133] pending: do_cache_clean, neigh_periodic_work, check_lifetime [14728.765435] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14728.767948] pending: neigh_periodic_work [14728.770382] workqueue mm_percpu_wq: flags=0x8 [14728.772493] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14728.776335] pending: vmstat_update, lru_add_drain_per_cpu [14728.779423] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14728.781901] pending: vmstat_update [14728.783322] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14728.785516] pending: lru_add_drain_per_cpu [14728.787151] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [14728.789586] pending: lru_add_drain_per_cpu BAR(48) [14728.791874] workqueue writeback: flags=0x4a [14728.793507] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [14728.796063] pending: wb_workfn [14728.797349] workqueue xprtiod: flags=0xa [14728.798908] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14728.802491] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [14728.806778] workqueue ipv6_addrconf: flags=0x40008 [14728.809619] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14728.813478] pending: addrconf_verify_work [14728.815582] workqueue knbd0-recv: flags=0x1a [14728.817075] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14728.819181] in-flight: 658:recv_work [14728.821647] workqueue ptlrpc_pinger: flags=0x2 [14728.824095] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14728.826855] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14728.830568] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12798s workers=4 idle: 32984 37 8 [14728.834370] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14713s workers=2 idle: 661 [14740.057029] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [14740.060261] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14740.096681] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14740.104043] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14740.109999] RIP: 0010:native_safe_halt+0x2e/0x50 [14740.114936] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14740.127221] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14740.131914] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14740.139974] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [14740.145640] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [14740.151300] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [14740.154509] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [14740.159325] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [14740.163788] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14740.167026] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [14740.176472] Call Trace: [14740.177529] [14740.178512] ? show_regs.cold.9+0x22/0x2f [14740.182546] ? watchdog_timer_fn.cold.6+0x62/0x10a [14740.184292] ? __hrtimer_run_queues+0x1c2/0x4d0 [14740.196550] ? watchdog+0x40/0x40 [14740.198978] ? hrtimer_interrupt+0x128/0x2c0 [14740.202832] ? smp_apic_timer_interrupt+0xa6/0x270 [14740.206542] ? apic_timer_interrupt+0xf/0x20 [14740.209765] [14740.211831] ? native_safe_halt+0x2e/0x50 [14740.215036] kvm_wait+0xcd/0xe0 [14740.215996] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14740.217457] do_raw_spin_lock+0x177/0x190 [14740.220947] _raw_spin_lock+0x21/0x30 [14740.224783] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14740.227453] ? do_raw_spin_unlock+0x75/0x190 [14740.229655] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14740.232534] ? do_wait_intr+0xf0/0xf0 [14740.234943] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14740.237667] mdc_close+0x53b/0xd70 [mdc] [14740.239472] lmv_close+0x18d/0x490 [lmv] [14740.246107] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14740.250336] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14740.254412] ll_md_real_close+0xa5/0x2a0 [lustre] [14740.256601] ll_md_close+0x274/0x800 [lustre] [14740.258579] ll_file_release+0x128/0xc50 [lustre] [14740.261822] __fput+0x10d/0x440 [14740.262948] ____fput+0x12/0x20 [14740.264858] task_work_run+0xc6/0x110 [14740.266525] exit_to_usermode_loop+0x1dd/0x1f0 [14740.267984] do_syscall_64+0x3d6/0x440 [14740.272347] entry_SYSCALL_64_after_hwframe+0x49/0xae [14740.277307] RIP: 0033:0x7f151bee3cd8 [14740.278847] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14740.290047] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14740.294527] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [14740.299812] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [14740.303949] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14740.309876] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14740.315765] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14752.055089] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [14752.056105] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14752.060415] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) [14752.070183] Modules linked in: [14752.070194] lmv(O) [14752.076708] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) [14752.078343] ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14752.078395] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14752.078401] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14752.078404] RIP: 0010:native_safe_halt+0x2e/0x50 [14752.078416] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14752.078421] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14752.078425] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [14752.078428] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [14752.078430] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [14752.078432] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [14752.078434] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14752.078436] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [14752.078438] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14752.078440] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14752.078447] Call Trace: [14752.078587] [14752.078594] ? show_regs.cold.9+0x22/0x2f [14752.078602] ? watchdog_timer_fn.cold.6+0x62/0x10a [14752.078607] ? __hrtimer_run_queues+0x1c2/0x4d0 [14752.078610] ? watchdog+0x40/0x40 [14752.078612] ? hrtimer_interrupt+0x128/0x2c0 [14752.078615] ? smp_apic_timer_interrupt+0xa6/0x270 [14752.078620] ? apic_timer_interrupt+0xf/0x20 [14752.078624] [14752.078626] ? native_safe_halt+0x2e/0x50 [14752.078628] kvm_wait+0xcd/0xe0 [14752.078634] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14752.078639] do_raw_spin_lock+0x177/0x190 [14752.078641] _raw_spin_lock+0x21/0x30 [14752.078645] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14752.079647] obdclass(O) [14752.086429] obd_mod_free+0x46/0x140 [ptlrpc] [14752.086688] mdc_commit_open+0x91/0xb0 [mdc] [14752.086717] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14752.086841] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14752.087005] after_reply+0xe2c/0x1780 [ptlrpc] [14752.087208] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14752.087402] ? __next_timer_interrupt+0x160/0x160 [14752.087411] ptlrpcd+0xa82/0xe40 [ptlrpc] [14752.087619] ? do_wait_intr+0xf0/0xf0 [14752.087629] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14752.103478] ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14752.103540] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14752.103548] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14752.103554] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14752.108917] kthread+0x1d1/0x200 [14752.108948] ? set_kthread_struct+0x70/0x70 [14752.108953] ret_from_fork+0x1f/0x30 [14752.286126] [14752.286696] RIP: 0010:native_safe_halt+0x2e/0x50 [14752.288379] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14752.295807] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14752.298880] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14752.301615] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [14752.304540] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [14752.307432] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [14752.310530] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [14752.313421] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [14752.316536] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14752.318686] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14752.321451] Call Trace: [14752.322488] [14752.323282] ? show_regs.cold.9+0x22/0x2f [14752.324676] ? watchdog_timer_fn.cold.6+0x62/0x10a [14752.326767] ? __hrtimer_run_queues+0x1c2/0x4d0 [14752.329333] ? watchdog+0x40/0x40 [14752.331219] ? hrtimer_interrupt+0x128/0x2c0 [14752.332818] ? smp_apic_timer_interrupt+0xa6/0x270 [14752.335216] ? apic_timer_interrupt+0xf/0x20 [14752.337417] [14752.338325] ? native_safe_halt+0x2e/0x50 [14752.340214] kvm_wait+0xcd/0xe0 [14752.341652] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14752.344250] do_raw_spin_lock+0x177/0x190 [14752.346241] _raw_spin_lock+0x21/0x30 [14752.347816] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14752.350776] process_one_work+0x2c8/0x700 [14752.352742] worker_thread+0x296/0x6e0 [14752.354834] ? rescuer_thread+0x570/0x570 [14752.356893] kthread+0x1d1/0x200 [14752.358875] ? set_kthread_struct+0x70/0x70 [14752.361461] ret_from_fork+0x1f/0x30 [14756.053210] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [14756.055865] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14756.069976] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14756.073752] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14756.076188] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14756.077844] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14756.084256] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14756.086793] RAX: 0000000000004397 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14756.088753] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [14756.090838] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [14756.094100] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [14756.097150] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [14756.100680] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [14756.104314] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14756.106655] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [14756.109913] Call Trace: [14756.110793] [14756.111482] ? show_regs.cold.9+0x22/0x2f [14756.113220] ? watchdog_timer_fn.cold.6+0x62/0x10a [14756.114850] ? __hrtimer_run_queues+0x1c2/0x4d0 [14756.116404] ? watchdog+0x40/0x40 [14756.117479] ? hrtimer_interrupt+0x128/0x2c0 [14756.119195] ? smp_apic_timer_interrupt+0xa6/0x270 [14756.121839] ? apic_timer_interrupt+0xf/0x20 [14756.123989] [14756.124813] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14756.127084] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14756.129072] do_raw_spin_lock+0x177/0x190 [14756.130673] _raw_spin_lock+0x21/0x30 [14756.131926] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14756.134132] ? do_raw_spin_unlock+0x75/0x190 [14756.135889] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14756.137763] ? do_wait_intr+0xf0/0xf0 [14756.139023] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14756.140783] mdc_reint+0x2d/0x110 [mdc] [14756.142290] mdc_unlink+0x658/0x870 [mdc] [14756.143787] lmv_unlink+0x34d/0xbd0 [lmv] [14756.145136] ll_unlink+0x24d/0x7f0 [lustre] [14756.146522] vfs_unlink+0x202/0x330 [14756.147479] do_unlinkat+0x2fa/0x460 [14756.148909] __x64_sys_unlink+0x37/0x50 [14756.150294] do_syscall_64+0xc1/0x440 [14756.151538] entry_SYSCALL_64_after_hwframe+0x49/0xae [14756.154085] RIP: 0033:0x7f151bdee10b [14756.156140] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [14756.163102] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [14756.165508] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [14756.167857] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [14756.170412] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14756.172653] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14756.174738] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14758.883121] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12828s! [14758.887152] Showing busy workqueues and worker pools: [14758.889824] workqueue events: flags=0x0 [14758.891611] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14758.894610] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [14758.898735] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14758.902375] pending: kfree_rcu_monitor [14758.904478] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14758.907336] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [14758.911234] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14758.913742] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [14758.916825] workqueue events_long: flags=0x0 [14758.918938] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14758.921647] pending: lu_objects_flush [obdclass] [14758.923783] workqueue events_unbound: flags=0x2 [14758.925706] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14758.928417] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14758.932490] workqueue events_power_efficient: flags=0x80 [14758.934382] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14758.936685] pending: do_cache_clean, neigh_periodic_work, check_lifetime [14758.939252] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14758.941764] pending: neigh_periodic_work [14758.943258] workqueue mm_percpu_wq: flags=0x8 [14758.944673] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14758.947174] pending: vmstat_update, lru_add_drain_per_cpu [14758.949242] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14758.951869] pending: vmstat_update [14758.953397] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14758.955663] pending: lru_add_drain_per_cpu [14758.957346] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [14758.959522] pending: lru_add_drain_per_cpu BAR(48) [14758.961228] workqueue writeback: flags=0x4a [14758.962848] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [14758.965240] pending: wb_workfn [14758.966561] workqueue xprtiod: flags=0xa [14758.968244] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14758.970609] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [14758.973540] workqueue ipv6_addrconf: flags=0x40008 [14758.975459] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14758.977652] pending: addrconf_verify_work [14758.979333] workqueue knbd0-recv: flags=0x1a [14758.981020] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14758.983306] in-flight: 658:recv_work [14758.984632] workqueue ptlrpc_pinger: flags=0x2 [14758.986332] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14758.988376] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14758.991404] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12828s workers=4 idle: 32984 37 8 [14758.993775] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14743s workers=2 idle: 661 [14764.920404] rcu: INFO: rcu_sched self-detected stall on CPU [14764.924217] rcu: 1-...!: (7613013 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [14764.928511] (t=12840273 jiffies g=249657 q=6332) [14764.930501] rcu: rcu_sched kthread starved for 12835225 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [14764.934775] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [14764.938849] rcu: RCU grace-period kthread stack dump: [14764.940987] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [14764.945182] Call Trace: [14764.946405] __schedule+0x351/0xcb0 [14764.947845] schedule+0xc0/0x180 [14764.949456] schedule_timeout+0xb4/0x190 [14764.951199] ? do_raw_spin_unlock+0x75/0x190 [14764.953290] ? __next_timer_interrupt+0x160/0x160 [14764.955271] ? prepare_to_swait_event+0xf3/0x140 [14764.957231] rcu_gp_kthread+0x8dd/0x15b0 [14764.959999] ? force_qs_rnp+0x2c0/0x2c0 [14764.961195] kthread+0x1d1/0x200 [14764.962337] ? set_kthread_struct+0x70/0x70 [14764.963743] ret_from_fork+0x1f/0x30 [14764.965996] rcu: Stack dump where RCU GP kthread last ran: [14764.969212] Sending NMI from CPU 1 to CPUs 3: [14764.974613] NMI backtrace for cpu 3 [14764.974620] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14764.974622] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14764.974624] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14764.974625] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14764.974627] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 [14764.974630] RAX: 000000000000559e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14764.974631] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [14764.974633] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [14764.974634] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [14764.974636] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [14764.974637] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [14764.974639] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14764.974641] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [14764.974642] Call Trace: [14764.974643] [14764.974644] ? show_regs.cold.9+0x22/0x2f [14764.974645] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [14764.974647] ? nmi_cpu_backtrace_handler+0x15/0x40 [14764.974648] ? nmi_handle+0xbd/0x210 [14764.974649] ? default_do_nmi+0x4b/0x1b0 [14764.974650] ? do_nmi+0x280/0x380 [14764.974651] ? end_repeat_nmi+0x16/0x1f [14764.974652] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14764.974653] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14764.974654] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14764.974655] [14764.974656] do_raw_spin_lock+0x177/0x190 [14764.974657] _raw_spin_lock+0x21/0x30 [14764.974659] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14764.974660] ? do_raw_spin_unlock+0x75/0x190 [14764.974661] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14764.974662] ? do_wait_intr+0xf0/0xf0 [14764.974663] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14764.974665] mdc_close+0x53b/0xd70 [mdc] [14764.974666] lmv_close+0x18d/0x490 [lmv] [14764.974667] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14764.974668] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14764.974669] ll_md_real_close+0xa5/0x2a0 [lustre] [14764.974670] ll_md_close+0x274/0x800 [lustre] [14764.974671] ll_file_release+0x128/0xc50 [lustre] [14764.974672] __fput+0x10d/0x440 [14764.974673] ____fput+0x12/0x20 [14764.974674] task_work_run+0xc6/0x110 [14764.974675] exit_to_usermode_loop+0x1dd/0x1f0 [14764.974676] do_syscall_64+0x3d6/0x440 [14764.974677] entry_SYSCALL_64_after_hwframe+0x49/0xae [14764.974679] RIP: 0033:0x7f151bee3cd8 [14764.974680] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14764.974682] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14764.974685] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [14764.974686] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [14764.974688] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14764.974689] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14764.974690] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14764.975475] NMI backtrace for cpu 1 [14765.252534] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14765.267037] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14765.275742] Call Trace: [14765.278349] [14765.280763] ? dump_stack+0xbb/0x10e [14765.286683] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [14765.289264] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [14765.294984] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [14765.299483] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [14765.305641] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [14765.308272] ? rcu_dump_cpu_stacks+0x11b/0x171 [14765.311072] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [14765.313579] ? update_process_times+0x6d/0xc0 [14765.315793] ? tick_sched_handle.isra.18+0x27/0x90 [14765.318461] ? tick_sched_timer+0x53/0xb0 [14765.320372] ? __hrtimer_run_queues+0x1c2/0x4d0 [14765.322866] ? tick_sched_handle.isra.18+0x90/0x90 [14765.324536] ? hrtimer_interrupt+0x128/0x2c0 [14765.327112] ? smp_apic_timer_interrupt+0xa6/0x270 [14765.329957] ? apic_timer_interrupt+0xf/0x20 [14765.332179] [14765.334049] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14765.337236] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14765.340466] ? do_raw_spin_lock+0x177/0x190 [14765.343374] ? _raw_spin_lock+0x21/0x30 [14765.346115] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14765.351439] ? obd_mod_free+0x46/0x140 [ptlrpc] [14765.354147] ? mdc_commit_open+0x91/0xb0 [mdc] [14765.356501] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14765.360033] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14765.364277] ? after_reply+0xe2c/0x1780 [ptlrpc] [14765.367577] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14765.371083] ? __next_timer_interrupt+0x160/0x160 [14765.373846] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [14765.377059] ? do_wait_intr+0xf0/0xf0 [14765.379405] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14765.382962] ? kthread+0x1d1/0x200 [14765.385428] ? set_kthread_struct+0x70/0x70 [14765.387501] ? ret_from_fork+0x1f/0x30 [14780.057477] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [14780.062287] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14780.098223] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14780.105521] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14780.111319] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14780.120438] RIP: 0010:native_safe_halt+0x2e/0x50 [14780.127389] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14780.142783] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14780.148567] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14780.152179] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [14780.155408] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [14780.158625] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [14780.162483] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [14780.166093] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [14780.178227] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14780.181347] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14780.186893] Call Trace: [14780.187989] [14780.189158] ? show_regs.cold.9+0x22/0x2f [14780.190934] ? watchdog_timer_fn.cold.6+0x62/0x10a [14780.193015] ? __hrtimer_run_queues+0x1c2/0x4d0 [14780.194699] ? watchdog+0x40/0x40 [14780.195755] ? hrtimer_interrupt+0x128/0x2c0 [14780.197388] ? smp_apic_timer_interrupt+0xa6/0x270 [14780.199577] ? apic_timer_interrupt+0xf/0x20 [14780.200928] [14780.201585] ? native_safe_halt+0x2e/0x50 [14780.202794] kvm_wait+0xcd/0xe0 [14780.203776] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14780.205411] do_raw_spin_lock+0x177/0x190 [14780.206607] _raw_spin_lock+0x21/0x30 [14780.207928] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14780.210200] process_one_work+0x2c8/0x700 [14780.211885] worker_thread+0x296/0x6e0 [14780.213352] ? rescuer_thread+0x570/0x570 [14780.215022] kthread+0x1d1/0x200 [14780.216302] ? set_kthread_struct+0x70/0x70 [14780.217860] ret_from_fork+0x1f/0x30 [14784.052428] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [14784.055913] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14784.078774] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14784.083533] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14784.087279] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14784.089609] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14784.098439] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14784.101773] RAX: 000000000000410c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14784.104881] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [14784.108238] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [14784.111351] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [14784.114872] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [14784.118144] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [14784.122256] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14784.124519] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [14784.127499] Call Trace: [14784.128817] [14784.130000] ? show_regs.cold.9+0x22/0x2f [14784.132246] ? watchdog_timer_fn.cold.6+0x62/0x10a [14784.134272] ? __hrtimer_run_queues+0x1c2/0x4d0 [14784.136492] ? watchdog+0x40/0x40 [14784.138203] ? hrtimer_interrupt+0x128/0x2c0 [14784.140268] ? smp_apic_timer_interrupt+0xa6/0x270 [14784.142895] ? apic_timer_interrupt+0xf/0x20 [14784.145037] [14784.145848] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14784.148515] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14784.150982] do_raw_spin_lock+0x177/0x190 [14784.152983] _raw_spin_lock+0x21/0x30 [14784.154597] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14784.157927] ? do_raw_spin_unlock+0x75/0x190 [14784.159931] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14784.163008] ? do_wait_intr+0xf0/0xf0 [14784.165226] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14784.167546] mdc_reint+0x2d/0x110 [mdc] [14784.169271] mdc_unlink+0x658/0x870 [mdc] [14784.171238] lmv_unlink+0x34d/0xbd0 [lmv] [14784.173319] ll_unlink+0x24d/0x7f0 [lustre] [14784.175572] vfs_unlink+0x202/0x330 [14784.176807] do_unlinkat+0x2fa/0x460 [14784.177942] __x64_sys_unlink+0x37/0x50 [14784.179884] do_syscall_64+0xc1/0x440 [14784.181809] entry_SYSCALL_64_after_hwframe+0x49/0xae [14784.184030] RIP: 0033:0x7f151bdee10b [14784.185601] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [14784.193883] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [14784.197207] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [14784.200643] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [14784.203801] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14784.207045] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14784.210316] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14789.096525] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12858s! [14789.106746] Showing busy workqueues and worker pools: [14789.112983] workqueue events: flags=0x0 [14789.118386] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14789.136142] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [14789.142289] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14789.145998] pending: kfree_rcu_monitor [14789.151754] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14789.158352] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [14789.170574] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14789.184302] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [14789.193806] workqueue events_long: flags=0x0 [14789.203398] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14789.215003] pending: lu_objects_flush [obdclass] [14789.220727] workqueue events_unbound: flags=0x2 [14789.223180] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14789.231231] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14789.244763] workqueue events_power_efficient: flags=0x80 [14789.253249] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14789.265130] pending: do_cache_clean, neigh_periodic_work, check_lifetime [14789.275782] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14789.287481] pending: neigh_periodic_work [14789.297985] workqueue mm_percpu_wq: flags=0x8 [14789.304186] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14789.312208] pending: vmstat_update, lru_add_drain_per_cpu [14789.319101] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14789.329869] pending: vmstat_update [14789.333113] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14789.342938] pending: lru_add_drain_per_cpu [14789.347561] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [14789.354532] pending: lru_add_drain_per_cpu BAR(48) [14789.357289] workqueue writeback: flags=0x4a [14789.359755] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [14789.364985] pending: wb_workfn [14789.371253] workqueue xprtiod: flags=0xa [14789.380259] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14789.383045] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [14789.394168] workqueue ipv6_addrconf: flags=0x40008 [14789.399878] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14789.406951] pending: addrconf_verify_work [14789.413747] workqueue knbd0-recv: flags=0x1a [14789.418018] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14789.422041] in-flight: 658:recv_work [14789.430243] workqueue ptlrpc_pinger: flags=0x2 [14789.437017] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14789.443350] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14789.448964] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12858s workers=4 idle: 32984 37 8 [14789.456184] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14774s workers=2 idle: 661 [14792.055009] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [14792.057904] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14792.060319] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [14792.123052] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14792.126103] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14792.126178] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14792.126186] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14792.126189] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14792.126203] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14792.126208] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14792.126213] RAX: 000000000000552c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14792.126215] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [14792.126218] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [14792.126220] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [14792.126223] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [14792.126226] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [14792.126229] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14792.126232] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [14792.126240] Call Trace: [14792.126246] [14792.126253] ? show_regs.cold.9+0x22/0x2f [14792.126263] ? watchdog_timer_fn.cold.6+0x62/0x10a [14792.126270] ? __hrtimer_run_queues+0x1c2/0x4d0 [14792.126275] ? watchdog+0x40/0x40 [14792.126278] ? hrtimer_interrupt+0x128/0x2c0 [14792.126281] ? smp_apic_timer_interrupt+0xa6/0x270 [14792.126288] ? apic_timer_interrupt+0xf/0x20 [14792.126293] [14792.126296] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14792.126303] do_raw_spin_lock+0x177/0x190 [14792.126309] _raw_spin_lock+0x21/0x30 [14792.126314] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14792.126583] ? do_raw_spin_unlock+0x75/0x190 [14792.126590] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14792.126994] ? do_wait_intr+0xf0/0xf0 [14792.127002] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14792.142499] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14792.186614] mdc_close+0x53b/0xd70 [mdc] [14792.187427] lmv_close+0x18d/0x490 [lmv] [14792.187457] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14792.187637] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14792.188051] ll_md_real_close+0xa5/0x2a0 [lustre] [14792.188216] ll_md_close+0x274/0x800 [lustre] [14792.188359] ll_file_release+0x128/0xc50 [lustre] [14792.188447] __fput+0x10d/0x440 [14792.188457] ____fput+0x12/0x20 [14792.188462] task_work_run+0xc6/0x110 [14792.188470] exit_to_usermode_loop+0x1dd/0x1f0 [14792.188477] do_syscall_64+0x3d6/0x440 [14792.188482] entry_SYSCALL_64_after_hwframe+0x49/0xae [14792.188490] RIP: 0033:0x7f151bee3cd8 [14792.188498] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14792.188503] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14792.188508] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [14792.188510] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [14792.188512] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14792.188515] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14792.188517] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14792.545301] RIP: 0010:__pv_queued_spin_lock_slowpath+0x17c/0x6d0 [14792.550508] Code: 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 48 83 05 5c d6 dc 02 01 <83> ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 75 e6 48 83 05 2c d6 dc [14792.579264] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14792.584997] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [14792.594282] RDX: 0000000000000e46 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [14792.600124] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [14792.608226] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [14792.613321] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14792.617125] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [14792.625549] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14792.631783] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14792.639310] Call Trace: [14792.640401] [14792.642520] ? show_regs.cold.9+0x22/0x2f [14792.650018] ? watchdog_timer_fn.cold.6+0x62/0x10a [14792.652092] ? __hrtimer_run_queues+0x1c2/0x4d0 [14792.660012] ? watchdog+0x40/0x40 [14792.663906] ? hrtimer_interrupt+0x128/0x2c0 [14792.666202] ? smp_apic_timer_interrupt+0xa6/0x270 [14792.671993] ? apic_timer_interrupt+0xf/0x20 [14792.677502] [14792.681215] ? __pv_queued_spin_lock_slowpath+0x17c/0x6d0 [14792.686439] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14792.689502] do_raw_spin_lock+0x177/0x190 [14792.691583] _raw_spin_lock+0x21/0x30 [14792.693903] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14792.697500] obd_mod_free+0x46/0x140 [ptlrpc] [14792.700510] mdc_commit_open+0x91/0xb0 [mdc] [14792.703102] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14792.706822] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14792.724821] after_reply+0xe2c/0x1780 [ptlrpc] [14792.730289] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14792.741022] ? __next_timer_interrupt+0x160/0x160 [14792.754851] ptlrpcd+0xa82/0xe40 [ptlrpc] [14792.766843] ? do_wait_intr+0xf0/0xf0 [14792.770345] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14792.788508] kthread+0x1d1/0x200 [14792.797398] ? set_kthread_struct+0x70/0x70 [14792.810363] ret_from_fork+0x1f/0x30 [14808.056026] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [14808.059430] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14808.078204] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14808.082843] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14808.086413] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14808.089239] RIP: 0010:native_safe_halt+0x2e/0x50 [14808.090764] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14808.098075] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [14808.101314] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14808.104152] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [14808.107024] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [14808.109923] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [14808.113709] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [14808.116579] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [14808.119850] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14808.121855] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14808.125891] Call Trace: [14808.126887] [14808.127881] ? show_regs.cold.9+0x22/0x2f [14808.129507] ? watchdog_timer_fn.cold.6+0x62/0x10a [14808.131523] ? __hrtimer_run_queues+0x1c2/0x4d0 [14808.133723] ? watchdog+0x40/0x40 [14808.134996] ? hrtimer_interrupt+0x128/0x2c0 [14808.136987] ? smp_apic_timer_interrupt+0xa6/0x270 [14808.138952] ? apic_timer_interrupt+0xf/0x20 [14808.143928] [14808.144797] ? native_safe_halt+0x2e/0x50 [14808.146730] kvm_wait+0xcd/0xe0 [14808.148229] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14808.150693] do_raw_spin_lock+0x177/0x190 [14808.152303] _raw_spin_lock+0x21/0x30 [14808.154131] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14808.156410] process_one_work+0x2c8/0x700 [14808.158146] worker_thread+0x296/0x6e0 [14808.159902] ? rescuer_thread+0x570/0x570 [14808.161741] kthread+0x1d1/0x200 [14808.163856] ? set_kthread_struct+0x70/0x70 [14808.165968] ret_from_fork+0x1f/0x30 [14816.055661] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [14816.060972] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14816.101289] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14816.109147] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14816.117574] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14816.124074] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14816.158642] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14816.165091] RAX: 0000000000007585 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14816.175080] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [14816.180154] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [14816.187059] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [14816.199846] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [14816.204834] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [14816.214093] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14816.217791] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [14816.221638] Call Trace: [14816.222751] [14816.223974] ? show_regs.cold.9+0x22/0x2f [14816.240755] ? watchdog_timer_fn.cold.6+0x62/0x10a [14816.246104] ? __hrtimer_run_queues+0x1c2/0x4d0 [14816.257237] ? watchdog+0x40/0x40 [14816.258480] ? hrtimer_interrupt+0x128/0x2c0 [14816.262969] ? smp_apic_timer_interrupt+0xa6/0x270 [14816.265438] ? apic_timer_interrupt+0xf/0x20 [14816.268244] [14816.269241] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14816.273185] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14816.277101] do_raw_spin_lock+0x177/0x190 [14816.282163] _raw_spin_lock+0x21/0x30 [14816.285110] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14816.288584] ? do_raw_spin_unlock+0x75/0x190 [14816.295854] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14816.305306] ? do_wait_intr+0xf0/0xf0 [14816.310558] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14816.315150] mdc_reint+0x2d/0x110 [mdc] [14816.317307] mdc_unlink+0x658/0x870 [mdc] [14816.328037] lmv_unlink+0x34d/0xbd0 [lmv] [14816.333502] ll_unlink+0x24d/0x7f0 [lustre] [14816.335204] vfs_unlink+0x202/0x330 [14816.336625] do_unlinkat+0x2fa/0x460 [14816.345287] __x64_sys_unlink+0x37/0x50 [14816.352718] do_syscall_64+0xc1/0x440 [14816.354439] entry_SYSCALL_64_after_hwframe+0x49/0xae [14816.364196] RIP: 0033:0x7f151bdee10b [14816.365764] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [14816.381120] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [14816.395713] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [14816.405980] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [14816.409037] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14816.412066] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14816.414670] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14819.809153] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12889s! [14819.812152] Showing busy workqueues and worker pools: [14819.815408] workqueue events: flags=0x0 [14819.819197] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14819.823009] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [14819.835778] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14819.838247] pending: kfree_rcu_monitor [14819.846120] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14819.852685] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [14819.858919] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14819.864355] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [14819.868207] workqueue events_long: flags=0x0 [14819.871014] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14819.878311] pending: lu_objects_flush [obdclass] [14819.882061] workqueue events_unbound: flags=0x2 [14819.887269] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14819.895225] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14819.904099] workqueue events_power_efficient: flags=0x80 [14819.918709] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14819.922876] pending: do_cache_clean, neigh_periodic_work, check_lifetime [14819.929866] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14819.936873] pending: neigh_periodic_work [14819.947430] workqueue mm_percpu_wq: flags=0x8 [14819.950183] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14819.958424] pending: vmstat_update, lru_add_drain_per_cpu [14819.963888] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14819.971002] pending: vmstat_update [14819.974435] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14819.986207] pending: lru_add_drain_per_cpu [14819.993643] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [14820.000121] pending: lru_add_drain_per_cpu BAR(48) [14820.004778] workqueue writeback: flags=0x4a [14820.006175] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [14820.023643] pending: wb_workfn [14820.029032] workqueue xprtiod: flags=0xa [14820.034165] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14820.043015] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [14820.051691] workqueue ipv6_addrconf: flags=0x40008 [14820.055323] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [14820.056049] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14820.056064] pending: addrconf_verify_work [14820.056084] workqueue knbd0-recv: flags=0x1a [14820.056087] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14820.056095] in-flight: 658:recv_work [14820.056105] workqueue ptlrpc_pinger: flags=0x2 [14820.056108] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14820.056114] in-flight: [14820.057015] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [14820.056116] 38:ptlrpc_pinger_main [ptlrpc] [14820.057222] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12889s workers=4 idle: 32984 37 8 [14820.057236] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14804s workers=2 idle: 661 [14820.061655] Modules linked in: [14820.063968] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic [14820.065372] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14820.065450] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14820.065457] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14820.065460] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14820.065475] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14820.065480] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14820.065485] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [14820.065487] RDX: 000000000000243c RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [14820.065489] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [14820.065491] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [14820.065494] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14820.065496] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [14820.065499] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14820.065501] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14820.065509] Call Trace: [14820.065514] [14820.065519] ? show_regs.cold.9+0x22/0x2f [14820.065527] ? watchdog_timer_fn.cold.6+0x62/0x10a [14820.065533] ? __hrtimer_run_queues+0x1c2/0x4d0 [14820.065537] ? watchdog+0x40/0x40 [14820.065542] ? hrtimer_interrupt+0x128/0x2c0 [14820.065546] ? smp_apic_timer_interrupt+0xa6/0x270 [14820.065555] ? apic_timer_interrupt+0xf/0x20 [14820.065559] [14820.065562] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14820.065567] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14820.065572] do_raw_spin_lock+0x177/0x190 [14820.065577] _raw_spin_lock+0x21/0x30 [14820.065583] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14820.065853] obd_mod_free+0x46/0x140 [ptlrpc] [14820.066043] mdc_commit_open+0x91/0xb0 [mdc] [14820.066081] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14820.066273] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14820.066480] after_reply+0xe2c/0x1780 [ptlrpc] [14820.066671] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14820.066864] ? __next_timer_interrupt+0x160/0x160 [14820.066872] ptlrpcd+0xa82/0xe40 [ptlrpc] [14820.067070] ? do_wait_intr+0xf0/0xf0 [14820.067078] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14820.067274] kthread+0x1d1/0x200 [14820.067279] ? set_kthread_struct+0x70/0x70 [14820.067283] ret_from_fork+0x1f/0x30 [14820.375300] crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14820.384951] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14820.392903] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14820.399490] RIP: 0010:native_safe_halt+0x2e/0x50 [14820.403335] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14820.421106] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14820.428223] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14820.436791] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [14820.446982] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [14820.452086] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [14820.456553] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [14820.463512] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [14820.467551] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14820.478835] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [14820.483125] Call Trace: [14820.485830] [14820.487841] ? show_regs.cold.9+0x22/0x2f [14820.493131] ? watchdog_timer_fn.cold.6+0x62/0x10a [14820.497583] ? __hrtimer_run_queues+0x1c2/0x4d0 [14820.503047] ? watchdog+0x40/0x40 [14820.506614] ? hrtimer_interrupt+0x128/0x2c0 [14820.507710] ? smp_apic_timer_interrupt+0xa6/0x270 [14820.508956] ? apic_timer_interrupt+0xf/0x20 [14820.513117] [14820.513858] ? native_safe_halt+0x2e/0x50 [14820.516048] kvm_wait+0xcd/0xe0 [14820.519501] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14820.522003] do_raw_spin_lock+0x177/0x190 [14820.524296] _raw_spin_lock+0x21/0x30 [14820.525564] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14820.530788] ? do_raw_spin_unlock+0x75/0x190 [14820.533130] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14820.542284] ? do_wait_intr+0xf0/0xf0 [14820.544162] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14820.547404] mdc_close+0x53b/0xd70 [mdc] [14820.551728] lmv_close+0x18d/0x490 [lmv] [14820.553162] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14820.555698] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14820.558000] ll_md_real_close+0xa5/0x2a0 [lustre] [14820.560536] ll_md_close+0x274/0x800 [lustre] [14820.562494] ll_file_release+0x128/0xc50 [lustre] [14820.567338] __fput+0x10d/0x440 [14820.570770] ____fput+0x12/0x20 [14820.572576] task_work_run+0xc6/0x110 [14820.575626] exit_to_usermode_loop+0x1dd/0x1f0 [14820.578111] do_syscall_64+0x3d6/0x440 [14820.581461] entry_SYSCALL_64_after_hwframe+0x49/0xae [14820.585603] RIP: 0033:0x7f151bee3cd8 [14820.587796] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14820.601721] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14820.606964] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [14820.611666] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [14820.614359] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14820.617258] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14820.620524] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14836.056136] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [14836.059656] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14836.097531] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14836.104896] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14836.118049] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14836.132323] RIP: 0010:native_safe_halt+0x2e/0x50 [14836.134018] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14836.147714] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14836.154914] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14836.162088] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [14836.168202] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [14836.170945] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [14836.173374] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [14836.178145] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [14836.184522] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14836.187667] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14836.190710] Call Trace: [14836.191779] [14836.192711] ? show_regs.cold.9+0x22/0x2f [14836.194210] ? watchdog_timer_fn.cold.6+0x62/0x10a [14836.197949] ? __hrtimer_run_queues+0x1c2/0x4d0 [14836.200320] ? watchdog+0x40/0x40 [14836.201713] ? hrtimer_interrupt+0x128/0x2c0 [14836.205034] ? smp_apic_timer_interrupt+0xa6/0x270 [14836.207567] ? apic_timer_interrupt+0xf/0x20 [14836.210033] [14836.210864] ? native_safe_halt+0x2e/0x50 [14836.212612] kvm_wait+0xcd/0xe0 [14836.214611] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14836.216931] do_raw_spin_lock+0x177/0x190 [14836.219864] _raw_spin_lock+0x21/0x30 [14836.224648] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14836.229735] process_one_work+0x2c8/0x700 [14836.231194] worker_thread+0x296/0x6e0 [14836.235017] ? rescuer_thread+0x570/0x570 [14836.237362] kthread+0x1d1/0x200 [14836.238876] ? set_kthread_struct+0x70/0x70 [14836.247108] ret_from_fork+0x1f/0x30 [14848.052030] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [14848.054359] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14848.057024] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [14848.058187] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [14848.058196] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14848.058294] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14848.058301] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14848.058304] RIP: 0010:native_safe_halt+0x2e/0x50 [14848.058368] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14848.058372] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14848.058377] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [14848.058380] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [14848.058382] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [14848.058384] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [14848.058387] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14848.058390] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [14848.058393] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14848.058396] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14848.058409] Call Trace: [14848.058430] [14848.058440] ? show_regs.cold.9+0x22/0x2f [14848.058454] ? watchdog_timer_fn.cold.6+0x62/0x10a [14848.058467] ? __hrtimer_run_queues+0x1c2/0x4d0 [14848.058489] ? watchdog+0x40/0x40 [14848.058497] ? hrtimer_interrupt+0x128/0x2c0 [14848.058506] ? smp_apic_timer_interrupt+0xa6/0x270 [14848.058519] ? apic_timer_interrupt+0xf/0x20 [14848.058528] [14848.058531] ? native_safe_halt+0x2e/0x50 [14848.058535] kvm_wait+0xcd/0xe0 [14848.058545] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14848.058553] do_raw_spin_lock+0x177/0x190 [14848.058559] _raw_spin_lock+0x21/0x30 [14848.058566] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14848.059081] obd_mod_free+0x46/0x140 [ptlrpc] [14848.059344] mdc_commit_open+0x91/0xb0 [mdc] [14848.059397] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14848.059731] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14848.060016] after_reply+0xe2c/0x1780 [ptlrpc] [14848.060223] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14848.060452] ? __next_timer_interrupt+0x160/0x160 [14848.060465] ptlrpcd+0xa82/0xe40 [ptlrpc] [14848.060704] ? do_wait_intr+0xf0/0xf0 [14848.060714] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14848.060947] kthread+0x1d1/0x200 [14848.060953] ? set_kthread_struct+0x70/0x70 [14848.060956] ret_from_fork+0x1f/0x30 [14848.080589] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14848.080601] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14848.080604] RIP: 0010:native_safe_halt+0x2e/0x50 [14848.080617] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14848.080621] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14848.080626] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14848.080629] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [14848.080630] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [14848.080632] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [14848.080635] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [14848.080638] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [14848.080640] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14848.080642] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [14848.080650] Call Trace: [14848.080656] [14848.080662] ? show_regs.cold.9+0x22/0x2f [14848.080672] ? watchdog_timer_fn.cold.6+0x62/0x10a [14848.080679] ? __hrtimer_run_queues+0x1c2/0x4d0 [14848.080682] ? watchdog+0x40/0x40 [14848.080686] ? hrtimer_interrupt+0x128/0x2c0 [14848.080690] ? smp_apic_timer_interrupt+0xa6/0x270 [14848.080696] ? apic_timer_interrupt+0xf/0x20 [14848.080700] [14848.080703] ? native_safe_halt+0x2e/0x50 [14848.080707] kvm_wait+0xcd/0xe0 [14848.080716] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14848.080721] do_raw_spin_lock+0x177/0x190 [14848.080724] _raw_spin_lock+0x21/0x30 [14848.080729] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14848.080932] ? do_raw_spin_unlock+0x75/0x190 [14848.080936] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14848.083450] Modules linked in: [14848.087408] ? do_wait_intr+0xf0/0xf0 [14848.087426] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14848.087720] mdc_reint+0x2d/0x110 [mdc] [14848.087761] mdc_unlink+0x658/0x870 [mdc] [14848.087789] lmv_unlink+0x34d/0xbd0 [lmv] [14848.087820] ll_unlink+0x24d/0x7f0 [lustre] [14848.088033] vfs_unlink+0x202/0x330 [14848.088041] do_unlinkat+0x2fa/0x460 [14848.088046] __x64_sys_unlink+0x37/0x50 [14848.088049] do_syscall_64+0xc1/0x440 [14848.088055] entry_SYSCALL_64_after_hwframe+0x49/0xae [14848.088064] RIP: 0033:0x7f151bdee10b [14848.088112] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [14848.088117] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [14848.088122] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [14848.088124] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [14848.088126] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14848.088129] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14848.088133] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14848.251745] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14848.264991] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14848.267744] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14848.270129] RIP: 0010:native_safe_halt+0x2e/0x50 [14848.271368] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14848.276316] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14848.278346] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14848.280211] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [14848.281997] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [14848.283951] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [14848.285808] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [14848.287972] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [14848.290415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14848.292001] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [14848.293930] Call Trace: [14848.294617] [14848.295312] ? show_regs.cold.9+0x22/0x2f [14848.296451] ? watchdog_timer_fn.cold.6+0x62/0x10a [14848.297885] ? __hrtimer_run_queues+0x1c2/0x4d0 [14848.299301] ? watchdog+0x40/0x40 [14848.300352] ? hrtimer_interrupt+0x128/0x2c0 [14848.301507] ? smp_apic_timer_interrupt+0xa6/0x270 [14848.302732] ? apic_timer_interrupt+0xf/0x20 [14848.304110] [14848.304621] ? native_safe_halt+0x2e/0x50 [14848.305732] kvm_wait+0xcd/0xe0 [14848.306803] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14848.308374] do_raw_spin_lock+0x177/0x190 [14848.309521] _raw_spin_lock+0x21/0x30 [14848.310626] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14848.312582] ? do_raw_spin_unlock+0x75/0x190 [14848.314061] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14848.315547] ? do_wait_intr+0xf0/0xf0 [14848.316606] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14848.318322] mdc_close+0x53b/0xd70 [mdc] [14848.319412] lmv_close+0x18d/0x490 [lmv] [14848.320516] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14848.322122] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14848.323728] ll_md_real_close+0xa5/0x2a0 [lustre] [14848.325274] ll_md_close+0x274/0x800 [lustre] [14848.326580] ll_file_release+0x128/0xc50 [lustre] [14848.327949] __fput+0x10d/0x440 [14848.328764] ____fput+0x12/0x20 [14848.329623] task_work_run+0xc6/0x110 [14848.330616] exit_to_usermode_loop+0x1dd/0x1f0 [14848.331796] do_syscall_64+0x3d6/0x440 [14848.332793] entry_SYSCALL_64_after_hwframe+0x49/0xae [14848.334235] RIP: 0033:0x7f151bee3cd8 [14848.335365] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14848.340794] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14848.343153] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [14848.345185] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [14848.347122] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14848.349119] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14848.351404] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14850.531828] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12919s! [14850.541608] Showing busy workqueues and worker pools: [14850.546804] workqueue events: flags=0x0 [14850.547964] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14850.549713] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [14850.556568] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14850.562167] pending: kfree_rcu_monitor [14850.563467] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14850.570843] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [14850.579591] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14850.584298] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [14850.589613] workqueue events_long: flags=0x0 [14850.591527] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14850.598308] pending: lu_objects_flush [obdclass] [14850.601260] workqueue events_unbound: flags=0x2 [14850.603490] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14850.606690] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14850.611493] workqueue events_power_efficient: flags=0x80 [14850.613473] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14850.615836] pending: do_cache_clean, neigh_periodic_work, check_lifetime [14850.618596] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14850.621640] pending: neigh_periodic_work [14850.623410] workqueue mm_percpu_wq: flags=0x8 [14850.625524] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14850.628344] pending: vmstat_update, lru_add_drain_per_cpu [14850.630469] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14850.635191] pending: vmstat_update [14850.637129] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14850.639265] pending: lru_add_drain_per_cpu [14850.641388] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [14850.643829] pending: lru_add_drain_per_cpu BAR(48) [14850.645681] workqueue writeback: flags=0x4a [14850.647203] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [14850.649212] pending: wb_workfn [14850.650345] workqueue xprtiod: flags=0xa [14850.651648] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14850.653775] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [14850.656618] workqueue ipv6_addrconf: flags=0x40008 [14850.658235] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14850.660663] pending: addrconf_verify_work [14850.662447] workqueue knbd0-recv: flags=0x1a [14850.664083] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14850.666382] in-flight: 658:recv_work [14850.667663] workqueue ptlrpc_pinger: flags=0x2 [14850.669158] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14850.671084] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14850.673256] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12920s workers=4 idle: 32984 37 8 [14850.675685] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14835s workers=2 idle: 661 [14864.059678] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [14864.064727] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14864.099060] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14864.114339] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14864.119500] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14864.123686] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14864.130791] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14864.146209] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14864.149665] RAX: 0000000000007e0f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14864.158763] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [14864.164326] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [14864.167561] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [14864.170726] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [14864.176269] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [14864.178560] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14864.180361] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14864.185736] Call Trace: [14864.186563] [14864.187350] ? show_regs.cold.9+0x22/0x2f [14864.188633] ? watchdog_timer_fn.cold.6+0x62/0x10a [14864.191779] ? __hrtimer_run_queues+0x1c2/0x4d0 [14864.193696] ? watchdog+0x40/0x40 [14864.196225] ? hrtimer_interrupt+0x128/0x2c0 [14864.197220] ? smp_apic_timer_interrupt+0xa6/0x270 [14864.198275] ? apic_timer_interrupt+0xf/0x20 [14864.201218] [14864.201675] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14864.203773] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14864.208225] do_raw_spin_lock+0x177/0x190 [14864.209173] _raw_spin_lock+0x21/0x30 [14864.209889] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14864.212304] process_one_work+0x2c8/0x700 [14864.213515] worker_thread+0x296/0x6e0 [14864.216487] ? rescuer_thread+0x570/0x570 [14864.218008] kthread+0x1d1/0x200 [14864.218813] ? set_kthread_struct+0x70/0x70 [14864.220017] ret_from_fork+0x1f/0x30 [14876.052361] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [14876.055021] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [14876.057023] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [14876.057033] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14876.057121] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14876.057127] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14876.057130] RIP: 0010:native_safe_halt+0x2e/0x50 [14876.057144] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14876.057149] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14876.057154] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14876.057156] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [14876.057158] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [14876.057161] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [14876.057163] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [14876.057167] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [14876.057171] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14876.057173] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [14876.057186] Call Trace: [14876.057191] [14876.057196] ? show_regs.cold.9+0x22/0x2f [14876.057205] ? watchdog_timer_fn.cold.6+0x62/0x10a [14876.057214] ? __hrtimer_run_queues+0x1c2/0x4d0 [14876.057219] ? watchdog+0x40/0x40 [14876.057223] ? hrtimer_interrupt+0x128/0x2c0 [14876.057228] ? smp_apic_timer_interrupt+0xa6/0x270 [14876.057234] ? apic_timer_interrupt+0xf/0x20 [14876.057239] [14876.057242] ? native_safe_halt+0x2e/0x50 [14876.057246] kvm_wait+0xcd/0xe0 [14876.057253] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14876.057261] do_raw_spin_lock+0x177/0x190 [14876.057265] _raw_spin_lock+0x21/0x30 [14876.057269] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14876.057590] ? do_raw_spin_unlock+0x75/0x190 [14876.057596] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14876.057860] ? do_wait_intr+0xf0/0xf0 [14876.057877] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14876.058071] mdc_close+0x53b/0xd70 [mdc] [14876.058116] lmv_close+0x18d/0x490 [lmv] [14876.058143] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14876.058319] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14876.058502] ll_md_real_close+0xa5/0x2a0 [lustre] [14876.058670] ll_md_close+0x274/0x800 [lustre] [14876.058711] Modules linked in: lustre(O) [14876.058839] ll_file_release+0x128/0xc50 [lustre] [14876.058972] __fput+0x10d/0x440 [14876.058980] ____fput+0x12/0x20 [14876.058984] task_work_run+0xc6/0x110 [14876.058991] exit_to_usermode_loop+0x1dd/0x1f0 [14876.058999] do_syscall_64+0x3d6/0x440 [14876.059003] entry_SYSCALL_64_after_hwframe+0x49/0xae [14876.059009] RIP: 0033:0x7f151bee3cd8 [14876.059015] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14876.059019] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14876.059023] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [14876.059025] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [14876.059027] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14876.059029] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14876.059031] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14876.061129] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14876.061210] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14876.061218] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14876.061221] RIP: 0010:native_safe_halt+0x2e/0x50 [14876.061231] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14876.061236] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14876.061240] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [14876.061243] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [14876.061245] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [14876.061247] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [14876.061250] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14876.061253] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [14876.061256] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14876.061258] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14876.061266] Call Trace: [14876.061271] [14876.061275] ? show_regs.cold.9+0x22/0x2f [14876.061283] ? watchdog_timer_fn.cold.6+0x62/0x10a [14876.061290] ? __hrtimer_run_queues+0x1c2/0x4d0 [14876.061294] ? watchdog+0x40/0x40 [14876.061299] ? hrtimer_interrupt+0x128/0x2c0 [14876.061303] ? smp_apic_timer_interrupt+0xa6/0x270 [14876.061310] ? apic_timer_interrupt+0xf/0x20 [14876.061320] [14876.061323] ? native_safe_halt+0x2e/0x50 [14876.061327] kvm_wait+0xcd/0xe0 [14876.061334] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14876.061341] do_raw_spin_lock+0x177/0x190 [14876.061346] _raw_spin_lock+0x21/0x30 [14876.061351] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14876.061629] obd_mod_free+0x46/0x140 [ptlrpc] [14876.061810] mdc_commit_open+0x91/0xb0 [mdc] [14876.061844] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14876.068546] osp(O) [14876.084408] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14876.095467] ofd(O) [14876.100176] after_reply+0xe2c/0x1780 [ptlrpc] [14876.100493] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14876.100739] ? __next_timer_interrupt+0x160/0x160 [14876.100749] ptlrpcd+0xa82/0xe40 [ptlrpc] [14876.103628] lod(O) [14876.110285] ? do_wait_intr+0xf0/0xf0 [14876.110306] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14876.114303] mdt(O) [14876.117490] kthread+0x1d1/0x200 [14876.117510] ? set_kthread_struct+0x70/0x70 [14876.117516] ret_from_fork+0x1f/0x30 [14876.471654] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14876.491264] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14876.494583] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14876.497833] RIP: 0010:native_safe_halt+0x2e/0x50 [14876.499244] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14876.505644] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14876.508742] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14876.511170] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [14876.513776] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [14876.518128] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [14876.520361] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [14876.522564] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [14876.528478] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14876.535182] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [14876.536791] Call Trace: [14876.537625] [14876.539534] ? show_regs.cold.9+0x22/0x2f [14876.542197] ? watchdog_timer_fn.cold.6+0x62/0x10a [14876.546519] ? __hrtimer_run_queues+0x1c2/0x4d0 [14876.549585] ? watchdog+0x40/0x40 [14876.550786] ? hrtimer_interrupt+0x128/0x2c0 [14876.552425] ? smp_apic_timer_interrupt+0xa6/0x270 [14876.554162] ? apic_timer_interrupt+0xf/0x20 [14876.555358] [14876.556097] ? native_safe_halt+0x2e/0x50 [14876.557598] kvm_wait+0xcd/0xe0 [14876.558603] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14876.560319] do_raw_spin_lock+0x177/0x190 [14876.562658] _raw_spin_lock+0x21/0x30 [14876.564402] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14876.566570] ? do_raw_spin_unlock+0x75/0x190 [14876.568208] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14876.570618] ? do_wait_intr+0xf0/0xf0 [14876.571746] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14876.574709] mdc_reint+0x2d/0x110 [mdc] [14876.576517] mdc_unlink+0x658/0x870 [mdc] [14876.578328] lmv_unlink+0x34d/0xbd0 [lmv] [14876.580171] ll_unlink+0x24d/0x7f0 [lustre] [14876.582266] vfs_unlink+0x202/0x330 [14876.583690] do_unlinkat+0x2fa/0x460 [14876.585276] __x64_sys_unlink+0x37/0x50 [14876.586838] do_syscall_64+0xc1/0x440 [14876.588463] entry_SYSCALL_64_after_hwframe+0x49/0xae [14876.590015] RIP: 0033:0x7f151bdee10b [14876.591140] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [14876.599687] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [14876.603482] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [14876.609041] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [14876.616703] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14876.624210] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14876.630548] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14880.739774] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12950s! [14880.750085] Showing busy workqueues and worker pools: [14880.754405] workqueue events: flags=0x0 [14880.757053] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14880.763201] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [14880.770956] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14880.775993] pending: kfree_rcu_monitor [14880.779094] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14880.783852] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [14880.790936] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14880.795977] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [14880.802413] workqueue events_long: flags=0x0 [14880.805587] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14880.810504] pending: lu_objects_flush [obdclass] [14880.814629] workqueue events_unbound: flags=0x2 [14880.817447] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14880.821015] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14880.828774] workqueue events_power_efficient: flags=0x80 [14880.832594] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14880.836929] pending: do_cache_clean, neigh_periodic_work, check_lifetime [14880.841384] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14880.845730] pending: neigh_periodic_work [14880.848441] workqueue mm_percpu_wq: flags=0x8 [14880.850959] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14880.854820] pending: vmstat_update, lru_add_drain_per_cpu [14880.858631] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14880.862566] pending: vmstat_update [14880.864052] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14880.867603] pending: lru_add_drain_per_cpu [14880.869629] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [14880.872488] pending: lru_add_drain_per_cpu BAR(48) [14880.875791] workqueue writeback: flags=0x4a [14880.877604] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [14880.880641] pending: wb_workfn [14880.882501] workqueue xprtiod: flags=0xa [14880.884422] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14880.887176] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [14880.891483] workqueue ipv6_addrconf: flags=0x40008 [14880.893594] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14880.896905] pending: addrconf_verify_work [14880.898935] workqueue knbd0-recv: flags=0x1a [14880.900932] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14880.903938] in-flight: 658:recv_work [14880.905752] workqueue ptlrpc_pinger: flags=0x2 [14880.907859] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14880.910550] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14880.913674] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12950s workers=4 idle: 32984 37 8 [14880.917239] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14865s workers=2 idle: 661 [14892.057112] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [14892.063275] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14892.088416] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14892.093477] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14892.098156] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14892.100471] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14892.103301] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14892.110739] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14892.113095] RAX: 0000000000007f28 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14892.115248] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [14892.117859] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [14892.120242] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [14892.122466] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [14892.124748] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [14892.129813] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14892.131662] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14892.134103] Call Trace: [14892.134964] [14892.135730] ? show_regs.cold.9+0x22/0x2f [14892.140563] ? watchdog_timer_fn.cold.6+0x62/0x10a [14892.141873] ? __hrtimer_run_queues+0x1c2/0x4d0 [14892.143512] ? watchdog+0x40/0x40 [14892.144402] ? hrtimer_interrupt+0x128/0x2c0 [14892.146014] ? smp_apic_timer_interrupt+0xa6/0x270 [14892.147469] ? apic_timer_interrupt+0xf/0x20 [14892.149145] [14892.149843] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14892.151123] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14892.152868] do_raw_spin_lock+0x177/0x190 [14892.154123] _raw_spin_lock+0x21/0x30 [14892.155321] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14892.158163] process_one_work+0x2c8/0x700 [14892.159939] worker_thread+0x296/0x6e0 [14892.161095] ? rescuer_thread+0x570/0x570 [14892.162748] kthread+0x1d1/0x200 [14892.163947] ? set_kthread_struct+0x70/0x70 [14892.165177] ret_from_fork+0x1f/0x30 [14904.057286] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [14904.058262] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [14904.060429] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14904.060569] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14904.060577] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14904.060581] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14904.060594] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14904.060606] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14904.060611] RAX: 0000000000001550 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14904.060614] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [14904.060616] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [14904.060619] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [14904.060621] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [14904.060628] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [14904.060631] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14904.060633] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [14904.060640] Call Trace: [14904.060646] [14904.060653] ? show_regs.cold.9+0x22/0x2f [14904.060662] ? watchdog_timer_fn.cold.6+0x62/0x10a [14904.060669] ? __hrtimer_run_queues+0x1c2/0x4d0 [14904.060673] ? watchdog+0x40/0x40 [14904.060678] ? hrtimer_interrupt+0x128/0x2c0 [14904.060682] ? smp_apic_timer_interrupt+0xa6/0x270 [14904.060690] ? apic_timer_interrupt+0xf/0x20 [14904.060697] [14904.060699] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14904.060706] do_raw_spin_lock+0x177/0x190 [14904.060711] _raw_spin_lock+0x21/0x30 [14904.060716] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14904.061013] ? do_raw_spin_unlock+0x75/0x190 [14904.061019] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14904.061231] ? do_wait_intr+0xf0/0xf0 [14904.061239] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14904.065656] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14904.065741] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14904.065749] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14904.065752] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14904.065780] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14904.065785] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14904.065790] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [14904.065792] RDX: 0000000000007e60 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [14904.065795] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [14904.065798] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [14904.065800] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14904.065803] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [14904.065806] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14904.065809] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14904.065817] Call Trace: [14904.065822] [14904.065828] ? show_regs.cold.9+0x22/0x2f [14904.065839] ? watchdog_timer_fn.cold.6+0x62/0x10a [14904.065846] ? __hrtimer_run_queues+0x1c2/0x4d0 [14904.065850] ? watchdog+0x40/0x40 [14904.065854] ? hrtimer_interrupt+0x128/0x2c0 [14904.065858] ? smp_apic_timer_interrupt+0xa6/0x270 [14904.065866] ? apic_timer_interrupt+0xf/0x20 [14904.065870] [14904.065873] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14904.065879] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14904.065883] do_raw_spin_lock+0x177/0x190 [14904.065888] _raw_spin_lock+0x21/0x30 [14904.065894] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14904.066171] obd_mod_free+0x46/0x140 [ptlrpc] [14904.066762] mdc_commit_open+0x91/0xb0 [mdc] [14904.066806] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14904.067021] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14904.067237] after_reply+0xe2c/0x1780 [ptlrpc] [14904.067445] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14904.091584] mdc_close+0x53b/0xd70 [mdc] [14904.091646] lmv_close+0x18d/0x490 [lmv] [14904.091676] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14904.091847] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14904.091965] ll_md_real_close+0xa5/0x2a0 [lustre] [14904.092596] ll_md_close+0x274/0x800 [lustre] [14904.092697] ll_file_release+0x128/0xc50 [lustre] [14904.092785] __fput+0x10d/0x440 [14904.092794] ____fput+0x12/0x20 [14904.092798] task_work_run+0xc6/0x110 [14904.092806] exit_to_usermode_loop+0x1dd/0x1f0 [14904.092813] do_syscall_64+0x3d6/0x440 [14904.092818] entry_SYSCALL_64_after_hwframe+0x49/0xae [14904.092827] RIP: 0033:0x7f151bee3cd8 [14904.092834] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14904.092838] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14904.092843] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [14904.092846] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [14904.092848] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14904.092850] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14904.092852] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14904.375832] ? __next_timer_interrupt+0x160/0x160 [14904.377961] ptlrpcd+0xa82/0xe40 [ptlrpc] [14904.380801] ? do_wait_intr+0xf0/0xf0 [14904.383627] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14904.386781] kthread+0x1d1/0x200 [14904.388097] ? set_kthread_struct+0x70/0x70 [14904.390837] ret_from_fork+0x1f/0x30 [14908.054336] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [14908.060404] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14908.112377] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14908.123723] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14908.127843] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14908.135766] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14908.152313] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14908.160195] RAX: 0000000000005f97 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14908.169381] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [14908.177996] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [14908.184851] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [14908.191856] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [14908.197575] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [14908.203782] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14908.205902] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [14908.209912] Call Trace: [14908.211269] [14908.212157] ? show_regs.cold.9+0x22/0x2f [14908.213762] ? watchdog_timer_fn.cold.6+0x62/0x10a [14908.215799] ? __hrtimer_run_queues+0x1c2/0x4d0 [14908.217781] ? watchdog+0x40/0x40 [14908.219356] ? hrtimer_interrupt+0x128/0x2c0 [14908.221260] ? smp_apic_timer_interrupt+0xa6/0x270 [14908.223376] ? apic_timer_interrupt+0xf/0x20 [14908.225598] [14908.226412] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14908.228755] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14908.230966] do_raw_spin_lock+0x177/0x190 [14908.232808] _raw_spin_lock+0x21/0x30 [14908.234801] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14908.237237] ? do_raw_spin_unlock+0x75/0x190 [14908.239141] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14908.241365] ? do_wait_intr+0xf0/0xf0 [14908.242888] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14908.245353] mdc_reint+0x2d/0x110 [mdc] [14908.247296] mdc_unlink+0x658/0x870 [mdc] [14908.249941] lmv_unlink+0x34d/0xbd0 [lmv] [14908.252219] ll_unlink+0x24d/0x7f0 [lustre] [14908.254829] vfs_unlink+0x202/0x330 [14908.258130] do_unlinkat+0x2fa/0x460 [14908.260307] __x64_sys_unlink+0x37/0x50 [14908.262200] do_syscall_64+0xc1/0x440 [14908.263578] entry_SYSCALL_64_after_hwframe+0x49/0xae [14908.266989] RIP: 0033:0x7f151bdee10b [14908.268350] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [14908.276848] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [14908.283532] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [14908.287154] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [14908.288970] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14908.291243] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14908.293016] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14910.945856] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12980s! [14910.955676] Showing busy workqueues and worker pools: [14910.962077] workqueue events: flags=0x0 [14910.965943] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14910.969715] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [14910.982808] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14910.988788] pending: kfree_rcu_monitor [14910.994051] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14910.998510] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [14911.013321] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14911.017955] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [14911.021892] workqueue events_long: flags=0x0 [14911.023323] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14911.026646] pending: lu_objects_flush [obdclass] [14911.028934] workqueue events_unbound: flags=0x2 [14911.031261] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14911.036925] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14911.045871] workqueue events_power_efficient: flags=0x80 [14911.053845] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14911.057616] pending: do_cache_clean, neigh_periodic_work, check_lifetime [14911.064851] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14911.073027] pending: neigh_periodic_work [14911.076013] workqueue mm_percpu_wq: flags=0x8 [14911.080929] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14911.083519] pending: vmstat_update, lru_add_drain_per_cpu [14911.087071] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14911.092698] pending: vmstat_update [14911.094614] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14911.097161] pending: lru_add_drain_per_cpu [14911.099216] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [14911.101881] pending: lru_add_drain_per_cpu BAR(48) [14911.104380] workqueue writeback: flags=0x4a [14911.106907] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [14911.110022] pending: wb_workfn [14911.114261] workqueue xprtiod: flags=0xa [14911.116018] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14911.120925] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [14911.124978] workqueue ipv6_addrconf: flags=0x40008 [14911.127139] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14911.129135] pending: addrconf_verify_work [14911.131512] workqueue knbd0-recv: flags=0x1a [14911.133187] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14911.137262] in-flight: 658:recv_work [14911.142728] workqueue ptlrpc_pinger: flags=0x2 [14911.144745] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14911.147051] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14911.149545] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12980s workers=4 idle: 32984 37 8 [14911.152528] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14895s workers=2 idle: 661 [14920.056027] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14920.059659] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14920.099898] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14920.103964] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14920.107221] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14920.109461] RIP: 0010:native_safe_halt+0x2e/0x50 [14920.111186] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14920.122084] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14920.124305] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14920.129680] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [14920.133450] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [14920.136591] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [14920.142025] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [14920.149104] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [14920.155212] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14920.161672] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14920.166118] Call Trace: [14920.168811] [14920.170754] ? show_regs.cold.9+0x22/0x2f [14920.174896] ? watchdog_timer_fn.cold.6+0x62/0x10a [14920.177695] ? __hrtimer_run_queues+0x1c2/0x4d0 [14920.180623] ? watchdog+0x40/0x40 [14920.182566] ? hrtimer_interrupt+0x128/0x2c0 [14920.184979] ? smp_apic_timer_interrupt+0xa6/0x270 [14920.187544] ? apic_timer_interrupt+0xf/0x20 [14920.188944] [14920.190010] ? native_safe_halt+0x2e/0x50 [14920.191566] kvm_wait+0xcd/0xe0 [14920.193008] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14920.198580] do_raw_spin_lock+0x177/0x190 [14920.201621] _raw_spin_lock+0x21/0x30 [14920.204187] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14920.207034] process_one_work+0x2c8/0x700 [14920.209023] worker_thread+0x296/0x6e0 [14920.210914] ? rescuer_thread+0x570/0x570 [14920.213090] kthread+0x1d1/0x200 [14920.214779] ? set_kthread_struct+0x70/0x70 [14920.216863] ret_from_fork+0x1f/0x30 [14932.055499] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [14932.057028] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [14932.061186] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14932.061390] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14932.061398] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14932.061401] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14932.061417] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14932.061421] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14932.061426] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [14932.061428] RDX: 000000000000677e RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [14932.061430] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [14932.061433] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [14932.061436] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14932.061439] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [14932.061442] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14932.061445] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14932.074805] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14932.074882] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14932.074889] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14932.074892] RIP: 0010:native_safe_halt+0x2e/0x50 [14932.074905] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14932.074909] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14932.074914] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14932.074916] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [14932.074918] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [14932.074920] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [14932.074922] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [14932.074925] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [14932.074928] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14932.074930] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [14932.074939] Call Trace: [14932.074944] [14932.074950] ? show_regs.cold.9+0x22/0x2f [14932.074959] ? watchdog_timer_fn.cold.6+0x62/0x10a [14932.074964] ? __hrtimer_run_queues+0x1c2/0x4d0 [14932.074968] ? watchdog+0x40/0x40 [14932.074972] ? hrtimer_interrupt+0x128/0x2c0 [14932.074976] ? smp_apic_timer_interrupt+0xa6/0x270 [14932.074982] ? apic_timer_interrupt+0xf/0x20 [14932.074986] [14932.074989] ? native_safe_halt+0x2e/0x50 [14932.074993] kvm_wait+0xcd/0xe0 [14932.075000] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14932.075009] do_raw_spin_lock+0x177/0x190 [14932.075680] _raw_spin_lock+0x21/0x30 [14932.075685] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14932.075937] ? do_raw_spin_unlock+0x75/0x190 [14932.075943] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14932.122822] Call Trace: [14932.122835] [14932.122841] ? show_regs.cold.9+0x22/0x2f [14932.123563] ? watchdog_timer_fn.cold.6+0x62/0x10a [14932.123569] ? __hrtimer_run_queues+0x1c2/0x4d0 [14932.123574] ? watchdog+0x40/0x40 [14932.123578] ? hrtimer_interrupt+0x128/0x2c0 [14932.123583] ? smp_apic_timer_interrupt+0xa6/0x270 [14932.123590] ? apic_timer_interrupt+0xf/0x20 [14932.123595] [14932.123598] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14932.123605] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14932.123610] do_raw_spin_lock+0x177/0x190 [14932.123615] _raw_spin_lock+0x21/0x30 [14932.123622] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14932.123902] obd_mod_free+0x46/0x140 [ptlrpc] [14932.124318] mdc_commit_open+0x91/0xb0 [mdc] [14932.124366] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14932.124584] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14932.124816] after_reply+0xe2c/0x1780 [ptlrpc] [14932.135976] ? do_wait_intr+0xf0/0xf0 [14932.135999] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14932.147797] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14932.148049] ? __next_timer_interrupt+0x160/0x160 [14932.148058] ptlrpcd+0xa82/0xe40 [ptlrpc] [14932.148217] ? do_wait_intr+0xf0/0xf0 [14932.148225] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14932.148365] kthread+0x1d1/0x200 [14932.148369] ? set_kthread_struct+0x70/0x70 [14932.148372] ret_from_fork+0x1f/0x30 [14932.438085] mdc_close+0x53b/0xd70 [mdc] [14932.439283] lmv_close+0x18d/0x490 [lmv] [14932.440767] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14932.442482] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14932.444286] ll_md_real_close+0xa5/0x2a0 [lustre] [14932.445637] ll_md_close+0x274/0x800 [lustre] [14932.446840] ll_file_release+0x128/0xc50 [lustre] [14932.448761] __fput+0x10d/0x440 [14932.449687] ____fput+0x12/0x20 [14932.450570] task_work_run+0xc6/0x110 [14932.451742] exit_to_usermode_loop+0x1dd/0x1f0 [14932.452958] do_syscall_64+0x3d6/0x440 [14932.454480] entry_SYSCALL_64_after_hwframe+0x49/0xae [14932.460161] RIP: 0033:0x7f151bee3cd8 [14932.462562] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14932.470826] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14932.473230] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [14932.475006] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [14932.478652] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14932.483574] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14932.487627] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14936.052026] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [14936.055122] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14936.077361] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14936.082721] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14936.086400] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14936.088760] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14936.095707] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14936.098661] RAX: 000000000000792b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14936.101405] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [14936.108995] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [14936.113246] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [14936.116378] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [14936.119232] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [14936.121996] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14936.124335] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [14936.127078] Call Trace: [14936.128937] [14936.129749] ? show_regs.cold.9+0x22/0x2f [14936.131018] ? watchdog_timer_fn.cold.6+0x62/0x10a [14936.133505] ? __hrtimer_run_queues+0x1c2/0x4d0 [14936.135139] ? watchdog+0x40/0x40 [14936.136418] ? hrtimer_interrupt+0x128/0x2c0 [14936.138684] ? smp_apic_timer_interrupt+0xa6/0x270 [14936.140470] ? apic_timer_interrupt+0xf/0x20 [14936.141976] [14936.142798] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14936.144553] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14936.146235] do_raw_spin_lock+0x177/0x190 [14936.147731] _raw_spin_lock+0x21/0x30 [14936.148962] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14936.151131] ? do_raw_spin_unlock+0x75/0x190 [14936.152528] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14936.155001] ? do_wait_intr+0xf0/0xf0 [14936.156563] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14936.159058] mdc_reint+0x2d/0x110 [mdc] [14936.160697] mdc_unlink+0x658/0x870 [mdc] [14936.162482] lmv_unlink+0x34d/0xbd0 [lmv] [14936.164283] ll_unlink+0x24d/0x7f0 [lustre] [14936.166268] vfs_unlink+0x202/0x330 [14936.167807] do_unlinkat+0x2fa/0x460 [14936.169112] __x64_sys_unlink+0x37/0x50 [14936.170987] do_syscall_64+0xc1/0x440 [14936.172533] entry_SYSCALL_64_after_hwframe+0x49/0xae [14936.174819] RIP: 0033:0x7f151bdee10b [14936.176077] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [14936.183743] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [14936.186810] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [14936.189764] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [14936.192609] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14936.195383] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14936.197813] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14941.665010] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13011s! [14941.671628] Showing busy workqueues and worker pools: [14941.675568] workqueue events: flags=0x0 [14941.679839] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14941.688618] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [14941.696411] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14941.700570] pending: kfree_rcu_monitor [14941.702731] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14941.708490] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [14941.714632] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14941.722732] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [14941.732697] workqueue events_long: flags=0x0 [14941.736072] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14941.745548] pending: lu_objects_flush [obdclass] [14941.748571] workqueue events_unbound: flags=0x2 [14941.751484] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14941.758161] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14941.776711] workqueue events_power_efficient: flags=0x80 [14941.778546] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14941.784384] pending: do_cache_clean, neigh_periodic_work, check_lifetime [14941.788760] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14941.800956] pending: neigh_periodic_work [14941.802482] workqueue mm_percpu_wq: flags=0x8 [14941.812079] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14941.824661] pending: vmstat_update, lru_add_drain_per_cpu [14941.830067] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14941.837066] pending: vmstat_update [14941.840503] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14941.846440] pending: lru_add_drain_per_cpu [14941.850408] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [14941.855952] pending: lru_add_drain_per_cpu BAR(48) [14941.859834] workqueue writeback: flags=0x4a [14941.863465] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [14941.869788] pending: wb_workfn [14941.872041] workqueue xprtiod: flags=0xa [14941.878393] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14941.892650] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [14941.897683] workqueue ipv6_addrconf: flags=0x40008 [14941.902647] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14941.909799] pending: addrconf_verify_work [14941.914372] workqueue knbd0-recv: flags=0x1a [14941.918881] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14941.929570] in-flight: 658:recv_work [14941.932687] workqueue ptlrpc_pinger: flags=0x2 [14941.941044] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14941.948441] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14941.954849] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13011s workers=4 idle: 32984 37 8 [14941.964041] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14926s workers=2 idle: 661 [14944.924056] rcu: INFO: rcu_sched self-detected stall on CPU [14944.933340] rcu: 1-...!: (7733750 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [14944.943442] (t=13020277 jiffies g=249657 q=6364) [14944.949218] rcu: rcu_sched kthread starved for 13015229 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [14944.966700] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [14944.982162] rcu: RCU grace-period kthread stack dump: [14944.984844] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [14944.994818] Call Trace: [14944.996428] __schedule+0x351/0xcb0 [14944.998629] schedule+0xc0/0x180 [14945.000410] schedule_timeout+0xb4/0x190 [14945.003212] ? do_raw_spin_unlock+0x75/0x190 [14945.007073] ? __next_timer_interrupt+0x160/0x160 [14945.009937] ? prepare_to_swait_event+0xf3/0x140 [14945.012299] rcu_gp_kthread+0x8dd/0x15b0 [14945.013917] ? force_qs_rnp+0x2c0/0x2c0 [14945.015809] kthread+0x1d1/0x200 [14945.018006] ? set_kthread_struct+0x70/0x70 [14945.022159] ret_from_fork+0x1f/0x30 [14945.023829] rcu: Stack dump where RCU GP kthread last ran: [14945.026679] NMI backtrace for cpu 1 [14945.028868] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14945.035522] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14945.040595] Call Trace: [14945.041893] [14945.042935] ? dump_stack+0xbb/0x10e [14945.044845] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [14945.047962] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [14945.050029] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [14945.052647] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [14945.055608] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [14945.058336] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [14945.061187] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [14945.063986] ? update_process_times+0x6d/0xc0 [14945.066321] ? tick_sched_handle.isra.18+0x27/0x90 [14945.069325] ? tick_sched_timer+0x53/0xb0 [14945.071582] ? __hrtimer_run_queues+0x1c2/0x4d0 [14945.075282] ? tick_sched_handle.isra.18+0x90/0x90 [14945.077804] ? hrtimer_interrupt+0x128/0x2c0 [14945.080339] ? smp_apic_timer_interrupt+0xa6/0x270 [14945.083563] ? apic_timer_interrupt+0xf/0x20 [14945.086758] [14945.093280] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14945.101658] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14945.111677] ? do_raw_spin_lock+0x177/0x190 [14945.115521] ? _raw_spin_lock+0x21/0x30 [14945.121362] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14945.128378] ? obd_mod_free+0x46/0x140 [ptlrpc] [14945.138007] ? mdc_commit_open+0x91/0xb0 [mdc] [14945.141106] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14945.148679] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14945.155948] ? after_reply+0xe2c/0x1780 [ptlrpc] [14945.161125] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14945.164348] ? __next_timer_interrupt+0x160/0x160 [14945.167936] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [14945.170313] ? do_wait_intr+0xf0/0xf0 [14945.171983] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14945.175106] ? kthread+0x1d1/0x200 [14945.177535] ? set_kthread_struct+0x70/0x70 [14945.179951] ? ret_from_fork+0x1f/0x30 [14945.185644] NMI backtrace for cpu 1 [14945.186604] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14945.196897] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14945.201733] Call Trace: [14945.203002] [14945.204078] ? dump_stack+0xbb/0x10e [14945.206025] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [14945.208704] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [14945.211088] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [14945.213727] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [14945.216875] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [14945.219372] ? rcu_dump_cpu_stacks+0x11b/0x171 [14945.221511] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [14945.223786] ? update_process_times+0x6d/0xc0 [14945.225856] ? tick_sched_handle.isra.18+0x27/0x90 [14945.228252] ? tick_sched_timer+0x53/0xb0 [14945.230214] ? __hrtimer_run_queues+0x1c2/0x4d0 [14945.232243] ? tick_sched_handle.isra.18+0x90/0x90 [14945.236180] ? hrtimer_interrupt+0x128/0x2c0 [14945.238577] ? smp_apic_timer_interrupt+0xa6/0x270 [14945.240541] ? apic_timer_interrupt+0xf/0x20 [14945.243041] [14945.244212] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14945.246715] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14945.248906] ? do_raw_spin_lock+0x177/0x190 [14945.251069] ? _raw_spin_lock+0x21/0x30 [14945.253006] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14945.255472] ? obd_mod_free+0x46/0x140 [ptlrpc] [14945.257787] ? mdc_commit_open+0x91/0xb0 [mdc] [14945.260429] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14945.263049] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14945.266227] ? after_reply+0xe2c/0x1780 [ptlrpc] [14945.269242] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14945.272061] ? __next_timer_interrupt+0x160/0x160 [14945.274896] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [14945.276978] ? do_wait_intr+0xf0/0xf0 [14945.278756] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14945.281552] ? kthread+0x1d1/0x200 [14945.282956] ? set_kthread_struct+0x70/0x70 [14945.288139] ? ret_from_fork+0x1f/0x30 [14948.056045] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14948.058559] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14948.102383] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14948.116486] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14948.127208] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14948.131155] RIP: 0010:native_safe_halt+0x2e/0x50 [14948.136270] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14948.159776] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14948.165643] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14948.170752] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [14948.182811] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [14948.193871] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [14948.200477] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [14948.207857] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [14948.217981] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14948.219668] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14948.231492] Call Trace: [14948.232978] [14948.234352] ? show_regs.cold.9+0x22/0x2f [14948.239006] ? watchdog_timer_fn.cold.6+0x62/0x10a [14948.246557] ? __hrtimer_run_queues+0x1c2/0x4d0 [14948.249489] ? watchdog+0x40/0x40 [14948.252466] ? hrtimer_interrupt+0x128/0x2c0 [14948.257293] ? smp_apic_timer_interrupt+0xa6/0x270 [14948.259745] ? apic_timer_interrupt+0xf/0x20 [14948.262133] [14948.263384] ? native_safe_halt+0x2e/0x50 [14948.265373] kvm_wait+0xcd/0xe0 [14948.267336] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14948.269465] do_raw_spin_lock+0x177/0x190 [14948.274323] _raw_spin_lock+0x21/0x30 [14948.276774] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14948.279204] process_one_work+0x2c8/0x700 [14948.281099] worker_thread+0x296/0x6e0 [14948.285640] ? rescuer_thread+0x570/0x570 [14948.287072] kthread+0x1d1/0x200 [14948.291998] ? set_kthread_struct+0x70/0x70 [14948.293535] ret_from_fork+0x1f/0x30 [14960.057499] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [14960.069442] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14960.114494] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14960.128674] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14960.134538] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14960.138338] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14960.153465] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14960.158106] RAX: 0000000000007be8 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14960.164551] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [14960.170764] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [14960.174291] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [14960.177708] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [14960.181257] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [14960.189400] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14960.192478] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [14960.199451] Call Trace: [14960.204085] [14960.206190] ? show_regs.cold.9+0x22/0x2f [14960.210508] ? watchdog_timer_fn.cold.6+0x62/0x10a [14960.214263] ? __hrtimer_run_queues+0x1c2/0x4d0 [14960.217488] ? watchdog+0x40/0x40 [14960.219708] ? hrtimer_interrupt+0x128/0x2c0 [14960.222094] ? smp_apic_timer_interrupt+0xa6/0x270 [14960.229471] ? apic_timer_interrupt+0xf/0x20 [14960.231142] [14960.231884] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14960.236338] do_raw_spin_lock+0x177/0x190 [14960.240536] _raw_spin_lock+0x21/0x30 [14960.242519] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14960.248565] ? do_raw_spin_unlock+0x75/0x190 [14960.251148] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14960.255522] ? do_wait_intr+0xf0/0xf0 [14960.258124] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14960.262481] mdc_close+0x53b/0xd70 [mdc] [14960.265140] lmv_close+0x18d/0x490 [lmv] [14960.267346] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14960.272803] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14960.277380] ll_md_real_close+0xa5/0x2a0 [lustre] [14960.282449] ll_md_close+0x274/0x800 [lustre] [14960.285877] ll_file_release+0x128/0xc50 [lustre] [14960.289623] __fput+0x10d/0x440 [14960.291533] ____fput+0x12/0x20 [14960.292798] task_work_run+0xc6/0x110 [14960.296490] exit_to_usermode_loop+0x1dd/0x1f0 [14960.305218] do_syscall_64+0x3d6/0x440 [14960.309825] entry_SYSCALL_64_after_hwframe+0x49/0xae [14960.313641] RIP: 0033:0x7f151bee3cd8 [14960.318735] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14960.339468] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14960.343722] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [14960.349727] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [14960.353017] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14960.360576] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14960.365118] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14968.054876] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [14968.058350] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14968.091769] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14968.105492] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14968.115408] RIP: 0010:native_safe_halt+0x2e/0x50 [14968.119273] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14968.128350] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [14968.131436] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14968.133282] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [14968.134760] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [14968.141174] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [14968.145234] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [14968.148547] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [14968.152507] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14968.154949] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [14968.158197] Call Trace: [14968.159353] [14968.160218] ? show_regs.cold.9+0x22/0x2f [14968.161883] ? watchdog_timer_fn.cold.6+0x62/0x10a [14968.164292] ? __hrtimer_run_queues+0x1c2/0x4d0 [14968.166165] ? watchdog+0x40/0x40 [14968.167594] ? hrtimer_interrupt+0x128/0x2c0 [14968.169466] ? smp_apic_timer_interrupt+0xa6/0x270 [14968.172924] ? apic_timer_interrupt+0xf/0x20 [14968.174516] [14968.183653] ? native_safe_halt+0x2e/0x50 [14968.187220] kvm_wait+0xcd/0xe0 [14968.195973] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14968.197573] do_raw_spin_lock+0x177/0x190 [14968.199334] _raw_spin_lock+0x21/0x30 [14968.200949] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14968.203521] ? do_raw_spin_unlock+0x75/0x190 [14968.205405] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14968.207483] ? do_wait_intr+0xf0/0xf0 [14968.208991] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14968.211613] mdc_reint+0x2d/0x110 [mdc] [14968.213019] mdc_unlink+0x658/0x870 [mdc] [14968.214462] lmv_unlink+0x34d/0xbd0 [lmv] [14968.216557] ll_unlink+0x24d/0x7f0 [lustre] [14968.218726] vfs_unlink+0x202/0x330 [14968.220446] do_unlinkat+0x2fa/0x460 [14968.221903] __x64_sys_unlink+0x37/0x50 [14968.223530] do_syscall_64+0xc1/0x440 [14968.224769] entry_SYSCALL_64_after_hwframe+0x49/0xae [14968.228297] RIP: 0033:0x7f151bdee10b [14968.229839] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [14968.245603] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [14968.252835] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [14968.262679] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [14968.269033] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14968.272479] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14968.276174] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [14972.055060] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [14972.064824] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14972.100522] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14972.104666] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14972.109318] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14972.113011] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14972.122642] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14972.128282] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [14972.131922] RDX: 0000000000004c96 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [14972.136872] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [14972.140040] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [14972.145287] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14972.153178] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [14972.159151] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14972.162945] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14972.166419] Call Trace: [14972.167348] [14972.168425] ? show_regs.cold.9+0x22/0x2f [14972.171766] ? watchdog_timer_fn.cold.6+0x62/0x10a [14972.173563] ? __hrtimer_run_queues+0x1c2/0x4d0 [14972.176399] ? watchdog+0x40/0x40 [14972.179474] ? hrtimer_interrupt+0x128/0x2c0 [14972.181734] ? smp_apic_timer_interrupt+0xa6/0x270 [14972.183520] ? apic_timer_interrupt+0xf/0x20 [14972.188546] [14972.192764] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14972.195454] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14972.198693] do_raw_spin_lock+0x177/0x190 [14972.201378] _raw_spin_lock+0x21/0x30 [14972.202688] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14972.204886] obd_mod_free+0x46/0x140 [ptlrpc] [14972.206757] mdc_commit_open+0x91/0xb0 [mdc] [14972.209913] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14972.212483] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14972.215372] after_reply+0xe2c/0x1780 [ptlrpc] [14972.217913] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14972.220218] ? __next_timer_interrupt+0x160/0x160 [14972.225572] ptlrpcd+0xa82/0xe40 [ptlrpc] [14972.228760] ? do_wait_intr+0xf0/0xf0 [14972.231211] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14972.237058] kthread+0x1d1/0x200 [14972.239246] ? set_kthread_struct+0x70/0x70 [14972.242792] ret_from_fork+0x1f/0x30 [14972.384051] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13041s! [14972.387365] Showing busy workqueues and worker pools: [14972.389164] workqueue events: flags=0x0 [14972.390596] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14972.393049] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [14972.398316] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14972.401103] pending: kfree_rcu_monitor [14972.403611] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14972.408212] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [14972.413085] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14972.415195] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [14972.419640] workqueue events_long: flags=0x0 [14972.422630] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14972.425177] pending: lu_objects_flush [obdclass] [14972.427557] workqueue events_unbound: flags=0x2 [14972.429161] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [14972.437593] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [14972.447787] workqueue events_power_efficient: flags=0x80 [14972.449550] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14972.454619] pending: do_cache_clean, neigh_periodic_work, check_lifetime [14972.460665] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14972.466130] pending: neigh_periodic_work [14972.468528] workqueue mm_percpu_wq: flags=0x8 [14972.470057] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14972.473390] pending: vmstat_update, lru_add_drain_per_cpu [14972.475285] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14972.479457] pending: vmstat_update [14972.483725] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14972.493766] pending: lru_add_drain_per_cpu [14972.498498] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [14972.505200] pending: lru_add_drain_per_cpu BAR(48) [14972.506625] workqueue writeback: flags=0x4a [14972.512323] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [14972.514010] pending: wb_workfn [14972.517033] workqueue xprtiod: flags=0xa [14972.518628] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [14972.522642] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [14972.527511] workqueue ipv6_addrconf: flags=0x40008 [14972.529258] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14972.536371] pending: addrconf_verify_work [14972.540222] workqueue knbd0-recv: flags=0x1a [14972.542703] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14972.549358] in-flight: 658:recv_work [14972.551634] workqueue ptlrpc_pinger: flags=0x2 [14972.556085] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14972.558142] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [14972.561674] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13041s workers=4 idle: 32984 37 8 [14972.563664] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14957s workers=2 idle: 661 [14976.056373] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [14976.063470] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14976.103219] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14976.113719] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14976.120198] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14976.123076] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14976.136021] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14976.150047] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14976.154827] RAX: 0000000000006b83 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14976.159096] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [14976.165147] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [14976.172672] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [14976.179192] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [14976.190967] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [14976.195556] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14976.207983] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [14976.216347] Call Trace: [14976.217958] [14976.219176] ? show_regs.cold.9+0x22/0x2f [14976.222913] ? watchdog_timer_fn.cold.6+0x62/0x10a [14976.224566] ? __hrtimer_run_queues+0x1c2/0x4d0 [14976.232511] ? watchdog+0x40/0x40 [14976.234440] ? hrtimer_interrupt+0x128/0x2c0 [14976.240549] ? smp_apic_timer_interrupt+0xa6/0x270 [14976.245254] ? apic_timer_interrupt+0xf/0x20 [14976.247834] [14976.248488] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14976.250573] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14976.254521] do_raw_spin_lock+0x177/0x190 [14976.255622] _raw_spin_lock+0x21/0x30 [14976.257724] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14976.259534] process_one_work+0x2c8/0x700 [14976.261253] worker_thread+0x296/0x6e0 [14976.263646] ? rescuer_thread+0x570/0x570 [14976.264560] kthread+0x1d1/0x200 [14976.265343] ? set_kthread_struct+0x70/0x70 [14976.266393] ret_from_fork+0x1f/0x30 [14988.059688] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [14988.070317] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14988.104533] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14988.113142] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14988.118333] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14988.121241] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14988.129732] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14988.132405] RAX: 0000000000007d13 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [14988.136000] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [14988.141565] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [14988.148984] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [14988.153171] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [14988.156652] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [14988.161448] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14988.164199] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [14988.168348] Call Trace: [14988.169506] [14988.170524] ? show_regs.cold.9+0x22/0x2f [14988.172537] ? watchdog_timer_fn.cold.6+0x62/0x10a [14988.176300] ? __hrtimer_run_queues+0x1c2/0x4d0 [14988.179438] ? watchdog+0x40/0x40 [14988.183121] ? hrtimer_interrupt+0x128/0x2c0 [14988.186011] ? smp_apic_timer_interrupt+0xa6/0x270 [14988.189342] ? apic_timer_interrupt+0xf/0x20 [14988.193577] [14988.196161] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14988.199343] do_raw_spin_lock+0x177/0x190 [14988.201460] _raw_spin_lock+0x21/0x30 [14988.203614] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14988.206649] ? do_raw_spin_unlock+0x75/0x190 [14988.212074] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14988.218576] ? do_wait_intr+0xf0/0xf0 [14988.220508] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14988.224428] mdc_close+0x53b/0xd70 [mdc] [14988.226614] lmv_close+0x18d/0x490 [lmv] [14988.228808] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [14988.231514] ll_close_inode_openhandle+0x493/0xea0 [lustre] [14988.234060] ll_md_real_close+0xa5/0x2a0 [lustre] [14988.236596] ll_md_close+0x274/0x800 [lustre] [14988.238118] ll_file_release+0x128/0xc50 [lustre] [14988.240737] __fput+0x10d/0x440 [14988.241885] ____fput+0x12/0x20 [14988.243245] task_work_run+0xc6/0x110 [14988.246461] exit_to_usermode_loop+0x1dd/0x1f0 [14988.249129] do_syscall_64+0x3d6/0x440 [14988.250510] entry_SYSCALL_64_after_hwframe+0x49/0xae [14988.252682] RIP: 0033:0x7f151bee3cd8 [14988.253985] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [14988.263574] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [14988.267820] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [14988.283258] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [14988.287041] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [14988.290370] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [14988.294624] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15000.052400] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [15000.055083] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [15000.056984] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15000.057127] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15000.057135] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15000.057138] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15000.062443] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15000.062522] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15000.062530] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15000.062765] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15000.062779] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15000.062784] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15000.062788] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [15000.062790] RDX: 0000000000005936 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [15000.062793] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [15000.062795] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [15000.062797] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15000.062800] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [15000.062803] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15000.062805] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15000.062813] Call Trace: [15000.062818] [15000.062824] ? show_regs.cold.9+0x22/0x2f [15000.062833] ? watchdog_timer_fn.cold.6+0x62/0x10a [15000.062839] ? __hrtimer_run_queues+0x1c2/0x4d0 [15000.062843] ? watchdog+0x40/0x40 [15000.062846] ? hrtimer_interrupt+0x128/0x2c0 [15000.062850] ? smp_apic_timer_interrupt+0xa6/0x270 [15000.062857] ? apic_timer_interrupt+0xf/0x20 [15000.062862] [15000.062864] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15000.063284] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15000.063289] do_raw_spin_lock+0x177/0x190 [15000.063294] _raw_spin_lock+0x21/0x30 [15000.063300] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15000.063815] obd_mod_free+0x46/0x140 [ptlrpc] [15000.105448] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15000.105463] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15000.105470] RAX: 0000000000005f83 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15000.105472] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [15000.105474] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [15000.105476] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [15000.105478] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [15000.105481] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [15000.105484] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15000.105486] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [15000.105494] Call Trace: [15000.105500] [15000.105506] ? show_regs.cold.9+0x22/0x2f [15000.105517] ? watchdog_timer_fn.cold.6+0x62/0x10a [15000.105523] ? __hrtimer_run_queues+0x1c2/0x4d0 [15000.105528] ? watchdog+0x40/0x40 [15000.105596] ? hrtimer_interrupt+0x128/0x2c0 [15000.105602] ? smp_apic_timer_interrupt+0xa6/0x270 [15000.105612] ? apic_timer_interrupt+0xf/0x20 [15000.105617] [15000.105620] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15000.105627] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15000.105632] do_raw_spin_lock+0x177/0x190 [15000.105637] _raw_spin_lock+0x21/0x30 [15000.105645] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15000.105935] ? do_raw_spin_unlock+0x75/0x190 [15000.105941] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15000.106275] ? do_wait_intr+0xf0/0xf0 [15000.106283] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15000.106470] mdc_reint+0x2d/0x110 [mdc] [15000.106516] mdc_unlink+0x658/0x870 [mdc] [15000.106616] lmv_unlink+0x34d/0xbd0 [lmv] [15000.106647] ll_unlink+0x24d/0x7f0 [lustre] [15000.106772] vfs_unlink+0x202/0x330 [15000.106781] do_unlinkat+0x2fa/0x460 [15000.106786] __x64_sys_unlink+0x37/0x50 [15000.106789] do_syscall_64+0xc1/0x440 [15000.106797] entry_SYSCALL_64_after_hwframe+0x49/0xae [15000.106804] RIP: 0033:0x7f151bdee10b [15000.106811] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [15000.106815] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [15000.106819] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [15000.106821] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [15000.106823] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15000.106825] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15000.106827] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15000.611511] mdc_commit_open+0x91/0xb0 [mdc] [15000.614748] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15000.627844] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15000.633677] after_reply+0xe2c/0x1780 [ptlrpc] [15000.640565] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15000.644753] ? __next_timer_interrupt+0x160/0x160 [15000.652244] ptlrpcd+0xa82/0xe40 [ptlrpc] [15000.657386] ? do_wait_intr+0xf0/0xf0 [15000.664405] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15000.671573] kthread+0x1d1/0x200 [15000.673951] ? set_kthread_struct+0x70/0x70 [15000.681264] ret_from_fork+0x1f/0x30 [15002.594158] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13071s! [15002.598012] Showing busy workqueues and worker pools: [15002.606733] workqueue events: flags=0x0 [15002.610248] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15002.614294] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [15002.621191] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15002.627965] pending: kfree_rcu_monitor [15002.632970] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15002.636172] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [15002.640969] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15002.646375] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [15002.649973] workqueue events_long: flags=0x0 [15002.651591] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15002.655880] pending: lu_objects_flush [obdclass] [15002.658282] workqueue events_unbound: flags=0x2 [15002.661259] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15002.664702] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15002.669621] workqueue events_power_efficient: flags=0x80 [15002.672638] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15002.675590] pending: do_cache_clean, neigh_periodic_work, check_lifetime [15002.679308] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15002.682786] pending: neigh_periodic_work [15002.685269] workqueue mm_percpu_wq: flags=0x8 [15002.686546] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15002.688310] pending: vmstat_update, lru_add_drain_per_cpu [15002.691930] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15002.697944] pending: vmstat_update [15002.700917] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15002.710340] pending: lru_add_drain_per_cpu [15002.712401] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [15002.715439] pending: lru_add_drain_per_cpu BAR(48) [15002.717766] workqueue writeback: flags=0x4a [15002.721403] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [15002.728336] pending: wb_workfn [15002.733021] workqueue xprtiod: flags=0xa [15002.734772] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15002.739984] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [15002.744487] workqueue ipv6_addrconf: flags=0x40008 [15002.747438] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15002.758295] pending: addrconf_verify_work [15002.761115] workqueue knbd0-recv: flags=0x1a [15002.764147] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15002.767416] in-flight: 658:recv_work [15002.770353] workqueue ptlrpc_pinger: flags=0x2 [15002.772738] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15002.777446] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15002.784080] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13072s workers=4 idle: 32984 37 8 [15002.787481] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14987s workers=2 idle: 661 [15004.056039] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [15004.059141] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15004.085001] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15004.091284] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15004.097457] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15004.100756] RIP: 0010:native_safe_halt+0x2e/0x50 [15004.103766] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15004.114784] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [15004.118955] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15004.122444] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [15004.126826] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [15004.131249] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [15004.135748] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [15004.140530] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [15004.145312] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15004.149306] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15004.152940] Call Trace: [15004.154877] [15004.155948] ? show_regs.cold.9+0x22/0x2f [15004.158224] ? watchdog_timer_fn.cold.6+0x62/0x10a [15004.161656] ? __hrtimer_run_queues+0x1c2/0x4d0 [15004.164871] ? watchdog+0x40/0x40 [15004.166337] ? hrtimer_interrupt+0x128/0x2c0 [15004.169252] ? smp_apic_timer_interrupt+0xa6/0x270 [15004.172222] ? apic_timer_interrupt+0xf/0x20 [15004.175206] [15004.176412] ? native_safe_halt+0x2e/0x50 [15004.178676] kvm_wait+0xcd/0xe0 [15004.180639] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15004.183157] do_raw_spin_lock+0x177/0x190 [15004.185907] _raw_spin_lock+0x21/0x30 [15004.188013] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15004.191101] process_one_work+0x2c8/0x700 [15004.193359] worker_thread+0x296/0x6e0 [15004.196010] ? rescuer_thread+0x570/0x570 [15004.197390] kthread+0x1d1/0x200 [15004.199090] ? set_kthread_struct+0x70/0x70 [15004.201329] ret_from_fork+0x1f/0x30 [15016.057076] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [15016.060979] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15016.083131] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15016.087461] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15016.090893] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [15016.093284] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [15016.101871] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15016.106643] RAX: 0000000000004ec7 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15016.111269] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [15016.114783] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [15016.117678] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [15016.120977] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [15016.124292] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [15016.128063] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15016.130797] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [15016.134100] Call Trace: [15016.135044] [15016.135787] ? show_regs.cold.9+0x22/0x2f [15016.137425] ? watchdog_timer_fn.cold.6+0x62/0x10a [15016.139497] ? __hrtimer_run_queues+0x1c2/0x4d0 [15016.140975] ? watchdog+0x40/0x40 [15016.142284] ? hrtimer_interrupt+0x128/0x2c0 [15016.144670] ? smp_apic_timer_interrupt+0xa6/0x270 [15016.147014] ? apic_timer_interrupt+0xf/0x20 [15016.148372] [15016.149202] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [15016.151215] do_raw_spin_lock+0x177/0x190 [15016.152521] _raw_spin_lock+0x21/0x30 [15016.153760] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15016.156704] ? do_raw_spin_unlock+0x75/0x190 [15016.158188] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15016.160362] ? do_wait_intr+0xf0/0xf0 [15016.161807] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15016.164312] mdc_close+0x53b/0xd70 [mdc] [15016.166167] lmv_close+0x18d/0x490 [lmv] [15016.167731] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15016.169841] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15016.172250] ll_md_real_close+0xa5/0x2a0 [lustre] [15016.174268] ll_md_close+0x274/0x800 [lustre] [15016.175871] ll_file_release+0x128/0xc50 [lustre] [15016.177970] __fput+0x10d/0x440 [15016.179098] ____fput+0x12/0x20 [15016.180203] task_work_run+0xc6/0x110 [15016.181849] exit_to_usermode_loop+0x1dd/0x1f0 [15016.184343] do_syscall_64+0x3d6/0x440 [15016.185692] entry_SYSCALL_64_after_hwframe+0x49/0xae [15016.187732] RIP: 0033:0x7f151bee3cd8 [15016.189324] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15016.196855] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15016.199594] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [15016.202622] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [15016.205221] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15016.208601] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15016.211858] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15028.059085] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [15028.061482] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [15028.071102] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15028.071180] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15028.071188] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15028.071191] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15028.071203] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15028.071206] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15028.071209] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [15028.071211] RDX: 000000000000323d RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [15028.071212] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [15028.071214] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [15028.071215] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15028.071218] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [15028.071219] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15028.071221] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15028.071228] Call Trace: [15028.071232] [15028.071236] ? show_regs.cold.9+0x22/0x2f [15028.071243] ? watchdog_timer_fn.cold.6+0x62/0x10a [15028.071247] ? __hrtimer_run_queues+0x1c2/0x4d0 [15028.071249] ? watchdog+0x40/0x40 [15028.071252] ? hrtimer_interrupt+0x128/0x2c0 [15028.071255] ? smp_apic_timer_interrupt+0xa6/0x270 [15028.074059] ? apic_timer_interrupt+0xf/0x20 [15028.074066] [15028.074070] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15028.074077] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15028.074082] do_raw_spin_lock+0x177/0x190 [15028.074087] _raw_spin_lock+0x21/0x30 [15028.074094] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15028.076180] obd_mod_free+0x46/0x140 [ptlrpc] [15028.085589] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15028.085673] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15028.085682] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15028.085685] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15028.085699] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15028.085704] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15028.085709] RAX: 000000000000790f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15028.085712] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [15028.085714] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [15028.085717] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [15028.085719] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [15028.085723] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [15028.085726] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15028.085729] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [15028.086178] Call Trace: [15028.086186] [15028.086192] ? show_regs.cold.9+0x22/0x2f [15028.086203] ? watchdog_timer_fn.cold.6+0x62/0x10a [15028.086210] ? __hrtimer_run_queues+0x1c2/0x4d0 [15028.086215] ? watchdog+0x40/0x40 [15028.086220] ? hrtimer_interrupt+0x128/0x2c0 [15028.086224] ? smp_apic_timer_interrupt+0xa6/0x270 [15028.086235] ? apic_timer_interrupt+0xf/0x20 [15028.086240] [15028.086243] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15028.086248] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15028.086253] do_raw_spin_lock+0x177/0x190 [15028.086258] _raw_spin_lock+0x21/0x30 [15028.086365] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15028.086727] ? do_raw_spin_unlock+0x75/0x190 [15028.086734] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15028.089388] ? do_wait_intr+0xf0/0xf0 [15028.149614] mdc_commit_open+0x91/0xb0 [mdc] [15028.149784] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15028.150062] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15028.150548] after_reply+0xe2c/0x1780 [ptlrpc] [15028.150832] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15028.168134] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15028.176019] ? __next_timer_interrupt+0x160/0x160 [15028.176044] ptlrpcd+0xa82/0xe40 [ptlrpc] [15028.176434] ? do_wait_intr+0xf0/0xf0 [15028.176443] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15028.176878] kthread+0x1d1/0x200 [15028.176884] ? set_kthread_struct+0x70/0x70 [15028.176889] ret_from_fork+0x1f/0x30 [15028.427776] mdc_reint+0x2d/0x110 [mdc] [15028.429747] mdc_unlink+0x658/0x870 [mdc] [15028.431934] lmv_unlink+0x34d/0xbd0 [lmv] [15028.434039] ll_unlink+0x24d/0x7f0 [lustre] [15028.436328] vfs_unlink+0x202/0x330 [15028.437817] do_unlinkat+0x2fa/0x460 [15028.439563] __x64_sys_unlink+0x37/0x50 [15028.441349] do_syscall_64+0xc1/0x440 [15028.444309] entry_SYSCALL_64_after_hwframe+0x49/0xae [15028.446532] RIP: 0033:0x7f151bdee10b [15028.449903] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [15028.458526] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [15028.462186] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [15028.466504] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [15028.469915] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15028.473058] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15028.476076] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15032.056033] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [15032.060772] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15032.090097] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15032.095531] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15032.102109] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15032.105189] RIP: 0010:native_safe_halt+0x2e/0x50 [15032.106887] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15032.120688] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15032.124954] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15032.129001] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [15032.133117] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [15032.137483] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [15032.142723] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [15032.146184] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [15032.151583] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15032.159394] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15032.166275] Call Trace: [15032.166954] [15032.168387] ? show_regs.cold.9+0x22/0x2f [15032.170351] ? watchdog_timer_fn.cold.6+0x62/0x10a [15032.173146] ? __hrtimer_run_queues+0x1c2/0x4d0 [15032.178960] ? watchdog+0x40/0x40 [15032.180578] ? hrtimer_interrupt+0x128/0x2c0 [15032.183127] ? smp_apic_timer_interrupt+0xa6/0x270 [15032.185413] ? apic_timer_interrupt+0xf/0x20 [15032.188099] [15032.194574] ? native_safe_halt+0x2e/0x50 [15032.198013] kvm_wait+0xcd/0xe0 [15032.199090] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15032.202753] do_raw_spin_lock+0x177/0x190 [15032.204053] _raw_spin_lock+0x21/0x30 [15032.207954] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15032.215974] process_one_work+0x2c8/0x700 [15032.224307] worker_thread+0x296/0x6e0 [15032.226229] ? rescuer_thread+0x570/0x570 [15032.228851] kthread+0x1d1/0x200 [15032.232721] ? set_kthread_struct+0x70/0x70 [15032.234307] ret_from_fork+0x1f/0x30 [15032.800088] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13102s! [15032.804151] Showing busy workqueues and worker pools: [15032.808964] workqueue events: flags=0x0 [15032.811391] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15032.815999] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [15032.823879] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15032.830992] pending: kfree_rcu_monitor [15032.834856] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15032.843971] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [15032.853141] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15032.860194] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [15032.862832] workqueue events_long: flags=0x0 [15032.865330] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15032.870666] pending: lu_objects_flush [obdclass] [15032.873093] workqueue events_unbound: flags=0x2 [15032.878927] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15032.885415] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15032.891527] workqueue events_power_efficient: flags=0x80 [15032.897747] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15032.900346] pending: do_cache_clean, neigh_periodic_work, check_lifetime [15032.904816] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15032.908086] pending: neigh_periodic_work [15032.910554] workqueue mm_percpu_wq: flags=0x8 [15032.913747] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15032.918702] pending: vmstat_update, lru_add_drain_per_cpu [15032.921335] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15032.923678] pending: vmstat_update [15032.925743] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15032.929830] pending: lru_add_drain_per_cpu [15032.932283] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [15032.936006] pending: lru_add_drain_per_cpu BAR(48) [15032.939518] workqueue writeback: flags=0x4a [15032.942880] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [15032.946680] pending: wb_workfn [15032.948325] workqueue xprtiod: flags=0xa [15032.949909] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15032.954532] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [15032.959443] workqueue ipv6_addrconf: flags=0x40008 [15032.966134] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15032.969212] pending: addrconf_verify_work [15032.973860] workqueue knbd0-recv: flags=0x1a [15032.985542] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15032.990441] in-flight: 658:recv_work [15032.992298] workqueue ptlrpc_pinger: flags=0x2 [15032.994103] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15032.997145] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15033.006110] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13102s workers=4 idle: 32984 37 8 [15033.014942] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15017s workers=2 idle: 661 [15044.061277] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [15044.067018] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15044.104087] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15044.110274] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15044.116519] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15044.120686] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15044.130542] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [15044.138933] RAX: 000000000000724b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15044.141806] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [15044.144042] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [15044.146572] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [15044.151346] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [15044.158070] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [15044.161660] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15044.166945] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [15044.171662] Call Trace: [15044.173304] [15044.173960] ? show_regs.cold.9+0x22/0x2f [15044.179797] ? watchdog_timer_fn.cold.6+0x62/0x10a [15044.184099] ? __hrtimer_run_queues+0x1c2/0x4d0 [15044.186821] ? watchdog+0x40/0x40 [15044.187996] ? hrtimer_interrupt+0x128/0x2c0 [15044.192490] ? smp_apic_timer_interrupt+0xa6/0x270 [15044.194520] ? apic_timer_interrupt+0xf/0x20 [15044.196491] [15044.197373] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15044.199265] do_raw_spin_lock+0x177/0x190 [15044.200520] _raw_spin_lock+0x21/0x30 [15044.201658] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15044.205945] ? do_raw_spin_unlock+0x75/0x190 [15044.209730] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15044.215364] ? do_wait_intr+0xf0/0xf0 [15044.216744] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15044.220622] mdc_close+0x53b/0xd70 [mdc] [15044.222846] lmv_close+0x18d/0x490 [lmv] [15044.223978] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15044.226251] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15044.228419] ll_md_real_close+0xa5/0x2a0 [lustre] [15044.230157] ll_md_close+0x274/0x800 [lustre] [15044.231851] ll_file_release+0x128/0xc50 [lustre] [15044.234218] __fput+0x10d/0x440 [15044.235201] ____fput+0x12/0x20 [15044.236067] task_work_run+0xc6/0x110 [15044.237462] exit_to_usermode_loop+0x1dd/0x1f0 [15044.239325] do_syscall_64+0x3d6/0x440 [15044.240610] entry_SYSCALL_64_after_hwframe+0x49/0xae [15044.242767] RIP: 0033:0x7f151bee3cd8 [15044.245180] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15044.250525] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15044.253035] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [15044.257856] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [15044.262631] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15044.265441] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15044.270540] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15056.055048] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [15056.057795] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15056.073494] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15056.077939] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15056.080912] RIP: 0010:native_safe_halt+0x2e/0x50 [15056.082900] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15056.089375] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [15056.091769] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [15056.093940] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [15056.096332] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [15056.098459] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [15056.100733] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15056.102947] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [15056.105582] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15056.107477] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15056.109698] Call Trace: [15056.110555] [15056.111104] ? show_regs.cold.9+0x22/0x2f [15056.112479] ? watchdog_timer_fn.cold.6+0x62/0x10a [15056.113979] ? __hrtimer_run_queues+0x1c2/0x4d0 [15056.115484] ? watchdog+0x40/0x40 [15056.116500] ? hrtimer_interrupt+0x128/0x2c0 [15056.118790] ? smp_apic_timer_interrupt+0xa6/0x270 [15056.120381] ? apic_timer_interrupt+0xf/0x20 [15056.121853] [15056.122609] ? native_safe_halt+0x2e/0x50 [15056.123786] kvm_wait+0xcd/0xe0 [15056.125102] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15056.126782] do_raw_spin_lock+0x177/0x190 [15056.128260] _raw_spin_lock+0x21/0x30 [15056.129646] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15056.131829] obd_mod_free+0x46/0x140 [ptlrpc] [15056.133852] mdc_commit_open+0x91/0xb0 [mdc] [15056.135668] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15056.137967] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15056.140728] after_reply+0xe2c/0x1780 [ptlrpc] [15056.142871] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15056.145305] ? __next_timer_interrupt+0x160/0x160 [15056.147250] ptlrpcd+0xa82/0xe40 [ptlrpc] [15056.149632] ? do_wait_intr+0xf0/0xf0 [15056.151116] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15056.153624] kthread+0x1d1/0x200 [15056.155016] ? set_kthread_struct+0x70/0x70 [15056.156920] ret_from_fork+0x1f/0x30 [15060.052911] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [15060.056021] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [15060.063073] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15060.063166] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15060.063174] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15060.063177] RIP: 0010:native_safe_halt+0x2e/0x50 [15060.063190] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15060.065822] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15060.065900] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15060.065907] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15060.065912] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15060.066330] RIP: 0010:native_safe_halt+0x2e/0x50 [15060.066340] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15060.066344] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15060.066347] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15060.066350] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [15060.066352] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [15060.066354] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [15060.066356] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [15060.066359] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [15060.066362] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15060.066365] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15060.066373] Call Trace: [15060.066379] [15060.066385] ? show_regs.cold.9+0x22/0x2f [15060.066394] ? watchdog_timer_fn.cold.6+0x62/0x10a [15060.066400] ? __hrtimer_run_queues+0x1c2/0x4d0 [15060.066405] ? watchdog+0x40/0x40 [15060.066409] ? hrtimer_interrupt+0x128/0x2c0 [15060.066414] ? smp_apic_timer_interrupt+0xa6/0x270 [15060.066425] ? apic_timer_interrupt+0xf/0x20 [15060.066431] [15060.066434] ? native_safe_halt+0x2e/0x50 [15060.066438] kvm_wait+0xcd/0xe0 [15060.066445] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15060.066453] do_raw_spin_lock+0x177/0x190 [15060.066458] _raw_spin_lock+0x21/0x30 [15060.066462] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15060.066672] process_one_work+0x2c8/0x700 [15060.066679] worker_thread+0x296/0x6e0 [15060.066683] ? rescuer_thread+0x570/0x570 [15060.066687] kthread+0x1d1/0x200 [15060.066691] ? set_kthread_struct+0x70/0x70 [15060.066694] ret_from_fork+0x1f/0x30 [15060.314846] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15060.318930] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15060.323310] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [15060.325586] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [15060.328374] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [15060.334097] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [15060.336444] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [15060.344851] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15060.352922] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [15060.356228] Call Trace: [15060.356902] [15060.358860] ? show_regs.cold.9+0x22/0x2f [15060.361588] ? watchdog_timer_fn.cold.6+0x62/0x10a [15060.363293] ? __hrtimer_run_queues+0x1c2/0x4d0 [15060.365023] ? watchdog+0x40/0x40 [15060.366258] ? hrtimer_interrupt+0x128/0x2c0 [15060.367557] ? smp_apic_timer_interrupt+0xa6/0x270 [15060.369065] ? apic_timer_interrupt+0xf/0x20 [15060.371822] [15060.372694] ? native_safe_halt+0x2e/0x50 [15060.373993] kvm_wait+0xcd/0xe0 [15060.375149] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15060.376929] do_raw_spin_lock+0x177/0x190 [15060.378813] _raw_spin_lock+0x21/0x30 [15060.380216] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15060.382510] ? do_raw_spin_unlock+0x75/0x190 [15060.385475] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15060.387377] ? do_wait_intr+0xf0/0xf0 [15060.390075] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15060.391836] mdc_reint+0x2d/0x110 [mdc] [15060.393054] mdc_unlink+0x658/0x870 [mdc] [15060.394277] lmv_unlink+0x34d/0xbd0 [lmv] [15060.395693] ll_unlink+0x24d/0x7f0 [lustre] [15060.397406] vfs_unlink+0x202/0x330 [15060.398476] do_unlinkat+0x2fa/0x460 [15060.400058] __x64_sys_unlink+0x37/0x50 [15060.401506] do_syscall_64+0xc1/0x440 [15060.404953] entry_SYSCALL_64_after_hwframe+0x49/0xae [15060.407190] RIP: 0033:0x7f151bdee10b [15060.409191] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [15060.426126] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [15060.432405] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [15060.435283] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [15060.441532] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15060.445371] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15060.449638] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15063.524895] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13132s! [15063.530125] Showing busy workqueues and worker pools: [15063.532806] workqueue events: flags=0x0 [15063.535095] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15063.541542] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [15063.550157] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15063.553764] pending: kfree_rcu_monitor [15063.555915] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15063.559431] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [15063.563836] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15063.567020] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [15063.570776] workqueue events_long: flags=0x0 [15063.572802] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15063.576237] pending: lu_objects_flush [obdclass] [15063.581584] workqueue events_unbound: flags=0x2 [15063.587087] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15063.596686] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15063.606207] workqueue events_power_efficient: flags=0x80 [15063.615198] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15063.621800] pending: do_cache_clean, neigh_periodic_work, check_lifetime [15063.628438] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15063.634741] pending: neigh_periodic_work [15063.636783] workqueue mm_percpu_wq: flags=0x8 [15063.640079] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15063.644281] pending: vmstat_update, lru_add_drain_per_cpu [15063.649039] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15063.652099] pending: vmstat_update [15063.653628] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15063.658784] pending: lru_add_drain_per_cpu [15063.670751] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [15063.673123] pending: lru_add_drain_per_cpu BAR(48) [15063.674880] workqueue writeback: flags=0x4a [15063.676778] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [15063.679470] pending: wb_workfn [15063.681153] workqueue xprtiod: flags=0xa [15063.687028] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15063.692850] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [15063.705812] workqueue ipv6_addrconf: flags=0x40008 [15063.712693] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15063.723433] pending: addrconf_verify_work [15063.727018] workqueue knbd0-recv: flags=0x1a [15063.728899] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15063.735595] in-flight: 658:recv_work [15063.743535] workqueue ptlrpc_pinger: flags=0x2 [15063.756225] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15063.763633] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15063.772785] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13133s workers=4 idle: 32984 37 8 [15063.779578] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15048s workers=2 idle: 661 [15072.058476] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [15072.060933] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15072.087642] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15072.093629] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15072.102066] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15072.105676] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15072.115726] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15072.119310] RAX: 000000000000183b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15072.124703] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [15072.129385] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [15072.131808] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [15072.135133] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [15072.143168] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [15072.148281] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15072.151438] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [15072.154652] Call Trace: [15072.155823] [15072.156744] ? show_regs.cold.9+0x22/0x2f [15072.159048] ? watchdog_timer_fn.cold.6+0x62/0x10a [15072.161381] ? __hrtimer_run_queues+0x1c2/0x4d0 [15072.163775] ? watchdog+0x40/0x40 [15072.165403] ? hrtimer_interrupt+0x128/0x2c0 [15072.167528] ? smp_apic_timer_interrupt+0xa6/0x270 [15072.169919] ? apic_timer_interrupt+0xf/0x20 [15072.172221] [15072.173392] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15072.176369] do_raw_spin_lock+0x177/0x190 [15072.178929] _raw_spin_lock+0x21/0x30 [15072.180503] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15072.183282] ? do_raw_spin_unlock+0x75/0x190 [15072.185603] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15072.188654] ? do_wait_intr+0xf0/0xf0 [15072.190732] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15072.193051] mdc_close+0x53b/0xd70 [mdc] [15072.194609] lmv_close+0x18d/0x490 [lmv] [15072.197009] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15072.200343] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15072.203529] ll_md_real_close+0xa5/0x2a0 [lustre] [15072.205732] ll_md_close+0x274/0x800 [lustre] [15072.208505] ll_file_release+0x128/0xc50 [lustre] [15072.211398] __fput+0x10d/0x440 [15072.212758] ____fput+0x12/0x20 [15072.214606] task_work_run+0xc6/0x110 [15072.216325] exit_to_usermode_loop+0x1dd/0x1f0 [15072.219176] do_syscall_64+0x3d6/0x440 [15072.221132] entry_SYSCALL_64_after_hwframe+0x49/0xae [15072.224206] RIP: 0033:0x7f151bee3cd8 [15072.226454] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15072.237058] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15072.240742] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [15072.244920] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [15072.248304] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15072.252305] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15072.256819] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15084.055041] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [15084.057154] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15084.080268] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15084.085404] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15084.089339] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15084.091972] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15084.098509] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15084.102457] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [15084.105014] RDX: 00000000000062f5 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [15084.107992] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [15084.111168] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [15084.114161] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15084.119128] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [15084.124288] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15084.130886] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15084.134444] Call Trace: [15084.136245] [15084.137250] ? show_regs.cold.9+0x22/0x2f [15084.141360] ? watchdog_timer_fn.cold.6+0x62/0x10a [15084.144223] ? __hrtimer_run_queues+0x1c2/0x4d0 [15084.151541] ? watchdog+0x40/0x40 [15084.153037] ? hrtimer_interrupt+0x128/0x2c0 [15084.155376] ? smp_apic_timer_interrupt+0xa6/0x270 [15084.157395] ? apic_timer_interrupt+0xf/0x20 [15084.159416] [15084.160405] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15084.162880] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15084.165584] do_raw_spin_lock+0x177/0x190 [15084.167289] _raw_spin_lock+0x21/0x30 [15084.168965] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15084.171411] obd_mod_free+0x46/0x140 [ptlrpc] [15084.173591] mdc_commit_open+0x91/0xb0 [mdc] [15084.176033] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15084.178584] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15084.181233] after_reply+0xe2c/0x1780 [ptlrpc] [15084.183348] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15084.185844] ? __next_timer_interrupt+0x160/0x160 [15084.187793] ptlrpcd+0xa82/0xe40 [ptlrpc] [15084.190109] ? do_wait_intr+0xf0/0xf0 [15084.191609] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15084.194134] kthread+0x1d1/0x200 [15084.196061] ? set_kthread_struct+0x70/0x70 [15084.197713] ret_from_fork+0x1f/0x30 [15088.052019] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [15088.054375] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15088.056618] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [15088.096320] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15088.096335] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15088.096338] RIP: 0010:native_safe_halt+0x2e/0x50 [15088.096352] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15088.096355] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15088.096361] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15088.096363] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [15088.096365] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [15088.096367] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [15088.096379] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [15088.096382] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [15088.096385] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15088.096387] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [15088.096395] Call Trace: [15088.096401] [15088.096407] ? show_regs.cold.9+0x22/0x2f [15088.096417] ? watchdog_timer_fn.cold.6+0x62/0x10a [15088.096423] ? __hrtimer_run_queues+0x1c2/0x4d0 [15088.096427] ? watchdog+0x40/0x40 [15088.096432] ? hrtimer_interrupt+0x128/0x2c0 [15088.096436] ? smp_apic_timer_interrupt+0xa6/0x270 [15088.096444] ? apic_timer_interrupt+0xf/0x20 [15088.096449] [15088.096452] ? native_safe_halt+0x2e/0x50 [15088.096456] kvm_wait+0xcd/0xe0 [15088.096463] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15088.096470] do_raw_spin_lock+0x177/0x190 [15088.096474] _raw_spin_lock+0x21/0x30 [15088.096481] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15088.101127] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15088.101212] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15088.101219] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15088.101225] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15088.101486] RIP: 0010:native_safe_halt+0x2e/0x50 [15088.101496] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15088.101500] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15088.101505] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15088.101507] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [15088.101510] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [15088.101512] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [15088.101515] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [15088.101518] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [15088.101521] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15088.101524] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15088.101533] Call Trace: [15088.101539] [15088.101543] ? show_regs.cold.9+0x22/0x2f [15088.101552] ? watchdog_timer_fn.cold.6+0x62/0x10a [15088.101557] ? __hrtimer_run_queues+0x1c2/0x4d0 [15088.101561] ? watchdog+0x40/0x40 [15088.101565] ? hrtimer_interrupt+0x128/0x2c0 [15088.101568] ? smp_apic_timer_interrupt+0xa6/0x270 [15088.101574] ? apic_timer_interrupt+0xf/0x20 [15088.101578] [15088.101581] ? native_safe_halt+0x2e/0x50 [15088.101584] kvm_wait+0xcd/0xe0 [15088.101591] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15088.101598] do_raw_spin_lock+0x177/0x190 [15088.101601] _raw_spin_lock+0x21/0x30 [15088.101605] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15088.101825] process_one_work+0x2c8/0x700 [15088.101834] worker_thread+0x296/0x6e0 [15088.101838] ? rescuer_thread+0x570/0x570 [15088.101842] kthread+0x1d1/0x200 [15088.101847] ? set_kthread_struct+0x70/0x70 [15088.101851] ret_from_fork+0x1f/0x30 [15088.502883] ? do_raw_spin_unlock+0x75/0x190 [15088.507652] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15088.514836] ? do_wait_intr+0xf0/0xf0 [15088.516836] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15088.523076] mdc_reint+0x2d/0x110 [mdc] [15088.526268] mdc_unlink+0x658/0x870 [mdc] [15088.528528] lmv_unlink+0x34d/0xbd0 [lmv] [15088.532793] ll_unlink+0x24d/0x7f0 [lustre] [15088.536356] vfs_unlink+0x202/0x330 [15088.538646] do_unlinkat+0x2fa/0x460 [15088.541241] __x64_sys_unlink+0x37/0x50 [15088.550689] do_syscall_64+0xc1/0x440 [15088.553928] entry_SYSCALL_64_after_hwframe+0x49/0xae [15088.562248] RIP: 0033:0x7f151bdee10b [15088.565515] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [15088.584130] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [15088.588531] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [15088.594985] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [15088.600623] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15088.608785] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15088.613477] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15094.247567] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13163s! [15094.252561] Showing busy workqueues and worker pools: [15094.254530] workqueue events: flags=0x0 [15094.255864] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15094.258622] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [15094.262921] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15094.265823] pending: kfree_rcu_monitor [15094.267488] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15094.270522] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [15094.274838] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15094.278430] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [15094.283189] workqueue events_long: flags=0x0 [15094.285516] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15094.288940] pending: lu_objects_flush [obdclass] [15094.291441] workqueue events_unbound: flags=0x2 [15094.293937] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15094.298831] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15094.311029] workqueue events_power_efficient: flags=0x80 [15094.316253] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15094.323534] pending: do_cache_clean, neigh_periodic_work, check_lifetime [15094.328260] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15094.336050] pending: neigh_periodic_work [15094.344985] workqueue mm_percpu_wq: flags=0x8 [15094.347965] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15094.361380] pending: vmstat_update, lru_add_drain_per_cpu [15094.378333] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15094.389945] pending: vmstat_update [15094.393507] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15094.411338] pending: lru_add_drain_per_cpu [15094.420857] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [15094.438542] pending: lru_add_drain_per_cpu BAR(48) [15094.446708] workqueue writeback: flags=0x4a [15094.448208] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [15094.461583] pending: wb_workfn [15094.470839] workqueue xprtiod: flags=0xa [15094.481221] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15094.484488] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [15094.493016] workqueue ipv6_addrconf: flags=0x40008 [15094.501771] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15094.509715] pending: addrconf_verify_work [15094.511018] workqueue knbd0-recv: flags=0x1a [15094.513450] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15094.524161] in-flight: 658:recv_work [15094.529426] workqueue ptlrpc_pinger: flags=0x2 [15094.532016] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15094.540390] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15094.545428] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13163s workers=4 idle: 32984 37 8 [15094.548384] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15079s workers=2 idle: 661 [15100.064695] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [15100.072232] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15100.102776] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15100.108860] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15100.113018] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15100.115523] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15100.124705] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15100.128384] RAX: 0000000000006105 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15100.132441] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [15100.136256] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [15100.140008] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [15100.143666] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [15100.147155] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [15100.151213] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15100.153647] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [15100.164816] Call Trace: [15100.166719] [15100.167371] ? show_regs.cold.9+0x22/0x2f [15100.171892] ? watchdog_timer_fn.cold.6+0x62/0x10a [15100.176954] ? __hrtimer_run_queues+0x1c2/0x4d0 [15100.179835] ? watchdog+0x40/0x40 [15100.183434] ? hrtimer_interrupt+0x128/0x2c0 [15100.186499] ? smp_apic_timer_interrupt+0xa6/0x270 [15100.188795] ? apic_timer_interrupt+0xf/0x20 [15100.193034] [15100.194373] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15100.199366] do_raw_spin_lock+0x177/0x190 [15100.203147] _raw_spin_lock+0x21/0x30 [15100.206440] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15100.209835] ? do_raw_spin_unlock+0x75/0x190 [15100.211643] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15100.217526] ? do_wait_intr+0xf0/0xf0 [15100.220116] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15100.223565] mdc_close+0x53b/0xd70 [mdc] [15100.231986] lmv_close+0x18d/0x490 [lmv] [15100.233693] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15100.237107] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15100.240259] ll_md_real_close+0xa5/0x2a0 [lustre] [15100.246378] ll_md_close+0x274/0x800 [lustre] [15100.248954] ll_file_release+0x128/0xc50 [lustre] [15100.254348] __fput+0x10d/0x440 [15100.255801] ____fput+0x12/0x20 [15100.257706] task_work_run+0xc6/0x110 [15100.259723] exit_to_usermode_loop+0x1dd/0x1f0 [15100.268670] do_syscall_64+0x3d6/0x440 [15100.273236] entry_SYSCALL_64_after_hwframe+0x49/0xae [15100.279170] RIP: 0033:0x7f151bee3cd8 [15100.282132] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15100.298817] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15100.304597] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [15100.313782] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [15100.323781] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15100.328516] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15100.332948] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15112.055304] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [15112.063424] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15112.087842] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15112.096120] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15112.104324] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15112.109726] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15112.124286] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15112.130173] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [15112.135479] RDX: 0000000000005096 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [15112.140021] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [15112.145013] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [15112.150101] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15112.153337] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [15112.157756] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15112.159629] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15112.161794] Call Trace: [15112.162428] [15112.163175] ? show_regs.cold.9+0x22/0x2f [15112.164832] ? watchdog_timer_fn.cold.6+0x62/0x10a [15112.166313] ? __hrtimer_run_queues+0x1c2/0x4d0 [15112.167537] ? watchdog+0x40/0x40 [15112.169354] ? hrtimer_interrupt+0x128/0x2c0 [15112.171299] ? smp_apic_timer_interrupt+0xa6/0x270 [15112.173320] ? apic_timer_interrupt+0xf/0x20 [15112.175085] [15112.176148] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15112.178411] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15112.182046] do_raw_spin_lock+0x177/0x190 [15112.183929] _raw_spin_lock+0x21/0x30 [15112.185430] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15112.188285] obd_mod_free+0x46/0x140 [ptlrpc] [15112.191332] mdc_commit_open+0x91/0xb0 [mdc] [15112.193369] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15112.197685] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15112.201048] after_reply+0xe2c/0x1780 [ptlrpc] [15112.203396] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15112.206673] ? __next_timer_interrupt+0x160/0x160 [15112.208660] ptlrpcd+0xa82/0xe40 [ptlrpc] [15112.210916] ? do_wait_intr+0xf0/0xf0 [15112.212879] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15112.214394] kthread+0x1d1/0x200 [15112.215217] ? set_kthread_struct+0x70/0x70 [15112.217708] ret_from_fork+0x1f/0x30 [15116.059619] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [15116.068144] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15116.115585] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15116.130151] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15116.137204] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15116.147658] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15116.153384] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15116.171219] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [15116.178327] RAX: 0000000000007d9b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15116.183711] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [15116.191082] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [15116.200097] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [15116.203670] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [15116.211645] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [15116.221042] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15116.224895] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15116.232049] Call Trace: [15116.233701] [15116.234593] ? show_regs.cold.9+0x22/0x2f [15116.236229] ? watchdog_timer_fn.cold.6+0x62/0x10a [15116.238324] ? __hrtimer_run_queues+0x1c2/0x4d0 [15116.239908] ? watchdog+0x40/0x40 [15116.241692] ? hrtimer_interrupt+0x128/0x2c0 [15116.244062] ? smp_apic_timer_interrupt+0xa6/0x270 [15116.246116] ? apic_timer_interrupt+0xf/0x20 [15116.248081] [15116.249116] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15116.259295] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15116.261921] do_raw_spin_lock+0x177/0x190 [15116.274314] _raw_spin_lock+0x21/0x30 [15116.281274] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15116.287212] process_one_work+0x2c8/0x700 [15116.290197] worker_thread+0x296/0x6e0 [15116.298407] ? rescuer_thread+0x570/0x570 [15116.307883] kthread+0x1d1/0x200 [15116.309316] ? set_kthread_struct+0x70/0x70 [15116.315014] ret_from_fork+0x1f/0x30 [15120.052166] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [15120.055138] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15120.075698] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15120.080731] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15120.084897] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15120.087600] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15120.096002] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15120.100198] RAX: 0000000000005188 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15120.104015] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [15120.108894] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [15120.112704] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [15120.116344] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [15120.120749] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [15120.124026] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15120.127497] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [15120.130696] Call Trace: [15120.131318] [15120.131964] ? show_regs.cold.9+0x22/0x2f [15120.133517] ? watchdog_timer_fn.cold.6+0x62/0x10a [15120.134980] ? __hrtimer_run_queues+0x1c2/0x4d0 [15120.136605] ? watchdog+0x40/0x40 [15120.137616] ? hrtimer_interrupt+0x128/0x2c0 [15120.138895] ? smp_apic_timer_interrupt+0xa6/0x270 [15120.140766] ? apic_timer_interrupt+0xf/0x20 [15120.143357] [15120.144443] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15120.147351] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15120.149911] do_raw_spin_lock+0x177/0x190 [15120.151644] _raw_spin_lock+0x21/0x30 [15120.153488] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15120.156560] ? do_raw_spin_unlock+0x75/0x190 [15120.158547] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15120.160811] ? do_wait_intr+0xf0/0xf0 [15120.162162] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15120.164220] mdc_reint+0x2d/0x110 [mdc] [15120.165815] mdc_unlink+0x658/0x870 [mdc] [15120.167600] lmv_unlink+0x34d/0xbd0 [lmv] [15120.168859] ll_unlink+0x24d/0x7f0 [lustre] [15120.170619] vfs_unlink+0x202/0x330 [15120.171910] do_unlinkat+0x2fa/0x460 [15120.173034] __x64_sys_unlink+0x37/0x50 [15120.174663] do_syscall_64+0xc1/0x440 [15120.176167] entry_SYSCALL_64_after_hwframe+0x49/0xae [15120.177851] RIP: 0033:0x7f151bdee10b [15120.179365] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [15120.186038] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [15120.189086] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [15120.191675] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [15120.194211] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15120.196790] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15120.199165] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15124.927011] rcu: INFO: rcu_sched self-detected stall on CPU [15124.929980] rcu: 1-...!: (7860466 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [15124.936623] (t=13200280 jiffies g=249657 q=6403) [15124.946490] rcu: rcu_sched kthread starved for 13195232 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [15124.954422] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [15124.963592] rcu: RCU grace-period kthread stack dump: [15124.967104] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [15124.974439] Call Trace: [15124.975773] __schedule+0x351/0xcb0 [15124.980646] schedule+0xc0/0x180 [15124.983946] schedule_timeout+0xb4/0x190 [15124.987708] ? do_raw_spin_unlock+0x75/0x190 [15124.990218] ? __next_timer_interrupt+0x160/0x160 [15124.993608] ? prepare_to_swait_event+0xf3/0x140 [15124.998142] rcu_gp_kthread+0x8dd/0x15b0 [15125.002594] ? force_qs_rnp+0x2c0/0x2c0 [15125.005444] kthread+0x1d1/0x200 [15125.006691] ? set_kthread_struct+0x70/0x70 [15125.009496] ret_from_fork+0x1f/0x30 [15125.014234] rcu: Stack dump where RCU GP kthread last ran: [15125.019003] Sending NMI from CPU 1 to CPUs 3: [15125.028199] NMI backtrace for cpu 3 [15125.028206] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15125.028208] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15125.028209] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15125.028211] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15125.028213] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000216 [15125.028215] RAX: 0000000000007e94 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15125.028217] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [15125.028218] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [15125.028219] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [15125.028220] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [15125.028222] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [15125.028223] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15125.028224] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [15125.028226] Call Trace: [15125.028226] [15125.028227] ? show_regs.cold.9+0x22/0x2f [15125.028228] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [15125.028229] ? nmi_cpu_backtrace_handler+0x15/0x40 [15125.028230] ? nmi_handle+0xbd/0x210 [15125.028231] ? default_do_nmi+0x4b/0x1b0 [15125.028232] ? do_nmi+0x280/0x380 [15125.028233] ? end_repeat_nmi+0x16/0x1f [15125.028234] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15125.028235] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15125.028236] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15125.028237] [15125.028238] do_raw_spin_lock+0x177/0x190 [15125.028239] _raw_spin_lock+0x21/0x30 [15125.028240] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15125.028242] ? do_raw_spin_unlock+0x75/0x190 [15125.028243] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15125.028244] ? do_wait_intr+0xf0/0xf0 [15125.028245] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15125.028246] mdc_close+0x53b/0xd70 [mdc] [15125.028247] lmv_close+0x18d/0x490 [lmv] [15125.028248] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15125.028249] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15125.028250] ll_md_real_close+0xa5/0x2a0 [lustre] [15125.028251] ll_md_close+0x274/0x800 [lustre] [15125.028253] ll_file_release+0x128/0xc50 [lustre] [15125.028254] __fput+0x10d/0x440 [15125.028255] ____fput+0x12/0x20 [15125.028256] task_work_run+0xc6/0x110 [15125.028257] exit_to_usermode_loop+0x1dd/0x1f0 [15125.028258] do_syscall_64+0x3d6/0x440 [15125.028259] entry_SYSCALL_64_after_hwframe+0x49/0xae [15125.028261] RIP: 0033:0x7f151bee3cd8 [15125.028262] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15125.028264] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15125.028267] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [15125.028269] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [15125.028270] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15125.028271] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15125.028273] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15125.032781] NMI backtrace for cpu 1 [15125.352912] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15125.362547] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15125.370117] Call Trace: [15125.371619] [15125.372304] ? dump_stack+0xbb/0x10e [15125.374184] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [15125.379648] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [15125.382544] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [15125.388107] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [15125.390942] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [15125.398216] ? rcu_dump_cpu_stacks+0x11b/0x171 [15125.401012] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [15125.404553] ? update_process_times+0x6d/0xc0 [15125.407991] ? tick_sched_handle.isra.18+0x27/0x90 [15125.417281] ? tick_sched_timer+0x53/0xb0 [15125.422024] ? __hrtimer_run_queues+0x1c2/0x4d0 [15125.426897] ? tick_sched_handle.isra.18+0x90/0x90 [15125.434994] ? hrtimer_interrupt+0x128/0x2c0 [15125.440367] ? smp_apic_timer_interrupt+0xa6/0x270 [15125.444969] ? apic_timer_interrupt+0xf/0x20 [15125.448234] [15125.450571] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15125.454902] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15125.459636] ? do_raw_spin_lock+0x177/0x190 [15125.463975] ? _raw_spin_lock+0x21/0x30 [15125.467795] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15125.470196] ? obd_mod_free+0x46/0x140 [ptlrpc] [15125.476388] ? mdc_commit_open+0x91/0xb0 [mdc] [15125.479613] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15125.487176] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15125.493609] ? after_reply+0xe2c/0x1780 [ptlrpc] [15125.499538] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15125.506867] ? __next_timer_interrupt+0x160/0x160 [15125.511068] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [15125.514978] ? do_wait_intr+0xf0/0xf0 [15125.522086] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15125.525402] ? kthread+0x1d1/0x200 [15125.528605] ? set_kthread_struct+0x70/0x70 [15125.530188] ? ret_from_fork+0x1f/0x30 [15125.537171] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13194s! [15125.543268] Showing busy workqueues and worker pools: [15125.547639] workqueue events: flags=0x0 [15125.549785] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15125.554788] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [15125.562706] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15125.567996] pending: kfree_rcu_monitor [15125.572105] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15125.581246] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [15125.589117] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15125.594180] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [15125.604880] workqueue events_long: flags=0x0 [15125.610071] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15125.615394] pending: lu_objects_flush [obdclass] [15125.618798] workqueue events_unbound: flags=0x2 [15125.623548] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15125.628811] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15125.635607] workqueue events_power_efficient: flags=0x80 [15125.638876] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15125.642737] pending: do_cache_clean, neigh_periodic_work, check_lifetime [15125.647096] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15125.651143] pending: neigh_periodic_work [15125.653463] workqueue mm_percpu_wq: flags=0x8 [15125.655899] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15125.660084] pending: vmstat_update, lru_add_drain_per_cpu [15125.663218] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15125.666871] pending: vmstat_update [15125.668880] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15125.672620] pending: lru_add_drain_per_cpu [15125.675286] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [15125.679344] pending: lru_add_drain_per_cpu BAR(48) [15125.683503] workqueue writeback: flags=0x4a [15125.685951] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [15125.690067] pending: wb_workfn [15125.692078] workqueue xprtiod: flags=0xa [15125.694008] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15125.697125] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [15125.701254] workqueue ipv6_addrconf: flags=0x40008 [15125.703685] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15125.706934] pending: addrconf_verify_work [15125.709385] workqueue knbd0-recv: flags=0x1a [15125.711372] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15125.715235] in-flight: 658:recv_work [15125.717167] workqueue ptlrpc_pinger: flags=0x2 [15125.719089] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15125.721877] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15125.724843] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13195s workers=4 idle: 32984 37 8 [15125.728666] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15110s workers=2 idle: 661 [15144.062231] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [15144.089739] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15144.155082] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15144.166787] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15144.178352] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15144.187544] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15144.197278] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15144.219472] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15144.228543] RAX: 0000000000007886 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15144.238329] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [15144.246369] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [15144.252534] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [15144.261959] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [15144.270450] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [15144.283072] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15144.296597] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15144.303121] Call Trace: [15144.308432] [15144.309493] ? show_regs.cold.9+0x22/0x2f [15144.315058] ? watchdog_timer_fn.cold.6+0x62/0x10a [15144.324159] ? __hrtimer_run_queues+0x1c2/0x4d0 [15144.331672] ? watchdog+0x40/0x40 [15144.333733] ? hrtimer_interrupt+0x128/0x2c0 [15144.343747] ? smp_apic_timer_interrupt+0xa6/0x270 [15144.348325] ? apic_timer_interrupt+0xf/0x20 [15144.357653] [15144.359462] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15144.365630] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15144.370197] do_raw_spin_lock+0x177/0x190 [15144.375218] _raw_spin_lock+0x21/0x30 [15144.382108] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15144.389202] process_one_work+0x2c8/0x700 [15144.398493] worker_thread+0x296/0x6e0 [15144.403586] ? rescuer_thread+0x570/0x570 [15144.408632] kthread+0x1d1/0x200 [15144.411518] ? set_kthread_struct+0x70/0x70 [15144.421118] ret_from_fork+0x1f/0x30 [15152.052721] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [15152.056526] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [15152.060948] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) [15152.060989] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [15152.060994] obdclass(O) [15152.060999] Modules linked in: [15152.061000] ksocklnd(O) lnet(O) [15152.061006] lustre(O) [15152.061010] libcfs(O) [15152.061012] osp(O) [15152.061014] rpcsec_gss_krb5 [15152.061019] ofd(O) [15152.061020] auth_rpcgss [15152.061025] lod(O) [15152.061026] nfsv4 [15152.061032] mdt(O) [15152.061032] dns_resolver [15152.061037] mdd(O) [15152.061038] intel_rapl_msr [15152.061043] mgs(O) [15152.061044] intel_rapl_common sb_edac [15152.061050] lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15152.061111] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15152.061117] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15152.061120] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15152.061134] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15152.061145] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15152.061150] RAX: 0000000000006b87 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15152.061153] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [15152.061155] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [15152.061158] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [15152.061161] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [15152.061164] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [15152.061167] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15152.061170] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [15152.061178] Call Trace: [15152.061184] [15152.061189] ? show_regs.cold.9+0x22/0x2f [15152.061213] ? watchdog_timer_fn.cold.6+0x62/0x10a [15152.061219] ? __hrtimer_run_queues+0x1c2/0x4d0 [15152.061224] ? watchdog+0x40/0x40 [15152.061229] ? hrtimer_interrupt+0x128/0x2c0 [15152.061233] ? smp_apic_timer_interrupt+0xa6/0x270 [15152.061242] ? apic_timer_interrupt+0xf/0x20 [15152.061247] [15152.061251] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15152.061258] do_raw_spin_lock+0x177/0x190 [15152.061264] _raw_spin_lock+0x21/0x30 [15152.061269] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15152.061555] ? do_raw_spin_unlock+0x75/0x190 [15152.061561] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15152.061758] ? do_wait_intr+0xf0/0xf0 [15152.061835] rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15152.061860] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15152.061866] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15152.061868] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15152.061876] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15152.061880] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15152.061884] RAX: 0000000000007184 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15152.061887] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [15152.061889] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [15152.061891] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [15152.061893] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [15152.061896] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [15152.061899] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15152.061901] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [15152.061908] Call Trace: [15152.061911] [15152.061914] ? show_regs.cold.9+0x22/0x2f [15152.061922] ? watchdog_timer_fn.cold.6+0x62/0x10a [15152.061928] ? __hrtimer_run_queues+0x1c2/0x4d0 [15152.061932] ? watchdog+0x40/0x40 [15152.061936] ? hrtimer_interrupt+0x128/0x2c0 [15152.061940] ? smp_apic_timer_interrupt+0xa6/0x270 [15152.061947] ? apic_timer_interrupt+0xf/0x20 [15152.061952] [15152.061955] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15152.061960] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15152.061965] do_raw_spin_lock+0x177/0x190 [15152.061970] _raw_spin_lock+0x21/0x30 [15152.061766] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15152.061975] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15152.063262] mdc_close+0x53b/0xd70 [mdc] [15152.063353] lmv_close+0x18d/0x490 [lmv] [15152.063381] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15152.063544] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15152.063658] ll_md_real_close+0xa5/0x2a0 [lustre] [15152.063735] ll_md_close+0x274/0x800 [lustre] [15152.063818] ll_file_release+0x128/0xc50 [lustre] [15152.064078] __fput+0x10d/0x440 [15152.064090] ____fput+0x12/0x20 [15152.064094] task_work_run+0xc6/0x110 [15152.064102] exit_to_usermode_loop+0x1dd/0x1f0 [15152.064110] do_syscall_64+0x3d6/0x440 [15152.064115] entry_SYSCALL_64_after_hwframe+0x49/0xae [15152.064123] RIP: 0033:0x7f151bee3cd8 [15152.064188] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15152.064203] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15152.064209] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [15152.064211] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [15152.064213] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15152.064216] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15152.064218] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15152.067338] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15152.067426] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15152.067433] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15152.067437] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15152.067451] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15152.067455] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15152.067460] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [15152.067462] RDX: 0000000000007983 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [15152.067465] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [15152.067467] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [15152.067469] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15152.067472] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [15152.067475] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15152.067478] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15152.067486] Call Trace: [15152.067492] [15152.067498] ? show_regs.cold.9+0x22/0x2f [15152.067506] ? watchdog_timer_fn.cold.6+0x62/0x10a [15152.067512] ? __hrtimer_run_queues+0x1c2/0x4d0 [15152.067516] ? watchdog+0x40/0x40 [15152.067521] ? hrtimer_interrupt+0x128/0x2c0 [15152.092467] ? do_raw_spin_unlock+0x75/0x190 [15152.092489] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15152.094089] ? do_wait_intr+0xf0/0xf0 [15152.094099] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15152.094825] mdc_reint+0x2d/0x110 [mdc] [15152.094866] mdc_unlink+0x658/0x870 [mdc] [15152.094893] lmv_unlink+0x34d/0xbd0 [lmv] [15152.094923] ll_unlink+0x24d/0x7f0 [lustre] [15152.095035] vfs_unlink+0x202/0x330 [15152.095043] do_unlinkat+0x2fa/0x460 [15152.095048] __x64_sys_unlink+0x37/0x50 [15152.095052] do_syscall_64+0xc1/0x440 [15152.095075] entry_SYSCALL_64_after_hwframe+0x49/0xae [15152.095084] RIP: 0033:0x7f151bdee10b [15152.095141] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [15152.095146] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [15152.095152] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [15152.095154] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [15152.095156] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15152.095159] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15152.095161] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15153.008776] ? smp_apic_timer_interrupt+0xa6/0x270 [15153.011735] ? apic_timer_interrupt+0xf/0x20 [15153.013088] [15153.019596] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15153.021510] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15153.031459] do_raw_spin_lock+0x177/0x190 [15153.034649] _raw_spin_lock+0x21/0x30 [15153.036989] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15153.041588] obd_mod_free+0x46/0x140 [ptlrpc] [15153.045663] mdc_commit_open+0x91/0xb0 [mdc] [15153.049717] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15153.053401] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15153.057429] after_reply+0xe2c/0x1780 [ptlrpc] [15153.064039] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15153.070102] ? __next_timer_interrupt+0x160/0x160 [15153.074600] ptlrpcd+0xa82/0xe40 [ptlrpc] [15153.083841] ? do_wait_intr+0xf0/0xf0 [15153.091136] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15153.099935] kthread+0x1d1/0x200 [15153.104714] ? set_kthread_struct+0x70/0x70 [15153.110496] ret_from_fork+0x1f/0x30 [15156.196044] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13225s! [15156.201849] Showing busy workqueues and worker pools: [15156.209225] workqueue events: flags=0x0 [15156.213081] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15156.233976] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [15156.242939] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15156.254393] pending: kfree_rcu_monitor [15156.260980] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15156.266702] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [15156.275904] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15156.281121] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [15156.295596] workqueue events_long: flags=0x0 [15156.302910] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15156.314864] pending: lu_objects_flush [obdclass] [15156.335691] workqueue events_unbound: flags=0x2 [15156.345528] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15156.352770] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15156.367753] workqueue events_power_efficient: flags=0x80 [15156.377798] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15156.382266] pending: do_cache_clean, neigh_periodic_work, check_lifetime [15156.394393] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15156.411085] pending: neigh_periodic_work [15156.412924] workqueue mm_percpu_wq: flags=0x8 [15156.434524] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15156.445836] pending: vmstat_update, lru_add_drain_per_cpu [15156.460376] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15156.468817] pending: vmstat_update [15156.479778] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15156.490179] pending: lru_add_drain_per_cpu [15156.510145] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [15156.521636] pending: lru_add_drain_per_cpu BAR(48) [15156.537636] workqueue writeback: flags=0x4a [15156.546578] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [15156.552685] pending: wb_workfn [15156.559701] workqueue xprtiod: flags=0xa [15156.563480] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15156.573699] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [15156.594959] workqueue ipv6_addrconf: flags=0x40008 [15156.610462] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15156.620412] pending: addrconf_verify_work [15156.622007] workqueue knbd0-recv: flags=0x1a [15156.636782] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15156.640000] in-flight: 658:recv_work [15156.644463] workqueue ptlrpc_pinger: flags=0x2 [15156.647432] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15156.659651] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15156.668933] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13226s workers=4 idle: 32984 37 8 [15156.677406] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15141s workers=2 idle: 661 [15172.058088] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [15172.066044] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15172.107191] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15172.114113] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15172.119792] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15172.123835] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15172.127798] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15172.142803] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [15172.146559] RAX: 00000000000058ab RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15172.153813] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [15172.159246] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [15172.162424] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [15172.165712] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [15172.171388] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [15172.178772] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15172.187305] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15172.190819] Call Trace: [15172.195720] [15172.196826] ? show_regs.cold.9+0x22/0x2f [15172.201735] ? watchdog_timer_fn.cold.6+0x62/0x10a [15172.204861] ? __hrtimer_run_queues+0x1c2/0x4d0 [15172.207077] ? watchdog+0x40/0x40 [15172.208537] ? hrtimer_interrupt+0x128/0x2c0 [15172.212813] ? smp_apic_timer_interrupt+0xa6/0x270 [15172.214525] ? apic_timer_interrupt+0xf/0x20 [15172.217801] [15172.220635] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15172.224436] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15172.226585] do_raw_spin_lock+0x177/0x190 [15172.229055] _raw_spin_lock+0x21/0x30 [15172.232861] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15172.236798] process_one_work+0x2c8/0x700 [15172.237791] worker_thread+0x296/0x6e0 [15172.240145] ? rescuer_thread+0x570/0x570 [15172.242575] kthread+0x1d1/0x200 [15172.244034] ? set_kthread_struct+0x70/0x70 [15172.250511] ret_from_fork+0x1f/0x30 [15180.055035] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [15180.060505] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [15180.063476] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15180.063551] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15180.063559] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15180.063562] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15180.063577] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15180.063582] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15180.063586] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [15180.063589] RDX: 0000000000004e21 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [15180.063592] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [15180.063594] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [15180.063596] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15180.063599] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [15180.063602] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15180.063604] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15180.063613] Call Trace: [15180.063618] [15180.063624] ? show_regs.cold.9+0x22/0x2f [15180.063633] ? watchdog_timer_fn.cold.6+0x62/0x10a [15180.063641] ? __hrtimer_run_queues+0x1c2/0x4d0 [15180.063646] ? watchdog+0x40/0x40 [15180.063650] ? hrtimer_interrupt+0x128/0x2c0 [15180.063654] ? smp_apic_timer_interrupt+0xa6/0x270 [15180.063663] ? apic_timer_interrupt+0xf/0x20 [15180.063668] [15180.063671] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15180.063676] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15180.063680] do_raw_spin_lock+0x177/0x190 [15180.063685] _raw_spin_lock+0x21/0x30 [15180.063690] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15180.064275] obd_mod_free+0x46/0x140 [ptlrpc] [15180.064476] mdc_commit_open+0x91/0xb0 [mdc] [15180.064515] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15180.065626] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15180.066545] after_reply+0xe2c/0x1780 [ptlrpc] [15180.075968] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15180.076045] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15180.076053] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15180.076056] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15180.076070] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15180.076075] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15180.076079] RAX: 0000000000006199 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15180.076082] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [15180.076084] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [15180.076086] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [15180.076089] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [15180.076093] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [15180.076097] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15180.076099] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [15180.076108] Call Trace: [15180.076114] [15180.076120] ? show_regs.cold.9+0x22/0x2f [15180.076129] ? watchdog_timer_fn.cold.6+0x62/0x10a [15180.076136] ? __hrtimer_run_queues+0x1c2/0x4d0 [15180.076141] ? watchdog+0x40/0x40 [15180.076146] ? hrtimer_interrupt+0x128/0x2c0 [15180.076151] ? smp_apic_timer_interrupt+0xa6/0x270 [15180.076159] ? apic_timer_interrupt+0xf/0x20 [15180.076164] [15180.076167] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15180.076174] do_raw_spin_lock+0x177/0x190 [15180.076179] _raw_spin_lock+0x21/0x30 [15180.076185] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15180.129216] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15180.142341] ? do_raw_spin_unlock+0x75/0x190 [15180.142366] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15180.142653] ? do_wait_intr+0xf0/0xf0 [15180.142662] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15180.142866] mdc_close+0x53b/0xd70 [mdc] [15180.142906] lmv_close+0x18d/0x490 [lmv] [15180.142947] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15180.143112] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15180.143222] ll_md_real_close+0xa5/0x2a0 [lustre] [15180.143304] ll_md_close+0x274/0x800 [lustre] [15180.143394] ll_file_release+0x128/0xc50 [lustre] [15180.143478] __fput+0x10d/0x440 [15180.143488] ____fput+0x12/0x20 [15180.143492] task_work_run+0xc6/0x110 [15180.143499] exit_to_usermode_loop+0x1dd/0x1f0 [15180.143506] do_syscall_64+0x3d6/0x440 [15180.143511] entry_SYSCALL_64_after_hwframe+0x49/0xae [15180.143520] RIP: 0033:0x7f151bee3cd8 [15180.143528] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15180.143533] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15180.143537] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [15180.143540] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [15180.143542] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15180.143544] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15180.143546] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15180.753844] ? __next_timer_interrupt+0x160/0x160 [15180.755465] ptlrpcd+0xa82/0xe40 [ptlrpc] [15180.761765] ? do_wait_intr+0xf0/0xf0 [15180.764129] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15180.766879] kthread+0x1d1/0x200 [15180.768187] ? set_kthread_struct+0x70/0x70 [15180.771049] ret_from_fork+0x1f/0x30 [15184.053909] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [15184.056019] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15184.072439] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15184.076501] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15184.079330] RIP: 0010:native_safe_halt+0x2e/0x50 [15184.080627] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15184.087503] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15184.092066] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15184.097016] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [15184.105345] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [15184.108776] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [15184.111458] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [15184.117213] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [15184.127289] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15184.131234] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [15184.134829] Call Trace: [15184.135849] [15184.137050] ? show_regs.cold.9+0x22/0x2f [15184.139539] ? watchdog_timer_fn.cold.6+0x62/0x10a [15184.148390] ? __hrtimer_run_queues+0x1c2/0x4d0 [15184.150174] ? watchdog+0x40/0x40 [15184.152848] ? hrtimer_interrupt+0x128/0x2c0 [15184.160553] ? smp_apic_timer_interrupt+0xa6/0x270 [15184.163521] ? apic_timer_interrupt+0xf/0x20 [15184.174810] [15184.178264] ? native_safe_halt+0x2e/0x50 [15184.182624] kvm_wait+0xcd/0xe0 [15184.185744] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15184.192693] do_raw_spin_lock+0x177/0x190 [15184.195787] _raw_spin_lock+0x21/0x30 [15184.197156] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15184.202679] ? do_raw_spin_unlock+0x75/0x190 [15184.206432] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15184.211727] ? do_wait_intr+0xf0/0xf0 [15184.216540] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15184.218998] mdc_reint+0x2d/0x110 [mdc] [15184.222063] mdc_unlink+0x658/0x870 [mdc] [15184.226203] lmv_unlink+0x34d/0xbd0 [lmv] [15184.228714] ll_unlink+0x24d/0x7f0 [lustre] [15184.231811] vfs_unlink+0x202/0x330 [15184.235144] do_unlinkat+0x2fa/0x460 [15184.237565] __x64_sys_unlink+0x37/0x50 [15184.240526] do_syscall_64+0xc1/0x440 [15184.242475] entry_SYSCALL_64_after_hwframe+0x49/0xae [15184.245657] RIP: 0033:0x7f151bdee10b [15184.249588] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [15184.259519] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [15184.263267] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [15184.276451] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [15184.293337] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15184.295830] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15184.303380] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15186.915406] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13256s! [15186.926019] Showing busy workqueues and worker pools: [15186.937071] workqueue events: flags=0x0 [15186.941256] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15186.955831] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [15186.969183] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15186.980329] pending: kfree_rcu_monitor [15186.981841] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15186.997696] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [15187.013611] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15187.022818] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [15187.030678] workqueue events_long: flags=0x0 [15187.033331] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15187.042362] pending: lu_objects_flush [obdclass] [15187.049399] workqueue events_unbound: flags=0x2 [15187.053710] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15187.058156] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15187.070794] workqueue events_power_efficient: flags=0x80 [15187.080576] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15187.087674] pending: do_cache_clean, neigh_periodic_work, check_lifetime [15187.093693] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15187.101671] pending: neigh_periodic_work [15187.105592] workqueue mm_percpu_wq: flags=0x8 [15187.110638] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15187.115336] pending: vmstat_update, lru_add_drain_per_cpu [15187.119549] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15187.123728] pending: vmstat_update [15187.126768] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15187.130973] pending: lru_add_drain_per_cpu [15187.134680] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [15187.139736] pending: lru_add_drain_per_cpu BAR(48) [15187.143640] workqueue writeback: flags=0x4a [15187.145782] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [15187.153033] pending: wb_workfn [15187.154262] workqueue xprtiod: flags=0xa [15187.157264] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15187.161981] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [15187.167582] workqueue ipv6_addrconf: flags=0x40008 [15187.174051] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15187.179803] pending: addrconf_verify_work [15187.182911] workqueue knbd0-recv: flags=0x1a [15187.185919] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15187.189641] in-flight: 658:recv_work [15187.191510] workqueue ptlrpc_pinger: flags=0x2 [15187.192740] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15187.196564] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15187.202611] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13256s workers=4 idle: 32984 37 8 [15187.207505] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15171s workers=2 idle: 661 [15200.056026] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [15200.059253] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15200.093161] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15200.103798] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15200.110021] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15200.115458] RIP: 0010:native_safe_halt+0x2e/0x50 [15200.117998] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15200.130418] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15200.134961] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15200.141637] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [15200.144826] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [15200.149418] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [15200.154600] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [15200.158954] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [15200.164586] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15200.169039] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15200.173817] Call Trace: [15200.175296] [15200.176469] ? show_regs.cold.9+0x22/0x2f [15200.180015] ? watchdog_timer_fn.cold.6+0x62/0x10a [15200.182756] ? __hrtimer_run_queues+0x1c2/0x4d0 [15200.185990] ? watchdog+0x40/0x40 [15200.188051] ? hrtimer_interrupt+0x128/0x2c0 [15200.190685] ? smp_apic_timer_interrupt+0xa6/0x270 [15200.193809] ? apic_timer_interrupt+0xf/0x20 [15200.196561] [15200.198433] ? native_safe_halt+0x2e/0x50 [15200.200916] kvm_wait+0xcd/0xe0 [15200.205563] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15200.209842] do_raw_spin_lock+0x177/0x190 [15200.212855] _raw_spin_lock+0x21/0x30 [15200.215455] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15200.220827] process_one_work+0x2c8/0x700 [15200.223735] worker_thread+0x296/0x6e0 [15200.226244] ? rescuer_thread+0x570/0x570 [15200.228533] kthread+0x1d1/0x200 [15200.230838] ? set_kthread_struct+0x70/0x70 [15200.233052] ret_from_fork+0x1f/0x30 [15208.055027] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [15208.057070] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [15208.059259] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15208.059342] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15208.059350] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15208.059354] RIP: 0010:native_safe_halt+0x2e/0x50 [15208.062390] Modules linked in: [15208.084758] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15208.089806] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15208.089891] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15208.093938] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 [15208.095899] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15208.097705] ORIG_RAX: ffffffffffffff13 [15208.105773] RIP: 0010:native_safe_halt+0x2e/0x50 [15208.128814] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [15208.140159] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15208.140174] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15208.140182] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15208.140185] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [15208.140188] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [15208.140191] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [15208.140194] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [15208.140197] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [15208.140201] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15208.140203] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [15208.140211] Call Trace: [15208.140216] [15208.140222] ? show_regs.cold.9+0x22/0x2f [15208.140248] ? watchdog_timer_fn.cold.6+0x62/0x10a [15208.140255] ? __hrtimer_run_queues+0x1c2/0x4d0 [15208.140259] ? watchdog+0x40/0x40 [15208.140263] ? hrtimer_interrupt+0x128/0x2c0 [15208.140268] ? smp_apic_timer_interrupt+0xa6/0x270 [15208.140277] ? apic_timer_interrupt+0xf/0x20 [15208.140281] [15208.140285] ? native_safe_halt+0x2e/0x50 [15208.140289] kvm_wait+0xcd/0xe0 [15208.140297] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15208.140307] do_raw_spin_lock+0x177/0x190 [15208.140311] _raw_spin_lock+0x21/0x30 [15208.140315] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15208.140785] ? do_raw_spin_unlock+0x75/0x190 [15208.140791] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15208.140981] ? do_wait_intr+0xf0/0xf0 [15208.140990] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15208.141177] mdc_close+0x53b/0xd70 [mdc] [15208.141222] lmv_close+0x18d/0x490 [lmv] [15208.141258] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15208.141420] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15208.141787] ll_md_real_close+0xa5/0x2a0 [lustre] [15208.141868] ll_md_close+0x274/0x800 [lustre] [15208.141955] ll_file_release+0x128/0xc50 [lustre] [15208.142046] __fput+0x10d/0x440 [15208.142055] ____fput+0x12/0x20 [15208.142059] task_work_run+0xc6/0x110 [15208.142067] exit_to_usermode_loop+0x1dd/0x1f0 [15208.142075] do_syscall_64+0x3d6/0x440 [15208.142080] entry_SYSCALL_64_after_hwframe+0x49/0xae [15208.142088] RIP: 0033:0x7f151bee3cd8 [15208.142095] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15208.142099] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15208.142103] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [15208.142106] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [15208.142109] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15208.142111] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15208.142114] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15208.357184] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [15208.361363] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [15208.366123] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [15208.370219] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15208.374911] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [15208.382763] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15208.385763] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15208.387789] Call Trace: [15208.389049] [15208.389859] ? show_regs.cold.9+0x22/0x2f [15208.391505] ? watchdog_timer_fn.cold.6+0x62/0x10a [15208.393362] ? __hrtimer_run_queues+0x1c2/0x4d0 [15208.396832] ? watchdog+0x40/0x40 [15208.398745] ? hrtimer_interrupt+0x128/0x2c0 [15208.400382] ? smp_apic_timer_interrupt+0xa6/0x270 [15208.403429] ? apic_timer_interrupt+0xf/0x20 [15208.405451] [15208.406297] ? native_safe_halt+0x2e/0x50 [15208.408866] kvm_wait+0xcd/0xe0 [15208.410330] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15208.414725] do_raw_spin_lock+0x177/0x190 [15208.417939] _raw_spin_lock+0x21/0x30 [15208.420469] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15208.424976] obd_mod_free+0x46/0x140 [ptlrpc] [15208.427313] mdc_commit_open+0x91/0xb0 [mdc] [15208.429135] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15208.434017] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15208.438697] after_reply+0xe2c/0x1780 [ptlrpc] [15208.441922] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15208.443524] ? __next_timer_interrupt+0x160/0x160 [15208.449406] ptlrpcd+0xa82/0xe40 [ptlrpc] [15208.452927] ? do_wait_intr+0xf0/0xf0 [15208.455860] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15208.460582] kthread+0x1d1/0x200 [15208.463071] ? set_kthread_struct+0x70/0x70 [15208.464554] ret_from_fork+0x1f/0x30 [15212.052128] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [15212.055762] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15212.105387] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15212.120391] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15212.138216] RIP: 0010:native_safe_halt+0x2e/0x50 [15212.143255] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15212.167125] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15212.175774] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15212.183700] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [15212.192588] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [15212.205802] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [15212.214310] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [15212.223137] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [15212.234371] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15212.242824] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [15212.254791] Call Trace: [15212.258014] [15212.263049] ? show_regs.cold.9+0x22/0x2f [15212.269117] ? watchdog_timer_fn.cold.6+0x62/0x10a [15212.276783] ? __hrtimer_run_queues+0x1c2/0x4d0 [15212.283375] ? watchdog+0x40/0x40 [15212.288276] ? hrtimer_interrupt+0x128/0x2c0 [15212.298123] ? smp_apic_timer_interrupt+0xa6/0x270 [15212.299827] ? apic_timer_interrupt+0xf/0x20 [15212.304757] [15212.307432] ? native_safe_halt+0x2e/0x50 [15212.314216] kvm_wait+0xcd/0xe0 [15212.315402] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15212.322068] do_raw_spin_lock+0x177/0x190 [15212.325387] _raw_spin_lock+0x21/0x30 [15212.331988] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15212.338271] ? do_raw_spin_unlock+0x75/0x190 [15212.342424] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15212.351744] ? do_wait_intr+0xf0/0xf0 [15212.353099] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15212.358535] mdc_reint+0x2d/0x110 [mdc] [15212.363233] mdc_unlink+0x658/0x870 [mdc] [15212.368956] lmv_unlink+0x34d/0xbd0 [lmv] [15212.375050] ll_unlink+0x24d/0x7f0 [lustre] [15212.380140] vfs_unlink+0x202/0x330 [15212.388049] do_unlinkat+0x2fa/0x460 [15212.392710] __x64_sys_unlink+0x37/0x50 [15212.394194] do_syscall_64+0xc1/0x440 [15212.405862] entry_SYSCALL_64_after_hwframe+0x49/0xae [15212.418351] RIP: 0033:0x7f151bdee10b [15212.419701] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [15212.441431] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [15212.453371] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [15212.459893] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [15212.467016] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15212.472837] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15212.481243] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15217.632139] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13287s! [15217.636300] Showing busy workqueues and worker pools: [15217.639077] workqueue events: flags=0x0 [15217.641408] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15217.646661] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [15217.653102] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15217.655522] pending: kfree_rcu_monitor [15217.657137] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15217.659329] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [15217.666322] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15217.668853] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [15217.672148] workqueue events_long: flags=0x0 [15217.673550] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15217.680676] pending: lu_objects_flush [obdclass] [15217.683211] workqueue events_unbound: flags=0x2 [15217.685059] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15217.687530] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15217.692063] workqueue events_power_efficient: flags=0x80 [15217.693835] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15217.696632] pending: do_cache_clean, neigh_periodic_work, check_lifetime [15217.700612] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15217.703216] pending: neigh_periodic_work [15217.705980] workqueue mm_percpu_wq: flags=0x8 [15217.710966] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15217.715812] pending: vmstat_update, lru_add_drain_per_cpu [15217.722461] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15217.726375] pending: vmstat_update [15217.728210] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15217.730405] pending: lru_add_drain_per_cpu [15217.732832] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [15217.735786] pending: lru_add_drain_per_cpu BAR(48) [15217.738222] workqueue writeback: flags=0x4a [15217.739600] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [15217.742990] pending: wb_workfn [15217.744370] workqueue xprtiod: flags=0xa [15217.747564] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15217.749755] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [15217.753698] workqueue ipv6_addrconf: flags=0x40008 [15217.756189] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15217.759181] pending: addrconf_verify_work [15217.763665] workqueue knbd0-recv: flags=0x1a [15217.765477] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15217.769693] in-flight: 658:recv_work [15217.772211] workqueue ptlrpc_pinger: flags=0x2 [15217.774495] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15217.779229] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15217.781951] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13287s workers=4 idle: 32984 37 8 [15217.786813] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15202s workers=2 idle: 661 [15228.056025] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [15228.063144] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15228.096199] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15228.106022] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15228.110001] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15228.113084] RIP: 0010:native_safe_halt+0x2e/0x50 [15228.115417] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15228.124444] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15228.128439] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15228.132546] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [15228.135784] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [15228.139276] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [15228.142730] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [15228.146233] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [15228.150281] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15228.153081] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15228.156798] Call Trace: [15228.158083] [15228.159005] ? show_regs.cold.9+0x22/0x2f [15228.160659] ? watchdog_timer_fn.cold.6+0x62/0x10a [15228.163100] ? __hrtimer_run_queues+0x1c2/0x4d0 [15228.165591] ? watchdog+0x40/0x40 [15228.167151] ? hrtimer_interrupt+0x128/0x2c0 [15228.169101] ? smp_apic_timer_interrupt+0xa6/0x270 [15228.171261] ? apic_timer_interrupt+0xf/0x20 [15228.173609] [15228.174652] ? native_safe_halt+0x2e/0x50 [15228.176941] kvm_wait+0xcd/0xe0 [15228.178445] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15228.181176] do_raw_spin_lock+0x177/0x190 [15228.183395] _raw_spin_lock+0x21/0x30 [15228.186070] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15228.189197] process_one_work+0x2c8/0x700 [15228.192057] worker_thread+0x296/0x6e0 [15228.194204] ? rescuer_thread+0x570/0x570 [15228.196429] kthread+0x1d1/0x200 [15228.197998] ? set_kthread_struct+0x70/0x70 [15228.200167] ret_from_fork+0x1f/0x30 [15236.055249] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [15236.057027] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [15236.068130] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) [15236.071227] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15236.071376] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15236.071396] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15236.071399] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15236.071415] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15236.071420] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15236.071425] RAX: 0000000000001d1d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15236.071428] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [15236.071430] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [15236.071433] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [15236.071436] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [15236.071440] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [15236.071443] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15236.071445] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [15236.071455] Call Trace: [15236.071461] [15236.071468] ? show_regs.cold.9+0x22/0x2f [15236.071481] ? watchdog_timer_fn.cold.6+0x62/0x10a [15236.071489] ? __hrtimer_run_queues+0x1c2/0x4d0 [15236.071495] ? watchdog+0x40/0x40 [15236.071499] ? hrtimer_interrupt+0x128/0x2c0 [15236.071504] ? smp_apic_timer_interrupt+0xa6/0x270 [15236.071515] ? apic_timer_interrupt+0xf/0x20 [15236.071520] [15236.071523] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15236.071530] do_raw_spin_lock+0x177/0x190 [15236.071535] _raw_spin_lock+0x21/0x30 [15236.071540] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15236.071924] ? do_raw_spin_unlock+0x75/0x190 [15236.071931] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15236.072147] ? do_wait_intr+0xf0/0xf0 [15236.072156] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15236.072531] mdc_close+0x53b/0xd70 [mdc] [15236.072656] lmv_close+0x18d/0x490 [lmv] [15236.072688] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15236.072859] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15236.072984] ll_md_real_close+0xa5/0x2a0 [lustre] [15236.073081] ll_md_close+0x274/0x800 [lustre] [15236.073172] ll_file_release+0x128/0xc50 [lustre] [15236.073398] __fput+0x10d/0x440 [15236.073407] ____fput+0x12/0x20 [15236.073411] task_work_run+0xc6/0x110 [15236.073421] exit_to_usermode_loop+0x1dd/0x1f0 [15236.073430] do_syscall_64+0x3d6/0x440 [15236.073435] entry_SYSCALL_64_after_hwframe+0x49/0xae [15236.073445] RIP: 0033:0x7f151bee3cd8 [15236.073454] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15236.073525] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15236.073530] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [15236.073533] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [15236.073535] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15236.073537] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15236.073539] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15236.516719] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15236.583129] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15236.589672] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15236.595965] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15236.599104] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15236.627743] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15236.638012] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [15236.644754] RDX: 0000000000006fd9 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [15236.655559] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [15236.662042] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [15236.667059] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15236.677779] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [15236.688682] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15236.695573] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15236.700649] Call Trace: [15236.701938] [15236.704408] ? show_regs.cold.9+0x22/0x2f [15236.714898] ? watchdog_timer_fn.cold.6+0x62/0x10a [15236.718118] ? __hrtimer_run_queues+0x1c2/0x4d0 [15236.727192] ? watchdog+0x40/0x40 [15236.732691] ? hrtimer_interrupt+0x128/0x2c0 [15236.737299] ? smp_apic_timer_interrupt+0xa6/0x270 [15236.744697] ? apic_timer_interrupt+0xf/0x20 [15236.748592] [15236.752510] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15236.755895] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15236.762158] do_raw_spin_lock+0x177/0x190 [15236.765137] _raw_spin_lock+0x21/0x30 [15236.770623] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15236.776062] obd_mod_free+0x46/0x140 [ptlrpc] [15236.780551] mdc_commit_open+0x91/0xb0 [mdc] [15236.782028] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15236.788025] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15236.791465] after_reply+0xe2c/0x1780 [ptlrpc] [15236.796006] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15236.803782] ? __next_timer_interrupt+0x160/0x160 [15236.814386] ptlrpcd+0xa82/0xe40 [ptlrpc] [15236.819828] ? do_wait_intr+0xf0/0xf0 [15236.828966] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15236.834045] kthread+0x1d1/0x200 [15236.839659] ? set_kthread_struct+0x70/0x70 [15236.843126] ret_from_fork+0x1f/0x30 [15244.057330] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [15244.066982] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15244.119641] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15244.133161] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15244.138921] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15244.149885] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15244.185144] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15244.194762] RAX: 0000000000007074 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15244.204438] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [15244.210534] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [15244.214485] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [15244.223674] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [15244.231307] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [15244.240710] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15244.248125] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [15244.258347] Call Trace: [15244.260028] [15244.261573] ? show_regs.cold.9+0x22/0x2f [15244.268777] ? watchdog_timer_fn.cold.6+0x62/0x10a [15244.273035] ? __hrtimer_run_queues+0x1c2/0x4d0 [15244.280112] ? watchdog+0x40/0x40 [15244.281265] ? hrtimer_interrupt+0x128/0x2c0 [15244.288437] ? smp_apic_timer_interrupt+0xa6/0x270 [15244.293972] ? apic_timer_interrupt+0xf/0x20 [15244.300738] [15244.301871] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15244.305830] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15244.309200] do_raw_spin_lock+0x177/0x190 [15244.313949] _raw_spin_lock+0x21/0x30 [15244.317542] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15244.323853] ? do_raw_spin_unlock+0x75/0x190 [15244.328366] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15244.334133] ? do_wait_intr+0xf0/0xf0 [15244.337489] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15244.342479] mdc_reint+0x2d/0x110 [mdc] [15244.345502] mdc_unlink+0x658/0x870 [mdc] [15244.350484] lmv_unlink+0x34d/0xbd0 [lmv] [15244.353800] ll_unlink+0x24d/0x7f0 [lustre] [15244.359084] vfs_unlink+0x202/0x330 [15244.363354] do_unlinkat+0x2fa/0x460 [15244.365926] __x64_sys_unlink+0x37/0x50 [15244.370464] do_syscall_64+0xc1/0x440 [15244.374640] entry_SYSCALL_64_after_hwframe+0x49/0xae [15244.380453] RIP: 0033:0x7f151bdee10b [15244.384599] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [15244.404309] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [15244.412903] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [15244.421362] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [15244.428791] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15244.436631] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15244.443861] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15247.842156] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13317s! [15247.845130] Showing busy workqueues and worker pools: [15247.850801] workqueue events: flags=0x0 [15247.854060] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15247.869016] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [15247.880487] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15247.885015] pending: kfree_rcu_monitor [15247.887108] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15247.890911] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [15247.907809] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15247.917868] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [15247.926683] workqueue events_long: flags=0x0 [15247.932015] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15247.940046] pending: lu_objects_flush [obdclass] [15247.943458] workqueue events_unbound: flags=0x2 [15247.944576] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15247.956834] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15247.962908] workqueue events_power_efficient: flags=0x80 [15247.965014] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15247.967664] pending: do_cache_clean, neigh_periodic_work, check_lifetime [15247.970560] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15247.973400] pending: neigh_periodic_work [15247.975077] workqueue mm_percpu_wq: flags=0x8 [15247.976886] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15247.980004] pending: vmstat_update, lru_add_drain_per_cpu [15247.982453] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15247.987449] pending: vmstat_update [15247.988908] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15247.992665] pending: lru_add_drain_per_cpu [15247.995864] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [15248.002740] pending: lru_add_drain_per_cpu BAR(48) [15248.008231] workqueue writeback: flags=0x4a [15248.011482] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [15248.013438] pending: wb_workfn [15248.018788] workqueue xprtiod: flags=0xa [15248.020970] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15248.023714] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [15248.030645] workqueue ipv6_addrconf: flags=0x40008 [15248.035017] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15248.039346] pending: addrconf_verify_work [15248.041371] workqueue knbd0-recv: flags=0x1a [15248.043348] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15248.046246] in-flight: 658:recv_work [15248.047737] workqueue ptlrpc_pinger: flags=0x2 [15248.049500] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15248.052409] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15248.054810] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13317s workers=4 idle: 32984 37 8 [15248.058838] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15232s workers=2 idle: 661 [15256.056137] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [15256.064544] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15256.088214] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15256.094657] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15256.102826] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15256.104945] RIP: 0010:native_safe_halt+0x2e/0x50 [15256.111611] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15256.127558] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15256.138770] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15256.145828] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [15256.155986] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [15256.163461] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [15256.173470] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [15256.181864] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [15256.187865] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15256.192855] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15256.201337] Call Trace: [15256.202330] [15256.203251] ? show_regs.cold.9+0x22/0x2f [15256.204701] ? watchdog_timer_fn.cold.6+0x62/0x10a [15256.209510] ? __hrtimer_run_queues+0x1c2/0x4d0 [15256.211353] ? watchdog+0x40/0x40 [15256.213035] ? hrtimer_interrupt+0x128/0x2c0 [15256.215033] ? smp_apic_timer_interrupt+0xa6/0x270 [15256.218990] ? apic_timer_interrupt+0xf/0x20 [15256.223353] [15256.227995] ? native_safe_halt+0x2e/0x50 [15256.229484] kvm_wait+0xcd/0xe0 [15256.230606] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15256.241299] do_raw_spin_lock+0x177/0x190 [15256.243555] _raw_spin_lock+0x21/0x30 [15256.246160] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15256.255193] process_one_work+0x2c8/0x700 [15256.257210] worker_thread+0x296/0x6e0 [15256.261543] ? rescuer_thread+0x570/0x570 [15256.264717] kthread+0x1d1/0x200 [15256.266354] ? set_kthread_struct+0x70/0x70 [15256.270442] ret_from_fork+0x1f/0x30 [15264.055026] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [15264.057142] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [15264.057849] Modules linked in: lustre(O) [15264.062540] Modules linked in: lustre(O) osp(O) [15264.064006] osp(O) [15264.064009] ofd(O) [15264.064014] lod(O) mdt(O) [15264.066024] ofd(O) [15264.067265] mdd(O) [15264.068135] lod(O) [15264.069589] mgs(O) [15264.070505] mdt(O) [15264.071922] lquota(O) [15264.072698] mdd(O) [15264.074154] lfsck(O) [15264.075185] mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15264.075254] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15264.075267] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15264.075270] RIP: 0010:native_safe_halt+0x2e/0x50 [15264.075289] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15264.075293] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15264.075298] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [15264.075300] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [15264.075303] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [15264.075305] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [15264.075315] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15264.075319] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [15264.075322] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15264.075324] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15264.075332] Call Trace: [15264.076813] obdecho(O) [15264.077607] [15264.078677] mgc(O) [15264.094221] ? show_regs.cold.9+0x22/0x2f [15264.094243] ? watchdog_timer_fn.cold.6+0x62/0x10a [15264.094251] ? __hrtimer_run_queues+0x1c2/0x4d0 [15264.094255] ? watchdog+0x40/0x40 [15264.094260] ? hrtimer_interrupt+0x128/0x2c0 [15264.094264] ? smp_apic_timer_interrupt+0xa6/0x270 [15264.094276] ? apic_timer_interrupt+0xf/0x20 [15264.094281] [15264.094286] ? native_safe_halt+0x2e/0x50 [15264.094291] kvm_wait+0xcd/0xe0 [15264.094300] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15264.094315] do_raw_spin_lock+0x177/0x190 [15264.094321] _raw_spin_lock+0x21/0x30 [15264.094328] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15264.094806] obd_mod_free+0x46/0x140 [ptlrpc] [15264.095009] mdc_commit_open+0x91/0xb0 [mdc] [15264.099917] mdc(O) [15264.103274] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15264.104823] lov(O) [15264.112424] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15264.112648] after_reply+0xe2c/0x1780 [ptlrpc] [15264.112783] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15264.112986] ? __next_timer_interrupt+0x160/0x160 [15264.112997] ptlrpcd+0xa82/0xe40 [ptlrpc] [15264.117805] osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15264.117864] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15264.117871] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15264.117874] RIP: 0010:native_safe_halt+0x2e/0x50 [15264.117887] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15264.117891] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15264.117896] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15264.117898] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [15264.117900] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [15264.117902] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [15264.117904] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [15264.117908] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [15264.117911] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15264.117913] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [15264.117922] Call Trace: [15264.117928] [15264.117934] ? show_regs.cold.9+0x22/0x2f [15264.117948] ? watchdog_timer_fn.cold.6+0x62/0x10a [15264.117955] ? __hrtimer_run_queues+0x1c2/0x4d0 [15264.117960] ? watchdog+0x40/0x40 [15264.117964] ? hrtimer_interrupt+0x128/0x2c0 [15264.117969] ? smp_apic_timer_interrupt+0xa6/0x270 [15264.117995] ? apic_timer_interrupt+0xf/0x20 [15264.118000] [15264.118003] ? native_safe_halt+0x2e/0x50 [15264.118006] kvm_wait+0xcd/0xe0 [15264.118014] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15264.118023] do_raw_spin_lock+0x177/0x190 [15264.118028] _raw_spin_lock+0x21/0x30 [15264.118033] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15264.118485] ? do_raw_spin_unlock+0x75/0x190 [15264.118491] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15264.118690] ? do_wait_intr+0xf0/0xf0 [15264.118699] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15264.118895] mdc_close+0x53b/0xd70 [mdc] [15264.118933] lmv_close+0x18d/0x490 [lmv] [15264.118961] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15264.119390] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15264.119506] ll_md_real_close+0xa5/0x2a0 [lustre] [15264.119588] ll_md_close+0x274/0x800 [lustre] [15264.119668] ll_file_release+0x128/0xc50 [lustre] [15264.119733] __fput+0x10d/0x440 [15264.119741] ____fput+0x12/0x20 [15264.119745] task_work_run+0xc6/0x110 [15264.119750] exit_to_usermode_loop+0x1dd/0x1f0 [15264.119757] do_syscall_64+0x3d6/0x440 [15264.119762] entry_SYSCALL_64_after_hwframe+0x49/0xae [15264.119770] RIP: 0033:0x7f151bee3cd8 [15264.119777] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15264.119782] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15264.119786] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [15264.119788] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [15264.119791] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15264.119793] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15264.119795] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15264.422820] ? do_wait_intr+0xf0/0xf0 [15264.424450] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15264.428364] kthread+0x1d1/0x200 [15264.430237] ? set_kthread_struct+0x70/0x70 [15264.433041] ret_from_fork+0x1f/0x30 [15276.052697] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [15276.063208] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15276.134186] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15276.152707] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15276.179913] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15276.196361] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15276.210261] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15276.214340] RAX: 0000000000005989 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15276.233842] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [15276.246671] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [15276.251499] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [15276.261835] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [15276.285771] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [15276.298845] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15276.303805] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [15276.308047] Call Trace: [15276.314407] [15276.317493] ? show_regs.cold.9+0x22/0x2f [15276.323849] ? watchdog_timer_fn.cold.6+0x62/0x10a [15276.327768] ? __hrtimer_run_queues+0x1c2/0x4d0 [15276.332005] ? watchdog+0x40/0x40 [15276.335919] ? hrtimer_interrupt+0x128/0x2c0 [15276.339720] ? smp_apic_timer_interrupt+0xa6/0x270 [15276.349302] ? apic_timer_interrupt+0xf/0x20 [15276.350833] [15276.364688] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15276.366304] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15276.380259] do_raw_spin_lock+0x177/0x190 [15276.383301] _raw_spin_lock+0x21/0x30 [15276.389025] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15276.402925] ? do_raw_spin_unlock+0x75/0x190 [15276.404464] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15276.408634] ? do_wait_intr+0xf0/0xf0 [15276.413775] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15276.416256] mdc_reint+0x2d/0x110 [mdc] [15276.421282] mdc_unlink+0x658/0x870 [mdc] [15276.426748] lmv_unlink+0x34d/0xbd0 [lmv] [15276.431025] ll_unlink+0x24d/0x7f0 [lustre] [15276.436436] vfs_unlink+0x202/0x330 [15276.437827] do_unlinkat+0x2fa/0x460 [15276.442000] __x64_sys_unlink+0x37/0x50 [15276.444167] do_syscall_64+0xc1/0x440 [15276.446215] entry_SYSCALL_64_after_hwframe+0x49/0xae [15276.449311] RIP: 0033:0x7f151bdee10b [15276.451154] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [15276.463282] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [15276.470371] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [15276.481838] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [15276.499250] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15276.503628] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15276.509826] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15278.564193] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13347s! [15278.573152] Showing busy workqueues and worker pools: [15278.576031] workqueue events: flags=0x0 [15278.577940] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15278.582643] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [15278.588393] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15278.593142] pending: kfree_rcu_monitor [15278.605639] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15278.612229] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [15278.617732] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15278.622229] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [15278.625616] workqueue events_long: flags=0x0 [15278.627572] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15278.631169] pending: lu_objects_flush [obdclass] [15278.634377] workqueue events_unbound: flags=0x2 [15278.637505] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15278.640673] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15278.645521] workqueue events_power_efficient: flags=0x80 [15278.648592] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15278.651741] pending: do_cache_clean, neigh_periodic_work, check_lifetime [15278.655678] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15278.659115] pending: neigh_periodic_work [15278.661287] workqueue mm_percpu_wq: flags=0x8 [15278.663433] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15278.667487] pending: vmstat_update, lru_add_drain_per_cpu [15278.670624] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15278.674544] pending: vmstat_update [15278.676403] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15278.683760] pending: lru_add_drain_per_cpu [15278.688610] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [15278.694634] pending: lru_add_drain_per_cpu BAR(48) [15278.699221] workqueue writeback: flags=0x4a [15278.701788] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [15278.705146] pending: wb_workfn [15278.707144] workqueue xprtiod: flags=0xa [15278.710475] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15278.718586] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [15278.732090] workqueue ipv6_addrconf: flags=0x40008 [15278.734183] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15278.737865] pending: addrconf_verify_work [15278.739661] workqueue knbd0-recv: flags=0x1a [15278.742275] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15278.744617] in-flight: 658:recv_work [15278.747282] workqueue ptlrpc_pinger: flags=0x2 [15278.750147] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15278.753390] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15278.761244] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13348s workers=4 idle: 32984 37 8 [15278.773103] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15263s workers=2 idle: 661 [15284.056074] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [15284.061564] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15284.096172] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15284.102464] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15284.107675] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15284.110448] RIP: 0010:native_safe_halt+0x2e/0x50 [15284.112428] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15284.120283] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15284.123587] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15284.126446] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [15284.135257] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [15284.137729] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [15284.140070] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [15284.142599] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [15284.145404] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15284.147068] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15284.149585] Call Trace: [15284.151586] [15284.152519] ? show_regs.cold.9+0x22/0x2f [15284.154300] ? watchdog_timer_fn.cold.6+0x62/0x10a [15284.156431] ? __hrtimer_run_queues+0x1c2/0x4d0 [15284.159311] ? watchdog+0x40/0x40 [15284.161093] ? hrtimer_interrupt+0x128/0x2c0 [15284.163121] ? smp_apic_timer_interrupt+0xa6/0x270 [15284.165241] ? apic_timer_interrupt+0xf/0x20 [15284.166891] [15284.167588] ? native_safe_halt+0x2e/0x50 [15284.169139] kvm_wait+0xcd/0xe0 [15284.170153] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15284.174066] do_raw_spin_lock+0x177/0x190 [15284.176454] _raw_spin_lock+0x21/0x30 [15284.178706] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15284.184594] process_one_work+0x2c8/0x700 [15284.224250] worker_thread+0x296/0x6e0 [15284.226167] ? rescuer_thread+0x570/0x570 [15284.228143] kthread+0x1d1/0x200 [15284.229561] ? set_kthread_struct+0x70/0x70 [15284.233349] ret_from_fork+0x1f/0x30 [15292.056053] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [15292.057005] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [15292.061389] Modules linked in: [15292.064623] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15292.064713] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15292.064721] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15292.064723] RIP: 0010:load_balance+0xdbc/0x1090 [15292.064737] Code: 83 8c 24 98 00 00 00 02 e9 54 f8 ff ff 48 83 05 11 b1 dd 02 01 48 89 c5 4d 89 e7 48 89 74 24 30 e9 41 f4 ff ff 48 8b 44 24 20 <48> 83 05 4c b3 dd 02 01 c7 40 64 00 00 00 00 e9 06 fe ff ff 48 83 [15292.064741] RSP: 0018:ffffb2cb80d8cde0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15292.064745] RAX: ffff8ee5412ed840 RBX: ffff8ee5481a80b0 RCX: ffff8ee5821b3440 [15292.064752] RDX: 0000000000000008 RSI: 0000000000000008 RDI: 0000000000000206 [15292.064755] RBP: ffffb2cb80d8ce90 R08: ffff8ee55218a7c0 R09: 0000000000000000 [15292.064757] R10: ffff8ee5481a8098 R11: 0000000000009b55 R12: ffff8ee5481a8000 [15292.064760] R13: ffff8ee582133e08 R14: 0000000000000008 R15: ffff8ee582133380 [15292.064764] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [15292.064767] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15292.064770] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [15292.064778] Call Trace: [15292.064784] [15292.064795] ? show_regs.cold.9+0x22/0x2f [15292.064807] ? watchdog_timer_fn.cold.6+0x62/0x10a [15292.064813] ? __hrtimer_run_queues+0x1c2/0x4d0 [15292.064816] ? watchdog+0x40/0x40 [15292.064820] ? hrtimer_interrupt+0x128/0x2c0 [15292.064825] ? smp_apic_timer_interrupt+0xa6/0x270 [15292.064850] ? apic_timer_interrupt+0xf/0x20 [15292.064855] ? apic_timer_interrupt+0xa/0x20 [15292.064861] ? load_balance+0xdbc/0x1090 [15292.064868] rebalance_domains+0x348/0x580 [15292.064872] ? kvm_clock_get_cycles+0x2c/0x50 [15292.064880] run_rebalance_domains+0x68/0xa0 [15292.064884] __do_softirq+0x134/0x4e8 [15292.064890] irq_exit_rcu+0x1c6/0x1e0 [15292.064896] irq_exit+0x12/0x30 [15292.064899] smp_apic_timer_interrupt+0xe0/0x270 [15292.064905] apic_timer_interrupt+0xf/0x20 [15292.064916] [15292.064918] RIP: 0010:native_safe_halt+0x2e/0x50 [15292.064922] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15292.064926] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [15292.064934] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15292.064936] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [15292.064938] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [15292.064940] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [15292.064942] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [15292.064947] kvm_wait+0xcd/0xe0 [15292.064955] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15292.064967] do_raw_spin_lock+0x177/0x190 [15292.064971] _raw_spin_lock+0x21/0x30 [15292.064975] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15292.065370] ? do_raw_spin_unlock+0x75/0x190 [15292.065376] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15292.065597] ? do_wait_intr+0xf0/0xf0 [15292.065605] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15292.067163] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15292.067465] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15292.067474] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15292.067476] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15292.067489] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15292.067494] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15292.067498] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [15292.067501] RDX: 00000000000052ee RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [15292.067504] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [15292.067506] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [15292.067508] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15292.067512] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [15292.067514] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15292.067516] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15292.067525] Call Trace: [15292.067530] [15292.067535] ? show_regs.cold.9+0x22/0x2f [15292.067544] ? watchdog_timer_fn.cold.6+0x62/0x10a [15292.067550] ? __hrtimer_run_queues+0x1c2/0x4d0 [15292.067555] ? watchdog+0x40/0x40 [15292.067558] ? hrtimer_interrupt+0x128/0x2c0 [15292.067562] ? smp_apic_timer_interrupt+0xa6/0x270 [15292.067570] ? apic_timer_interrupt+0xf/0x20 [15292.067575] [15292.067578] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15292.067583] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15292.067588] do_raw_spin_lock+0x177/0x190 [15292.067593] _raw_spin_lock+0x21/0x30 [15292.067599] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15292.091657] mdc_close+0x53b/0xd70 [mdc] [15292.091713] lmv_close+0x18d/0x490 [lmv] [15292.091752] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15292.091976] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15292.092489] ll_md_real_close+0xa5/0x2a0 [lustre] [15292.092706] ll_md_close+0x274/0x800 [lustre] [15292.092872] ll_file_release+0x128/0xc50 [lustre] [15292.092991] __fput+0x10d/0x440 [15292.093161] ____fput+0x12/0x20 [15292.093172] task_work_run+0xc6/0x110 [15292.093189] exit_to_usermode_loop+0x1dd/0x1f0 [15292.093197] do_syscall_64+0x3d6/0x440 [15292.093203] entry_SYSCALL_64_after_hwframe+0x49/0xae [15292.093221] RIP: 0033:0x7f151bee3cd8 [15292.093231] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15292.093236] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15292.093242] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [15292.093245] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [15292.093247] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15292.093250] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15292.093253] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15292.752178] obd_mod_free+0x46/0x140 [ptlrpc] [15292.755216] mdc_commit_open+0x91/0xb0 [mdc] [15292.760383] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15292.767203] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15292.773227] after_reply+0xe2c/0x1780 [ptlrpc] [15292.778958] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15292.786772] ? __next_timer_interrupt+0x160/0x160 [15292.790498] ptlrpcd+0xa82/0xe40 [ptlrpc] [15292.797591] ? do_wait_intr+0xf0/0xf0 [15292.806200] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15292.820860] kthread+0x1d1/0x200 [15292.824033] ? set_kthread_struct+0x70/0x70 [15292.827824] ret_from_fork+0x1f/0x30 [15304.052021] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [15304.054743] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15304.078445] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15304.088939] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15304.094510] RIP: 0010:native_safe_halt+0x2e/0x50 [15304.096387] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15304.103356] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [15304.106309] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15304.108776] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [15304.110600] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [15304.113994] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [15304.116687] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [15304.119361] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [15304.122323] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15304.124460] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [15304.127049] Call Trace: [15304.128121] [15304.128991] ? show_regs.cold.9+0x22/0x2f [15304.130817] ? watchdog_timer_fn.cold.6+0x62/0x10a [15304.133392] ? __hrtimer_run_queues+0x1c2/0x4d0 [15304.137889] ? watchdog+0x40/0x40 [15304.142620] ? hrtimer_interrupt+0x128/0x2c0 [15304.145611] ? smp_apic_timer_interrupt+0xa6/0x270 [15304.150753] ? apic_timer_interrupt+0xf/0x20 [15304.153399] [15304.154317] ? native_safe_halt+0x2e/0x50 [15304.156108] kvm_wait+0xcd/0xe0 [15304.157387] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15304.159274] do_raw_spin_lock+0x177/0x190 [15304.160800] _raw_spin_lock+0x21/0x30 [15304.162702] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15304.165208] ? do_raw_spin_unlock+0x75/0x190 [15304.171390] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15304.174677] ? do_wait_intr+0xf0/0xf0 [15304.176549] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15304.179324] mdc_reint+0x2d/0x110 [mdc] [15304.181438] mdc_unlink+0x658/0x870 [mdc] [15304.183915] lmv_unlink+0x34d/0xbd0 [lmv] [15304.185485] ll_unlink+0x24d/0x7f0 [lustre] [15304.190914] vfs_unlink+0x202/0x330 [15304.193223] do_unlinkat+0x2fa/0x460 [15304.198878] __x64_sys_unlink+0x37/0x50 [15304.203192] do_syscall_64+0xc1/0x440 [15304.206559] entry_SYSCALL_64_after_hwframe+0x49/0xae [15304.212971] RIP: 0033:0x7f151bdee10b [15304.214583] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [15304.223448] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [15304.227400] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [15304.230458] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [15304.233705] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15304.236445] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15304.242166] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15304.930038] rcu: INFO: rcu_sched self-detected stall on CPU [15304.933459] rcu: 1-...!: (7965943 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [15304.940221] (t=13380283 jiffies g=249657 q=6451) [15304.945284] rcu: rcu_sched kthread starved for 13375235 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [15304.950304] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [15304.953690] rcu: RCU grace-period kthread stack dump: [15304.956092] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [15304.959598] Call Trace: [15304.960279] __schedule+0x351/0xcb0 [15304.961648] schedule+0xc0/0x180 [15304.963389] schedule_timeout+0xb4/0x190 [15304.966101] ? do_raw_spin_unlock+0x75/0x190 [15304.968195] ? __next_timer_interrupt+0x160/0x160 [15304.970557] ? prepare_to_swait_event+0xf3/0x140 [15304.972996] rcu_gp_kthread+0x8dd/0x15b0 [15304.974492] ? force_qs_rnp+0x2c0/0x2c0 [15304.977289] kthread+0x1d1/0x200 [15304.978512] ? set_kthread_struct+0x70/0x70 [15304.980100] ret_from_fork+0x1f/0x30 [15304.981683] rcu: Stack dump where RCU GP kthread last ran: [15304.984495] Sending NMI from CPU 1 to CPUs 3: [15304.987152] NMI backtrace for cpu 3 skipped: idling at native_safe_halt+0x2e/0x50 [15304.991779] NMI backtrace for cpu 1 [15304.994893] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15305.001151] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15305.005626] Call Trace: [15305.009297] [15305.010357] ? dump_stack+0xbb/0x10e [15305.012382] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [15305.016649] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [15305.019160] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [15305.022906] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [15305.028302] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [15305.031476] ? rcu_dump_cpu_stacks+0x11b/0x171 [15305.033723] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [15305.037585] ? update_process_times+0x6d/0xc0 [15305.040144] ? tick_sched_handle.isra.18+0x27/0x90 [15305.041891] ? tick_sched_timer+0x53/0xb0 [15305.044457] ? __hrtimer_run_queues+0x1c2/0x4d0 [15305.050229] ? tick_sched_handle.isra.18+0x90/0x90 [15305.054693] ? hrtimer_interrupt+0x128/0x2c0 [15305.057330] ? smp_apic_timer_interrupt+0xa6/0x270 [15305.059936] ? apic_timer_interrupt+0xf/0x20 [15305.064517] [15305.066075] ? native_safe_halt+0x2e/0x50 [15305.068345] ? kvm_wait+0xcd/0xe0 [15305.070189] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15305.073203] ? do_raw_spin_lock+0x177/0x190 [15305.075783] ? _raw_spin_lock+0x21/0x30 [15305.077499] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15305.080375] ? obd_mod_free+0x46/0x140 [ptlrpc] [15305.083119] ? mdc_commit_open+0x91/0xb0 [mdc] [15305.085455] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15305.088512] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15305.091728] ? after_reply+0xe2c/0x1780 [ptlrpc] [15305.094882] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15305.097944] ? __next_timer_interrupt+0x160/0x160 [15305.101057] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [15305.105296] ? do_wait_intr+0xf0/0xf0 [15305.107674] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15305.109686] ? kthread+0x1d1/0x200 [15305.111414] ? set_kthread_struct+0x70/0x70 [15305.113519] ? ret_from_fork+0x1f/0x30 [15309.281041] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13378s! [15309.286110] Showing busy workqueues and worker pools: [15309.288666] workqueue events: flags=0x0 [15309.291701] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15309.294853] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [15309.299193] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15309.302175] pending: kfree_rcu_monitor [15309.303891] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15309.306531] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [15309.310252] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15309.313212] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [15309.316389] workqueue events_long: flags=0x0 [15309.317996] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15309.320864] pending: lu_objects_flush [obdclass] [15309.322981] workqueue events_unbound: flags=0x2 [15309.324648] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15309.326982] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15309.331339] workqueue events_power_efficient: flags=0x80 [15309.333193] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15309.335318] pending: do_cache_clean, neigh_periodic_work, check_lifetime [15309.337771] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15309.340433] pending: neigh_periodic_work [15309.341838] workqueue mm_percpu_wq: flags=0x8 [15309.343314] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15309.345493] pending: vmstat_update, lru_add_drain_per_cpu [15309.348047] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15309.350499] pending: vmstat_update [15309.351991] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15309.354840] pending: lru_add_drain_per_cpu [15309.356730] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [15309.358926] pending: lru_add_drain_per_cpu BAR(48) [15309.361262] workqueue writeback: flags=0x4a [15309.362738] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [15309.365170] pending: wb_workfn [15309.367301] workqueue xprtiod: flags=0xa [15309.369010] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15309.371789] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [15309.375233] workqueue ipv6_addrconf: flags=0x40008 [15309.377460] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15309.380657] pending: addrconf_verify_work [15309.382376] workqueue knbd0-recv: flags=0x1a [15309.384351] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15309.386696] in-flight: 658:recv_work [15309.387927] workqueue ptlrpc_pinger: flags=0x2 [15309.389193] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15309.391216] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15309.393494] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13378s workers=4 idle: 32984 37 8 [15309.396070] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15294s workers=2 idle: 661 [15312.056219] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [15312.063059] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15312.093224] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15312.099143] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15312.104180] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15312.109004] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15312.113497] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15312.124065] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [15312.128308] RAX: 0000000000005098 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15312.132284] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [15312.136304] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [15312.138554] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [15312.145563] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [15312.152340] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [15312.157207] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15312.160719] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15312.167416] Call Trace: [15312.168403] [15312.170133] ? show_regs.cold.9+0x22/0x2f [15312.175120] ? watchdog_timer_fn.cold.6+0x62/0x10a [15312.181552] ? __hrtimer_run_queues+0x1c2/0x4d0 [15312.184425] ? watchdog+0x40/0x40 [15312.185554] ? hrtimer_interrupt+0x128/0x2c0 [15312.188734] ? smp_apic_timer_interrupt+0xa6/0x270 [15312.191372] ? apic_timer_interrupt+0xf/0x20 [15312.193463] [15312.194334] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15312.198653] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15312.203627] do_raw_spin_lock+0x177/0x190 [15312.205843] _raw_spin_lock+0x21/0x30 [15312.207171] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15312.210075] process_one_work+0x2c8/0x700 [15312.211574] worker_thread+0x296/0x6e0 [15312.214453] ? rescuer_thread+0x570/0x570 [15312.215770] kthread+0x1d1/0x200 [15312.216905] ? set_kthread_struct+0x70/0x70 [15312.220250] ret_from_fork+0x1f/0x30 [15332.055082] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [15332.057165] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [15332.059035] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15332.059118] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15332.059127] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15332.059131] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15332.059156] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15332.059161] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15332.059165] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [15332.059168] RDX: 000000000000417c RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [15332.059171] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [15332.059173] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [15332.059176] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15332.059179] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [15332.059182] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15332.059184] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15332.059193] Call Trace: [15332.059200] [15332.059207] ? show_regs.cold.9+0x22/0x2f [15332.059219] ? watchdog_timer_fn.cold.6+0x62/0x10a [15332.059228] ? __hrtimer_run_queues+0x1c2/0x4d0 [15332.059233] ? watchdog+0x40/0x40 [15332.059238] ? hrtimer_interrupt+0x128/0x2c0 [15332.059242] ? smp_apic_timer_interrupt+0xa6/0x270 [15332.059376] ? apic_timer_interrupt+0xf/0x20 [15332.059434] [15332.059437] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15332.067428] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15332.067525] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15332.067534] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15332.067537] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15332.067554] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15332.067560] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15332.067565] RAX: 0000000000007d89 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15332.067568] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [15332.067571] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [15332.095668] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15332.095693] do_raw_spin_lock+0x177/0x190 [15332.095699] _raw_spin_lock+0x21/0x30 [15332.095709] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15332.096032] obd_mod_free+0x46/0x140 [ptlrpc] [15332.096228] mdc_commit_open+0x91/0xb0 [mdc] [15332.096276] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15332.096702] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15332.096969] after_reply+0xe2c/0x1780 [ptlrpc] [15332.097193] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15332.097738] ? __next_timer_interrupt+0x160/0x160 [15332.097749] ptlrpcd+0xa82/0xe40 [ptlrpc] [15332.111670] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [15332.118628] ? do_wait_intr+0xf0/0xf0 [15332.118653] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15332.126065] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [15332.126078] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [15332.126081] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15332.126084] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [15332.126093] Call Trace: [15332.126102] [15332.126108] ? show_regs.cold.9+0x22/0x2f [15332.126124] ? watchdog_timer_fn.cold.6+0x62/0x10a [15332.126130] ? __hrtimer_run_queues+0x1c2/0x4d0 [15332.126134] ? watchdog+0x40/0x40 [15332.126139] ? hrtimer_interrupt+0x128/0x2c0 [15332.126143] ? smp_apic_timer_interrupt+0xa6/0x270 [15332.126152] ? apic_timer_interrupt+0xf/0x20 [15332.126157] [15332.126160] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15332.126173] do_raw_spin_lock+0x177/0x190 [15332.126178] _raw_spin_lock+0x21/0x30 [15332.126183] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15332.138421] kthread+0x1d1/0x200 [15332.138442] ? set_kthread_struct+0x70/0x70 [15332.138458] ret_from_fork+0x1f/0x30 [15332.510521] ? do_raw_spin_unlock+0x75/0x190 [15332.514707] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15332.522156] ? do_wait_intr+0xf0/0xf0 [15332.526668] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15332.533513] mdc_close+0x53b/0xd70 [mdc] [15332.537744] lmv_close+0x18d/0x490 [lmv] [15332.541580] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15332.550690] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15332.559156] ll_md_real_close+0xa5/0x2a0 [lustre] [15332.564889] ll_md_close+0x274/0x800 [lustre] [15332.572086] ll_file_release+0x128/0xc50 [lustre] [15332.575491] __fput+0x10d/0x440 [15332.579101] ____fput+0x12/0x20 [15332.581939] task_work_run+0xc6/0x110 [15332.585699] exit_to_usermode_loop+0x1dd/0x1f0 [15332.589873] do_syscall_64+0x3d6/0x440 [15332.594106] entry_SYSCALL_64_after_hwframe+0x49/0xae [15332.596984] RIP: 0033:0x7f151bee3cd8 [15332.600085] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15332.616652] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15332.625781] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [15332.630935] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [15332.637624] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15332.644501] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15332.653155] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15336.052615] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [15336.059867] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15336.104795] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15336.115221] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15336.118631] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15336.120763] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15336.140418] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15336.145092] RAX: 00000000000077da RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15336.149348] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [15336.154949] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [15336.166610] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [15336.173234] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [15336.180284] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [15336.186093] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15336.190837] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [15336.199283] Call Trace: [15336.200813] [15336.201367] ? show_regs.cold.9+0x22/0x2f [15336.204920] ? watchdog_timer_fn.cold.6+0x62/0x10a [15336.207732] ? __hrtimer_run_queues+0x1c2/0x4d0 [15336.209735] ? watchdog+0x40/0x40 [15336.211252] ? hrtimer_interrupt+0x128/0x2c0 [15336.213933] ? smp_apic_timer_interrupt+0xa6/0x270 [15336.216229] ? apic_timer_interrupt+0xf/0x20 [15336.217471] [15336.218185] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15336.220223] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15336.222065] do_raw_spin_lock+0x177/0x190 [15336.223261] _raw_spin_lock+0x21/0x30 [15336.225054] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15336.231382] ? do_raw_spin_unlock+0x75/0x190 [15336.233015] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15336.235474] ? do_wait_intr+0xf0/0xf0 [15336.236999] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15336.239780] mdc_reint+0x2d/0x110 [mdc] [15336.241780] mdc_unlink+0x658/0x870 [mdc] [15336.243572] lmv_unlink+0x34d/0xbd0 [lmv] [15336.245058] ll_unlink+0x24d/0x7f0 [lustre] [15336.246794] vfs_unlink+0x202/0x330 [15336.248104] do_unlinkat+0x2fa/0x460 [15336.249560] __x64_sys_unlink+0x37/0x50 [15336.252148] do_syscall_64+0xc1/0x440 [15336.253733] entry_SYSCALL_64_after_hwframe+0x49/0xae [15336.255843] RIP: 0033:0x7f151bdee10b [15336.257699] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [15336.267487] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [15336.270644] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [15336.274135] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [15336.278985] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15336.281842] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15336.284809] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15339.498353] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13408s! [15339.505413] Showing busy workqueues and worker pools: [15339.507672] workqueue events: flags=0x0 [15339.509096] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15339.511632] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [15339.515726] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15339.518844] pending: kfree_rcu_monitor [15339.520874] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15339.524489] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [15339.529609] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15339.532915] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [15339.536416] workqueue events_long: flags=0x0 [15339.537994] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15339.540991] pending: lu_objects_flush [obdclass] [15339.544162] workqueue events_unbound: flags=0x2 [15339.545847] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15339.548551] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15339.553114] workqueue events_power_efficient: flags=0x80 [15339.556710] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15339.560526] pending: do_cache_clean, neigh_periodic_work, check_lifetime [15339.564115] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15339.567510] pending: neigh_periodic_work [15339.569652] workqueue mm_percpu_wq: flags=0x8 [15339.572814] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15339.577086] pending: vmstat_update, lru_add_drain_per_cpu [15339.580018] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15339.583463] pending: vmstat_update [15339.585293] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15339.588439] pending: lru_add_drain_per_cpu [15339.590809] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [15339.594066] pending: lru_add_drain_per_cpu BAR(48) [15339.596815] workqueue writeback: flags=0x4a [15339.598884] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [15339.601931] pending: wb_workfn [15339.603767] workqueue xprtiod: flags=0xa [15339.605547] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15339.608810] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [15339.612798] workqueue ipv6_addrconf: flags=0x40008 [15339.615600] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15339.618839] pending: addrconf_verify_work [15339.620325] workqueue knbd0-recv: flags=0x1a [15339.622372] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15339.625699] in-flight: 658:recv_work [15339.627104] workqueue ptlrpc_pinger: flags=0x2 [15339.629193] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15339.632084] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15339.635432] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13409s workers=4 idle: 32984 37 8 [15339.639791] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15324s workers=2 idle: 661 [15340.056066] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [15340.064160] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15340.118096] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15340.133954] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15340.146070] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15340.154435] RIP: 0010:native_safe_halt+0x2e/0x50 [15340.156833] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15340.169559] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15340.172643] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15340.176102] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [15340.181975] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [15340.185557] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [15340.189022] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [15340.193032] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [15340.196906] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15340.201884] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15340.213596] Call Trace: [15340.217960] [15340.219022] ? show_regs.cold.9+0x22/0x2f [15340.223449] ? watchdog_timer_fn.cold.6+0x62/0x10a [15340.228808] ? __hrtimer_run_queues+0x1c2/0x4d0 [15340.233405] ? watchdog+0x40/0x40 [15340.235828] ? hrtimer_interrupt+0x128/0x2c0 [15340.237492] ? smp_apic_timer_interrupt+0xa6/0x270 [15340.241819] ? apic_timer_interrupt+0xf/0x20 [15340.245981] [15340.246933] ? native_safe_halt+0x2e/0x50 [15340.255199] kvm_wait+0xcd/0xe0 [15340.257437] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15340.260552] do_raw_spin_lock+0x177/0x190 [15340.262070] _raw_spin_lock+0x21/0x30 [15340.264758] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15340.267816] process_one_work+0x2c8/0x700 [15340.269602] worker_thread+0x296/0x6e0 [15340.271797] ? rescuer_thread+0x570/0x570 [15340.273599] kthread+0x1d1/0x200 [15340.275039] ? set_kthread_struct+0x70/0x70 [15340.277011] ret_from_fork+0x1f/0x30 [15360.055038] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [15360.057028] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [15360.063210] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15360.063428] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15360.063435] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15360.063438] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15360.063454] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15360.063459] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15360.063464] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [15360.063466] RDX: 00000000000009c4 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [15360.063469] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [15360.063472] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [15360.063474] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15360.063477] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [15360.063480] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15360.063482] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15360.063491] Call Trace: [15360.063497] [15360.063503] ? show_regs.cold.9+0x22/0x2f [15360.063513] ? watchdog_timer_fn.cold.6+0x62/0x10a [15360.063520] ? __hrtimer_run_queues+0x1c2/0x4d0 [15360.068920] Modules linked in: [15360.108759] ? watchdog+0x40/0x40 [15360.108783] ? hrtimer_interrupt+0x128/0x2c0 [15360.108790] ? smp_apic_timer_interrupt+0xa6/0x270 [15360.108801] ? apic_timer_interrupt+0xf/0x20 [15360.108806] [15360.108811] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15360.108819] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15360.108823] do_raw_spin_lock+0x177/0x190 [15360.108829] _raw_spin_lock+0x21/0x30 [15360.108837] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15360.123429] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15360.123507] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15360.123515] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15360.123518] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15360.123532] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15360.123548] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15360.123553] RAX: 0000000000003d19 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15360.123555] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [15360.123557] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [15360.123559] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [15360.123562] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [15360.123566] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [15360.123569] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15360.123571] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [15360.123579] Call Trace: [15360.123584] [15360.123590] ? show_regs.cold.9+0x22/0x2f [15360.123947] ? watchdog_timer_fn.cold.6+0x62/0x10a [15360.123955] ? __hrtimer_run_queues+0x1c2/0x4d0 [15360.123959] ? watchdog+0x40/0x40 [15360.123964] ? hrtimer_interrupt+0x128/0x2c0 [15360.123967] ? smp_apic_timer_interrupt+0xa6/0x270 [15360.123975] ? apic_timer_interrupt+0xf/0x20 [15360.123979] [15360.123982] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15360.123989] do_raw_spin_lock+0x177/0x190 [15360.123995] _raw_spin_lock+0x21/0x30 [15360.124000] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15360.124302] ? do_raw_spin_unlock+0x75/0x190 [15360.124308] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15360.124512] ? do_wait_intr+0xf0/0xf0 [15360.124520] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15360.124930] mdc_close+0x53b/0xd70 [mdc] [15360.124974] lmv_close+0x18d/0x490 [lmv] [15360.124999] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15360.125346] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15360.125455] ll_md_real_close+0xa5/0x2a0 [lustre] [15360.125752] ll_md_close+0x274/0x800 [lustre] [15360.126215] ll_file_release+0x128/0xc50 [lustre] [15360.135591] obd_mod_free+0x46/0x140 [ptlrpc] [15360.136687] mdc_commit_open+0x91/0xb0 [mdc] [15360.148349] __fput+0x10d/0x440 [15360.182246] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15360.182869] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15360.183555] after_reply+0xe2c/0x1780 [ptlrpc] [15360.185628] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15360.185848] ? __next_timer_interrupt+0x160/0x160 [15360.185858] ptlrpcd+0xa82/0xe40 [ptlrpc] [15360.188385] ? do_wait_intr+0xf0/0xf0 [15360.188398] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15360.188999] kthread+0x1d1/0x200 [15360.189008] ? set_kthread_struct+0x70/0x70 [15360.189012] ret_from_fork+0x1f/0x30 [15360.529552] ____fput+0x12/0x20 [15360.533844] task_work_run+0xc6/0x110 [15360.537351] exit_to_usermode_loop+0x1dd/0x1f0 [15360.542318] do_syscall_64+0x3d6/0x440 [15360.548312] entry_SYSCALL_64_after_hwframe+0x49/0xae [15360.553327] RIP: 0033:0x7f151bee3cd8 [15360.556185] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15360.579337] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15360.584993] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [15360.589443] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [15360.592784] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15360.598987] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15360.601985] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15364.052033] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [15364.056530] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15364.077598] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15364.081966] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15364.084753] RIP: 0010:native_safe_halt+0x2e/0x50 [15364.086427] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15364.093088] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15364.096223] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15364.101813] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [15364.105921] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [15364.110507] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [15364.111577] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [15364.112592] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [15364.113971] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15364.117678] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [15364.122391] Call Trace: [15364.122926] [15364.124766] ? show_regs.cold.9+0x22/0x2f [15364.125645] ? watchdog_timer_fn.cold.6+0x62/0x10a [15364.126701] ? __hrtimer_run_queues+0x1c2/0x4d0 [15364.129096] ? watchdog+0x40/0x40 [15364.129990] ? hrtimer_interrupt+0x128/0x2c0 [15364.132269] ? smp_apic_timer_interrupt+0xa6/0x270 [15364.133831] ? apic_timer_interrupt+0xf/0x20 [15364.135751] [15364.137347] ? native_safe_halt+0x2e/0x50 [15364.138424] kvm_wait+0xcd/0xe0 [15364.139225] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15364.140768] do_raw_spin_lock+0x177/0x190 [15364.142621] _raw_spin_lock+0x21/0x30 [15364.143528] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15364.146800] ? do_raw_spin_unlock+0x75/0x190 [15364.148209] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15364.150188] ? do_wait_intr+0xf0/0xf0 [15364.151231] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15364.157414] mdc_reint+0x2d/0x110 [mdc] [15364.158561] mdc_unlink+0x658/0x870 [mdc] [15364.159614] lmv_unlink+0x34d/0xbd0 [lmv] [15364.161510] ll_unlink+0x24d/0x7f0 [lustre] [15364.165775] vfs_unlink+0x202/0x330 [15364.166589] do_unlinkat+0x2fa/0x460 [15364.167818] __x64_sys_unlink+0x37/0x50 [15364.169221] do_syscall_64+0xc1/0x440 [15364.170223] entry_SYSCALL_64_after_hwframe+0x49/0xae [15364.171922] RIP: 0033:0x7f151bdee10b [15364.173237] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [15364.184833] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [15364.188698] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [15364.190276] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [15364.198408] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15364.203034] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15364.206656] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15368.056057] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [15368.060662] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15368.102685] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15368.110093] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15368.114310] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15368.117057] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15368.123919] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15368.134794] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15368.137693] RAX: 000000000000643e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15368.140011] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [15368.142902] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [15368.145429] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [15368.149773] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [15368.152943] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [15368.157542] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15368.160507] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15368.163516] Call Trace: [15368.164400] [15368.165846] ? show_regs.cold.9+0x22/0x2f [15368.167849] ? watchdog_timer_fn.cold.6+0x62/0x10a [15368.169832] ? __hrtimer_run_queues+0x1c2/0x4d0 [15368.171519] ? watchdog+0x40/0x40 [15368.172733] ? hrtimer_interrupt+0x128/0x2c0 [15368.175844] ? smp_apic_timer_interrupt+0xa6/0x270 [15368.179951] ? apic_timer_interrupt+0xf/0x20 [15368.181491] [15368.182240] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15368.183894] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15368.185901] do_raw_spin_lock+0x177/0x190 [15368.187241] _raw_spin_lock+0x21/0x30 [15368.188680] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15368.191711] process_one_work+0x2c8/0x700 [15368.193366] worker_thread+0x296/0x6e0 [15368.194836] ? rescuer_thread+0x570/0x570 [15368.195958] kthread+0x1d1/0x200 [15368.197177] ? set_kthread_struct+0x70/0x70 [15368.198848] ret_from_fork+0x1f/0x30 [15369.697221] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13439s! [15369.700483] Showing busy workqueues and worker pools: [15369.710449] workqueue events: flags=0x0 [15369.714579] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15369.720542] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [15369.726451] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15369.730147] pending: kfree_rcu_monitor [15369.731648] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15369.737226] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [15369.742348] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15369.752490] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [15369.759889] workqueue events_long: flags=0x0 [15369.761008] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15369.771226] pending: lu_objects_flush [obdclass] [15369.773113] workqueue events_unbound: flags=0x2 [15369.779765] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15369.788495] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15369.803783] workqueue events_power_efficient: flags=0x80 [15369.812686] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15369.827570] pending: do_cache_clean, neigh_periodic_work, check_lifetime [15369.835483] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15369.843887] pending: neigh_periodic_work [15369.850839] workqueue mm_percpu_wq: flags=0x8 [15369.859493] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15369.867165] pending: vmstat_update, lru_add_drain_per_cpu [15369.869369] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15369.879773] pending: vmstat_update [15369.883803] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15369.890429] pending: lru_add_drain_per_cpu [15369.894096] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [15369.897507] pending: lru_add_drain_per_cpu BAR(48) [15369.899774] workqueue writeback: flags=0x4a [15369.901730] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [15369.904463] pending: wb_workfn [15369.906150] workqueue xprtiod: flags=0xa [15369.907538] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15369.910052] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [15369.913628] workqueue ipv6_addrconf: flags=0x40008 [15369.915667] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15369.918650] pending: addrconf_verify_work [15369.920644] workqueue knbd0-recv: flags=0x1a [15369.922925] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15369.926541] in-flight: 658:recv_work [15369.928465] workqueue ptlrpc_pinger: flags=0x2 [15369.930252] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15369.932792] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15369.936048] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13439s workers=4 idle: 32984 37 8 [15369.939894] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15354s workers=2 idle: 661 [15388.055010] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [15388.057077] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [15388.059087] Modules linked in: [15388.062703] Modules linked in: [15388.062707] lustre(O) osp(O) [15388.064443] lustre(O) [15388.070530] ofd(O) [15388.071977] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15388.072048] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15388.087333] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15388.087478] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15388.087486] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15388.087489] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15388.087503] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15388.087507] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15388.087511] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [15388.087513] RDX: 00000000000008c0 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [15388.087515] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [15388.087517] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [15388.087519] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15388.087523] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [15388.087526] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15388.087528] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15388.087538] Call Trace: [15388.087544] [15388.087550] ? show_regs.cold.9+0x22/0x2f [15388.087559] ? watchdog_timer_fn.cold.6+0x62/0x10a [15388.087565] ? __hrtimer_run_queues+0x1c2/0x4d0 [15388.087569] ? watchdog+0x40/0x40 [15388.087574] ? hrtimer_interrupt+0x128/0x2c0 [15388.087579] ? smp_apic_timer_interrupt+0xa6/0x270 [15388.087587] ? apic_timer_interrupt+0xf/0x20 [15388.087592] [15388.087595] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15388.087600] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15388.087605] do_raw_spin_lock+0x177/0x190 [15388.087609] _raw_spin_lock+0x21/0x30 [15388.087615] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15388.094104] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15388.094113] RIP: 0010:native_safe_halt+0x2e/0x50 [15388.094126] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15388.094130] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15388.094136] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15388.094138] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [15388.094141] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [15388.094143] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [15388.094145] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [15388.094148] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [15388.094151] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15388.094153] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [15388.094163] Call Trace: [15388.094169] [15388.094174] ? show_regs.cold.9+0x22/0x2f [15388.094184] ? watchdog_timer_fn.cold.6+0x62/0x10a [15388.094191] ? __hrtimer_run_queues+0x1c2/0x4d0 [15388.094196] ? watchdog+0x40/0x40 [15388.094200] ? hrtimer_interrupt+0x128/0x2c0 [15388.094204] ? smp_apic_timer_interrupt+0xa6/0x270 [15388.094209] ? apic_timer_interrupt+0xf/0x20 [15388.094213] [15388.094216] ? native_safe_halt+0x2e/0x50 [15388.094220] kvm_wait+0xcd/0xe0 [15388.094227] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15388.094236] do_raw_spin_lock+0x177/0x190 [15388.094241] _raw_spin_lock+0x21/0x30 [15388.094245] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15388.094538] ? do_raw_spin_unlock+0x75/0x190 [15388.094544] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15388.094741] ? do_wait_intr+0xf0/0xf0 [15388.094749] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15388.095087] mdc_close+0x53b/0xd70 [mdc] [15388.095127] lmv_close+0x18d/0x490 [lmv] [15388.095159] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15388.095328] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15388.176979] obd_mod_free+0x46/0x140 [ptlrpc] [15388.179734] mdc_commit_open+0x91/0xb0 [mdc] [15388.191516] ll_md_real_close+0xa5/0x2a0 [lustre] [15388.191637] ll_md_close+0x274/0x800 [lustre] [15388.191722] ll_file_release+0x128/0xc50 [lustre] [15388.191821] __fput+0x10d/0x440 [15388.191830] ____fput+0x12/0x20 [15388.191835] task_work_run+0xc6/0x110 [15388.191842] exit_to_usermode_loop+0x1dd/0x1f0 [15388.191849] do_syscall_64+0x3d6/0x440 [15388.191853] entry_SYSCALL_64_after_hwframe+0x49/0xae [15388.191862] RIP: 0033:0x7f151bee3cd8 [15388.191869] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15388.191873] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15388.191878] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [15388.191881] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [15388.191883] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15388.191885] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15388.191888] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15388.616092] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15388.617870] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15388.620721] after_reply+0xe2c/0x1780 [ptlrpc] [15388.623142] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15388.625542] ? __next_timer_interrupt+0x160/0x160 [15388.627606] ptlrpcd+0xa82/0xe40 [ptlrpc] [15388.629763] ? do_wait_intr+0xf0/0xf0 [15388.631384] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15388.634027] kthread+0x1d1/0x200 [15388.635430] ? set_kthread_struct+0x70/0x70 [15388.637308] ret_from_fork+0x1f/0x30 [15396.052014] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [15396.057649] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15396.058751] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [15396.095138] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15396.095150] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15396.095153] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15396.095166] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15396.095476] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15396.095482] RAX: 000000000000648a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15396.095484] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [15396.095486] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [15396.095488] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [15396.095489] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [15396.095492] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [15396.095495] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15396.095497] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [15396.095505] Call Trace: [15396.095511] [15396.095516] ? show_regs.cold.9+0x22/0x2f [15396.095525] ? watchdog_timer_fn.cold.6+0x62/0x10a [15396.095531] ? __hrtimer_run_queues+0x1c2/0x4d0 [15396.095535] ? watchdog+0x40/0x40 [15396.095539] ? hrtimer_interrupt+0x128/0x2c0 [15396.095542] ? smp_apic_timer_interrupt+0xa6/0x270 [15396.095550] ? apic_timer_interrupt+0xf/0x20 [15396.095554] [15396.095557] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15396.095562] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15396.095567] do_raw_spin_lock+0x177/0x190 [15396.095572] _raw_spin_lock+0x21/0x30 [15396.095580] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15396.096071] ? do_raw_spin_unlock+0x75/0x190 [15396.096077] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15396.096431] ? do_wait_intr+0xf0/0xf0 [15396.096439] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15396.096645] mdc_reint+0x2d/0x110 [mdc] [15396.096683] mdc_unlink+0x658/0x870 [mdc] [15396.097042] lmv_unlink+0x34d/0xbd0 [lmv] [15396.097071] ll_unlink+0x24d/0x7f0 [lustre] [15396.097201] vfs_unlink+0x202/0x330 [15396.097211] do_unlinkat+0x2fa/0x460 [15396.097216] __x64_sys_unlink+0x37/0x50 [15396.097220] do_syscall_64+0xc1/0x440 [15396.097228] entry_SYSCALL_64_after_hwframe+0x49/0xae [15396.097236] RIP: 0033:0x7f151bdee10b [15396.097244] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [15396.097249] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [15396.097253] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [15396.097255] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [15396.097257] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15396.097259] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15396.097261] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15396.343224] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15396.374470] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15396.381000] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15396.384637] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15396.391065] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15396.397349] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15396.418286] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15396.430418] RAX: 0000000000005cac RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15396.436458] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [15396.447227] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [15396.450818] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [15396.454680] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [15396.459269] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [15396.469122] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15396.476218] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15396.482041] Call Trace: [15396.482891] [15396.483438] ? show_regs.cold.9+0x22/0x2f [15396.486701] ? watchdog_timer_fn.cold.6+0x62/0x10a [15396.494049] ? __hrtimer_run_queues+0x1c2/0x4d0 [15396.500781] ? watchdog+0x40/0x40 [15396.502541] ? hrtimer_interrupt+0x128/0x2c0 [15396.508076] ? smp_apic_timer_interrupt+0xa6/0x270 [15396.509436] ? apic_timer_interrupt+0xf/0x20 [15396.510561] [15396.515235] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15396.522659] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15396.534511] do_raw_spin_lock+0x177/0x190 [15396.537608] _raw_spin_lock+0x21/0x30 [15396.540400] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15396.542147] process_one_work+0x2c8/0x700 [15396.545543] worker_thread+0x296/0x6e0 [15396.548419] ? rescuer_thread+0x570/0x570 [15396.556389] kthread+0x1d1/0x200 [15396.557411] ? set_kthread_struct+0x70/0x70 [15396.558741] ret_from_fork+0x1f/0x30 [15400.417035] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13469s! [15400.424239] Showing busy workqueues and worker pools: [15400.426563] workqueue events: flags=0x0 [15400.429010] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15400.431611] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [15400.437137] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15400.442442] pending: kfree_rcu_monitor [15400.444431] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15400.450605] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [15400.454359] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15400.459097] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [15400.462563] workqueue events_long: flags=0x0 [15400.464328] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15400.467680] pending: lu_objects_flush [obdclass] [15400.470278] workqueue events_unbound: flags=0x2 [15400.472459] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15400.475189] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15400.479937] workqueue events_power_efficient: flags=0x80 [15400.483528] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15400.493801] pending: do_cache_clean, neigh_periodic_work, check_lifetime [15400.497867] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15400.504457] pending: neigh_periodic_work [15400.506285] workqueue mm_percpu_wq: flags=0x8 [15400.511494] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15400.518639] pending: vmstat_update, lru_add_drain_per_cpu [15400.525475] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15400.529527] pending: vmstat_update [15400.532181] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15400.536239] pending: lru_add_drain_per_cpu [15400.539307] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [15400.546293] pending: lru_add_drain_per_cpu BAR(48) [15400.549471] workqueue writeback: flags=0x4a [15400.557575] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [15400.568373] pending: wb_workfn [15400.571251] workqueue xprtiod: flags=0xa [15400.575531] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15400.586590] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [15400.605443] workqueue ipv6_addrconf: flags=0x40008 [15400.607149] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15400.615548] pending: addrconf_verify_work [15400.625661] workqueue knbd0-recv: flags=0x1a [15400.629313] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15400.631533] in-flight: 658:recv_work [15400.640896] workqueue ptlrpc_pinger: flags=0x2 [15400.645604] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15400.654028] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15400.660094] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13470s workers=4 idle: 32984 37 8 [15400.668516] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15385s workers=2 idle: 661 [15416.055033] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [15416.057163] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [15416.062602] Modules linked in: [15416.066415] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) [15416.069260] lustre(O) osp(O) ofd(O) lod(O) [15416.072689] mdd(O) mgs(O) [15416.075552] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15416.075623] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15416.075630] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15416.075644] RIP: 0010:native_safe_halt+0x2e/0x50 [15416.075657] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15416.075662] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15416.075666] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [15416.075668] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [15416.075671] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [15416.075673] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [15416.075676] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15416.075679] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [15416.075682] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15416.075684] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15416.075692] Call Trace: [15416.075699] [15416.075705] ? show_regs.cold.9+0x22/0x2f [15416.075717] ? watchdog_timer_fn.cold.6+0x62/0x10a [15416.075723] ? __hrtimer_run_queues+0x1c2/0x4d0 [15416.075727] ? watchdog+0x40/0x40 [15416.075731] ? hrtimer_interrupt+0x128/0x2c0 [15416.075734] ? smp_apic_timer_interrupt+0xa6/0x270 [15416.075741] ? apic_timer_interrupt+0xf/0x20 [15416.075746] [15416.075749] ? native_safe_halt+0x2e/0x50 [15416.075752] kvm_wait+0xcd/0xe0 [15416.075760] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15416.075768] do_raw_spin_lock+0x177/0x190 [15416.075772] _raw_spin_lock+0x21/0x30 [15416.075778] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15416.077138] lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr [15416.114856] obd_mod_free+0x46/0x140 [ptlrpc] [15416.130865] squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15416.130898] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15416.130906] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15416.130909] RIP: 0010:native_safe_halt+0x2e/0x50 [15416.130922] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15416.130927] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15416.130931] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15416.130934] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [15416.130936] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [15416.130938] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [15416.130940] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [15416.130943] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [15416.130946] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15416.130948] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [15416.130957] Call Trace: [15416.130963] [15416.130968] ? show_regs.cold.9+0x22/0x2f [15416.130978] ? watchdog_timer_fn.cold.6+0x62/0x10a [15416.130984] ? __hrtimer_run_queues+0x1c2/0x4d0 [15416.130988] ? watchdog+0x40/0x40 [15416.130992] ? hrtimer_interrupt+0x128/0x2c0 [15416.130996] ? smp_apic_timer_interrupt+0xa6/0x270 [15416.131003] ? apic_timer_interrupt+0xf/0x20 [15416.131007] [15416.131011] ? native_safe_halt+0x2e/0x50 [15416.131014] kvm_wait+0xcd/0xe0 [15416.131022] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15416.131732] do_raw_spin_lock+0x177/0x190 [15416.136771] mdc_commit_open+0x91/0xb0 [mdc] [15416.139009] _raw_spin_lock+0x21/0x30 [15416.139032] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15416.139366] ? do_raw_spin_unlock+0x75/0x190 [15416.139375] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15416.165675] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15416.165987] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15416.166187] after_reply+0xe2c/0x1780 [ptlrpc] [15416.166382] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15416.186969] ? do_wait_intr+0xf0/0xf0 [15416.186994] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15416.187284] mdc_close+0x53b/0xd70 [mdc] [15416.187329] lmv_close+0x18d/0x490 [lmv] [15416.187357] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15416.187524] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15416.187650] ll_md_real_close+0xa5/0x2a0 [lustre] [15416.187730] ll_md_close+0x274/0x800 [lustre] [15416.187815] ll_file_release+0x128/0xc50 [lustre] [15416.187896] __fput+0x10d/0x440 [15416.187906] ____fput+0x12/0x20 [15416.187909] task_work_run+0xc6/0x110 [15416.187916] exit_to_usermode_loop+0x1dd/0x1f0 [15416.187923] do_syscall_64+0x3d6/0x440 [15416.187927] entry_SYSCALL_64_after_hwframe+0x49/0xae [15416.187935] RIP: 0033:0x7f151bee3cd8 [15416.187943] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15416.187947] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15416.187952] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [15416.187955] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [15416.187957] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15416.187960] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15416.187962] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15416.596109] ? __next_timer_interrupt+0x160/0x160 [15416.605065] ptlrpcd+0xa82/0xe40 [ptlrpc] [15416.607518] ? do_wait_intr+0xf0/0xf0 [15416.608855] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15416.619344] kthread+0x1d1/0x200 [15416.620429] ? set_kthread_struct+0x70/0x70 [15416.628648] ret_from_fork+0x1f/0x30 [15424.059291] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [15424.075488] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15424.126359] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15424.132425] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15424.137323] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15424.141186] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15424.143976] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15424.164354] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15424.169336] RAX: 000000000000758a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15424.173166] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [15424.177015] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [15424.182236] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [15424.189255] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [15424.193246] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [15424.198543] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15424.201921] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15424.205470] Call Trace: [15424.206767] [15424.207679] ? show_regs.cold.9+0x22/0x2f [15424.209879] ? watchdog_timer_fn.cold.6+0x62/0x10a [15424.212134] ? __hrtimer_run_queues+0x1c2/0x4d0 [15424.213924] ? watchdog+0x40/0x40 [15424.215616] ? hrtimer_interrupt+0x128/0x2c0 [15424.217630] ? smp_apic_timer_interrupt+0xa6/0x270 [15424.219686] ? apic_timer_interrupt+0xf/0x20 [15424.221300] [15424.222307] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15424.224299] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15424.226903] do_raw_spin_lock+0x177/0x190 [15424.228247] _raw_spin_lock+0x21/0x30 [15424.229963] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15424.232493] process_one_work+0x2c8/0x700 [15424.238571] worker_thread+0x296/0x6e0 [15424.244057] ? rescuer_thread+0x570/0x570 [15424.248376] kthread+0x1d1/0x200 [15424.250327] ? set_kthread_struct+0x70/0x70 [15424.253066] ret_from_fork+0x1f/0x30 [15428.052048] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [15428.057894] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15428.080580] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15428.086481] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15428.091192] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15428.094289] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15428.104496] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15428.109636] RAX: 0000000000000723 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15428.112947] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [15428.116612] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [15428.119952] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [15428.123785] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [15428.127648] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [15428.131707] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15428.135041] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [15428.138054] Call Trace: [15428.139159] [15428.140474] ? show_regs.cold.9+0x22/0x2f [15428.142546] ? watchdog_timer_fn.cold.6+0x62/0x10a [15428.144993] ? __hrtimer_run_queues+0x1c2/0x4d0 [15428.147537] ? watchdog+0x40/0x40 [15428.149108] ? hrtimer_interrupt+0x128/0x2c0 [15428.152852] ? smp_apic_timer_interrupt+0xa6/0x270 [15428.156477] ? apic_timer_interrupt+0xf/0x20 [15428.158938] [15428.160563] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15428.164329] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15428.168362] do_raw_spin_lock+0x177/0x190 [15428.169731] _raw_spin_lock+0x21/0x30 [15428.172112] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15428.174938] ? do_raw_spin_unlock+0x75/0x190 [15428.176726] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15428.179371] ? do_wait_intr+0xf0/0xf0 [15428.181131] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15428.183741] mdc_reint+0x2d/0x110 [mdc] [15428.186082] mdc_unlink+0x658/0x870 [mdc] [15428.188092] lmv_unlink+0x34d/0xbd0 [lmv] [15428.189809] ll_unlink+0x24d/0x7f0 [lustre] [15428.191739] vfs_unlink+0x202/0x330 [15428.193378] do_unlinkat+0x2fa/0x460 [15428.195092] __x64_sys_unlink+0x37/0x50 [15428.196850] do_syscall_64+0xc1/0x440 [15428.198215] entry_SYSCALL_64_after_hwframe+0x49/0xae [15428.199748] RIP: 0033:0x7f151bdee10b [15428.201224] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [15428.210177] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [15428.213638] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [15428.215998] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [15428.219402] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15428.223251] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15428.227280] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15431.140138] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13500s! [15431.147494] Showing busy workqueues and worker pools: [15431.149747] workqueue events: flags=0x0 [15431.152019] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15431.155828] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [15431.159832] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15431.163685] pending: kfree_rcu_monitor [15431.165682] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15431.169883] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [15431.174994] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15431.177897] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [15431.181977] workqueue events_long: flags=0x0 [15431.184210] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15431.187941] pending: lu_objects_flush [obdclass] [15431.190703] workqueue events_unbound: flags=0x2 [15431.192878] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15431.195898] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15431.201804] workqueue events_power_efficient: flags=0x80 [15431.204383] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15431.207924] pending: do_cache_clean, neigh_periodic_work, check_lifetime [15431.211108] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15431.214483] pending: neigh_periodic_work [15431.216723] workqueue mm_percpu_wq: flags=0x8 [15431.219578] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15431.223058] pending: vmstat_update, lru_add_drain_per_cpu [15431.225838] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15431.229413] pending: vmstat_update [15431.234338] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15431.242693] pending: lru_add_drain_per_cpu [15431.246367] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [15431.251550] pending: lru_add_drain_per_cpu BAR(48) [15431.254046] workqueue writeback: flags=0x4a [15431.255836] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [15431.259060] pending: wb_workfn [15431.261174] workqueue xprtiod: flags=0xa [15431.263066] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15431.266061] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [15431.270266] workqueue ipv6_addrconf: flags=0x40008 [15431.273054] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15431.277173] pending: addrconf_verify_work [15431.279801] workqueue knbd0-recv: flags=0x1a [15431.282203] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15431.286504] in-flight: 658:recv_work [15431.288616] workqueue ptlrpc_pinger: flags=0x2 [15431.290736] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15431.293978] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15431.297022] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13500s workers=4 idle: 32984 37 8 [15431.301873] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15416s workers=2 idle: 661 [15444.055049] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [15444.057012] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [15444.059772] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15444.059850] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15444.059857] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15444.059860] RIP: 0010:native_safe_halt+0x2e/0x50 [15444.059965] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15444.059973] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15444.059978] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [15444.059980] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [15444.059983] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [15444.059985] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [15444.059987] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15444.059990] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [15444.059993] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15444.059995] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15444.060005] Call Trace: [15444.060012] [15444.060019] ? show_regs.cold.9+0x22/0x2f [15444.060030] ? watchdog_timer_fn.cold.6+0x62/0x10a [15444.060037] ? __hrtimer_run_queues+0x1c2/0x4d0 [15444.060042] ? watchdog+0x40/0x40 [15444.060046] ? hrtimer_interrupt+0x128/0x2c0 [15444.060052] ? smp_apic_timer_interrupt+0xa6/0x270 [15444.060131] ? apic_timer_interrupt+0xf/0x20 [15444.060137] [15444.060140] ? native_safe_halt+0x2e/0x50 [15444.060143] kvm_wait+0xcd/0xe0 [15444.060154] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15444.060161] do_raw_spin_lock+0x177/0x190 [15444.060485] _raw_spin_lock+0x21/0x30 [15444.060494] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15444.061316] obd_mod_free+0x46/0x140 [ptlrpc] [15444.061644] mdc_commit_open+0x91/0xb0 [mdc] [15444.061683] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15444.061899] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15444.062179] after_reply+0xe2c/0x1780 [ptlrpc] [15444.062459] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15444.062675] ? __next_timer_interrupt+0x160/0x160 [15444.062684] ptlrpcd+0xa82/0xe40 [ptlrpc] [15444.066152] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15444.066212] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15444.066218] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15444.066220] RIP: 0010:native_safe_halt+0x2e/0x50 [15444.066237] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15444.066241] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15444.066244] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15444.066246] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [15444.066247] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [15444.066249] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [15444.066250] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [15444.066253] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [15444.066254] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15444.066256] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [15444.066262] Call Trace: [15444.066266] [15444.066270] ? show_regs.cold.9+0x22/0x2f [15444.066277] ? watchdog_timer_fn.cold.6+0x62/0x10a [15444.066281] ? __hrtimer_run_queues+0x1c2/0x4d0 [15444.066284] ? watchdog+0x40/0x40 [15444.066287] ? hrtimer_interrupt+0x128/0x2c0 [15444.066289] ? smp_apic_timer_interrupt+0xa6/0x270 [15444.066294] ? apic_timer_interrupt+0xf/0x20 [15444.066297] [15444.066299] ? native_safe_halt+0x2e/0x50 [15444.066301] kvm_wait+0xcd/0xe0 [15444.066306] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15444.066312] do_raw_spin_lock+0x177/0x190 [15444.066315] _raw_spin_lock+0x21/0x30 [15444.066320] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15444.066745] ? do_raw_spin_unlock+0x75/0x190 [15444.066751] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15444.086588] ? do_wait_intr+0xf0/0xf0 [15444.086609] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15444.086924] kthread+0x1d1/0x200 [15444.086929] ? set_kthread_struct+0x70/0x70 [15444.086931] ret_from_fork+0x1f/0x30 [15444.272984] ? do_wait_intr+0xf0/0xf0 [15444.275264] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15444.277940] mdc_close+0x53b/0xd70 [mdc] [15444.279451] lmv_close+0x18d/0x490 [lmv] [15444.281624] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15444.284433] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15444.286566] ll_md_real_close+0xa5/0x2a0 [lustre] [15444.288475] ll_md_close+0x274/0x800 [lustre] [15444.290602] ll_file_release+0x128/0xc50 [lustre] [15444.294272] __fput+0x10d/0x440 [15444.295340] ____fput+0x12/0x20 [15444.296523] task_work_run+0xc6/0x110 [15444.297889] exit_to_usermode_loop+0x1dd/0x1f0 [15444.299513] do_syscall_64+0x3d6/0x440 [15444.300812] entry_SYSCALL_64_after_hwframe+0x49/0xae [15444.302711] RIP: 0033:0x7f151bee3cd8 [15444.304154] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15444.311512] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15444.314319] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [15444.317082] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [15444.318960] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15444.321588] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15444.323512] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15452.059971] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [15452.070725] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15452.113260] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15452.121974] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15452.128008] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15452.138178] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15452.140554] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15452.151592] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15452.155968] RAX: 0000000000005d87 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15452.159677] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [15452.165348] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [15452.180584] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [15452.185877] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [15452.193219] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [15452.197500] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15452.200446] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15452.204988] Call Trace: [15452.206156] [15452.207006] ? show_regs.cold.9+0x22/0x2f [15452.209634] ? watchdog_timer_fn.cold.6+0x62/0x10a [15452.212780] ? __hrtimer_run_queues+0x1c2/0x4d0 [15452.221831] ? watchdog+0x40/0x40 [15452.223528] ? hrtimer_interrupt+0x128/0x2c0 [15452.226616] ? smp_apic_timer_interrupt+0xa6/0x270 [15452.229006] ? apic_timer_interrupt+0xf/0x20 [15452.231066] [15452.234853] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15452.244944] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15452.248635] do_raw_spin_lock+0x177/0x190 [15452.250558] _raw_spin_lock+0x21/0x30 [15452.252709] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15452.255481] process_one_work+0x2c8/0x700 [15452.257466] worker_thread+0x296/0x6e0 [15452.259655] ? rescuer_thread+0x570/0x570 [15452.261427] kthread+0x1d1/0x200 [15452.262972] ? set_kthread_struct+0x70/0x70 [15452.265488] ret_from_fork+0x1f/0x30 [15456.052058] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [15456.055825] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15456.077459] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15456.083539] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15456.088360] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15456.092188] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15456.102382] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15456.106320] RAX: 0000000000003e89 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15456.109674] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [15456.113579] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [15456.116765] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [15456.119961] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [15456.122796] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [15456.127125] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15456.130223] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [15456.133490] Call Trace: [15456.134580] [15456.135622] ? show_regs.cold.9+0x22/0x2f [15456.138056] ? watchdog_timer_fn.cold.6+0x62/0x10a [15456.140704] ? __hrtimer_run_queues+0x1c2/0x4d0 [15456.142699] ? watchdog+0x40/0x40 [15456.144591] ? hrtimer_interrupt+0x128/0x2c0 [15456.146679] ? smp_apic_timer_interrupt+0xa6/0x270 [15456.148551] ? apic_timer_interrupt+0xf/0x20 [15456.150634] [15456.151653] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15456.154557] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15456.157473] do_raw_spin_lock+0x177/0x190 [15456.159595] _raw_spin_lock+0x21/0x30 [15456.161132] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15456.163964] ? do_raw_spin_unlock+0x75/0x190 [15456.165974] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15456.169352] ? do_wait_intr+0xf0/0xf0 [15456.170882] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15456.173602] mdc_reint+0x2d/0x110 [mdc] [15456.175650] mdc_unlink+0x658/0x870 [mdc] [15456.177866] lmv_unlink+0x34d/0xbd0 [lmv] [15456.180162] ll_unlink+0x24d/0x7f0 [lustre] [15456.182155] vfs_unlink+0x202/0x330 [15456.183376] do_unlinkat+0x2fa/0x460 [15456.185202] __x64_sys_unlink+0x37/0x50 [15456.186571] do_syscall_64+0xc1/0x440 [15456.188323] entry_SYSCALL_64_after_hwframe+0x49/0xae [15456.191580] RIP: 0033:0x7f151bdee10b [15456.193026] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [15456.201307] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [15456.205491] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [15456.208725] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [15456.212075] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15456.215769] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15456.219494] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15461.361045] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13530s! [15461.364032] Showing busy workqueues and worker pools: [15461.366506] workqueue events: flags=0x0 [15461.368011] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15461.371338] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [15461.375591] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15461.378925] pending: kfree_rcu_monitor [15461.381017] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15461.384011] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [15461.388559] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15461.392239] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [15461.395896] workqueue events_long: flags=0x0 [15461.398121] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15461.401956] pending: lu_objects_flush [obdclass] [15461.404140] workqueue events_unbound: flags=0x2 [15461.406265] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15461.409571] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15461.415332] workqueue events_power_efficient: flags=0x80 [15461.418619] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15461.422051] pending: do_cache_clean, neigh_periodic_work, check_lifetime [15461.425745] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15461.428760] pending: neigh_periodic_work [15461.430938] workqueue mm_percpu_wq: flags=0x8 [15461.432632] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15461.435914] pending: vmstat_update, lru_add_drain_per_cpu [15461.438831] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15461.442344] pending: vmstat_update [15461.444159] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15461.447535] pending: lru_add_drain_per_cpu [15461.449804] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [15461.453129] pending: lru_add_drain_per_cpu BAR(48) [15461.455805] workqueue writeback: flags=0x4a [15461.457622] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [15461.459978] pending: wb_workfn [15461.461706] workqueue xprtiod: flags=0xa [15461.463446] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15461.465985] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [15461.469736] workqueue ipv6_addrconf: flags=0x40008 [15461.471678] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15461.474910] pending: addrconf_verify_work [15461.477011] workqueue knbd0-recv: flags=0x1a [15461.478981] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15461.481757] in-flight: 658:recv_work [15461.483902] workqueue ptlrpc_pinger: flags=0x2 [15461.486047] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15461.488984] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15461.491871] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13530s workers=4 idle: 32984 37 8 [15461.496011] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15446s workers=2 idle: 661 [15472.055041] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [15472.057306] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [15472.060042] Modules linked in: [15472.064042] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15472.064139] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15472.064149] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15472.064152] RIP: 0010:native_safe_halt+0x2e/0x50 [15472.064166] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15472.064169] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [15472.064173] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15472.064179] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [15472.064182] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [15472.064187] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [15472.067242] lustre(O) [15472.099723] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [15472.105059] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15472.105154] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15472.105162] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15472.105165] RIP: 0010:native_safe_halt+0x2e/0x50 [15472.105178] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15472.105183] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15472.105187] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [15472.105190] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [15472.105193] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [15472.112113] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [15472.112126] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15472.112129] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [15472.112139] Call Trace: [15472.112149] [15472.112156] ? show_regs.cold.9+0x22/0x2f [15472.112172] ? watchdog_timer_fn.cold.6+0x62/0x10a [15472.112179] ? __hrtimer_run_queues+0x1c2/0x4d0 [15472.112183] ? watchdog+0x40/0x40 [15472.112187] ? hrtimer_interrupt+0x128/0x2c0 [15472.112191] ? smp_apic_timer_interrupt+0xa6/0x270 [15472.112211] ? apic_timer_interrupt+0xf/0x20 [15472.112215] [15472.112218] ? native_safe_halt+0x2e/0x50 [15472.112223] kvm_wait+0xcd/0xe0 [15472.112231] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15472.112241] do_raw_spin_lock+0x177/0x190 [15472.112245] _raw_spin_lock+0x21/0x30 [15472.112251] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15472.115783] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [15472.115793] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15472.115796] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [15472.115799] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15472.115802] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15472.115810] Call Trace: [15472.115816] [15472.115821] ? show_regs.cold.9+0x22/0x2f [15472.115836] ? watchdog_timer_fn.cold.6+0x62/0x10a [15472.115843] ? __hrtimer_run_queues+0x1c2/0x4d0 [15472.115847] ? watchdog+0x40/0x40 [15472.115961] ? hrtimer_interrupt+0x128/0x2c0 [15472.115967] ? smp_apic_timer_interrupt+0xa6/0x270 [15472.115976] ? apic_timer_interrupt+0xf/0x20 [15472.115980] [15472.115983] ? native_safe_halt+0x2e/0x50 [15472.115986] kvm_wait+0xcd/0xe0 [15472.115994] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15472.116002] do_raw_spin_lock+0x177/0x190 [15472.116007] _raw_spin_lock+0x21/0x30 [15472.116014] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15472.116311] obd_mod_free+0x46/0x140 [ptlrpc] [15472.116509] mdc_commit_open+0x91/0xb0 [mdc] [15472.116546] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15472.116743] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15472.118491] after_reply+0xe2c/0x1780 [ptlrpc] [15472.118689] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15472.118900] ? __next_timer_interrupt+0x160/0x160 [15472.118910] ptlrpcd+0xa82/0xe40 [ptlrpc] [15472.119190] ? do_wait_intr+0xf0/0xf0 [15472.119201] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15472.119405] kthread+0x1d1/0x200 [15472.119411] ? set_kthread_struct+0x70/0x70 [15472.119415] ret_from_fork+0x1f/0x30 [15472.483037] ? do_raw_spin_unlock+0x75/0x190 [15472.487888] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15472.495783] ? do_wait_intr+0xf0/0xf0 [15472.503268] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15472.510475] mdc_close+0x53b/0xd70 [mdc] [15472.513474] lmv_close+0x18d/0x490 [lmv] [15472.514917] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15472.521507] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15472.528395] ll_md_real_close+0xa5/0x2a0 [lustre] [15472.532281] ll_md_close+0x274/0x800 [lustre] [15472.534619] ll_file_release+0x128/0xc50 [lustre] [15472.538663] __fput+0x10d/0x440 [15472.544239] ____fput+0x12/0x20 [15472.546267] task_work_run+0xc6/0x110 [15472.549513] exit_to_usermode_loop+0x1dd/0x1f0 [15472.555354] do_syscall_64+0x3d6/0x440 [15472.556812] entry_SYSCALL_64_after_hwframe+0x49/0xae [15472.562395] RIP: 0033:0x7f151bee3cd8 [15472.564317] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15472.586113] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15472.590248] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [15472.593569] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [15472.596556] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15472.600478] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15472.604935] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15480.056072] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [15480.059100] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15480.105376] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15480.111452] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15480.117606] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15480.120573] RIP: 0010:native_safe_halt+0x2e/0x50 [15480.122575] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15480.130524] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15480.133742] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15480.136659] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [15480.139527] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [15480.143325] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [15480.149120] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [15480.158500] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [15480.165289] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15480.174527] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15480.178556] Call Trace: [15480.184033] [15480.184739] ? show_regs.cold.9+0x22/0x2f [15480.187800] ? watchdog_timer_fn.cold.6+0x62/0x10a [15480.195335] ? __hrtimer_run_queues+0x1c2/0x4d0 [15480.198835] ? watchdog+0x40/0x40 [15480.201045] ? hrtimer_interrupt+0x128/0x2c0 [15480.207434] ? smp_apic_timer_interrupt+0xa6/0x270 [15480.212568] ? apic_timer_interrupt+0xf/0x20 [15480.225533] [15480.228037] ? native_safe_halt+0x2e/0x50 [15480.234017] kvm_wait+0xcd/0xe0 [15480.237240] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15480.244741] do_raw_spin_lock+0x177/0x190 [15480.247563] _raw_spin_lock+0x21/0x30 [15480.250572] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15480.258764] process_one_work+0x2c8/0x700 [15480.264298] worker_thread+0x296/0x6e0 [15480.265658] ? rescuer_thread+0x570/0x570 [15480.266920] kthread+0x1d1/0x200 [15480.269201] ? set_kthread_struct+0x70/0x70 [15480.274809] ret_from_fork+0x1f/0x30 [15484.933103] rcu: INFO: rcu_sched self-detected stall on CPU [15484.935416] rcu: 1-...!: (8085419 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [15484.941383] (t=13560286 jiffies g=249657 q=6494) [15484.950172] rcu: rcu_sched kthread starved for 13555238 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [15484.955719] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [15484.962960] rcu: RCU grace-period kthread stack dump: [15484.964754] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [15484.979109] Call Trace: [15484.981241] __schedule+0x351/0xcb0 [15484.986322] schedule+0xc0/0x180 [15484.987814] schedule_timeout+0xb4/0x190 [15484.990543] ? do_raw_spin_unlock+0x75/0x190 [15484.994730] ? __next_timer_interrupt+0x160/0x160 [15485.000505] ? prepare_to_swait_event+0xf3/0x140 [15485.004797] rcu_gp_kthread+0x8dd/0x15b0 [15485.007991] ? force_qs_rnp+0x2c0/0x2c0 [15485.009677] kthread+0x1d1/0x200 [15485.010710] ? set_kthread_struct+0x70/0x70 [15485.012514] ret_from_fork+0x1f/0x30 [15485.014193] rcu: Stack dump where RCU GP kthread last ran: [15485.023141] NMI backtrace for cpu 1 [15485.025779] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15485.038026] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15485.047478] Call Trace: [15485.049381] [15485.051133] ? dump_stack+0xbb/0x10e [15485.053353] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [15485.058601] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [15485.062554] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [15485.066708] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [15485.069247] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [15485.078646] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [15485.082684] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [15485.088215] ? update_process_times+0x6d/0xc0 [15485.089773] ? tick_sched_handle.isra.18+0x27/0x90 [15485.091626] ? tick_sched_timer+0x53/0xb0 [15485.094430] ? __hrtimer_run_queues+0x1c2/0x4d0 [15485.098981] ? tick_sched_handle.isra.18+0x90/0x90 [15485.104031] ? hrtimer_interrupt+0x128/0x2c0 [15485.106571] ? smp_apic_timer_interrupt+0xa6/0x270 [15485.111642] ? apic_timer_interrupt+0xf/0x20 [15485.113597] [15485.114628] ? native_safe_halt+0x2e/0x50 [15485.121537] ? kvm_wait+0xcd/0xe0 [15485.125684] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15485.129678] ? do_raw_spin_lock+0x177/0x190 [15485.136036] ? _raw_spin_lock+0x21/0x30 [15485.141477] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15485.146458] ? obd_mod_free+0x46/0x140 [ptlrpc] [15485.154985] ? mdc_commit_open+0x91/0xb0 [mdc] [15485.156677] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15485.162480] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15485.165681] ? after_reply+0xe2c/0x1780 [ptlrpc] [15485.173618] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15485.179061] ? __next_timer_interrupt+0x160/0x160 [15485.180292] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [15485.181887] ? do_wait_intr+0xf0/0xf0 [15485.192022] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15485.195608] ? kthread+0x1d1/0x200 [15485.196920] ? set_kthread_struct+0x70/0x70 [15485.199592] ? ret_from_fork+0x1f/0x30 [15485.203773] NMI backtrace for cpu 1 [15485.206811] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15485.220937] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15485.230481] Call Trace: [15485.232616] [15485.234417] ? dump_stack+0xbb/0x10e [15485.238249] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [15485.242646] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [15485.250029] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [15485.253710] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [15485.257712] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [15485.264717] ? rcu_dump_cpu_stacks+0x11b/0x171 [15485.273650] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [15485.283664] ? update_process_times+0x6d/0xc0 [15485.285257] ? tick_sched_handle.isra.18+0x27/0x90 [15485.295339] ? tick_sched_timer+0x53/0xb0 [15485.299615] ? __hrtimer_run_queues+0x1c2/0x4d0 [15485.309421] ? tick_sched_handle.isra.18+0x90/0x90 [15485.317397] ? hrtimer_interrupt+0x128/0x2c0 [15485.323446] ? smp_apic_timer_interrupt+0xa6/0x270 [15485.333053] ? apic_timer_interrupt+0xf/0x20 [15485.334665] [15485.335427] ? native_safe_halt+0x2e/0x50 [15485.343915] ? kvm_wait+0xcd/0xe0 [15485.345171] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15485.350826] ? do_raw_spin_lock+0x177/0x190 [15485.355051] ? _raw_spin_lock+0x21/0x30 [15485.360605] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15485.371221] ? obd_mod_free+0x46/0x140 [ptlrpc] [15485.374431] ? mdc_commit_open+0x91/0xb0 [mdc] [15485.378804] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15485.384487] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15485.394514] ? after_reply+0xe2c/0x1780 [ptlrpc] [15485.400190] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15485.408687] ? __next_timer_interrupt+0x160/0x160 [15485.417685] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [15485.426087] ? do_wait_intr+0xf0/0xf0 [15485.427427] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15485.434283] ? kthread+0x1d1/0x200 [15485.440654] ? set_kthread_struct+0x70/0x70 [15485.445481] ? ret_from_fork+0x1f/0x30 [15488.052013] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [15488.054297] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15488.069443] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15488.073408] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15488.077024] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15488.079424] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15488.087910] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15488.091372] RAX: 0000000000001031 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15488.094342] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [15488.097677] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [15488.100703] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [15488.104350] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [15488.106972] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [15488.109945] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15488.111778] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [15488.115642] Call Trace: [15488.117778] [15488.118591] ? show_regs.cold.9+0x22/0x2f [15488.120094] ? watchdog_timer_fn.cold.6+0x62/0x10a [15488.121672] ? __hrtimer_run_queues+0x1c2/0x4d0 [15488.123392] ? watchdog+0x40/0x40 [15488.124659] ? hrtimer_interrupt+0x128/0x2c0 [15488.126689] ? smp_apic_timer_interrupt+0xa6/0x270 [15488.128686] ? apic_timer_interrupt+0xf/0x20 [15488.130389] [15488.131450] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15488.133247] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15488.135161] do_raw_spin_lock+0x177/0x190 [15488.136362] _raw_spin_lock+0x21/0x30 [15488.137890] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15488.139929] ? do_raw_spin_unlock+0x75/0x190 [15488.141420] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15488.143351] ? do_wait_intr+0xf0/0xf0 [15488.144769] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15488.147088] mdc_reint+0x2d/0x110 [mdc] [15488.148393] mdc_unlink+0x658/0x870 [mdc] [15488.150211] lmv_unlink+0x34d/0xbd0 [lmv] [15488.151575] ll_unlink+0x24d/0x7f0 [lustre] [15488.152901] vfs_unlink+0x202/0x330 [15488.154244] do_unlinkat+0x2fa/0x460 [15488.155506] __x64_sys_unlink+0x37/0x50 [15488.156763] do_syscall_64+0xc1/0x440 [15488.157909] entry_SYSCALL_64_after_hwframe+0x49/0xae [15488.159537] RIP: 0033:0x7f151bdee10b [15488.160819] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [15488.166978] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [15488.169617] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [15488.171977] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [15488.174512] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15488.177073] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15488.179964] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15491.554530] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13560s! [15491.562551] Showing busy workqueues and worker pools: [15491.566256] workqueue events: flags=0x0 [15491.572552] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15491.575472] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [15491.580399] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15491.588661] pending: kfree_rcu_monitor [15491.591478] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15491.594923] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [15491.604569] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15491.612838] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [15491.628399] workqueue events_long: flags=0x0 [15491.639762] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15491.647663] pending: lu_objects_flush [obdclass] [15491.656141] workqueue events_unbound: flags=0x2 [15491.659169] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15491.663878] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15491.680146] workqueue events_power_efficient: flags=0x80 [15491.688086] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15491.697329] pending: do_cache_clean, neigh_periodic_work, check_lifetime [15491.700846] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15491.716473] pending: neigh_periodic_work [15491.719204] workqueue mm_percpu_wq: flags=0x8 [15491.721550] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15491.734408] pending: vmstat_update, lru_add_drain_per_cpu [15491.739090] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15491.743917] pending: vmstat_update [15491.749466] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15491.760755] pending: lru_add_drain_per_cpu [15491.766896] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [15491.785886] pending: lru_add_drain_per_cpu BAR(48) [15491.794176] workqueue writeback: flags=0x4a [15491.799396] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [15491.804555] pending: wb_workfn [15491.806719] workqueue xprtiod: flags=0xa [15491.809517] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15491.817517] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [15491.826471] workqueue ipv6_addrconf: flags=0x40008 [15491.829523] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15491.834046] pending: addrconf_verify_work [15491.840025] workqueue knbd0-recv: flags=0x1a [15491.844742] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15491.851284] in-flight: 658:recv_work [15491.859833] workqueue ptlrpc_pinger: flags=0x2 [15491.866639] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15491.876430] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15491.881121] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13561s workers=4 idle: 32984 37 8 [15491.892098] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15476s workers=2 idle: 661 [15500.058835] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [15500.062350] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15500.093260] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15500.102234] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15500.118134] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15500.124874] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15500.146762] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15500.150602] RAX: 0000000000007f08 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15500.153665] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [15500.164041] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [15500.172494] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [15500.180313] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [15500.190257] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [15500.196276] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15500.203627] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [15500.211284] Call Trace: [15500.213476] [15500.215162] ? show_regs.cold.9+0x22/0x2f [15500.217106] ? watchdog_timer_fn.cold.6+0x62/0x10a [15500.220118] ? __hrtimer_run_queues+0x1c2/0x4d0 [15500.226089] ? watchdog+0x40/0x40 [15500.229565] ? hrtimer_interrupt+0x128/0x2c0 [15500.232275] ? smp_apic_timer_interrupt+0xa6/0x270 [15500.240481] ? apic_timer_interrupt+0xf/0x20 [15500.242549] [15500.243942] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15500.246559] do_raw_spin_lock+0x177/0x190 [15500.249347] _raw_spin_lock+0x21/0x30 [15500.251229] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15500.253564] ? do_raw_spin_unlock+0x75/0x190 [15500.257490] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15500.261246] ? do_wait_intr+0xf0/0xf0 [15500.263002] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15500.265403] mdc_close+0x53b/0xd70 [mdc] [15500.268862] lmv_close+0x18d/0x490 [lmv] [15500.273696] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15500.276761] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15500.278354] ll_md_real_close+0xa5/0x2a0 [lustre] [15500.279721] ll_md_close+0x274/0x800 [lustre] [15500.283319] ll_file_release+0x128/0xc50 [lustre] [15500.287916] __fput+0x10d/0x440 [15500.291474] ____fput+0x12/0x20 [15500.292636] task_work_run+0xc6/0x110 [15500.297388] exit_to_usermode_loop+0x1dd/0x1f0 [15500.298986] do_syscall_64+0x3d6/0x440 [15500.302121] entry_SYSCALL_64_after_hwframe+0x49/0xae [15500.304974] RIP: 0033:0x7f151bee3cd8 [15500.307143] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15500.328213] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15500.333640] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [15500.346082] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [15500.350162] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15500.352957] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15500.355917] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15508.057372] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [15508.061564] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15508.100712] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15508.110735] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15508.125273] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15508.129579] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15508.131239] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15508.144232] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15508.147483] RAX: 0000000000007f84 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15508.149833] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [15508.154164] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [15508.157710] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [15508.160730] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [15508.164236] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [15508.167208] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15508.169954] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15508.172689] Call Trace: [15508.173478] [15508.174313] ? show_regs.cold.9+0x22/0x2f [15508.176148] ? watchdog_timer_fn.cold.6+0x62/0x10a [15508.178460] ? __hrtimer_run_queues+0x1c2/0x4d0 [15508.180509] ? watchdog+0x40/0x40 [15508.182016] ? hrtimer_interrupt+0x128/0x2c0 [15508.183929] ? smp_apic_timer_interrupt+0xa6/0x270 [15508.185577] ? apic_timer_interrupt+0xf/0x20 [15508.187156] [15508.188092] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15508.190539] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15508.192890] do_raw_spin_lock+0x177/0x190 [15508.197550] _raw_spin_lock+0x21/0x30 [15508.199450] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15508.202666] process_one_work+0x2c8/0x700 [15508.204937] worker_thread+0x296/0x6e0 [15508.206937] ? rescuer_thread+0x570/0x570 [15508.210725] kthread+0x1d1/0x200 [15508.212704] ? set_kthread_struct+0x70/0x70 [15508.215087] ret_from_fork+0x1f/0x30 [15512.055088] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [15512.057625] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15512.079857] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15512.084262] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15512.095145] RIP: 0010:native_safe_halt+0x2e/0x50 [15512.098246] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15512.110105] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15512.115979] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [15512.119817] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [15512.124745] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [15512.130131] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [15512.134548] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15512.138237] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [15512.144231] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15512.147412] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15512.151647] Call Trace: [15512.152654] [15512.153651] ? show_regs.cold.9+0x22/0x2f [15512.156062] ? watchdog_timer_fn.cold.6+0x62/0x10a [15512.159745] ? __hrtimer_run_queues+0x1c2/0x4d0 [15512.164257] ? watchdog+0x40/0x40 [15512.167259] ? hrtimer_interrupt+0x128/0x2c0 [15512.171309] ? smp_apic_timer_interrupt+0xa6/0x270 [15512.173783] ? apic_timer_interrupt+0xf/0x20 [15512.176624] [15512.177507] ? native_safe_halt+0x2e/0x50 [15512.179429] kvm_wait+0xcd/0xe0 [15512.181314] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15512.182800] do_raw_spin_lock+0x177/0x190 [15512.190125] _raw_spin_lock+0x21/0x30 [15512.193323] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15512.203517] obd_mod_free+0x46/0x140 [ptlrpc] [15512.210864] mdc_commit_open+0x91/0xb0 [mdc] [15512.214629] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15512.219551] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15512.222340] after_reply+0xe2c/0x1780 [ptlrpc] [15512.234244] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15512.239709] ? __next_timer_interrupt+0x160/0x160 [15512.244328] ptlrpcd+0xa82/0xe40 [ptlrpc] [15512.248123] ? do_wait_intr+0xf0/0xf0 [15512.249187] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15512.251348] kthread+0x1d1/0x200 [15512.253383] ? set_kthread_struct+0x70/0x70 [15512.254507] ret_from_fork+0x1f/0x30 [15516.052456] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [15516.056294] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15516.077619] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15516.083431] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15516.087769] RIP: 0010:native_safe_halt+0x2e/0x50 [15516.090054] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15516.099769] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15516.103630] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15516.107406] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [15516.110871] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [15516.114332] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [15516.118165] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [15516.121456] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [15516.125257] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15516.128346] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [15516.131736] Call Trace: [15516.132504] [15516.133677] ? show_regs.cold.9+0x22/0x2f [15516.135346] ? watchdog_timer_fn.cold.6+0x62/0x10a [15516.137675] ? __hrtimer_run_queues+0x1c2/0x4d0 [15516.139967] ? watchdog+0x40/0x40 [15516.141594] ? hrtimer_interrupt+0x128/0x2c0 [15516.143403] ? smp_apic_timer_interrupt+0xa6/0x270 [15516.145447] ? apic_timer_interrupt+0xf/0x20 [15516.147365] [15516.148501] ? native_safe_halt+0x2e/0x50 [15516.150147] kvm_wait+0xcd/0xe0 [15516.152038] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15516.154932] do_raw_spin_lock+0x177/0x190 [15516.156628] _raw_spin_lock+0x21/0x30 [15516.158817] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15516.160882] ? do_raw_spin_unlock+0x75/0x190 [15516.163178] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15516.165777] ? do_wait_intr+0xf0/0xf0 [15516.167590] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15516.170465] mdc_reint+0x2d/0x110 [mdc] [15516.172477] mdc_unlink+0x658/0x870 [mdc] [15516.174103] lmv_unlink+0x34d/0xbd0 [lmv] [15516.176296] ll_unlink+0x24d/0x7f0 [lustre] [15516.178417] vfs_unlink+0x202/0x330 [15516.180138] do_unlinkat+0x2fa/0x460 [15516.181842] __x64_sys_unlink+0x37/0x50 [15516.183710] do_syscall_64+0xc1/0x440 [15516.185380] entry_SYSCALL_64_after_hwframe+0x49/0xae [15516.187431] RIP: 0033:0x7f151bdee10b [15516.189364] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [15516.197515] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [15516.201126] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [15516.203916] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [15516.206973] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15516.209936] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15516.213648] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15522.280381] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13591s! [15522.282641] Showing busy workqueues and worker pools: [15522.292896] workqueue events: flags=0x0 [15522.296890] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15522.300513] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [15522.323929] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15522.330164] pending: kfree_rcu_monitor [15522.337632] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15522.348162] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [15522.360647] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15522.365759] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [15522.368874] workqueue events_long: flags=0x0 [15522.373695] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15522.382507] pending: lu_objects_flush [obdclass] [15522.384539] workqueue events_unbound: flags=0x2 [15522.395174] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15522.400390] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15522.409959] workqueue events_power_efficient: flags=0x80 [15522.418146] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15522.431113] pending: do_cache_clean, neigh_periodic_work, check_lifetime [15522.440791] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15522.450912] pending: neigh_periodic_work [15522.453820] workqueue mm_percpu_wq: flags=0x8 [15522.458654] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15522.466257] pending: vmstat_update, lru_add_drain_per_cpu [15522.472453] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15522.482217] pending: vmstat_update [15522.483572] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15522.494186] pending: lru_add_drain_per_cpu [15522.500634] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [15522.508271] pending: lru_add_drain_per_cpu BAR(48) [15522.509937] workqueue writeback: flags=0x4a [15522.520424] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [15522.525129] pending: wb_workfn [15522.528874] workqueue xprtiod: flags=0xa [15522.534336] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15522.538954] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [15522.548095] workqueue ipv6_addrconf: flags=0x40008 [15522.551893] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15522.564753] pending: addrconf_verify_work [15522.571976] workqueue knbd0-recv: flags=0x1a [15522.578728] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15522.588283] in-flight: 658:recv_work [15522.595223] workqueue ptlrpc_pinger: flags=0x2 [15522.599048] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15522.611165] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15522.616145] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13592s workers=4 idle: 32984 37 8 [15522.625773] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15507s workers=2 idle: 661 [15528.060057] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [15528.068947] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15528.145336] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15528.157476] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15528.168466] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15528.177341] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15528.210195] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15528.219164] RAX: 000000000000766f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15528.223999] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [15528.227441] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [15528.239333] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [15528.247040] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [15528.255095] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [15528.260530] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15528.265780] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [15528.273346] Call Trace: [15528.274750] [15528.279354] ? show_regs.cold.9+0x22/0x2f [15528.284071] ? watchdog_timer_fn.cold.6+0x62/0x10a [15528.292822] ? __hrtimer_run_queues+0x1c2/0x4d0 [15528.296091] ? watchdog+0x40/0x40 [15528.300150] ? hrtimer_interrupt+0x128/0x2c0 [15528.302881] ? smp_apic_timer_interrupt+0xa6/0x270 [15528.314258] ? apic_timer_interrupt+0xf/0x20 [15528.321974] [15528.324149] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15528.327235] do_raw_spin_lock+0x177/0x190 [15528.336326] _raw_spin_lock+0x21/0x30 [15528.340458] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15528.342631] ? do_raw_spin_unlock+0x75/0x190 [15528.356886] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15528.361049] ? do_wait_intr+0xf0/0xf0 [15528.363196] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15528.369967] mdc_close+0x53b/0xd70 [mdc] [15528.378027] lmv_close+0x18d/0x490 [lmv] [15528.381451] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15528.386331] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15528.388477] ll_md_real_close+0xa5/0x2a0 [lustre] [15528.393178] ll_md_close+0x274/0x800 [lustre] [15528.395394] ll_file_release+0x128/0xc50 [lustre] [15528.397018] __fput+0x10d/0x440 [15528.397931] ____fput+0x12/0x20 [15528.401980] task_work_run+0xc6/0x110 [15528.404135] exit_to_usermode_loop+0x1dd/0x1f0 [15528.407671] do_syscall_64+0x3d6/0x440 [15528.409160] entry_SYSCALL_64_after_hwframe+0x49/0xae [15528.411794] RIP: 0033:0x7f151bee3cd8 [15528.415185] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15528.440368] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15528.447825] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [15528.456080] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [15528.460851] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15528.464220] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15528.474779] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15536.058438] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [15536.065024] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15536.126776] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15536.140641] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15536.145838] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15536.155625] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15536.170737] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15536.189043] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15536.201068] RAX: 000000000000757a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15536.210205] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [15536.213803] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [15536.218050] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [15536.222256] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [15536.229284] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [15536.236169] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15536.240034] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15536.243102] Call Trace: [15536.244987] [15536.246025] ? show_regs.cold.9+0x22/0x2f [15536.248164] ? watchdog_timer_fn.cold.6+0x62/0x10a [15536.250764] ? __hrtimer_run_queues+0x1c2/0x4d0 [15536.253696] ? watchdog+0x40/0x40 [15536.256009] ? hrtimer_interrupt+0x128/0x2c0 [15536.258597] ? smp_apic_timer_interrupt+0xa6/0x270 [15536.260365] ? apic_timer_interrupt+0xf/0x20 [15536.267187] [15536.269176] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15536.272758] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15536.276926] do_raw_spin_lock+0x177/0x190 [15536.279250] _raw_spin_lock+0x21/0x30 [15536.282972] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15536.295056] process_one_work+0x2c8/0x700 [15536.301650] worker_thread+0x296/0x6e0 [15536.306326] ? rescuer_thread+0x570/0x570 [15536.321274] kthread+0x1d1/0x200 [15536.331755] ? set_kthread_struct+0x70/0x70 [15536.335317] ret_from_fork+0x1f/0x30 [15540.055743] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [15540.069117] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15540.118964] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15540.144634] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15540.158150] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15540.160267] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15540.197719] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15540.209179] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [15540.216512] RDX: 0000000000003175 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [15540.223883] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [15540.235199] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [15540.247835] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15540.250084] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [15540.259873] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15540.265255] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15540.277330] Call Trace: [15540.278273] [15540.279035] ? show_regs.cold.9+0x22/0x2f [15540.284840] ? watchdog_timer_fn.cold.6+0x62/0x10a [15540.290888] ? __hrtimer_run_queues+0x1c2/0x4d0 [15540.300928] ? watchdog+0x40/0x40 [15540.302786] ? hrtimer_interrupt+0x128/0x2c0 [15540.307500] ? smp_apic_timer_interrupt+0xa6/0x270 [15540.309223] ? apic_timer_interrupt+0xf/0x20 [15540.321025] [15540.321882] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15540.328655] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15540.344738] do_raw_spin_lock+0x177/0x190 [15540.350710] _raw_spin_lock+0x21/0x30 [15540.352107] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15540.360168] obd_mod_free+0x46/0x140 [ptlrpc] [15540.370423] mdc_commit_open+0x91/0xb0 [mdc] [15540.372848] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15540.381536] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15540.392699] after_reply+0xe2c/0x1780 [ptlrpc] [15540.394247] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15540.402983] ? __next_timer_interrupt+0x160/0x160 [15540.409099] ptlrpcd+0xa82/0xe40 [ptlrpc] [15540.417128] ? do_wait_intr+0xf0/0xf0 [15540.421601] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15540.431036] kthread+0x1d1/0x200 [15540.437267] ? set_kthread_struct+0x70/0x70 [15540.440835] ret_from_fork+0x1f/0x30 [15548.059611] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [15548.067646] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15548.108519] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15548.122404] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15548.131984] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15548.140479] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15548.162927] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15548.169544] RAX: 000000000000090d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15548.178016] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [15548.183665] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [15548.195856] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [15548.199967] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [15548.205193] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [15548.209280] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15548.212332] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [15548.220642] Call Trace: [15548.221596] [15548.222234] ? show_regs.cold.9+0x22/0x2f [15548.228919] ? watchdog_timer_fn.cold.6+0x62/0x10a [15548.234923] ? __hrtimer_run_queues+0x1c2/0x4d0 [15548.239561] ? watchdog+0x40/0x40 [15548.242027] ? hrtimer_interrupt+0x128/0x2c0 [15548.247577] ? smp_apic_timer_interrupt+0xa6/0x270 [15548.251459] ? apic_timer_interrupt+0xf/0x20 [15548.256075] [15548.257148] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15548.261883] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15548.269278] do_raw_spin_lock+0x177/0x190 [15548.274187] _raw_spin_lock+0x21/0x30 [15548.277989] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15548.283172] ? do_raw_spin_unlock+0x75/0x190 [15548.285145] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15548.293120] ? do_wait_intr+0xf0/0xf0 [15548.299261] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15548.307358] mdc_reint+0x2d/0x110 [mdc] [15548.309469] mdc_unlink+0x658/0x870 [mdc] [15548.313890] lmv_unlink+0x34d/0xbd0 [lmv] [15548.315854] ll_unlink+0x24d/0x7f0 [lustre] [15548.318049] vfs_unlink+0x202/0x330 [15548.319826] do_unlinkat+0x2fa/0x460 [15548.326908] __x64_sys_unlink+0x37/0x50 [15548.330421] do_syscall_64+0xc1/0x440 [15548.332611] entry_SYSCALL_64_after_hwframe+0x49/0xae [15548.334300] RIP: 0033:0x7f151bdee10b [15548.339582] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [15548.357922] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [15548.362113] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [15548.364939] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [15548.368181] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15548.372377] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15548.375884] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15552.998464] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13622s! [15553.010452] Showing busy workqueues and worker pools: [15553.023764] workqueue events: flags=0x0 [15553.030683] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15553.047279] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [15553.066478] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15553.074702] pending: kfree_rcu_monitor [15553.077626] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15553.081867] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [15553.088740] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15553.090932] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [15553.093595] workqueue events_long: flags=0x0 [15553.094797] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15553.096804] pending: lu_objects_flush [obdclass] [15553.099019] workqueue events_unbound: flags=0x2 [15553.101573] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15553.103523] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15553.113876] workqueue events_power_efficient: flags=0x80 [15553.119057] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15553.130598] pending: do_cache_clean, neigh_periodic_work, check_lifetime [15553.138087] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15553.143777] pending: neigh_periodic_work [15553.146196] workqueue mm_percpu_wq: flags=0x8 [15553.151349] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15553.159122] pending: vmstat_update, lru_add_drain_per_cpu [15553.167382] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15553.169827] pending: vmstat_update [15553.175056] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15553.189367] pending: lru_add_drain_per_cpu [15553.192730] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [15553.201184] pending: lru_add_drain_per_cpu BAR(48) [15553.205546] workqueue writeback: flags=0x4a [15553.209624] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [15553.216499] pending: wb_workfn [15553.218616] workqueue xprtiod: flags=0xa [15553.220331] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15553.229717] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [15553.242498] workqueue ipv6_addrconf: flags=0x40008 [15553.246545] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15553.260703] pending: addrconf_verify_work [15553.264395] workqueue knbd0-recv: flags=0x1a [15553.269297] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15553.276106] in-flight: 658:recv_work [15553.280767] workqueue ptlrpc_pinger: flags=0x2 [15553.285954] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15553.292643] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15553.296453] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13622s workers=4 idle: 32984 37 8 [15553.305689] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15538s workers=2 idle: 661 [15556.064735] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [15556.079849] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15556.151710] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15556.170987] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15556.174075] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15556.185678] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15556.232878] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15556.237625] RAX: 000000000000753a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15556.250071] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [15556.262195] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [15556.268385] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [15556.277772] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [15556.292823] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [15556.299520] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15556.312957] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [15556.325620] Call Trace: [15556.326410] [15556.327082] ? show_regs.cold.9+0x22/0x2f [15556.341034] ? watchdog_timer_fn.cold.6+0x62/0x10a [15556.353426] ? __hrtimer_run_queues+0x1c2/0x4d0 [15556.362010] ? watchdog+0x40/0x40 [15556.364192] ? hrtimer_interrupt+0x128/0x2c0 [15556.369087] ? smp_apic_timer_interrupt+0xa6/0x270 [15556.377427] ? apic_timer_interrupt+0xf/0x20 [15556.378768] [15556.382091] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15556.389855] do_raw_spin_lock+0x177/0x190 [15556.395889] _raw_spin_lock+0x21/0x30 [15556.410055] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15556.412105] ? do_raw_spin_unlock+0x75/0x190 [15556.418774] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15556.424937] ? do_wait_intr+0xf0/0xf0 [15556.426859] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15556.437501] mdc_close+0x53b/0xd70 [mdc] [15556.447868] lmv_close+0x18d/0x490 [lmv] [15556.458428] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15556.468853] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15556.479680] ll_md_real_close+0xa5/0x2a0 [lustre] [15556.487437] ll_md_close+0x274/0x800 [lustre] [15556.493678] ll_file_release+0x128/0xc50 [lustre] [15556.501320] __fput+0x10d/0x440 [15556.502595] ____fput+0x12/0x20 [15556.503843] task_work_run+0xc6/0x110 [15556.515953] exit_to_usermode_loop+0x1dd/0x1f0 [15556.521514] do_syscall_64+0x3d6/0x440 [15556.524579] entry_SYSCALL_64_after_hwframe+0x49/0xae [15556.532043] RIP: 0033:0x7f151bee3cd8 [15556.536239] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15556.577312] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15556.580103] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [15556.593199] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [15556.603650] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15556.607556] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15556.623047] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15564.056747] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [15564.060351] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15564.084518] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15564.090019] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15564.094379] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15564.096515] RIP: 0010:native_safe_halt+0x2e/0x50 [15564.097859] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15564.104392] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15564.106626] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15564.108737] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [15564.110960] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [15564.113336] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [15564.115585] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [15564.117767] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [15564.120258] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15564.121921] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15564.124025] Call Trace: [15564.124781] [15564.125442] ? show_regs.cold.9+0x22/0x2f [15564.126734] ? watchdog_timer_fn.cold.6+0x62/0x10a [15564.128020] ? __hrtimer_run_queues+0x1c2/0x4d0 [15564.129447] ? watchdog+0x40/0x40 [15564.130566] ? hrtimer_interrupt+0x128/0x2c0 [15564.131735] ? smp_apic_timer_interrupt+0xa6/0x270 [15564.133194] ? apic_timer_interrupt+0xf/0x20 [15564.134492] [15564.135457] ? native_safe_halt+0x2e/0x50 [15564.136774] kvm_wait+0xcd/0xe0 [15564.138282] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15564.140524] do_raw_spin_lock+0x177/0x190 [15564.141886] _raw_spin_lock+0x21/0x30 [15564.143026] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15564.145682] process_one_work+0x2c8/0x700 [15564.146920] worker_thread+0x296/0x6e0 [15564.148737] ? rescuer_thread+0x570/0x570 [15564.150528] kthread+0x1d1/0x200 [15564.151899] ? set_kthread_struct+0x70/0x70 [15564.153813] ret_from_fork+0x1f/0x30 [15568.055154] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [15568.058896] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15568.075939] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15568.081169] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15568.089122] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15568.094570] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15568.102173] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15568.105551] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [15568.108947] RDX: 0000000000004077 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [15568.112263] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [15568.114711] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [15568.117827] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15568.120759] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [15568.124135] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15568.126522] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15568.129364] Call Trace: [15568.130438] [15568.131209] ? show_regs.cold.9+0x22/0x2f [15568.132854] ? watchdog_timer_fn.cold.6+0x62/0x10a [15568.134871] ? __hrtimer_run_queues+0x1c2/0x4d0 [15568.136531] ? watchdog+0x40/0x40 [15568.137939] ? hrtimer_interrupt+0x128/0x2c0 [15568.139581] ? smp_apic_timer_interrupt+0xa6/0x270 [15568.142286] ? apic_timer_interrupt+0xf/0x20 [15568.143707] [15568.144466] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15568.146459] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15568.148850] do_raw_spin_lock+0x177/0x190 [15568.150606] _raw_spin_lock+0x21/0x30 [15568.152412] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15568.155109] obd_mod_free+0x46/0x140 [ptlrpc] [15568.156981] mdc_commit_open+0x91/0xb0 [mdc] [15568.159246] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15568.161511] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15568.166189] after_reply+0xe2c/0x1780 [ptlrpc] [15568.168519] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15568.171427] ? __next_timer_interrupt+0x160/0x160 [15568.173509] ptlrpcd+0xa82/0xe40 [ptlrpc] [15568.175800] ? do_wait_intr+0xf0/0xf0 [15568.177578] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15568.180106] kthread+0x1d1/0x200 [15568.181323] ? set_kthread_struct+0x70/0x70 [15568.184663] ret_from_fork+0x1f/0x30 [15580.052053] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [15580.055416] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15580.118230] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15580.137761] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15580.149848] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15580.169119] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15580.193181] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15580.199766] RAX: 0000000000004ba6 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15580.203834] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [15580.207765] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [15580.211729] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [15580.214761] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [15580.218159] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [15580.222158] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15580.225738] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [15580.229394] Call Trace: [15580.230507] [15580.231503] ? show_regs.cold.9+0x22/0x2f [15580.233592] ? watchdog_timer_fn.cold.6+0x62/0x10a [15580.242277] ? __hrtimer_run_queues+0x1c2/0x4d0 [15580.245524] ? watchdog+0x40/0x40 [15580.247594] ? hrtimer_interrupt+0x128/0x2c0 [15580.251525] ? smp_apic_timer_interrupt+0xa6/0x270 [15580.256515] ? apic_timer_interrupt+0xf/0x20 [15580.259577] [15580.260542] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15580.263530] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15580.266487] do_raw_spin_lock+0x177/0x190 [15580.269447] _raw_spin_lock+0x21/0x30 [15580.271581] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15580.276742] ? do_raw_spin_unlock+0x75/0x190 [15580.279371] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15580.284806] ? do_wait_intr+0xf0/0xf0 [15580.287023] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15580.289439] mdc_reint+0x2d/0x110 [mdc] [15580.293825] mdc_unlink+0x658/0x870 [mdc] [15580.296481] lmv_unlink+0x34d/0xbd0 [lmv] [15580.299772] ll_unlink+0x24d/0x7f0 [lustre] [15580.302336] vfs_unlink+0x202/0x330 [15580.304897] do_unlinkat+0x2fa/0x460 [15580.309891] __x64_sys_unlink+0x37/0x50 [15580.312388] do_syscall_64+0xc1/0x440 [15580.314427] entry_SYSCALL_64_after_hwframe+0x49/0xae [15580.316846] RIP: 0033:0x7f151bdee10b [15580.321274] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [15580.335248] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [15580.346677] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [15580.350451] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [15580.357844] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15580.362430] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15580.366209] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15583.712141] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13653s! [15583.715658] Showing busy workqueues and worker pools: [15583.719801] workqueue events: flags=0x0 [15583.725938] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15583.735164] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [15583.745067] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15583.751055] pending: kfree_rcu_monitor [15583.754519] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15583.767602] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [15583.787589] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15583.792050] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [15583.799478] workqueue events_long: flags=0x0 [15583.805153] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15583.826040] pending: lu_objects_flush [obdclass] [15583.837176] workqueue events_unbound: flags=0x2 [15583.845234] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15583.854372] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15583.865853] workqueue events_power_efficient: flags=0x80 [15583.873170] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15583.884362] pending: do_cache_clean, neigh_periodic_work, check_lifetime [15583.889481] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15583.900059] pending: neigh_periodic_work [15583.910259] workqueue mm_percpu_wq: flags=0x8 [15583.916047] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15583.924061] pending: vmstat_update, lru_add_drain_per_cpu [15583.934698] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15583.945092] pending: vmstat_update [15583.950235] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15583.959327] pending: lru_add_drain_per_cpu [15583.963507] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [15583.972789] pending: lru_add_drain_per_cpu BAR(48) [15583.980132] workqueue writeback: flags=0x4a [15583.990450] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [15584.004054] pending: wb_workfn [15584.013966] workqueue xprtiod: flags=0xa [15584.019863] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15584.026924] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [15584.042815] workqueue ipv6_addrconf: flags=0x40008 [15584.049067] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15584.052923] pending: addrconf_verify_work [15584.057037] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [15584.062251] workqueue knbd0-recv: flags=0x1a [15584.062264] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15584.062277] in-flight: 658:recv_work [15584.062292] workqueue ptlrpc_pinger: flags=0x2 [15584.062295] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15584.062302] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15584.062934] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13653s workers=4 idle: 32984 37 8 [15584.062948] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15568s workers=2 idle: 661 [15584.151807] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15584.187754] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15584.199182] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15584.218583] RIP: 0010:native_safe_halt+0x2e/0x50 [15584.220375] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15584.246847] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15584.264401] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15584.275594] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [15584.283774] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [15584.293371] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [15584.303201] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [15584.305696] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [15584.327098] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15584.335507] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [15584.355930] Call Trace: [15584.363854] [15584.364681] ? show_regs.cold.9+0x22/0x2f [15584.369717] ? watchdog_timer_fn.cold.6+0x62/0x10a [15584.374413] ? __hrtimer_run_queues+0x1c2/0x4d0 [15584.381893] ? watchdog+0x40/0x40 [15584.383236] ? hrtimer_interrupt+0x128/0x2c0 [15584.389645] ? smp_apic_timer_interrupt+0xa6/0x270 [15584.396864] ? apic_timer_interrupt+0xf/0x20 [15584.405309] [15584.406531] ? native_safe_halt+0x2e/0x50 [15584.415161] kvm_wait+0xcd/0xe0 [15584.416576] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15584.424348] do_raw_spin_lock+0x177/0x190 [15584.425856] _raw_spin_lock+0x21/0x30 [15584.434788] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15584.442429] ? do_raw_spin_unlock+0x75/0x190 [15584.449577] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15584.456382] ? do_wait_intr+0xf0/0xf0 [15584.459230] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15584.465791] mdc_close+0x53b/0xd70 [mdc] [15584.467247] lmv_close+0x18d/0x490 [lmv] [15584.470360] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15584.474357] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15584.480759] ll_md_real_close+0xa5/0x2a0 [lustre] [15584.486572] ll_md_close+0x274/0x800 [lustre] [15584.490605] ll_file_release+0x128/0xc50 [lustre] [15584.495538] __fput+0x10d/0x440 [15584.499647] ____fput+0x12/0x20 [15584.502509] task_work_run+0xc6/0x110 [15584.509110] exit_to_usermode_loop+0x1dd/0x1f0 [15584.517337] do_syscall_64+0x3d6/0x440 [15584.530896] entry_SYSCALL_64_after_hwframe+0x49/0xae [15584.546090] RIP: 0033:0x7f151bee3cd8 [15584.547432] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15584.579202] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15584.584705] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [15584.598625] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [15584.611408] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15584.621681] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15584.628103] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15592.056057] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [15592.063985] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15592.088479] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15592.095103] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15592.103512] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15592.110842] RIP: 0010:native_safe_halt+0x2e/0x50 [15592.114502] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15592.125609] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15592.130990] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15592.135760] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [15592.140410] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [15592.146293] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [15592.149534] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [15592.152674] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [15592.159018] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15592.161655] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15592.168698] Call Trace: [15592.170369] [15592.173511] ? show_regs.cold.9+0x22/0x2f [15592.175472] ? watchdog_timer_fn.cold.6+0x62/0x10a [15592.180274] ? __hrtimer_run_queues+0x1c2/0x4d0 [15592.185693] ? watchdog+0x40/0x40 [15592.189241] ? hrtimer_interrupt+0x128/0x2c0 [15592.190739] ? smp_apic_timer_interrupt+0xa6/0x270 [15592.194520] ? apic_timer_interrupt+0xf/0x20 [15592.196740] [15592.198115] ? native_safe_halt+0x2e/0x50 [15592.200302] kvm_wait+0xcd/0xe0 [15592.201762] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15592.204295] do_raw_spin_lock+0x177/0x190 [15592.205930] _raw_spin_lock+0x21/0x30 [15592.207583] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15592.210111] process_one_work+0x2c8/0x700 [15592.211641] worker_thread+0x296/0x6e0 [15592.216710] ? rescuer_thread+0x570/0x570 [15592.218089] kthread+0x1d1/0x200 [15592.220602] ? set_kthread_struct+0x70/0x70 [15592.223173] ret_from_fork+0x1f/0x30 [15596.056416] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [15596.060584] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15596.075619] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15596.079233] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15596.081821] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15596.083544] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15596.088674] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15596.090679] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [15596.092626] RDX: 0000000000007cac RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [15596.094628] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [15596.096568] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [15596.098259] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15596.100360] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [15596.103237] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15596.105325] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15596.108000] Call Trace: [15596.108894] [15596.109613] ? show_regs.cold.9+0x22/0x2f [15596.111235] ? watchdog_timer_fn.cold.6+0x62/0x10a [15596.113309] ? __hrtimer_run_queues+0x1c2/0x4d0 [15596.114980] ? watchdog+0x40/0x40 [15596.116269] ? hrtimer_interrupt+0x128/0x2c0 [15596.117996] ? smp_apic_timer_interrupt+0xa6/0x270 [15596.119730] ? apic_timer_interrupt+0xf/0x20 [15596.121406] [15596.122093] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15596.124510] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15596.126600] do_raw_spin_lock+0x177/0x190 [15596.127924] _raw_spin_lock+0x21/0x30 [15596.129582] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15596.131875] obd_mod_free+0x46/0x140 [ptlrpc] [15596.133921] mdc_commit_open+0x91/0xb0 [mdc] [15596.135660] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15596.137930] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15596.140615] after_reply+0xe2c/0x1780 [ptlrpc] [15596.142477] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15596.144329] ? __next_timer_interrupt+0x160/0x160 [15596.145963] ptlrpcd+0xa82/0xe40 [ptlrpc] [15596.147830] ? do_wait_intr+0xf0/0xf0 [15596.149627] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15596.151737] kthread+0x1d1/0x200 [15596.152940] ? set_kthread_struct+0x70/0x70 [15596.154337] ret_from_fork+0x1f/0x30 [15612.053230] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [15612.057221] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [15612.058545] Modules linked in: [15612.063549] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15612.064716] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15612.064725] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15612.064728] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15612.064773] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15612.064778] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15612.064783] RAX: 00000000000011b3 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15612.064785] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [15612.064788] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [15612.064790] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [15612.064793] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [15612.064796] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [15612.064799] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15612.064801] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [15612.064811] Call Trace: [15612.064817] [15612.064823] ? show_regs.cold.9+0x22/0x2f [15612.064834] ? watchdog_timer_fn.cold.6+0x62/0x10a [15612.064841] ? __hrtimer_run_queues+0x1c2/0x4d0 [15612.064846] ? watchdog+0x40/0x40 [15612.064850] ? hrtimer_interrupt+0x128/0x2c0 [15612.064854] ? smp_apic_timer_interrupt+0xa6/0x270 [15612.064863] ? apic_timer_interrupt+0xf/0x20 [15612.064878] [15612.064882] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15612.064890] do_raw_spin_lock+0x177/0x190 [15612.064895] _raw_spin_lock+0x21/0x30 [15612.064900] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15612.065196] ? do_raw_spin_unlock+0x75/0x190 [15612.065202] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15612.065405] ? do_wait_intr+0xf0/0xf0 [15612.065414] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15612.065619] mdc_close+0x53b/0xd70 [mdc] [15612.065663] lmv_close+0x18d/0x490 [lmv] [15612.065696] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15612.066557] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15612.066809] ll_md_real_close+0xa5/0x2a0 [lustre] [15612.066908] ll_md_close+0x274/0x800 [lustre] [15612.066992] ll_file_release+0x128/0xc50 [lustre] [15612.067073] __fput+0x10d/0x440 [15612.067082] ____fput+0x12/0x20 [15612.067087] task_work_run+0xc6/0x110 [15612.067094] exit_to_usermode_loop+0x1dd/0x1f0 [15612.067101] do_syscall_64+0x3d6/0x440 [15612.067106] entry_SYSCALL_64_after_hwframe+0x49/0xae [15612.067115] RIP: 0033:0x7f151bee3cd8 [15612.067123] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15612.067127] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15612.067131] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [15612.067133] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [15612.067136] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15612.067138] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15612.067140] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15612.320990] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15612.353243] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15612.364299] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15612.370199] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15612.373552] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15612.390437] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15612.396501] RAX: 0000000000007a46 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15612.402204] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [15612.412367] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [15612.417427] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [15612.422118] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [15612.433558] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [15612.440471] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15612.449414] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [15612.453955] Call Trace: [15612.456182] [15612.457548] ? show_regs.cold.9+0x22/0x2f [15612.460516] ? watchdog_timer_fn.cold.6+0x62/0x10a [15612.464272] ? __hrtimer_run_queues+0x1c2/0x4d0 [15612.467970] ? watchdog+0x40/0x40 [15612.472230] ? hrtimer_interrupt+0x128/0x2c0 [15612.474228] ? smp_apic_timer_interrupt+0xa6/0x270 [15612.476089] ? apic_timer_interrupt+0xf/0x20 [15612.480465] [15612.481208] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15612.492996] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15612.495421] do_raw_spin_lock+0x177/0x190 [15612.497496] _raw_spin_lock+0x21/0x30 [15612.499889] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15612.511986] ? do_raw_spin_unlock+0x75/0x190 [15612.513648] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15612.530017] ? do_wait_intr+0xf0/0xf0 [15612.531411] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15612.541337] mdc_reint+0x2d/0x110 [mdc] [15612.544489] mdc_unlink+0x658/0x870 [mdc] [15612.547953] lmv_unlink+0x34d/0xbd0 [lmv] [15612.550202] ll_unlink+0x24d/0x7f0 [lustre] [15612.557756] vfs_unlink+0x202/0x330 [15612.565274] do_unlinkat+0x2fa/0x460 [15612.567441] __x64_sys_unlink+0x37/0x50 [15612.569859] do_syscall_64+0xc1/0x440 [15612.573220] entry_SYSCALL_64_after_hwframe+0x49/0xae [15612.576574] RIP: 0033:0x7f151bdee10b [15612.580094] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [15612.590584] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [15612.596908] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [15612.602183] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [15612.615366] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15612.624261] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15612.630665] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15614.433610] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13683s! [15614.440157] Showing busy workqueues and worker pools: [15614.442607] workqueue events: flags=0x0 [15614.445942] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15614.451783] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [15614.459778] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15614.462586] pending: kfree_rcu_monitor [15614.464605] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15614.467110] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [15614.474970] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15614.481478] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [15614.489233] workqueue events_long: flags=0x0 [15614.490743] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15614.494527] pending: lu_objects_flush [obdclass] [15614.497200] workqueue events_unbound: flags=0x2 [15614.500409] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15614.503774] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15614.509272] workqueue events_power_efficient: flags=0x80 [15614.515170] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15614.522534] pending: do_cache_clean, neigh_periodic_work, check_lifetime [15614.527454] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15614.533239] pending: neigh_periodic_work [15614.535356] workqueue mm_percpu_wq: flags=0x8 [15614.537266] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15614.539441] pending: vmstat_update, lru_add_drain_per_cpu [15614.541730] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15614.544323] pending: vmstat_update [15614.545530] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15614.548226] pending: lru_add_drain_per_cpu [15614.549739] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [15614.552727] pending: lru_add_drain_per_cpu BAR(48) [15614.555124] workqueue writeback: flags=0x4a [15614.556427] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [15614.558395] pending: wb_workfn [15614.561287] workqueue xprtiod: flags=0xa [15614.563197] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15614.565963] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [15614.575523] workqueue ipv6_addrconf: flags=0x40008 [15614.580436] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15614.589026] pending: addrconf_verify_work [15614.593491] workqueue knbd0-recv: flags=0x1a [15614.595520] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15614.598827] in-flight: 658:recv_work [15614.600888] workqueue ptlrpc_pinger: flags=0x2 [15614.602937] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15614.608679] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15614.612470] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13684s workers=4 idle: 32984 37 8 [15614.615889] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15599s workers=2 idle: 661 [15620.056221] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [15620.058670] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15620.094508] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15620.100907] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15620.105448] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15620.108132] RIP: 0010:native_safe_halt+0x2e/0x50 [15620.110633] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15620.120298] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15620.124317] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15620.130098] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [15620.133957] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [15620.138201] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [15620.141626] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [15620.145022] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [15620.148465] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15620.151170] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15620.154233] Call Trace: [15620.155566] [15620.156413] ? show_regs.cold.9+0x22/0x2f [15620.158223] ? watchdog_timer_fn.cold.6+0x62/0x10a [15620.160285] ? __hrtimer_run_queues+0x1c2/0x4d0 [15620.162630] ? watchdog+0x40/0x40 [15620.164582] ? hrtimer_interrupt+0x128/0x2c0 [15620.166368] ? smp_apic_timer_interrupt+0xa6/0x270 [15620.168901] ? apic_timer_interrupt+0xf/0x20 [15620.170784] [15620.171849] ? native_safe_halt+0x2e/0x50 [15620.173642] kvm_wait+0xcd/0xe0 [15620.175068] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15620.177465] do_raw_spin_lock+0x177/0x190 [15620.180672] _raw_spin_lock+0x21/0x30 [15620.182339] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15620.184919] process_one_work+0x2c8/0x700 [15620.186721] worker_thread+0x296/0x6e0 [15620.188557] ? rescuer_thread+0x570/0x570 [15620.190405] kthread+0x1d1/0x200 [15620.192090] ? set_kthread_struct+0x70/0x70 [15620.193987] ret_from_fork+0x1f/0x30 [15624.055305] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [15624.060322] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15624.096838] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15624.105076] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15624.111840] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15624.115719] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15624.127574] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15624.132082] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [15624.136154] RDX: 0000000000005c8a RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [15624.140575] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [15624.144945] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [15624.153732] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15624.156574] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [15624.159935] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15624.162929] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15624.166216] Call Trace: [15624.167399] [15624.168398] ? show_regs.cold.9+0x22/0x2f [15624.170360] ? watchdog_timer_fn.cold.6+0x62/0x10a [15624.172742] ? __hrtimer_run_queues+0x1c2/0x4d0 [15624.174929] ? watchdog+0x40/0x40 [15624.176848] ? hrtimer_interrupt+0x128/0x2c0 [15624.179007] ? smp_apic_timer_interrupt+0xa6/0x270 [15624.181349] ? apic_timer_interrupt+0xf/0x20 [15624.183393] [15624.184273] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15624.187693] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15624.195865] do_raw_spin_lock+0x177/0x190 [15624.198397] _raw_spin_lock+0x21/0x30 [15624.199721] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15624.202822] obd_mod_free+0x46/0x140 [ptlrpc] [15624.205284] mdc_commit_open+0x91/0xb0 [mdc] [15624.206941] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15624.209380] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15624.212105] after_reply+0xe2c/0x1780 [ptlrpc] [15624.214053] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15624.216293] ? __next_timer_interrupt+0x160/0x160 [15624.217940] ptlrpcd+0xa82/0xe40 [ptlrpc] [15624.219850] ? do_wait_intr+0xf0/0xf0 [15624.221979] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15624.224904] kthread+0x1d1/0x200 [15624.227261] ? set_kthread_struct+0x70/0x70 [15624.228977] ret_from_fork+0x1f/0x30 [15640.058912] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [15640.060611] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [15640.066318] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15640.066977] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15640.066985] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15640.066988] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15640.067002] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15640.067007] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15640.067011] RAX: 0000000000007d84 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15640.067013] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [15640.067016] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [15640.067018] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [15640.067020] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [15640.067024] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [15640.067026] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15640.067029] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [15640.067038] Call Trace: [15640.067044] [15640.067051] ? show_regs.cold.9+0x22/0x2f [15640.067061] ? watchdog_timer_fn.cold.6+0x62/0x10a [15640.067068] ? __hrtimer_run_queues+0x1c2/0x4d0 [15640.067072] ? watchdog+0x40/0x40 [15640.067077] ? hrtimer_interrupt+0x128/0x2c0 [15640.067081] ? smp_apic_timer_interrupt+0xa6/0x270 [15640.067091] ? apic_timer_interrupt+0xf/0x20 [15640.067096] [15640.067099] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15640.067105] do_raw_spin_lock+0x177/0x190 [15640.067111] _raw_spin_lock+0x21/0x30 [15640.067116] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15640.068927] ? do_raw_spin_unlock+0x75/0x190 [15640.068939] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15640.069195] ? do_wait_intr+0xf0/0xf0 [15640.069203] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15640.075413] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15640.075510] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15640.075517] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15640.075520] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15640.075534] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15640.075538] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15640.075543] RAX: 0000000000007f09 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15640.075545] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [15640.075548] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [15640.075551] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [15640.075553] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [15640.075557] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [15640.075560] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15640.075563] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [15640.075570] Call Trace: [15640.075575] [15640.075581] ? show_regs.cold.9+0x22/0x2f [15640.075590] ? watchdog_timer_fn.cold.6+0x62/0x10a [15640.075596] ? __hrtimer_run_queues+0x1c2/0x4d0 [15640.075600] ? watchdog+0x40/0x40 [15640.075604] ? hrtimer_interrupt+0x128/0x2c0 [15640.075609] ? smp_apic_timer_interrupt+0xa6/0x270 [15640.075616] ? apic_timer_interrupt+0xf/0x20 [15640.075621] [15640.075624] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15640.075629] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15640.075633] do_raw_spin_lock+0x177/0x190 [15640.075638] _raw_spin_lock+0x21/0x30 [15640.075643] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15640.077149] ? do_raw_spin_unlock+0x75/0x190 [15640.077157] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15640.081256] ? do_wait_intr+0xf0/0xf0 [15640.081274] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15640.081844] mdc_reint+0x2d/0x110 [mdc] [15640.081889] mdc_unlink+0x658/0x870 [mdc] [15640.081922] lmv_unlink+0x34d/0xbd0 [lmv] [15640.081952] ll_unlink+0x24d/0x7f0 [lustre] [15640.082086] vfs_unlink+0x202/0x330 [15640.082094] do_unlinkat+0x2fa/0x460 [15640.082100] __x64_sys_unlink+0x37/0x50 [15640.082105] do_syscall_64+0xc1/0x440 [15640.082112] entry_SYSCALL_64_after_hwframe+0x49/0xae [15640.082119] RIP: 0033:0x7f151bdee10b [15640.082126] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [15640.082130] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [15640.082304] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [15640.082307] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [15640.082309] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15640.123221] mdc_close+0x53b/0xd70 [mdc] [15640.123266] lmv_close+0x18d/0x490 [lmv] [15640.123297] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15640.139162] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15640.139171] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15640.727631] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15640.729648] ll_md_real_close+0xa5/0x2a0 [lustre] [15640.738075] ll_md_close+0x274/0x800 [lustre] [15640.741214] ll_file_release+0x128/0xc50 [lustre] [15640.747445] __fput+0x10d/0x440 [15640.750957] ____fput+0x12/0x20 [15640.757105] task_work_run+0xc6/0x110 [15640.761731] exit_to_usermode_loop+0x1dd/0x1f0 [15640.769233] do_syscall_64+0x3d6/0x440 [15640.771548] entry_SYSCALL_64_after_hwframe+0x49/0xae [15640.774686] RIP: 0033:0x7f151bee3cd8 [15640.778133] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15640.802448] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15640.807487] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [15640.817183] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [15640.830131] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15640.843511] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15640.847267] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15644.641053] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13714s! [15644.643755] Showing busy workqueues and worker pools: [15644.646985] workqueue events: flags=0x0 [15644.648242] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15644.651434] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [15644.655112] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15644.658352] pending: kfree_rcu_monitor [15644.660225] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15644.663385] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [15644.668580] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15644.672711] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [15644.684575] workqueue events_long: flags=0x0 [15644.692796] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15644.702808] pending: lu_objects_flush [obdclass] [15644.716071] workqueue events_unbound: flags=0x2 [15644.728472] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15644.735208] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15644.749091] workqueue events_power_efficient: flags=0x80 [15644.755371] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15644.761979] pending: do_cache_clean, neigh_periodic_work, check_lifetime [15644.770146] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15644.774338] pending: neigh_periodic_work [15644.776151] workqueue mm_percpu_wq: flags=0x8 [15644.784369] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15644.791750] pending: vmstat_update, lru_add_drain_per_cpu [15644.795554] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15644.805153] pending: vmstat_update [15644.806894] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15644.808508] pending: lru_add_drain_per_cpu [15644.820545] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [15644.833721] pending: lru_add_drain_per_cpu BAR(48) [15644.839953] workqueue writeback: flags=0x4a [15644.852181] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [15644.858855] pending: wb_workfn [15644.860784] workqueue xprtiod: flags=0xa [15644.862227] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15644.867958] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [15644.872640] workqueue ipv6_addrconf: flags=0x40008 [15644.874858] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15644.884006] pending: addrconf_verify_work [15644.887719] workqueue knbd0-recv: flags=0x1a [15644.893569] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15644.901187] in-flight: 658:recv_work [15644.908081] workqueue ptlrpc_pinger: flags=0x2 [15644.912494] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15644.916782] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15644.923243] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13714s workers=4 idle: 32984 37 8 [15644.934016] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15629s workers=2 idle: 661 [15648.056029] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [15648.059571] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15648.078732] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15648.083110] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15648.086058] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15648.088560] RIP: 0010:native_safe_halt+0x2e/0x50 [15648.090192] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15648.098107] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [15648.101746] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15648.104928] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [15648.107522] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [15648.110517] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [15648.113311] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [15648.116672] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [15648.119978] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15648.122589] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15648.125594] Call Trace: [15648.126542] [15648.127212] ? show_regs.cold.9+0x22/0x2f [15648.129033] ? watchdog_timer_fn.cold.6+0x62/0x10a [15648.131163] ? __hrtimer_run_queues+0x1c2/0x4d0 [15648.132809] ? watchdog+0x40/0x40 [15648.134772] ? hrtimer_interrupt+0x128/0x2c0 [15648.136538] ? smp_apic_timer_interrupt+0xa6/0x270 [15648.139021] ? apic_timer_interrupt+0xf/0x20 [15648.140649] [15648.141637] ? native_safe_halt+0x2e/0x50 [15648.143416] kvm_wait+0xcd/0xe0 [15648.144735] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15648.146924] do_raw_spin_lock+0x177/0x190 [15648.148749] _raw_spin_lock+0x21/0x30 [15648.150148] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15648.153020] process_one_work+0x2c8/0x700 [15648.154741] worker_thread+0x296/0x6e0 [15648.156350] ? rescuer_thread+0x570/0x570 [15648.158224] kthread+0x1d1/0x200 [15648.159432] ? set_kthread_struct+0x70/0x70 [15648.161134] ret_from_fork+0x1f/0x30 [15652.055489] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [15652.063526] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15652.082819] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15652.087978] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15652.091384] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15652.093693] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15652.100915] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15652.103922] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [15652.106889] RDX: 0000000000000222 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [15652.109816] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [15652.112934] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [15652.116166] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15652.119197] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [15652.122394] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15652.124630] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15652.127574] Call Trace: [15652.129219] [15652.130525] ? show_regs.cold.9+0x22/0x2f [15652.133201] ? watchdog_timer_fn.cold.6+0x62/0x10a [15652.134767] ? __hrtimer_run_queues+0x1c2/0x4d0 [15652.136495] ? watchdog+0x40/0x40 [15652.137727] ? hrtimer_interrupt+0x128/0x2c0 [15652.139380] ? smp_apic_timer_interrupt+0xa6/0x270 [15652.140977] ? apic_timer_interrupt+0xf/0x20 [15652.142522] [15652.143583] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15652.145651] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15652.147992] do_raw_spin_lock+0x177/0x190 [15652.149456] _raw_spin_lock+0x21/0x30 [15652.150689] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15652.152529] obd_mod_free+0x46/0x140 [ptlrpc] [15652.154006] mdc_commit_open+0x91/0xb0 [mdc] [15652.155421] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15652.158109] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15652.161231] after_reply+0xe2c/0x1780 [ptlrpc] [15652.163048] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15652.165105] ? __next_timer_interrupt+0x160/0x160 [15652.167019] ptlrpcd+0xa82/0xe40 [ptlrpc] [15652.168810] ? do_wait_intr+0xf0/0xf0 [15652.170157] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15652.172487] kthread+0x1d1/0x200 [15652.173731] ? set_kthread_struct+0x70/0x70 [15652.175073] ret_from_fork+0x1f/0x30 [15664.936067] rcu: INFO: rcu_sched self-detected stall on CPU [15664.939447] rcu: 1-...!: (8195538 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [15664.943540] (t=13740289 jiffies g=249657 q=6589) [15664.946425] rcu: rcu_sched kthread starved for 13735241 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [15664.951200] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [15664.954325] rcu: RCU grace-period kthread stack dump: [15664.956258] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [15664.959769] Call Trace: [15664.960688] __schedule+0x351/0xcb0 [15664.961743] schedule+0xc0/0x180 [15664.962907] schedule_timeout+0xb4/0x190 [15664.964250] ? do_raw_spin_unlock+0x75/0x190 [15664.965783] ? __next_timer_interrupt+0x160/0x160 [15664.967447] ? prepare_to_swait_event+0xf3/0x140 [15664.969161] rcu_gp_kthread+0x8dd/0x15b0 [15664.970528] ? force_qs_rnp+0x2c0/0x2c0 [15664.971886] kthread+0x1d1/0x200 [15664.973249] ? set_kthread_struct+0x70/0x70 [15664.974840] ret_from_fork+0x1f/0x30 [15664.975980] rcu: Stack dump where RCU GP kthread last ran: [15664.977871] Sending NMI from CPU 1 to CPUs 2: [15664.979881] NMI backtrace for cpu 2 skipped: idling at native_safe_halt+0x2e/0x50 [15664.983631] NMI backtrace for cpu 1 [15664.984808] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15664.989945] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15664.993297] Call Trace: [15664.994022] [15664.994895] ? dump_stack+0xbb/0x10e [15664.996290] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [15664.998179] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [15665.000117] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [15665.001960] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [15665.004149] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [15665.006285] ? rcu_dump_cpu_stacks+0x11b/0x171 [15665.008072] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [15665.009833] ? update_process_times+0x6d/0xc0 [15665.011603] ? tick_sched_handle.isra.18+0x27/0x90 [15665.013754] ? tick_sched_timer+0x53/0xb0 [15665.015282] ? __hrtimer_run_queues+0x1c2/0x4d0 [15665.017010] ? tick_sched_handle.isra.18+0x90/0x90 [15665.018737] ? hrtimer_interrupt+0x128/0x2c0 [15665.020263] ? smp_apic_timer_interrupt+0xa6/0x270 [15665.021910] ? apic_timer_interrupt+0xf/0x20 [15665.023562] [15665.024085] ? native_safe_halt+0x2e/0x50 [15665.025692] ? kvm_wait+0xcd/0xe0 [15665.027072] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15665.028757] ? do_raw_spin_lock+0x177/0x190 [15665.030282] ? _raw_spin_lock+0x21/0x30 [15665.031778] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15665.034304] ? obd_mod_free+0x46/0x140 [ptlrpc] [15665.036451] ? mdc_commit_open+0x91/0xb0 [mdc] [15665.038280] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15665.040558] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15665.043307] ? after_reply+0xe2c/0x1780 [ptlrpc] [15665.045631] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15665.047608] ? __next_timer_interrupt+0x160/0x160 [15665.049563] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [15665.051561] ? do_wait_intr+0xf0/0xf0 [15665.053056] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15665.055409] ? kthread+0x1d1/0x200 [15665.056832] ? set_kthread_struct+0x70/0x70 [15665.058568] ? ret_from_fork+0x1f/0x30 [15668.058903] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [15668.065876] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15668.116184] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15668.126211] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15668.136042] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15668.138953] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15668.149196] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15668.151591] RAX: 0000000000007e19 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15668.153618] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [15668.159016] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [15668.163122] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [15668.168106] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [15668.172210] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [15668.177173] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15668.179434] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [15668.181898] Call Trace: [15668.182547] [15668.183270] ? show_regs.cold.9+0x22/0x2f [15668.185200] ? watchdog_timer_fn.cold.6+0x62/0x10a [15668.187843] ? __hrtimer_run_queues+0x1c2/0x4d0 [15668.189348] ? watchdog+0x40/0x40 [15668.191696] ? hrtimer_interrupt+0x128/0x2c0 [15668.194247] ? smp_apic_timer_interrupt+0xa6/0x270 [15668.197221] ? apic_timer_interrupt+0xf/0x20 [15668.201273] [15668.201920] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15668.203673] do_raw_spin_lock+0x177/0x190 [15668.204837] _raw_spin_lock+0x21/0x30 [15668.205973] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15668.207865] ? do_raw_spin_unlock+0x75/0x190 [15668.210921] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15668.212971] ? do_wait_intr+0xf0/0xf0 [15668.214618] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15668.216663] mdc_close+0x53b/0xd70 [mdc] [15668.223661] lmv_close+0x18d/0x490 [lmv] [15668.225594] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15668.229595] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15668.232668] ll_md_real_close+0xa5/0x2a0 [lustre] [15668.235147] ll_md_close+0x274/0x800 [lustre] [15668.239225] ll_file_release+0x128/0xc50 [lustre] [15668.241604] __fput+0x10d/0x440 [15668.244173] ____fput+0x12/0x20 [15668.246671] task_work_run+0xc6/0x110 [15668.248792] exit_to_usermode_loop+0x1dd/0x1f0 [15668.251117] do_syscall_64+0x3d6/0x440 [15668.253294] entry_SYSCALL_64_after_hwframe+0x49/0xae [15668.255904] RIP: 0033:0x7f151bee3cd8 [15668.257714] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15668.265458] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15668.271683] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [15668.277940] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [15668.280613] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15668.285831] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15668.289243] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15672.052094] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [15672.055288] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15672.074904] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15672.079767] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15672.083621] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15672.086304] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15672.094295] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [15672.097538] RAX: 0000000000001d02 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15672.100294] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [15672.102985] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [15672.105830] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [15672.108558] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [15672.111459] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [15672.114424] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15672.116504] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [15672.119774] Call Trace: [15672.120884] [15672.121461] ? show_regs.cold.9+0x22/0x2f [15672.123158] ? watchdog_timer_fn.cold.6+0x62/0x10a [15672.124935] ? __hrtimer_run_queues+0x1c2/0x4d0 [15672.126508] ? watchdog+0x40/0x40 [15672.128066] ? hrtimer_interrupt+0x128/0x2c0 [15672.129453] ? smp_apic_timer_interrupt+0xa6/0x270 [15672.131359] ? apic_timer_interrupt+0xf/0x20 [15672.133420] [15672.134448] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15672.136974] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15672.139001] do_raw_spin_lock+0x177/0x190 [15672.140508] _raw_spin_lock+0x21/0x30 [15672.142244] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15672.144353] ? do_raw_spin_unlock+0x75/0x190 [15672.146186] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15672.148893] ? do_wait_intr+0xf0/0xf0 [15672.150342] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15672.152558] mdc_reint+0x2d/0x110 [mdc] [15672.154264] mdc_unlink+0x658/0x870 [mdc] [15672.156316] lmv_unlink+0x34d/0xbd0 [lmv] [15672.157998] ll_unlink+0x24d/0x7f0 [lustre] [15672.160009] vfs_unlink+0x202/0x330 [15672.161455] do_unlinkat+0x2fa/0x460 [15672.163126] __x64_sys_unlink+0x37/0x50 [15672.164568] do_syscall_64+0xc1/0x440 [15672.165895] entry_SYSCALL_64_after_hwframe+0x49/0xae [15672.167822] RIP: 0033:0x7f151bdee10b [15672.169395] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [15672.176776] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [15672.179577] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [15672.182516] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [15672.185050] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15672.189067] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15672.192249] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15675.362574] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13744s! [15675.372320] Showing busy workqueues and worker pools: [15675.374746] workqueue events: flags=0x0 [15675.380258] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15675.387079] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [15675.395884] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15675.401585] pending: kfree_rcu_monitor [15675.403020] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15675.409559] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [15675.416644] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15675.423060] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [15675.430102] workqueue events_long: flags=0x0 [15675.435103] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15675.441939] pending: lu_objects_flush [obdclass] [15675.445780] workqueue events_unbound: flags=0x2 [15675.449833] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15675.456827] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15675.467569] workqueue events_power_efficient: flags=0x80 [15675.471071] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15675.477941] pending: do_cache_clean, neigh_periodic_work, check_lifetime [15675.484863] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15675.493084] pending: neigh_periodic_work [15675.494976] workqueue mm_percpu_wq: flags=0x8 [15675.500647] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15675.506691] pending: vmstat_update, lru_add_drain_per_cpu [15675.510933] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15675.517825] pending: vmstat_update [15675.522282] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15675.530933] pending: lru_add_drain_per_cpu [15675.532949] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [15675.538835] pending: lru_add_drain_per_cpu BAR(48) [15675.543759] workqueue writeback: flags=0x4a [15675.547262] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [15675.553585] pending: wb_workfn [15675.555794] workqueue xprtiod: flags=0xa [15675.560638] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15675.568130] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [15675.575423] workqueue ipv6_addrconf: flags=0x40008 [15675.581387] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15675.585598] pending: addrconf_verify_work [15675.587681] workqueue knbd0-recv: flags=0x1a [15675.596452] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15675.600077] in-flight: 658:recv_work [15675.601919] workqueue ptlrpc_pinger: flags=0x2 [15675.605243] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15675.608245] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15675.611279] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13745s workers=4 idle: 32984 37 8 [15675.614398] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15660s workers=2 idle: 661 [15692.056558] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [15692.058779] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15692.060577] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [15692.102120] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15692.102136] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15692.102144] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15692.102505] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15692.102516] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15692.102521] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15692.102526] RAX: 000000000000730f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15692.102528] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [15692.102531] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [15692.102533] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [15692.102536] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [15692.102539] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [15692.102543] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15692.102546] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15692.102554] Call Trace: [15692.102560] [15692.102566] ? show_regs.cold.9+0x22/0x2f [15692.102575] ? watchdog_timer_fn.cold.6+0x62/0x10a [15692.102582] ? __hrtimer_run_queues+0x1c2/0x4d0 [15692.102586] ? watchdog+0x40/0x40 [15692.114792] Modules linked in: [15692.122178] ? hrtimer_interrupt+0x128/0x2c0 [15692.127465] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15692.127548] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15692.127555] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15692.127559] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15692.127573] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15692.127578] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15692.127583] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [15692.127585] RDX: 0000000000001e84 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [15692.127588] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [15692.127590] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [15692.127593] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15692.127596] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [15692.127599] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15692.127601] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15692.127610] Call Trace: [15692.127616] [15692.127622] ? show_regs.cold.9+0x22/0x2f [15692.127633] ? watchdog_timer_fn.cold.6+0x62/0x10a [15692.127640] ? __hrtimer_run_queues+0x1c2/0x4d0 [15692.127644] ? watchdog+0x40/0x40 [15692.127648] ? hrtimer_interrupt+0x128/0x2c0 [15692.127652] ? smp_apic_timer_interrupt+0xa6/0x270 [15692.127663] ? apic_timer_interrupt+0xf/0x20 [15692.127668] [15692.127671] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15692.127676] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15692.127681] do_raw_spin_lock+0x177/0x190 [15692.127686] _raw_spin_lock+0x21/0x30 [15692.127693] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15692.128003] obd_mod_free+0x46/0x140 [ptlrpc] [15692.128206] mdc_commit_open+0x91/0xb0 [mdc] [15692.128249] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15692.128445] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15692.128642] after_reply+0xe2c/0x1780 [ptlrpc] [15692.129195] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15692.129389] ? __next_timer_interrupt+0x160/0x160 [15692.129400] ptlrpcd+0xa82/0xe40 [ptlrpc] [15692.130309] ? do_wait_intr+0xf0/0xf0 [15692.130330] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15692.144935] ? smp_apic_timer_interrupt+0xa6/0x270 [15692.144968] ? apic_timer_interrupt+0xf/0x20 [15692.144974] [15692.144979] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15692.144986] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15692.144994] do_raw_spin_lock+0x177/0x190 [15692.145000] _raw_spin_lock+0x21/0x30 [15692.145005] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15692.149287] kthread+0x1d1/0x200 [15692.149305] ? set_kthread_struct+0x70/0x70 [15692.149310] ret_from_fork+0x1f/0x30 [15692.649085] process_one_work+0x2c8/0x700 [15692.651240] worker_thread+0x296/0x6e0 [15692.652395] ? rescuer_thread+0x570/0x570 [15692.658643] kthread+0x1d1/0x200 [15692.662108] ? set_kthread_struct+0x70/0x70 [15692.665463] ret_from_fork+0x1f/0x30 [15696.060924] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [15696.066877] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15696.129762] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15696.140722] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15696.151620] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15696.162594] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15696.192637] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15696.197688] RAX: 0000000000001718 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15696.202369] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [15696.210597] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [15696.214533] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [15696.216901] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [15696.219840] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [15696.223118] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15696.225731] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [15696.228348] Call Trace: [15696.229358] [15696.230331] ? show_regs.cold.9+0x22/0x2f [15696.232130] ? watchdog_timer_fn.cold.6+0x62/0x10a [15696.234282] ? __hrtimer_run_queues+0x1c2/0x4d0 [15696.236492] ? watchdog+0x40/0x40 [15696.238012] ? hrtimer_interrupt+0x128/0x2c0 [15696.239924] ? smp_apic_timer_interrupt+0xa6/0x270 [15696.241989] ? apic_timer_interrupt+0xf/0x20 [15696.243975] [15696.244938] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15696.247226] do_raw_spin_lock+0x177/0x190 [15696.249072] _raw_spin_lock+0x21/0x30 [15696.250683] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15696.253247] ? do_raw_spin_unlock+0x75/0x190 [15696.255051] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15696.257382] ? do_wait_intr+0xf0/0xf0 [15696.258583] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15696.260329] mdc_close+0x53b/0xd70 [mdc] [15696.261584] lmv_close+0x18d/0x490 [lmv] [15696.262702] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15696.264371] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15696.266136] ll_md_real_close+0xa5/0x2a0 [lustre] [15696.267572] ll_md_close+0x274/0x800 [lustre] [15696.268772] ll_file_release+0x128/0xc50 [lustre] [15696.270157] __fput+0x10d/0x440 [15696.271043] ____fput+0x12/0x20 [15696.271948] task_work_run+0xc6/0x110 [15696.272779] exit_to_usermode_loop+0x1dd/0x1f0 [15696.274080] do_syscall_64+0x3d6/0x440 [15696.275183] entry_SYSCALL_64_after_hwframe+0x49/0xae [15696.276391] RIP: 0033:0x7f151bee3cd8 [15696.277332] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15696.282405] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15696.284369] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [15696.286236] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [15696.288157] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15696.289978] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15696.292121] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15700.055981] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [15700.070919] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15700.122350] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15700.147046] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15700.163230] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15700.173055] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15700.201362] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15700.212121] RAX: 0000000000005998 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15700.223200] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [15700.234168] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [15700.242602] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [15700.252933] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [15700.264253] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [15700.276366] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15700.285590] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [15700.296857] Call Trace: [15700.299027] [15700.303991] ? show_regs.cold.9+0x22/0x2f [15700.308299] ? watchdog_timer_fn.cold.6+0x62/0x10a [15700.315454] ? __hrtimer_run_queues+0x1c2/0x4d0 [15700.321159] ? watchdog+0x40/0x40 [15700.325647] ? hrtimer_interrupt+0x128/0x2c0 [15700.330961] ? smp_apic_timer_interrupt+0xa6/0x270 [15700.338327] ? apic_timer_interrupt+0xf/0x20 [15700.342286] [15700.345439] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15700.348284] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15700.357660] do_raw_spin_lock+0x177/0x190 [15700.360008] _raw_spin_lock+0x21/0x30 [15700.367280] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15700.372515] ? do_raw_spin_unlock+0x75/0x190 [15700.383864] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15700.391068] ? do_wait_intr+0xf0/0xf0 [15700.402399] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15700.409498] mdc_reint+0x2d/0x110 [mdc] [15700.411946] mdc_unlink+0x658/0x870 [mdc] [15700.420490] lmv_unlink+0x34d/0xbd0 [lmv] [15700.428592] ll_unlink+0x24d/0x7f0 [lustre] [15700.440854] vfs_unlink+0x202/0x330 [15700.447434] do_unlinkat+0x2fa/0x460 [15700.452681] __x64_sys_unlink+0x37/0x50 [15700.461009] do_syscall_64+0xc1/0x440 [15700.469033] entry_SYSCALL_64_after_hwframe+0x49/0xae [15700.473616] RIP: 0033:0x7f151bdee10b [15700.475447] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [15700.492862] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [15700.504529] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [15700.514546] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [15700.521242] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15700.535361] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15700.548405] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15706.084450] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13775s! [15706.100836] Showing busy workqueues and worker pools: [15706.111404] workqueue events: flags=0x0 [15706.114152] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15706.116245] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [15706.137919] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15706.146239] pending: kfree_rcu_monitor [15706.154351] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15706.161501] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [15706.180708] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15706.191310] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [15706.202055] workqueue events_long: flags=0x0 [15706.208641] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15706.220408] pending: lu_objects_flush [obdclass] [15706.229364] workqueue events_unbound: flags=0x2 [15706.231799] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15706.245707] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15706.257679] workqueue events_power_efficient: flags=0x80 [15706.265513] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15706.271317] pending: do_cache_clean, neigh_periodic_work, check_lifetime [15706.280466] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15706.289593] pending: neigh_periodic_work [15706.294652] workqueue mm_percpu_wq: flags=0x8 [15706.299453] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15706.306652] pending: vmstat_update, lru_add_drain_per_cpu [15706.319489] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15706.334803] pending: vmstat_update [15706.339540] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15706.348897] pending: lru_add_drain_per_cpu [15706.357449] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [15706.374805] pending: lru_add_drain_per_cpu BAR(48) [15706.386392] workqueue writeback: flags=0x4a [15706.390457] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [15706.395447] pending: wb_workfn [15706.399759] workqueue xprtiod: flags=0xa [15706.402303] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15706.406631] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [15706.412112] workqueue ipv6_addrconf: flags=0x40008 [15706.419435] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15706.423173] pending: addrconf_verify_work [15706.431086] workqueue knbd0-recv: flags=0x1a [15706.438862] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15706.450334] in-flight: 658:recv_work [15706.460010] workqueue ptlrpc_pinger: flags=0x2 [15706.465429] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15706.480553] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15706.494152] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13775s workers=4 idle: 32984 37 8 [15706.503491] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15691s workers=2 idle: 661 [15720.055021] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [15720.056030] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [15720.058266] Modules linked in: [15720.063841] Modules linked in: [15720.063843] lustre(O) osp(O) ofd(O) lod(O) [15720.066374] lustre(O) [15720.070281] mdt(O) [15720.075966] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15720.076045] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15720.076053] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15720.076059] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15720.077055] mdd(O) [15720.077814] [15720.077819] RIP: 0010:native_safe_halt+0x2e/0x50 [15720.115400] mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic [15720.125466] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15720.130864] crct10dif_pclmul [15720.139936] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 [15720.144529] crc32_pclmul [15720.150032] ORIG_RAX: ffffffffffffff13 [15720.150043] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15720.150046] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [15720.150048] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [15720.150050] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [15720.150052] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [15720.150056] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [15720.150058] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15720.150060] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15720.150068] Call Trace: [15720.150076] [15720.150082] ? show_regs.cold.9+0x22/0x2f [15720.150096] ? watchdog_timer_fn.cold.6+0x62/0x10a [15720.150102] ? __hrtimer_run_queues+0x1c2/0x4d0 [15720.150106] ? watchdog+0x40/0x40 [15720.150111] ? hrtimer_interrupt+0x128/0x2c0 [15720.150115] ? smp_apic_timer_interrupt+0xa6/0x270 [15720.150124] ? apic_timer_interrupt+0xf/0x20 [15720.150129] [15720.150132] ? native_safe_halt+0x2e/0x50 [15720.150136] kvm_wait+0xcd/0xe0 [15720.150142] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15720.150151] do_raw_spin_lock+0x177/0x190 [15720.150156] _raw_spin_lock+0x21/0x30 [15720.150160] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15720.150710] process_one_work+0x2c8/0x700 [15720.150718] worker_thread+0x296/0x6e0 [15720.150722] ? rescuer_thread+0x570/0x570 [15720.150726] kthread+0x1d1/0x200 [15720.150731] ? set_kthread_struct+0x70/0x70 [15720.150734] ret_from_fork+0x1f/0x30 [15720.314528] crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15720.325716] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15720.334239] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15720.347396] RIP: 0010:native_safe_halt+0x2e/0x50 [15720.349541] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15720.360254] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15720.366712] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [15720.371108] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [15720.380425] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [15720.385322] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [15720.389954] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15720.395389] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [15720.399877] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15720.404481] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15720.410455] Call Trace: [15720.417164] [15720.418158] ? show_regs.cold.9+0x22/0x2f [15720.421984] ? watchdog_timer_fn.cold.6+0x62/0x10a [15720.424195] ? __hrtimer_run_queues+0x1c2/0x4d0 [15720.430410] ? watchdog+0x40/0x40 [15720.432542] ? hrtimer_interrupt+0x128/0x2c0 [15720.435028] ? smp_apic_timer_interrupt+0xa6/0x270 [15720.440466] ? apic_timer_interrupt+0xf/0x20 [15720.444434] [15720.449144] ? native_safe_halt+0x2e/0x50 [15720.453124] kvm_wait+0xcd/0xe0 [15720.456142] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15720.459985] do_raw_spin_lock+0x177/0x190 [15720.462121] _raw_spin_lock+0x21/0x30 [15720.463501] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15720.466151] obd_mod_free+0x46/0x140 [ptlrpc] [15720.468322] mdc_commit_open+0x91/0xb0 [mdc] [15720.470147] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15720.474755] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15720.480462] after_reply+0xe2c/0x1780 [ptlrpc] [15720.482599] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15720.485390] ? __next_timer_interrupt+0x160/0x160 [15720.487206] ptlrpcd+0xa82/0xe40 [ptlrpc] [15720.490377] ? do_wait_intr+0xf0/0xf0 [15720.491730] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15720.495177] kthread+0x1d1/0x200 [15720.496866] ? set_kthread_struct+0x70/0x70 [15720.499059] ret_from_fork+0x1f/0x30 [15724.057551] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [15724.061757] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15724.084629] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15724.091015] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15724.098784] RIP: 0010:native_safe_halt+0x2e/0x50 [15724.101599] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15724.113736] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15724.116750] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15724.119665] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [15724.122821] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [15724.125974] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [15724.129154] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [15724.132044] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [15724.135771] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15724.138186] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [15724.141259] Call Trace: [15724.142639] [15724.143897] ? show_regs.cold.9+0x22/0x2f [15724.145810] ? watchdog_timer_fn.cold.6+0x62/0x10a [15724.147700] ? __hrtimer_run_queues+0x1c2/0x4d0 [15724.149748] ? watchdog+0x40/0x40 [15724.151319] ? hrtimer_interrupt+0x128/0x2c0 [15724.153768] ? smp_apic_timer_interrupt+0xa6/0x270 [15724.155639] ? apic_timer_interrupt+0xf/0x20 [15724.157446] [15724.158274] ? native_safe_halt+0x2e/0x50 [15724.159923] kvm_wait+0xcd/0xe0 [15724.162005] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15724.165027] do_raw_spin_lock+0x177/0x190 [15724.166674] _raw_spin_lock+0x21/0x30 [15724.168845] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15724.171105] ? do_raw_spin_unlock+0x75/0x190 [15724.174237] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15724.177271] ? do_wait_intr+0xf0/0xf0 [15724.178835] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15724.181124] mdc_close+0x53b/0xd70 [mdc] [15724.182928] lmv_close+0x18d/0x490 [lmv] [15724.185694] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15724.188304] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15724.191190] ll_md_real_close+0xa5/0x2a0 [lustre] [15724.193965] ll_md_close+0x274/0x800 [lustre] [15724.196460] ll_file_release+0x128/0xc50 [lustre] [15724.198711] __fput+0x10d/0x440 [15724.200329] ____fput+0x12/0x20 [15724.201887] task_work_run+0xc6/0x110 [15724.203586] exit_to_usermode_loop+0x1dd/0x1f0 [15724.205434] do_syscall_64+0x3d6/0x440 [15724.207260] entry_SYSCALL_64_after_hwframe+0x49/0xae [15724.209447] RIP: 0033:0x7f151bee3cd8 [15724.211185] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15724.219371] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15724.222496] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [15724.225313] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [15724.228472] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15724.231668] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15724.234421] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15732.052060] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [15732.057221] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15732.093545] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15732.099642] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15732.103875] RIP: 0010:native_safe_halt+0x2e/0x50 [15732.106765] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15732.115436] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15732.119019] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15732.123529] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [15732.127057] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [15732.132843] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [15732.136132] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [15732.139546] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [15732.145758] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15732.149023] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [15732.152646] Call Trace: [15732.154341] [15732.155781] ? show_regs.cold.9+0x22/0x2f [15732.157267] ? watchdog_timer_fn.cold.6+0x62/0x10a [15732.159072] ? __hrtimer_run_queues+0x1c2/0x4d0 [15732.160325] ? watchdog+0x40/0x40 [15732.162299] ? hrtimer_interrupt+0x128/0x2c0 [15732.164930] ? smp_apic_timer_interrupt+0xa6/0x270 [15732.168808] ? apic_timer_interrupt+0xf/0x20 [15732.171408] [15732.172355] ? native_safe_halt+0x2e/0x50 [15732.174916] kvm_wait+0xcd/0xe0 [15732.176756] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15732.179835] do_raw_spin_lock+0x177/0x190 [15732.182300] _raw_spin_lock+0x21/0x30 [15732.183871] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15732.187830] ? do_raw_spin_unlock+0x75/0x190 [15732.190339] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15732.192825] ? do_wait_intr+0xf0/0xf0 [15732.194993] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15732.196891] mdc_reint+0x2d/0x110 [mdc] [15732.198819] mdc_unlink+0x658/0x870 [mdc] [15732.202742] lmv_unlink+0x34d/0xbd0 [lmv] [15732.205553] ll_unlink+0x24d/0x7f0 [lustre] [15732.207023] vfs_unlink+0x202/0x330 [15732.208306] do_unlinkat+0x2fa/0x460 [15732.210934] __x64_sys_unlink+0x37/0x50 [15732.212918] do_syscall_64+0xc1/0x440 [15732.214716] entry_SYSCALL_64_after_hwframe+0x49/0xae [15732.217046] RIP: 0033:0x7f151bdee10b [15732.218411] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [15732.228779] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [15732.231884] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [15732.235209] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [15732.240488] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15732.246088] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15732.249909] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15736.801745] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13806s! [15736.813631] Showing busy workqueues and worker pools: [15736.818222] workqueue events: flags=0x0 [15736.821806] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15736.826287] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [15736.830780] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15736.836082] pending: kfree_rcu_monitor [15736.840109] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15736.849760] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [15736.862854] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15736.866724] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [15736.871842] workqueue events_long: flags=0x0 [15736.876092] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15736.882681] pending: lu_objects_flush [obdclass] [15736.890191] workqueue events_unbound: flags=0x2 [15736.892962] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15736.895112] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15736.907893] workqueue events_power_efficient: flags=0x80 [15736.911365] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15736.914980] pending: do_cache_clean, neigh_periodic_work, check_lifetime [15736.920728] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15736.928111] pending: neigh_periodic_work [15736.930990] workqueue mm_percpu_wq: flags=0x8 [15736.933513] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15736.936439] pending: vmstat_update, lru_add_drain_per_cpu [15736.942101] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15736.945058] pending: vmstat_update [15736.950849] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15736.953192] pending: lru_add_drain_per_cpu [15736.960308] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [15736.967183] pending: lru_add_drain_per_cpu BAR(48) [15736.977932] workqueue writeback: flags=0x4a [15736.980100] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [15736.982446] pending: wb_workfn [15736.984199] workqueue xprtiod: flags=0xa [15736.986340] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15736.989161] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [15736.998673] workqueue ipv6_addrconf: flags=0x40008 [15737.001148] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15737.004226] pending: addrconf_verify_work [15737.006172] workqueue knbd0-recv: flags=0x1a [15737.008262] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15737.014177] in-flight: 658:recv_work [15737.016849] workqueue ptlrpc_pinger: flags=0x2 [15737.025376] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15737.028804] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15737.033935] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13806s workers=4 idle: 32984 37 8 [15737.039711] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15721s workers=2 idle: 661 [15748.055577] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [15748.057246] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [15748.058772] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15748.058859] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15748.058866] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15748.059050] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15748.059064] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15748.059069] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15748.059073] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [15748.059076] RDX: 0000000000006a13 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [15748.059078] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [15748.059080] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [15748.059083] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15748.059086] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [15748.059088] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15748.059091] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15748.059099] Call Trace: [15748.059105] [15748.059110] ? show_regs.cold.9+0x22/0x2f [15748.059120] ? watchdog_timer_fn.cold.6+0x62/0x10a [15748.059126] ? __hrtimer_run_queues+0x1c2/0x4d0 [15748.059131] ? watchdog+0x40/0x40 [15748.059135] ? hrtimer_interrupt+0x128/0x2c0 [15748.059139] ? smp_apic_timer_interrupt+0xa6/0x270 [15748.059147] ? apic_timer_interrupt+0xf/0x20 [15748.059152] [15748.059155] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15748.059159] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15748.059164] do_raw_spin_lock+0x177/0x190 [15748.059169] _raw_spin_lock+0x21/0x30 [15748.059175] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15748.059470] obd_mod_free+0x46/0x140 [ptlrpc] [15748.059778] mdc_commit_open+0x91/0xb0 [mdc] [15748.059823] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15748.060121] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15748.060332] after_reply+0xe2c/0x1780 [ptlrpc] [15748.060560] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15748.061014] ? __next_timer_interrupt+0x160/0x160 [15748.061024] ptlrpcd+0xa82/0xe40 [ptlrpc] [15748.061340] ? do_wait_intr+0xf0/0xf0 [15748.061349] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15748.065358] Modules linked in: [15748.089605] kthread+0x1d1/0x200 [15748.089629] ? set_kthread_struct+0x70/0x70 [15748.089634] ret_from_fork+0x1f/0x30 [15748.231488] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15748.251890] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15748.258109] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15748.261718] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15748.264596] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15748.267360] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15748.274977] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [15748.277886] RAX: 00000000000020f0 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15748.280492] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [15748.283002] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [15748.285698] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [15748.288361] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [15748.291393] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [15748.294056] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15748.296010] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15748.298104] Call Trace: [15748.298979] [15748.299672] ? show_regs.cold.9+0x22/0x2f [15748.300842] ? watchdog_timer_fn.cold.6+0x62/0x10a [15748.302123] ? __hrtimer_run_queues+0x1c2/0x4d0 [15748.303672] ? watchdog+0x40/0x40 [15748.304698] ? hrtimer_interrupt+0x128/0x2c0 [15748.305907] ? smp_apic_timer_interrupt+0xa6/0x270 [15748.307335] ? apic_timer_interrupt+0xf/0x20 [15748.308800] [15748.309364] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15748.311063] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15748.313095] do_raw_spin_lock+0x177/0x190 [15748.314352] _raw_spin_lock+0x21/0x30 [15748.315534] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15748.317874] process_one_work+0x2c8/0x700 [15748.319154] worker_thread+0x296/0x6e0 [15748.320245] ? rescuer_thread+0x570/0x570 [15748.321874] kthread+0x1d1/0x200 [15748.322921] ? set_kthread_struct+0x70/0x70 [15748.324635] ret_from_fork+0x1f/0x30 [15752.058136] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [15752.062877] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15752.083265] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15752.089997] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15752.093135] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15752.097748] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15752.109028] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15752.115770] RAX: 0000000000004731 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15752.119689] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [15752.124766] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [15752.127948] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [15752.134001] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [15752.137123] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [15752.140053] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15752.142565] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [15752.146444] Call Trace: [15752.147171] [15752.147928] ? show_regs.cold.9+0x22/0x2f [15752.149405] ? watchdog_timer_fn.cold.6+0x62/0x10a [15752.151224] ? __hrtimer_run_queues+0x1c2/0x4d0 [15752.153854] ? watchdog+0x40/0x40 [15752.155473] ? hrtimer_interrupt+0x128/0x2c0 [15752.157493] ? smp_apic_timer_interrupt+0xa6/0x270 [15752.159210] ? apic_timer_interrupt+0xf/0x20 [15752.160748] [15752.161490] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15752.163909] do_raw_spin_lock+0x177/0x190 [15752.165313] _raw_spin_lock+0x21/0x30 [15752.166456] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15752.169105] ? do_raw_spin_unlock+0x75/0x190 [15752.170740] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15752.173102] ? do_wait_intr+0xf0/0xf0 [15752.175668] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15752.177881] mdc_close+0x53b/0xd70 [mdc] [15752.179694] lmv_close+0x18d/0x490 [lmv] [15752.182721] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15752.185563] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15752.187951] ll_md_real_close+0xa5/0x2a0 [lustre] [15752.190003] ll_md_close+0x274/0x800 [lustre] [15752.192036] ll_file_release+0x128/0xc50 [lustre] [15752.193713] __fput+0x10d/0x440 [15752.194917] ____fput+0x12/0x20 [15752.196528] task_work_run+0xc6/0x110 [15752.197938] exit_to_usermode_loop+0x1dd/0x1f0 [15752.199348] do_syscall_64+0x3d6/0x440 [15752.200714] entry_SYSCALL_64_after_hwframe+0x49/0xae [15752.202987] RIP: 0033:0x7f151bee3cd8 [15752.204251] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15752.211142] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15752.214092] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [15752.216738] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [15752.219112] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15752.221524] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15752.224024] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15764.058350] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [15764.073501] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15764.136938] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15764.160249] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15764.177957] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15764.179807] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15764.204942] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15764.217669] RAX: 0000000000005444 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15764.225297] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [15764.239480] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [15764.243774] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [15764.252985] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [15764.256640] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [15764.260747] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15764.263997] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [15764.281787] Call Trace: [15764.286701] [15764.292638] ? show_regs.cold.9+0x22/0x2f [15764.296670] ? watchdog_timer_fn.cold.6+0x62/0x10a [15764.303355] ? __hrtimer_run_queues+0x1c2/0x4d0 [15764.305003] ? watchdog+0x40/0x40 [15764.306199] ? hrtimer_interrupt+0x128/0x2c0 [15764.315727] ? smp_apic_timer_interrupt+0xa6/0x270 [15764.318748] ? apic_timer_interrupt+0xf/0x20 [15764.327499] [15764.331865] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15764.334928] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15764.346716] do_raw_spin_lock+0x177/0x190 [15764.348790] _raw_spin_lock+0x21/0x30 [15764.352387] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15764.365974] ? do_raw_spin_unlock+0x75/0x190 [15764.373945] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15764.385748] ? do_wait_intr+0xf0/0xf0 [15764.389426] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15764.395965] mdc_reint+0x2d/0x110 [mdc] [15764.397318] mdc_unlink+0x658/0x870 [mdc] [15764.402071] lmv_unlink+0x34d/0xbd0 [lmv] [15764.405237] ll_unlink+0x24d/0x7f0 [lustre] [15764.414443] vfs_unlink+0x202/0x330 [15764.419853] do_unlinkat+0x2fa/0x460 [15764.429027] __x64_sys_unlink+0x37/0x50 [15764.431625] do_syscall_64+0xc1/0x440 [15764.436754] entry_SYSCALL_64_after_hwframe+0x49/0xae [15764.443677] RIP: 0033:0x7f151bdee10b [15764.448766] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [15764.477982] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [15764.487759] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [15764.500846] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [15764.503001] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15764.514208] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15764.524759] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15767.521057] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13836s! [15767.531738] Showing busy workqueues and worker pools: [15767.535900] workqueue events: flags=0x0 [15767.539378] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15767.547096] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [15767.554801] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15767.559894] pending: kfree_rcu_monitor [15767.568326] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15767.575039] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [15767.581014] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15767.585537] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [15767.593505] workqueue events_long: flags=0x0 [15767.599116] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15767.602739] pending: lu_objects_flush [obdclass] [15767.608158] workqueue events_unbound: flags=0x2 [15767.610748] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15767.614208] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15767.626771] workqueue events_power_efficient: flags=0x80 [15767.632848] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15767.640170] pending: do_cache_clean, neigh_periodic_work, check_lifetime [15767.647335] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15767.653453] pending: neigh_periodic_work [15767.665315] workqueue mm_percpu_wq: flags=0x8 [15767.671062] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15767.678741] pending: vmstat_update, lru_add_drain_per_cpu [15767.683282] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15767.686011] pending: vmstat_update [15767.688012] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15767.691036] pending: lru_add_drain_per_cpu [15767.693369] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [15767.696277] pending: lru_add_drain_per_cpu BAR(48) [15767.698438] workqueue writeback: flags=0x4a [15767.700492] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [15767.702950] pending: wb_workfn [15767.705573] workqueue xprtiod: flags=0xa [15767.707037] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15767.710864] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [15767.713982] workqueue ipv6_addrconf: flags=0x40008 [15767.716599] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15767.719736] pending: addrconf_verify_work [15767.725524] workqueue knbd0-recv: flags=0x1a [15767.728013] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15767.731279] in-flight: 658:recv_work [15767.733141] workqueue ptlrpc_pinger: flags=0x2 [15767.734629] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15767.739552] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15767.742527] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13837s workers=4 idle: 32984 37 8 [15767.746580] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15752s workers=2 idle: 661 [15776.056674] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [15776.063101] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [15776.064423] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15776.064499] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15776.064507] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15776.064510] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15776.064524] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15776.064528] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15776.064533] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [15776.064535] RDX: 000000000000689b RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [15776.064538] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [15776.064541] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [15776.064543] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15776.064547] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [15776.064549] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15776.064552] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15776.064561] Call Trace: [15776.064567] [15776.064573] ? show_regs.cold.9+0x22/0x2f [15776.064583] ? watchdog_timer_fn.cold.6+0x62/0x10a [15776.064589] ? __hrtimer_run_queues+0x1c2/0x4d0 [15776.064594] ? watchdog+0x40/0x40 [15776.064598] ? hrtimer_interrupt+0x128/0x2c0 [15776.064602] ? smp_apic_timer_interrupt+0xa6/0x270 [15776.064610] ? apic_timer_interrupt+0xf/0x20 [15776.064616] [15776.064620] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15776.064625] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15776.064630] do_raw_spin_lock+0x177/0x190 [15776.064634] _raw_spin_lock+0x21/0x30 [15776.064640] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15776.064918] obd_mod_free+0x46/0x140 [ptlrpc] [15776.080400] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl [15776.128822] mdc_commit_open+0x91/0xb0 [mdc] [15776.142326] i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15776.142358] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15776.142366] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15776.142372] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15776.142915] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15776.142926] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15776.142930] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15776.142935] RAX: 0000000000005314 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15776.142937] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [15776.142940] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [15776.142942] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [15776.142945] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [15776.142948] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [15776.142952] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15776.142954] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15776.142962] Call Trace: [15776.142967] [15776.142982] ? show_regs.cold.9+0x22/0x2f [15776.142991] ? watchdog_timer_fn.cold.6+0x62/0x10a [15776.143000] ? __hrtimer_run_queues+0x1c2/0x4d0 [15776.143010] ? watchdog+0x40/0x40 [15776.143014] ? hrtimer_interrupt+0x128/0x2c0 [15776.143018] ? smp_apic_timer_interrupt+0xa6/0x270 [15776.143032] ? apic_timer_interrupt+0xf/0x20 [15776.143037] [15776.143041] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15776.143049] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15776.143054] do_raw_spin_lock+0x177/0x190 [15776.143058] _raw_spin_lock+0x21/0x30 [15776.143063] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15776.143370] process_one_work+0x2c8/0x700 [15776.143376] worker_thread+0x296/0x6e0 [15776.143381] ? rescuer_thread+0x570/0x570 [15776.143385] kthread+0x1d1/0x200 [15776.143388] ? set_kthread_struct+0x70/0x70 [15776.143392] ret_from_fork+0x1f/0x30 [15776.588314] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15776.592958] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15776.596816] after_reply+0xe2c/0x1780 [ptlrpc] [15776.598854] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15776.604907] ? __next_timer_interrupt+0x160/0x160 [15776.608703] ptlrpcd+0xa82/0xe40 [ptlrpc] [15776.612880] ? do_wait_intr+0xf0/0xf0 [15776.614686] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15776.616828] kthread+0x1d1/0x200 [15776.618089] ? set_kthread_struct+0x70/0x70 [15776.624661] ret_from_fork+0x1f/0x30 [15780.073935] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [15780.078549] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15780.110317] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15780.116641] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15780.121775] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15780.125507] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15780.142521] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15780.146955] RAX: 0000000000007a6d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15780.151059] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [15780.156027] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [15780.161496] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [15780.167050] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [15780.172006] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [15780.177131] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15780.180577] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [15780.184488] Call Trace: [15780.185620] [15780.186638] ? show_regs.cold.9+0x22/0x2f [15780.189120] ? watchdog_timer_fn.cold.6+0x62/0x10a [15780.193502] ? __hrtimer_run_queues+0x1c2/0x4d0 [15780.195876] ? watchdog+0x40/0x40 [15780.198226] ? hrtimer_interrupt+0x128/0x2c0 [15780.200765] ? smp_apic_timer_interrupt+0xa6/0x270 [15780.203684] ? apic_timer_interrupt+0xf/0x20 [15780.206077] [15780.207642] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15780.210945] do_raw_spin_lock+0x177/0x190 [15780.213809] _raw_spin_lock+0x21/0x30 [15780.223624] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15780.229003] ? do_raw_spin_unlock+0x75/0x190 [15780.230927] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15780.234238] ? do_wait_intr+0xf0/0xf0 [15780.236492] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15780.239696] mdc_close+0x53b/0xd70 [mdc] [15780.242314] lmv_close+0x18d/0x490 [lmv] [15780.244588] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15780.248093] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15780.251486] ll_md_real_close+0xa5/0x2a0 [lustre] [15780.254370] ll_md_close+0x274/0x800 [lustre] [15780.257235] ll_file_release+0x128/0xc50 [lustre] [15780.260056] __fput+0x10d/0x440 [15780.261691] ____fput+0x12/0x20 [15780.263745] task_work_run+0xc6/0x110 [15780.266005] exit_to_usermode_loop+0x1dd/0x1f0 [15780.269940] do_syscall_64+0x3d6/0x440 [15780.272322] entry_SYSCALL_64_after_hwframe+0x49/0xae [15780.276206] RIP: 0033:0x7f151bee3cd8 [15780.278123] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15780.290256] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15780.294132] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [15780.298091] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [15780.303186] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15780.307533] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15780.312015] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15792.052031] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [15792.055450] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15792.078733] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15792.084450] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15792.088964] RIP: 0010:native_safe_halt+0x2e/0x50 [15792.091180] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15792.099254] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15792.103128] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15792.106464] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [15792.110190] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [15792.113702] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [15792.117100] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [15792.120587] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [15792.124317] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15792.127603] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [15792.131314] Call Trace: [15792.132831] [15792.134099] ? show_regs.cold.9+0x22/0x2f [15792.136822] ? watchdog_timer_fn.cold.6+0x62/0x10a [15792.140120] ? __hrtimer_run_queues+0x1c2/0x4d0 [15792.141830] ? watchdog+0x40/0x40 [15792.144739] ? hrtimer_interrupt+0x128/0x2c0 [15792.148267] ? smp_apic_timer_interrupt+0xa6/0x270 [15792.150461] ? apic_timer_interrupt+0xf/0x20 [15792.152974] [15792.153831] ? native_safe_halt+0x2e/0x50 [15792.155708] kvm_wait+0xcd/0xe0 [15792.157244] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15792.160728] do_raw_spin_lock+0x177/0x190 [15792.162790] _raw_spin_lock+0x21/0x30 [15792.164785] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15792.168693] ? do_raw_spin_unlock+0x75/0x190 [15792.171220] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15792.174823] ? do_wait_intr+0xf0/0xf0 [15792.177197] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15792.180463] mdc_reint+0x2d/0x110 [mdc] [15792.182450] mdc_unlink+0x658/0x870 [mdc] [15792.184764] lmv_unlink+0x34d/0xbd0 [lmv] [15792.186827] ll_unlink+0x24d/0x7f0 [lustre] [15792.190063] vfs_unlink+0x202/0x330 [15792.192216] do_unlinkat+0x2fa/0x460 [15792.194022] __x64_sys_unlink+0x37/0x50 [15792.196223] do_syscall_64+0xc1/0x440 [15792.197972] entry_SYSCALL_64_after_hwframe+0x49/0xae [15792.200719] RIP: 0033:0x7f151bdee10b [15792.202596] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [15792.211719] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [15792.215222] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [15792.220997] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [15792.225057] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15792.229403] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15792.236720] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15798.242588] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13867s! [15798.246863] Showing busy workqueues and worker pools: [15798.249683] workqueue events: flags=0x0 [15798.255014] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15798.262900] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [15798.269459] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15798.272668] pending: kfree_rcu_monitor [15798.275157] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15798.278618] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [15798.283199] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15798.286161] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [15798.289523] workqueue events_long: flags=0x0 [15798.291142] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15798.293839] pending: lu_objects_flush [obdclass] [15798.296670] workqueue events_unbound: flags=0x2 [15798.299217] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15798.302605] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15798.307515] workqueue events_power_efficient: flags=0x80 [15798.310703] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15798.313553] pending: do_cache_clean, neigh_periodic_work, check_lifetime [15798.316895] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15798.319693] pending: neigh_periodic_work [15798.321217] workqueue mm_percpu_wq: flags=0x8 [15798.323105] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15798.325201] pending: vmstat_update, lru_add_drain_per_cpu [15798.327190] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15798.329490] pending: vmstat_update [15798.330704] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15798.333281] pending: lru_add_drain_per_cpu [15798.335079] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [15798.339051] pending: lru_add_drain_per_cpu BAR(48) [15798.341634] workqueue writeback: flags=0x4a [15798.343436] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [15798.346457] pending: wb_workfn [15798.347984] workqueue xprtiod: flags=0xa [15798.349521] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15798.352553] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [15798.356269] workqueue ipv6_addrconf: flags=0x40008 [15798.358368] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15798.361343] pending: addrconf_verify_work [15798.363875] workqueue knbd0-recv: flags=0x1a [15798.366172] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15798.369856] in-flight: 658:recv_work [15798.372273] workqueue ptlrpc_pinger: flags=0x2 [15798.374286] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15798.377155] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15798.380527] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13867s workers=4 idle: 32984 37 8 [15798.384953] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15783s workers=2 idle: 661 [15804.056027] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [15804.057961] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [15804.058908] Modules linked in: lustre(O) [15804.062403] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15804.062556] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15804.062564] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15804.062568] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15804.062582] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15804.062586] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15804.062590] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [15804.062624] RDX: 000000000000411b RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [15804.062628] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [15804.062630] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [15804.062635] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15804.062639] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [15804.062644] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15804.062647] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15804.062653] Call Trace: [15804.062659] [15804.062664] ? show_regs.cold.9+0x22/0x2f [15804.062677] ? watchdog_timer_fn.cold.6+0x62/0x10a [15804.062684] ? __hrtimer_run_queues+0x1c2/0x4d0 [15804.062688] ? watchdog+0x40/0x40 [15804.062692] ? hrtimer_interrupt+0x128/0x2c0 [15804.062697] ? smp_apic_timer_interrupt+0xa6/0x270 [15804.062705] ? apic_timer_interrupt+0xf/0x20 [15804.062710] [15804.062713] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15804.062721] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15804.062726] do_raw_spin_lock+0x177/0x190 [15804.062731] _raw_spin_lock+0x21/0x30 [15804.062737] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15804.064546] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15804.064621] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15804.064628] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15804.064634] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15804.065067] RIP: 0010:native_safe_halt+0x2e/0x50 [15804.065077] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15804.065080] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15804.065132] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15804.065135] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [15804.065137] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [15804.065139] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [15804.065142] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [15804.065145] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [15804.065148] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15804.065151] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15804.065158] Call Trace: [15804.065163] [15804.065168] ? show_regs.cold.9+0x22/0x2f [15804.065182] ? watchdog_timer_fn.cold.6+0x62/0x10a [15804.065187] ? __hrtimer_run_queues+0x1c2/0x4d0 [15804.065191] ? watchdog+0x40/0x40 [15804.065195] ? hrtimer_interrupt+0x128/0x2c0 [15804.065199] ? smp_apic_timer_interrupt+0xa6/0x270 [15804.065206] ? apic_timer_interrupt+0xf/0x20 [15804.065210] [15804.065213] ? native_safe_halt+0x2e/0x50 [15804.065218] kvm_wait+0xcd/0xe0 [15804.065225] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15804.065232] do_raw_spin_lock+0x177/0x190 [15804.065237] _raw_spin_lock+0x21/0x30 [15804.065241] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15804.084385] obd_mod_free+0x46/0x140 [ptlrpc] [15804.084933] mdc_commit_open+0x91/0xb0 [mdc] [15804.084964] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15804.085112] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15804.085288] after_reply+0xe2c/0x1780 [ptlrpc] [15804.085433] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15804.085868] ? __next_timer_interrupt+0x160/0x160 [15804.085910] ptlrpcd+0xa82/0xe40 [ptlrpc] [15804.091436] process_one_work+0x2c8/0x700 [15804.091452] worker_thread+0x296/0x6e0 [15804.091456] ? rescuer_thread+0x570/0x570 [15804.091458] kthread+0x1d1/0x200 [15804.091461] ? set_kthread_struct+0x70/0x70 [15804.091464] ret_from_fork+0x1f/0x30 [15804.250658] ? do_wait_intr+0xf0/0xf0 [15804.252175] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15804.254365] kthread+0x1d1/0x200 [15804.255631] ? set_kthread_struct+0x70/0x70 [15804.256995] ret_from_fork+0x1f/0x30 [15808.057111] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [15808.059652] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15808.075675] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15808.079599] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15808.082961] RIP: 0010:native_safe_halt+0x2e/0x50 [15808.084580] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15808.091191] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15808.093474] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15808.096305] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [15808.098931] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [15808.101389] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [15808.104032] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [15808.106413] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [15808.109293] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15808.111191] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [15808.113667] Call Trace: [15808.114564] [15808.115239] ? show_regs.cold.9+0x22/0x2f [15808.116602] ? watchdog_timer_fn.cold.6+0x62/0x10a [15808.118296] ? __hrtimer_run_queues+0x1c2/0x4d0 [15808.119937] ? watchdog+0x40/0x40 [15808.121089] ? hrtimer_interrupt+0x128/0x2c0 [15808.122552] ? smp_apic_timer_interrupt+0xa6/0x270 [15808.124108] ? apic_timer_interrupt+0xf/0x20 [15808.125679] [15808.126541] ? native_safe_halt+0x2e/0x50 [15808.127953] kvm_wait+0xcd/0xe0 [15808.129016] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15808.130854] do_raw_spin_lock+0x177/0x190 [15808.132271] _raw_spin_lock+0x21/0x30 [15808.133501] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15808.135611] ? do_raw_spin_unlock+0x75/0x190 [15808.137107] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15808.139120] ? do_wait_intr+0xf0/0xf0 [15808.140420] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15808.142402] mdc_close+0x53b/0xd70 [mdc] [15808.143927] lmv_close+0x18d/0x490 [lmv] [15808.145605] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15808.147510] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15808.149484] ll_md_real_close+0xa5/0x2a0 [lustre] [15808.151282] ll_md_close+0x274/0x800 [lustre] [15808.152899] ll_file_release+0x128/0xc50 [lustre] [15808.154579] __fput+0x10d/0x440 [15808.155647] ____fput+0x12/0x20 [15808.156866] task_work_run+0xc6/0x110 [15808.158071] exit_to_usermode_loop+0x1dd/0x1f0 [15808.159521] do_syscall_64+0x3d6/0x440 [15808.160922] entry_SYSCALL_64_after_hwframe+0x49/0xae [15808.162662] RIP: 0033:0x7f151bee3cd8 [15808.163968] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15808.170227] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15808.172757] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [15808.175117] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [15808.178004] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15808.180540] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15808.182989] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15824.053654] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [15824.058489] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15824.118369] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15824.131530] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15824.143024] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15824.151942] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15824.176084] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15824.177881] RAX: 000000000000662e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15824.190032] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [15824.199658] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [15824.208668] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [15824.214235] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [15824.222008] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [15824.233957] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15824.240134] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [15824.246177] Call Trace: [15824.247133] [15824.247896] ? show_regs.cold.9+0x22/0x2f [15824.250978] ? watchdog_timer_fn.cold.6+0x62/0x10a [15824.258303] ? __hrtimer_run_queues+0x1c2/0x4d0 [15824.265074] ? watchdog+0x40/0x40 [15824.269181] ? hrtimer_interrupt+0x128/0x2c0 [15824.271266] ? smp_apic_timer_interrupt+0xa6/0x270 [15824.278778] ? apic_timer_interrupt+0xf/0x20 [15824.282180] [15824.283140] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15824.288546] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15824.293695] do_raw_spin_lock+0x177/0x190 [15824.295393] _raw_spin_lock+0x21/0x30 [15824.300186] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15824.304852] ? do_raw_spin_unlock+0x75/0x190 [15824.307023] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15824.312287] ? do_wait_intr+0xf0/0xf0 [15824.314251] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15824.316576] mdc_reint+0x2d/0x110 [mdc] [15824.323158] mdc_unlink+0x658/0x870 [mdc] [15824.325882] lmv_unlink+0x34d/0xbd0 [lmv] [15824.327965] ll_unlink+0x24d/0x7f0 [lustre] [15824.332187] vfs_unlink+0x202/0x330 [15824.336167] do_unlinkat+0x2fa/0x460 [15824.337719] __x64_sys_unlink+0x37/0x50 [15824.341883] do_syscall_64+0xc1/0x440 [15824.345977] entry_SYSCALL_64_after_hwframe+0x49/0xae [15824.350382] RIP: 0033:0x7f151bdee10b [15824.352106] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [15824.381191] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [15824.391013] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [15824.395535] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [15824.402162] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15824.408996] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15824.413527] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15828.450364] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13897s! [15828.459957] Showing busy workqueues and worker pools: [15828.462797] workqueue events: flags=0x0 [15828.465056] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15828.469928] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [15828.479443] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15828.484978] pending: kfree_rcu_monitor [15828.487327] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15828.491081] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [15828.495909] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15828.500000] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [15828.503987] workqueue events_long: flags=0x0 [15828.509903] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15828.520710] pending: lu_objects_flush [obdclass] [15828.527935] workqueue events_unbound: flags=0x2 [15828.534745] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15828.540570] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15828.554863] workqueue events_power_efficient: flags=0x80 [15828.560873] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15828.570260] pending: do_cache_clean, neigh_periodic_work, check_lifetime [15828.578863] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15828.586163] pending: neigh_periodic_work [15828.588074] workqueue mm_percpu_wq: flags=0x8 [15828.590417] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15828.593174] pending: vmstat_update, lru_add_drain_per_cpu [15828.595736] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15828.598582] pending: vmstat_update [15828.600614] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15828.603056] pending: lru_add_drain_per_cpu [15828.605269] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [15828.608885] pending: lru_add_drain_per_cpu BAR(48) [15828.611121] workqueue writeback: flags=0x4a [15828.615445] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [15828.619617] pending: wb_workfn [15828.621119] workqueue xprtiod: flags=0xa [15828.623740] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15828.626272] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [15828.635148] workqueue ipv6_addrconf: flags=0x40008 [15828.638472] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15828.645309] pending: addrconf_verify_work [15828.648661] workqueue knbd0-recv: flags=0x1a [15828.651913] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15828.655733] in-flight: 658:recv_work [15828.658210] workqueue ptlrpc_pinger: flags=0x2 [15828.660539] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15828.663939] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15828.667202] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13898s workers=4 idle: 32984 37 8 [15828.670922] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15813s workers=2 idle: 661 [15832.056720] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [15832.057128] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [15832.062686] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15832.062766] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15832.062772] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15832.062774] RIP: 0010:native_safe_halt+0x2e/0x50 [15832.062787] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15832.062790] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15832.062802] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [15832.062805] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [15832.062807] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [15832.062809] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [15832.062812] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15832.062815] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [15832.062818] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15832.062821] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15832.071433] Modules linked in: lustre(O) [15832.103308] Call Trace: [15832.113110] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15832.113185] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15832.113193] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15832.113200] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15832.113747] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15832.113758] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15832.113762] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15832.113766] RAX: 0000000000006e9d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15832.113769] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [15832.113771] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [15832.113774] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [15832.113776] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [15832.113779] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [15832.113781] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15832.113784] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15832.113793] Call Trace: [15832.113797] [15832.113802] ? show_regs.cold.9+0x22/0x2f [15832.113813] ? watchdog_timer_fn.cold.6+0x62/0x10a [15832.113819] ? __hrtimer_run_queues+0x1c2/0x4d0 [15832.113824] ? watchdog+0x40/0x40 [15832.113829] ? hrtimer_interrupt+0x128/0x2c0 [15832.113833] ? smp_apic_timer_interrupt+0xa6/0x270 [15832.113840] ? apic_timer_interrupt+0xf/0x20 [15832.113845] [15832.113848] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15832.113852] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15832.113857] do_raw_spin_lock+0x177/0x190 [15832.113862] _raw_spin_lock+0x21/0x30 [15832.113867] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15832.114127] process_one_work+0x2c8/0x700 [15832.114134] worker_thread+0x296/0x6e0 [15832.114139] ? rescuer_thread+0x570/0x570 [15832.114143] kthread+0x1d1/0x200 [15832.114147] ? set_kthread_struct+0x70/0x70 [15832.114151] ret_from_fork+0x1f/0x30 [15832.447683] [15832.450564] ? show_regs.cold.9+0x22/0x2f [15832.455110] ? watchdog_timer_fn.cold.6+0x62/0x10a [15832.459661] ? __hrtimer_run_queues+0x1c2/0x4d0 [15832.461751] ? watchdog+0x40/0x40 [15832.463533] ? hrtimer_interrupt+0x128/0x2c0 [15832.466522] ? smp_apic_timer_interrupt+0xa6/0x270 [15832.469009] ? apic_timer_interrupt+0xf/0x20 [15832.471255] [15832.472002] ? native_safe_halt+0x2e/0x50 [15832.477080] kvm_wait+0xcd/0xe0 [15832.480342] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15832.486179] do_raw_spin_lock+0x177/0x190 [15832.488545] _raw_spin_lock+0x21/0x30 [15832.492424] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15832.501576] obd_mod_free+0x46/0x140 [ptlrpc] [15832.506963] mdc_commit_open+0x91/0xb0 [mdc] [15832.508392] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15832.515362] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15832.520292] after_reply+0xe2c/0x1780 [ptlrpc] [15832.528321] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15832.531248] ? __next_timer_interrupt+0x160/0x160 [15832.533153] ptlrpcd+0xa82/0xe40 [ptlrpc] [15832.546204] ? do_wait_intr+0xf0/0xf0 [15832.550804] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15832.565003] kthread+0x1d1/0x200 [15832.566350] ? set_kthread_struct+0x70/0x70 [15832.570220] ret_from_fork+0x1f/0x30 [15836.065438] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [15836.075847] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15836.112274] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15836.127131] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15836.130274] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15836.147251] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15836.164320] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15836.171172] RAX: 0000000000007c03 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15836.176606] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [15836.180364] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [15836.188391] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [15836.193312] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [15836.196777] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [15836.208880] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15836.216827] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [15836.221617] Call Trace: [15836.222603] [15836.223356] ? show_regs.cold.9+0x22/0x2f [15836.230283] ? watchdog_timer_fn.cold.6+0x62/0x10a [15836.234047] ? __hrtimer_run_queues+0x1c2/0x4d0 [15836.238330] ? watchdog+0x40/0x40 [15836.241649] ? hrtimer_interrupt+0x128/0x2c0 [15836.253493] ? smp_apic_timer_interrupt+0xa6/0x270 [15836.255367] ? apic_timer_interrupt+0xf/0x20 [15836.261213] [15836.264540] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15836.267975] do_raw_spin_lock+0x177/0x190 [15836.272569] _raw_spin_lock+0x21/0x30 [15836.274744] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15836.279381] ? do_raw_spin_unlock+0x75/0x190 [15836.284255] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15836.290825] ? do_wait_intr+0xf0/0xf0 [15836.292890] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15836.302072] mdc_close+0x53b/0xd70 [mdc] [15836.305149] lmv_close+0x18d/0x490 [lmv] [15836.308089] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15836.316423] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15836.325629] ll_md_real_close+0xa5/0x2a0 [lustre] [15836.329245] ll_md_close+0x274/0x800 [lustre] [15836.338372] ll_file_release+0x128/0xc50 [lustre] [15836.341643] __fput+0x10d/0x440 [15836.346516] ____fput+0x12/0x20 [15836.348230] task_work_run+0xc6/0x110 [15836.350710] exit_to_usermode_loop+0x1dd/0x1f0 [15836.356923] do_syscall_64+0x3d6/0x440 [15836.359129] entry_SYSCALL_64_after_hwframe+0x49/0xae [15836.365038] RIP: 0033:0x7f151bee3cd8 [15836.367404] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15836.389170] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15836.394197] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [15836.401107] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [15836.405596] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15836.414040] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15836.417966] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15844.939029] rcu: INFO: rcu_sched self-detected stall on CPU [15844.941906] rcu: 1-...!: (8307978 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [15844.947901] (t=13920292 jiffies g=249657 q=6640) [15844.951218] rcu: rcu_sched kthread starved for 13915244 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [15844.961027] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [15844.966800] rcu: RCU grace-period kthread stack dump: [15844.971597] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [15844.983966] Call Trace: [15844.985435] __schedule+0x351/0xcb0 [15844.992122] schedule+0xc0/0x180 [15844.994971] schedule_timeout+0xb4/0x190 [15844.996395] ? do_raw_spin_unlock+0x75/0x190 [15845.001980] ? __next_timer_interrupt+0x160/0x160 [15845.005461] ? prepare_to_swait_event+0xf3/0x140 [15845.010252] rcu_gp_kthread+0x8dd/0x15b0 [15845.012875] ? force_qs_rnp+0x2c0/0x2c0 [15845.015818] kthread+0x1d1/0x200 [15845.020518] ? set_kthread_struct+0x70/0x70 [15845.026428] ret_from_fork+0x1f/0x30 [15845.029620] rcu: Stack dump where RCU GP kthread last ran: [15845.040591] Sending NMI from CPU 1 to CPUs 2: [15845.042729] NMI backtrace for cpu 2 skipped: idling at native_safe_halt+0x2e/0x50 [15845.044258] NMI backtrace for cpu 1 [15845.061296] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15845.077274] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15845.085562] Call Trace: [15845.090632] [15845.092216] ? dump_stack+0xbb/0x10e [15845.095635] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [15845.101446] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [15845.104830] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [15845.110034] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [15845.114873] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [15845.120463] ? rcu_dump_cpu_stacks+0x11b/0x171 [15845.124457] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [15845.128159] ? update_process_times+0x6d/0xc0 [15845.131750] ? tick_sched_handle.isra.18+0x27/0x90 [15845.134522] ? tick_sched_timer+0x53/0xb0 [15845.139006] ? __hrtimer_run_queues+0x1c2/0x4d0 [15845.141981] ? tick_sched_handle.isra.18+0x90/0x90 [15845.145388] ? hrtimer_interrupt+0x128/0x2c0 [15845.147256] ? smp_apic_timer_interrupt+0xa6/0x270 [15845.154266] ? apic_timer_interrupt+0xf/0x20 [15845.158890] [15845.164565] ? native_safe_halt+0x2e/0x50 [15845.171044] ? kvm_wait+0xcd/0xe0 [15845.177669] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15845.186393] ? do_raw_spin_lock+0x177/0x190 [15845.193925] ? _raw_spin_lock+0x21/0x30 [15845.197556] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15845.200905] ? obd_mod_free+0x46/0x140 [ptlrpc] [15845.208444] ? mdc_commit_open+0x91/0xb0 [mdc] [15845.211555] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15845.218965] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15845.226393] ? after_reply+0xe2c/0x1780 [ptlrpc] [15845.232425] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15845.237417] ? __next_timer_interrupt+0x160/0x160 [15845.242487] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [15845.247305] ? do_wait_intr+0xf0/0xf0 [15845.248691] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15845.258936] ? kthread+0x1d1/0x200 [15845.262594] ? set_kthread_struct+0x70/0x70 [15845.267643] ? ret_from_fork+0x1f/0x30 [15856.052025] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [15856.054859] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15856.085087] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15856.090799] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15856.097898] RIP: 0010:native_safe_halt+0x2e/0x50 [15856.103194] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15856.116490] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15856.121911] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15856.125920] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [15856.128898] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [15856.131388] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [15856.133319] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [15856.136445] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [15856.140209] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15856.142642] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [15856.145615] Call Trace: [15856.146254] [15856.146969] ? show_regs.cold.9+0x22/0x2f [15856.149073] ? watchdog_timer_fn.cold.6+0x62/0x10a [15856.151073] ? __hrtimer_run_queues+0x1c2/0x4d0 [15856.152920] ? watchdog+0x40/0x40 [15856.154502] ? hrtimer_interrupt+0x128/0x2c0 [15856.156569] ? smp_apic_timer_interrupt+0xa6/0x270 [15856.158833] ? apic_timer_interrupt+0xf/0x20 [15856.160900] [15856.161915] ? native_safe_halt+0x2e/0x50 [15856.163840] kvm_wait+0xcd/0xe0 [15856.165396] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15856.167603] do_raw_spin_lock+0x177/0x190 [15856.169391] _raw_spin_lock+0x21/0x30 [15856.171066] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15856.173734] ? do_raw_spin_unlock+0x75/0x190 [15856.175636] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15856.177401] ? do_wait_intr+0xf0/0xf0 [15856.178488] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15856.180742] mdc_reint+0x2d/0x110 [mdc] [15856.183133] mdc_unlink+0x658/0x870 [mdc] [15856.187940] lmv_unlink+0x34d/0xbd0 [lmv] [15856.193374] ll_unlink+0x24d/0x7f0 [lustre] [15856.198316] vfs_unlink+0x202/0x330 [15856.201219] do_unlinkat+0x2fa/0x460 [15856.203412] __x64_sys_unlink+0x37/0x50 [15856.204735] do_syscall_64+0xc1/0x440 [15856.207218] entry_SYSCALL_64_after_hwframe+0x49/0xae [15856.209172] RIP: 0033:0x7f151bdee10b [15856.211209] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [15856.224901] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [15856.229895] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [15856.237854] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [15856.248652] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15856.255676] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15856.267411] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15859.169113] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13928s! [15859.173020] Showing busy workqueues and worker pools: [15859.175592] workqueue events: flags=0x0 [15859.177649] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15859.182174] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [15859.187883] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15859.190696] pending: kfree_rcu_monitor [15859.193092] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15859.196730] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [15859.202116] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15859.205732] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [15859.209830] workqueue events_long: flags=0x0 [15859.212119] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15859.215559] pending: lu_objects_flush [obdclass] [15859.218210] workqueue events_unbound: flags=0x2 [15859.220923] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15859.224551] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15859.230729] workqueue events_power_efficient: flags=0x80 [15859.233690] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15859.237995] pending: do_cache_clean, neigh_periodic_work, check_lifetime [15859.241906] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15859.247594] pending: neigh_periodic_work [15859.249533] workqueue mm_percpu_wq: flags=0x8 [15859.251776] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15859.255732] pending: vmstat_update, lru_add_drain_per_cpu [15859.258958] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15859.262548] pending: vmstat_update [15859.264485] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15859.267997] pending: lru_add_drain_per_cpu [15859.270799] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [15859.274239] pending: lru_add_drain_per_cpu BAR(48) [15859.277507] workqueue writeback: flags=0x4a [15859.280804] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [15859.287452] pending: wb_workfn [15859.290424] workqueue xprtiod: flags=0xa [15859.293414] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15859.296926] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [15859.303782] workqueue ipv6_addrconf: flags=0x40008 [15859.307194] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15859.311123] pending: addrconf_verify_work [15859.313392] workqueue knbd0-recv: flags=0x1a [15859.315542] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15859.319998] in-flight: 658:recv_work [15859.321993] workqueue ptlrpc_pinger: flags=0x2 [15859.324013] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15859.327485] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15859.331831] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13928s workers=4 idle: 32984 37 8 [15859.335699] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15844s workers=2 idle: 661 [15864.057556] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [15864.067808] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15864.095832] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15864.104682] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15864.112781] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15864.118584] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15864.132763] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15864.137009] RAX: 0000000000007011 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15864.141975] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [15864.147447] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [15864.151479] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [15864.156593] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [15864.160518] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [15864.164414] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15864.167906] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [15864.171829] Call Trace: [15864.174027] [15864.176106] ? show_regs.cold.9+0x22/0x2f [15864.178846] ? watchdog_timer_fn.cold.6+0x62/0x10a [15864.182587] ? __hrtimer_run_queues+0x1c2/0x4d0 [15864.187114] ? watchdog+0x40/0x40 [15864.188820] ? hrtimer_interrupt+0x128/0x2c0 [15864.190581] ? smp_apic_timer_interrupt+0xa6/0x270 [15864.192850] ? apic_timer_interrupt+0xf/0x20 [15864.195040] [15864.196083] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15864.198909] do_raw_spin_lock+0x177/0x190 [15864.201374] _raw_spin_lock+0x21/0x30 [15864.203149] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15864.206780] ? do_raw_spin_unlock+0x75/0x190 [15864.211104] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15864.213702] ? do_wait_intr+0xf0/0xf0 [15864.216183] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15864.221642] mdc_close+0x53b/0xd70 [mdc] [15864.224376] lmv_close+0x18d/0x490 [lmv] [15864.228649] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15864.232350] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15864.235844] ll_md_real_close+0xa5/0x2a0 [lustre] [15864.238732] ll_md_close+0x274/0x800 [lustre] [15864.240890] ll_file_release+0x128/0xc50 [lustre] [15864.243959] __fput+0x10d/0x440 [15864.245767] ____fput+0x12/0x20 [15864.250002] task_work_run+0xc6/0x110 [15864.252495] exit_to_usermode_loop+0x1dd/0x1f0 [15864.255877] do_syscall_64+0x3d6/0x440 [15864.257941] entry_SYSCALL_64_after_hwframe+0x49/0xae [15864.263773] RIP: 0033:0x7f151bee3cd8 [15864.266281] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15864.276734] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15864.282020] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [15864.287333] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [15864.290703] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15864.293976] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15864.298153] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15872.060853] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [15872.062419] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [15872.077680] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15872.077773] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15872.077782] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15872.077786] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15872.077799] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15872.077803] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15872.077808] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [15872.077810] RDX: 0000000000007e8b RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [15872.077812] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [15872.077814] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [15872.077817] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15872.077820] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [15872.077823] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15872.077825] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15872.077834] Call Trace: [15872.077839] [15872.077846] ? show_regs.cold.9+0x22/0x2f [15872.077856] ? watchdog_timer_fn.cold.6+0x62/0x10a [15872.077862] ? __hrtimer_run_queues+0x1c2/0x4d0 [15872.077867] ? watchdog+0x40/0x40 [15872.077872] ? hrtimer_interrupt+0x128/0x2c0 [15872.077877] ? smp_apic_timer_interrupt+0xa6/0x270 [15872.077886] ? apic_timer_interrupt+0xf/0x20 [15872.077891] [15872.077894] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15872.077912] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15872.077918] do_raw_spin_lock+0x177/0x190 [15872.077935] _raw_spin_lock+0x21/0x30 [15872.077940] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15872.078515] obd_mod_free+0x46/0x140 [ptlrpc] [15872.078709] mdc_commit_open+0x91/0xb0 [mdc] [15872.078745] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15872.078957] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15872.082047] after_reply+0xe2c/0x1780 [ptlrpc] [15872.082659] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15872.082867] ? __next_timer_interrupt+0x160/0x160 [15872.082878] ptlrpcd+0xa82/0xe40 [ptlrpc] [15872.083078] ? do_wait_intr+0xf0/0xf0 [15872.083089] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15872.083559] kthread+0x1d1/0x200 [15872.083566] ? set_kthread_struct+0x70/0x70 [15872.083570] ret_from_fork+0x1f/0x30 [15872.454060] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15872.482485] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15872.488697] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15872.493024] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15872.499032] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15872.501305] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15872.508268] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15872.511479] RAX: 000000000000161b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15872.514515] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [15872.517559] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [15872.520848] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [15872.524022] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [15872.527326] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [15872.530816] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15872.533478] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15872.537324] Call Trace: [15872.538316] [15872.538762] ? show_regs.cold.9+0x22/0x2f [15872.539925] ? watchdog_timer_fn.cold.6+0x62/0x10a [15872.541933] ? __hrtimer_run_queues+0x1c2/0x4d0 [15872.543904] ? watchdog+0x40/0x40 [15872.545482] ? hrtimer_interrupt+0x128/0x2c0 [15872.546980] ? smp_apic_timer_interrupt+0xa6/0x270 [15872.548960] ? apic_timer_interrupt+0xf/0x20 [15872.550430] [15872.550899] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15872.552611] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15872.554041] do_raw_spin_lock+0x177/0x190 [15872.554971] _raw_spin_lock+0x21/0x30 [15872.556930] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15872.559666] process_one_work+0x2c8/0x700 [15872.561588] worker_thread+0x296/0x6e0 [15872.563442] ? rescuer_thread+0x570/0x570 [15872.565283] kthread+0x1d1/0x200 [15872.567048] ? set_kthread_struct+0x70/0x70 [15872.568968] ret_from_fork+0x1f/0x30 [15884.052500] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [15884.058045] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15884.080444] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15884.084544] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15884.086860] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15884.088296] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15884.094081] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15884.096900] RAX: 0000000000006447 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15884.105414] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [15884.107582] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [15884.114309] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [15884.117863] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [15884.121818] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [15884.127835] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15884.129989] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [15884.132193] Call Trace: [15884.132827] [15884.133454] ? show_regs.cold.9+0x22/0x2f [15884.134660] ? watchdog_timer_fn.cold.6+0x62/0x10a [15884.136064] ? __hrtimer_run_queues+0x1c2/0x4d0 [15884.137678] ? watchdog+0x40/0x40 [15884.138908] ? hrtimer_interrupt+0x128/0x2c0 [15884.140277] ? smp_apic_timer_interrupt+0xa6/0x270 [15884.141723] ? apic_timer_interrupt+0xf/0x20 [15884.143328] [15884.144111] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15884.145782] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15884.147673] do_raw_spin_lock+0x177/0x190 [15884.149227] _raw_spin_lock+0x21/0x30 [15884.150347] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15884.152304] ? do_raw_spin_unlock+0x75/0x190 [15884.154159] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15884.156488] ? do_wait_intr+0xf0/0xf0 [15884.157752] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15884.159632] mdc_reint+0x2d/0x110 [mdc] [15884.161126] mdc_unlink+0x658/0x870 [mdc] [15884.162763] lmv_unlink+0x34d/0xbd0 [lmv] [15884.164178] ll_unlink+0x24d/0x7f0 [lustre] [15884.165928] vfs_unlink+0x202/0x330 [15884.167267] do_unlinkat+0x2fa/0x460 [15884.168521] __x64_sys_unlink+0x37/0x50 [15884.169924] do_syscall_64+0xc1/0x440 [15884.171362] entry_SYSCALL_64_after_hwframe+0x49/0xae [15884.173399] RIP: 0033:0x7f151bdee10b [15884.174456] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [15884.181933] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [15884.185061] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [15884.187485] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [15884.190897] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15884.194368] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15884.196685] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15889.380378] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13958s! [15889.384468] Showing busy workqueues and worker pools: [15889.386577] workqueue events: flags=0x0 [15889.388340] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15889.394255] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [15889.400073] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15889.403669] pending: kfree_rcu_monitor [15889.406931] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15889.410359] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [15889.414928] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15889.417837] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [15889.422683] workqueue events_long: flags=0x0 [15889.424887] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15889.428961] pending: lu_objects_flush [obdclass] [15889.431740] workqueue events_unbound: flags=0x2 [15889.434616] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15889.438634] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15889.444699] workqueue events_power_efficient: flags=0x80 [15889.447216] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15889.451177] pending: do_cache_clean, neigh_periodic_work, check_lifetime [15889.454952] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15889.458798] pending: neigh_periodic_work [15889.461383] workqueue mm_percpu_wq: flags=0x8 [15889.465587] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15889.469767] pending: vmstat_update, lru_add_drain_per_cpu [15889.473539] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15889.477231] pending: vmstat_update [15889.478989] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15889.482718] pending: lru_add_drain_per_cpu [15889.484859] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [15889.488823] pending: lru_add_drain_per_cpu BAR(48) [15889.491357] workqueue writeback: flags=0x4a [15889.493387] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [15889.496592] pending: wb_workfn [15889.498431] workqueue xprtiod: flags=0xa [15889.500271] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15889.503820] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [15889.507543] workqueue ipv6_addrconf: flags=0x40008 [15889.509653] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15889.512638] pending: addrconf_verify_work [15889.515028] workqueue knbd0-recv: flags=0x1a [15889.517095] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15889.521338] in-flight: 658:recv_work [15889.523337] workqueue ptlrpc_pinger: flags=0x2 [15889.525599] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15889.528785] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15889.531608] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13958s workers=4 idle: 32984 37 8 [15889.535300] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15874s workers=2 idle: 661 [15892.059268] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [15892.064786] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15892.108653] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15892.114278] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15892.122306] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15892.126141] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15892.144109] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [15892.148281] RAX: 0000000000000915 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15892.153615] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [15892.162059] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [15892.167162] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [15892.172692] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [15892.179678] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [15892.187008] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15892.190126] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [15892.193682] Call Trace: [15892.194989] [15892.196029] ? show_regs.cold.9+0x22/0x2f [15892.197777] ? watchdog_timer_fn.cold.6+0x62/0x10a [15892.200112] ? __hrtimer_run_queues+0x1c2/0x4d0 [15892.201859] ? watchdog+0x40/0x40 [15892.203492] ? hrtimer_interrupt+0x128/0x2c0 [15892.205255] ? smp_apic_timer_interrupt+0xa6/0x270 [15892.206498] ? apic_timer_interrupt+0xf/0x20 [15892.210629] [15892.214600] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15892.217736] do_raw_spin_lock+0x177/0x190 [15892.221777] _raw_spin_lock+0x21/0x30 [15892.228811] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15892.231695] ? do_raw_spin_unlock+0x75/0x190 [15892.233609] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15892.236291] ? do_wait_intr+0xf0/0xf0 [15892.237808] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15892.240160] mdc_close+0x53b/0xd70 [mdc] [15892.241722] lmv_close+0x18d/0x490 [lmv] [15892.243470] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15892.245307] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15892.246721] ll_md_real_close+0xa5/0x2a0 [lustre] [15892.248380] ll_md_close+0x274/0x800 [lustre] [15892.249629] ll_file_release+0x128/0xc50 [lustre] [15892.251033] __fput+0x10d/0x440 [15892.252329] ____fput+0x12/0x20 [15892.253665] task_work_run+0xc6/0x110 [15892.255086] exit_to_usermode_loop+0x1dd/0x1f0 [15892.256839] do_syscall_64+0x3d6/0x440 [15892.258146] entry_SYSCALL_64_after_hwframe+0x49/0xae [15892.259404] RIP: 0033:0x7f151bee3cd8 [15892.260600] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15892.266865] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15892.268669] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [15892.270358] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [15892.272089] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15892.273857] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15892.275599] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15900.055169] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [15900.056077] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [15900.063344] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15900.063432] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15900.063445] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15900.063448] RIP: 0010:native_safe_halt+0x2e/0x50 [15900.063463] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15900.063468] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [15900.063473] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [15900.063475] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [15900.063477] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [15900.063479] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [15900.063482] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15900.063485] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [15900.063488] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15900.063490] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15900.063498] Call Trace: [15900.063504] [15900.063510] ? show_regs.cold.9+0x22/0x2f [15900.063522] ? watchdog_timer_fn.cold.6+0x62/0x10a [15900.063529] ? __hrtimer_run_queues+0x1c2/0x4d0 [15900.063533] ? watchdog+0x40/0x40 [15900.063537] ? hrtimer_interrupt+0x128/0x2c0 [15900.063540] ? smp_apic_timer_interrupt+0xa6/0x270 [15900.063546] ? apic_timer_interrupt+0xf/0x20 [15900.063551] [15900.063553] ? native_safe_halt+0x2e/0x50 [15900.063557] kvm_wait+0xcd/0xe0 [15900.063565] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15900.063579] do_raw_spin_lock+0x177/0x190 [15900.063583] _raw_spin_lock+0x21/0x30 [15900.063589] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15900.075078] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15900.075154] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15900.075161] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15900.075168] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15900.075511] RIP: 0010:native_safe_halt+0x2e/0x50 [15900.075522] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15900.075526] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15900.075530] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15900.075532] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [15900.075535] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [15900.075537] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [15900.075539] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [15900.075544] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [15900.075547] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15900.075549] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15900.075557] Call Trace: [15900.075563] [15900.075569] ? show_regs.cold.9+0x22/0x2f [15900.075578] ? watchdog_timer_fn.cold.6+0x62/0x10a [15900.075585] ? __hrtimer_run_queues+0x1c2/0x4d0 [15900.075589] ? watchdog+0x40/0x40 [15900.075593] ? hrtimer_interrupt+0x128/0x2c0 [15900.075597] ? smp_apic_timer_interrupt+0xa6/0x270 [15900.075604] ? apic_timer_interrupt+0xf/0x20 [15900.075608] [15900.075611] ? native_safe_halt+0x2e/0x50 [15900.075614] kvm_wait+0xcd/0xe0 [15900.075623] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15900.075631] do_raw_spin_lock+0x177/0x190 [15900.075635] _raw_spin_lock+0x21/0x30 [15900.075648] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15900.076002] process_one_work+0x2c8/0x700 [15900.076009] worker_thread+0x296/0x6e0 [15900.076013] ? rescuer_thread+0x570/0x570 [15900.076017] kthread+0x1d1/0x200 [15900.076029] ? set_kthread_struct+0x70/0x70 [15900.076033] ret_from_fork+0x1f/0x30 [15900.318399] obd_mod_free+0x46/0x140 [ptlrpc] [15900.321153] mdc_commit_open+0x91/0xb0 [mdc] [15900.325350] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15900.328448] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15900.330295] after_reply+0xe2c/0x1780 [ptlrpc] [15900.332026] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15900.335763] ? __next_timer_interrupt+0x160/0x160 [15900.338338] ptlrpcd+0xa82/0xe40 [ptlrpc] [15900.341272] ? do_wait_intr+0xf0/0xf0 [15900.343527] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15900.346571] kthread+0x1d1/0x200 [15900.347517] ? set_kthread_struct+0x70/0x70 [15900.349695] ret_from_fork+0x1f/0x30 [15916.052103] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [15916.057991] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15916.086549] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15916.092801] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15916.098767] RIP: 0010:native_safe_halt+0x2e/0x50 [15916.101536] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15916.113934] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15916.118274] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15916.122699] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [15916.130442] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [15916.135261] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [15916.139397] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [15916.143561] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [15916.149275] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15916.152750] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [15916.157087] Call Trace: [15916.158474] [15916.159547] ? show_regs.cold.9+0x22/0x2f [15916.162009] ? watchdog_timer_fn.cold.6+0x62/0x10a [15916.164451] ? __hrtimer_run_queues+0x1c2/0x4d0 [15916.167153] ? watchdog+0x40/0x40 [15916.168992] ? hrtimer_interrupt+0x128/0x2c0 [15916.171532] ? smp_apic_timer_interrupt+0xa6/0x270 [15916.174215] ? apic_timer_interrupt+0xf/0x20 [15916.176636] [15916.178120] ? native_safe_halt+0x2e/0x50 [15916.180625] kvm_wait+0xcd/0xe0 [15916.182711] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15916.185925] do_raw_spin_lock+0x177/0x190 [15916.188295] _raw_spin_lock+0x21/0x30 [15916.190449] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15916.193892] ? do_raw_spin_unlock+0x75/0x190 [15916.196294] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15916.199628] ? do_wait_intr+0xf0/0xf0 [15916.202287] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15916.206312] mdc_reint+0x2d/0x110 [mdc] [15916.208527] mdc_unlink+0x658/0x870 [mdc] [15916.211388] lmv_unlink+0x34d/0xbd0 [lmv] [15916.213721] ll_unlink+0x24d/0x7f0 [lustre] [15916.216564] vfs_unlink+0x202/0x330 [15916.218974] do_unlinkat+0x2fa/0x460 [15916.221194] __x64_sys_unlink+0x37/0x50 [15916.223725] do_syscall_64+0xc1/0x440 [15916.225765] entry_SYSCALL_64_after_hwframe+0x49/0xae [15916.228795] RIP: 0033:0x7f151bdee10b [15916.230862] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [15916.242200] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [15916.247024] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [15916.251257] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [15916.256711] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15916.260285] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15916.264929] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15919.588220] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13988s! [15919.595899] Showing busy workqueues and worker pools: [15919.601990] workqueue events: flags=0x0 [15919.606309] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15919.608482] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [15919.621262] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15919.623473] pending: kfree_rcu_monitor [15919.635736] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15919.642316] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [15919.653183] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15919.659544] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [15919.665701] workqueue events_long: flags=0x0 [15919.667710] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15919.670173] pending: lu_objects_flush [obdclass] [15919.672567] workqueue events_unbound: flags=0x2 [15919.674512] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15919.677272] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15919.682236] workqueue events_power_efficient: flags=0x80 [15919.685492] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15919.690818] pending: do_cache_clean, neigh_periodic_work, check_lifetime [15919.694182] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15919.697216] pending: neigh_periodic_work [15919.699156] workqueue mm_percpu_wq: flags=0x8 [15919.700760] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15919.703784] pending: vmstat_update, lru_add_drain_per_cpu [15919.706988] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15919.715104] pending: vmstat_update [15919.716987] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15919.720486] pending: lru_add_drain_per_cpu [15919.724056] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [15919.729397] pending: lru_add_drain_per_cpu BAR(48) [15919.740333] workqueue writeback: flags=0x4a [15919.747718] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [15919.759590] pending: wb_workfn [15919.766367] workqueue xprtiod: flags=0xa [15919.774055] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15919.779341] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [15919.790833] workqueue ipv6_addrconf: flags=0x40008 [15919.800543] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15919.809302] pending: addrconf_verify_work [15919.814741] workqueue knbd0-recv: flags=0x1a [15919.818294] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15919.828205] in-flight: 658:recv_work [15919.829644] workqueue ptlrpc_pinger: flags=0x2 [15919.840789] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15919.851270] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15919.857347] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13989s workers=4 idle: 32984 37 8 [15919.863269] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15904s workers=2 idle: 661 [15920.060711] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [15920.066430] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15920.097311] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15920.102151] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15920.105888] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15920.108138] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15920.114681] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15920.117167] RAX: 0000000000004089 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15920.119929] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [15920.122372] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [15920.124854] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [15920.127073] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [15920.129354] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [15920.132158] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15920.134308] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [15920.136546] Call Trace: [15920.137399] [15920.138276] ? show_regs.cold.9+0x22/0x2f [15920.139901] ? watchdog_timer_fn.cold.6+0x62/0x10a [15920.142060] ? __hrtimer_run_queues+0x1c2/0x4d0 [15920.143799] ? watchdog+0x40/0x40 [15920.145116] ? hrtimer_interrupt+0x128/0x2c0 [15920.146783] ? smp_apic_timer_interrupt+0xa6/0x270 [15920.148387] ? apic_timer_interrupt+0xf/0x20 [15920.149956] [15920.150884] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15920.152922] do_raw_spin_lock+0x177/0x190 [15920.155020] _raw_spin_lock+0x21/0x30 [15920.156389] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15920.158586] ? do_raw_spin_unlock+0x75/0x190 [15920.160912] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15920.162906] ? do_wait_intr+0xf0/0xf0 [15920.164287] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15920.166401] mdc_close+0x53b/0xd70 [mdc] [15920.168346] lmv_close+0x18d/0x490 [lmv] [15920.169915] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15920.172152] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15920.174753] ll_md_real_close+0xa5/0x2a0 [lustre] [15920.176331] ll_md_close+0x274/0x800 [lustre] [15920.178043] ll_file_release+0x128/0xc50 [lustre] [15920.179722] __fput+0x10d/0x440 [15920.180752] ____fput+0x12/0x20 [15920.182091] task_work_run+0xc6/0x110 [15920.183382] exit_to_usermode_loop+0x1dd/0x1f0 [15920.185097] do_syscall_64+0x3d6/0x440 [15920.186759] entry_SYSCALL_64_after_hwframe+0x49/0xae [15920.188738] RIP: 0033:0x7f151bee3cd8 [15920.189983] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15920.196389] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15920.199013] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [15920.201885] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [15920.204863] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15920.207539] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15920.212341] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15928.055380] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [15928.056049] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [15928.061123] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15928.061195] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15928.061203] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15928.061206] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15928.061223] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15928.061226] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15928.061230] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [15928.061232] RDX: 000000000000690d RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [15928.061234] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [15928.061236] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [15928.061238] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15928.061241] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [15928.061244] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15928.061246] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15928.061255] Call Trace: [15928.061263] [15928.061271] ? show_regs.cold.9+0x22/0x2f [15928.061285] ? watchdog_timer_fn.cold.6+0x62/0x10a [15928.061294] ? __hrtimer_run_queues+0x1c2/0x4d0 [15928.061298] ? watchdog+0x40/0x40 [15928.061302] ? hrtimer_interrupt+0x128/0x2c0 [15928.061306] ? smp_apic_timer_interrupt+0xa6/0x270 [15928.061317] ? apic_timer_interrupt+0xf/0x20 [15928.061321] [15928.061324] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15928.061329] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15928.061334] do_raw_spin_lock+0x177/0x190 [15928.061339] _raw_spin_lock+0x21/0x30 [15928.061784] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15928.067079] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15928.067160] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15928.067168] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15928.067174] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15928.111869] obd_mod_free+0x46/0x140 [ptlrpc] [15928.112354] mdc_commit_open+0x91/0xb0 [mdc] [15928.112398] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15928.122313] [15928.122323] RIP: 0010:native_safe_halt+0x2e/0x50 [15928.125042] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15928.130187] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15928.148025] after_reply+0xe2c/0x1780 [ptlrpc] [15928.148437] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15928.148707] ? __next_timer_interrupt+0x160/0x160 [15928.148719] ptlrpcd+0xa82/0xe40 [ptlrpc] [15928.148920] ? do_wait_intr+0xf0/0xf0 [15928.148929] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15928.149130] kthread+0x1d1/0x200 [15928.149136] ? set_kthread_struct+0x70/0x70 [15928.149144] ret_from_fork+0x1f/0x30 [15928.356353] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [15928.361994] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15928.365308] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [15928.369776] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [15928.373188] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [15928.376214] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [15928.379482] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [15928.382857] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15928.385235] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15928.388372] Call Trace: [15928.391472] [15928.392517] ? show_regs.cold.9+0x22/0x2f [15928.397197] ? watchdog_timer_fn.cold.6+0x62/0x10a [15928.399342] ? __hrtimer_run_queues+0x1c2/0x4d0 [15928.401457] ? watchdog+0x40/0x40 [15928.403225] ? hrtimer_interrupt+0x128/0x2c0 [15928.405577] ? smp_apic_timer_interrupt+0xa6/0x270 [15928.408092] ? apic_timer_interrupt+0xf/0x20 [15928.409545] [15928.410503] ? native_safe_halt+0x2e/0x50 [15928.412285] kvm_wait+0xcd/0xe0 [15928.413475] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15928.415296] do_raw_spin_lock+0x177/0x190 [15928.417780] _raw_spin_lock+0x21/0x30 [15928.419389] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15928.422173] process_one_work+0x2c8/0x700 [15928.424608] worker_thread+0x296/0x6e0 [15928.426599] ? rescuer_thread+0x570/0x570 [15928.428432] kthread+0x1d1/0x200 [15928.430403] ? set_kthread_struct+0x70/0x70 [15928.432170] ret_from_fork+0x1f/0x30 [15944.052109] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [15944.059036] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15944.119856] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15944.134348] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15944.146824] RIP: 0010:native_safe_halt+0x2e/0x50 [15944.154745] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15944.178785] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15944.189190] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15944.194892] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [15944.202664] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [15944.213453] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [15944.224463] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [15944.231680] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [15944.245275] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15944.253518] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [15944.261907] Call Trace: [15944.264588] [15944.265808] ? show_regs.cold.9+0x22/0x2f [15944.270159] ? watchdog_timer_fn.cold.6+0x62/0x10a [15944.272409] ? __hrtimer_run_queues+0x1c2/0x4d0 [15944.277864] ? watchdog+0x40/0x40 [15944.283906] ? hrtimer_interrupt+0x128/0x2c0 [15944.294874] ? smp_apic_timer_interrupt+0xa6/0x270 [15944.301200] ? apic_timer_interrupt+0xf/0x20 [15944.307221] [15944.309134] ? native_safe_halt+0x2e/0x50 [15944.315098] kvm_wait+0xcd/0xe0 [15944.316126] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15944.337865] do_raw_spin_lock+0x177/0x190 [15944.339281] _raw_spin_lock+0x21/0x30 [15944.349873] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15944.355542] ? do_raw_spin_unlock+0x75/0x190 [15944.361110] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15944.368052] ? do_wait_intr+0xf0/0xf0 [15944.384070] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15944.386184] mdc_reint+0x2d/0x110 [mdc] [15944.403940] mdc_unlink+0x658/0x870 [mdc] [15944.405123] lmv_unlink+0x34d/0xbd0 [lmv] [15944.415878] ll_unlink+0x24d/0x7f0 [lustre] [15944.417139] vfs_unlink+0x202/0x330 [15944.418180] do_unlinkat+0x2fa/0x460 [15944.425791] __x64_sys_unlink+0x37/0x50 [15944.427824] do_syscall_64+0xc1/0x440 [15944.430276] entry_SYSCALL_64_after_hwframe+0x49/0xae [15944.432727] RIP: 0033:0x7f151bdee10b [15944.434788] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [15944.445707] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [15944.450741] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [15944.455060] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [15944.458924] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15944.463082] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15944.466918] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15948.057079] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [15948.063153] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15948.095607] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15948.102577] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15948.108626] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15948.113005] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15948.125153] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15948.130542] RAX: 0000000000002948 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15948.135956] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [15948.140589] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [15948.145243] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [15948.149713] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [15948.153639] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [15948.159069] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15948.162785] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [15948.171434] Call Trace: [15948.173139] [15948.173946] ? show_regs.cold.9+0x22/0x2f [15948.178822] ? watchdog_timer_fn.cold.6+0x62/0x10a [15948.184253] ? __hrtimer_run_queues+0x1c2/0x4d0 [15948.188113] ? watchdog+0x40/0x40 [15948.193150] ? hrtimer_interrupt+0x128/0x2c0 [15948.197080] ? smp_apic_timer_interrupt+0xa6/0x270 [15948.203738] ? apic_timer_interrupt+0xf/0x20 [15948.209199] [15948.210650] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15948.216747] do_raw_spin_lock+0x177/0x190 [15948.218653] _raw_spin_lock+0x21/0x30 [15948.223884] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15948.226765] ? do_raw_spin_unlock+0x75/0x190 [15948.229796] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15948.232822] ? do_wait_intr+0xf0/0xf0 [15948.235209] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15948.248182] mdc_close+0x53b/0xd70 [mdc] [15948.250110] lmv_close+0x18d/0x490 [lmv] [15948.257429] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15948.263951] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15948.268233] ll_md_real_close+0xa5/0x2a0 [lustre] [15948.271942] ll_md_close+0x274/0x800 [lustre] [15948.273401] ll_file_release+0x128/0xc50 [lustre] [15948.284307] __fput+0x10d/0x440 [15948.285140] ____fput+0x12/0x20 [15948.286956] task_work_run+0xc6/0x110 [15948.292592] exit_to_usermode_loop+0x1dd/0x1f0 [15948.294207] do_syscall_64+0x3d6/0x440 [15948.298899] entry_SYSCALL_64_after_hwframe+0x49/0xae [15948.301181] RIP: 0033:0x7f151bee3cd8 [15948.302984] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15948.311078] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15948.314919] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [15948.317068] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [15948.320095] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15948.322499] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15948.325732] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15950.305055] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14019s! [15950.309981] Showing busy workqueues and worker pools: [15950.312832] workqueue events: flags=0x0 [15950.315148] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15950.318926] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [15950.327298] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15950.334870] pending: kfree_rcu_monitor [15950.337893] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15950.343358] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [15950.349373] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15950.353120] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [15950.358091] workqueue events_long: flags=0x0 [15950.359974] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15950.362985] pending: lu_objects_flush [obdclass] [15950.365580] workqueue events_unbound: flags=0x2 [15950.367987] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15950.371104] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15950.376297] workqueue events_power_efficient: flags=0x80 [15950.378958] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15950.384426] pending: do_cache_clean, neigh_periodic_work, check_lifetime [15950.388992] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15950.392518] pending: neigh_periodic_work [15950.394389] workqueue mm_percpu_wq: flags=0x8 [15950.397094] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15950.400229] pending: vmstat_update, lru_add_drain_per_cpu [15950.403473] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15950.411380] pending: vmstat_update [15950.413078] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15950.416775] pending: lru_add_drain_per_cpu [15950.419152] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [15950.422937] pending: lru_add_drain_per_cpu BAR(48) [15950.424829] workqueue writeback: flags=0x4a [15950.426794] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [15950.429782] pending: wb_workfn [15950.431232] workqueue xprtiod: flags=0xa [15950.432829] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15950.436883] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [15950.441034] workqueue ipv6_addrconf: flags=0x40008 [15950.443225] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15950.446259] pending: addrconf_verify_work [15950.447851] workqueue knbd0-recv: flags=0x1a [15950.450151] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15950.459247] in-flight: 658:recv_work [15950.460605] workqueue ptlrpc_pinger: flags=0x2 [15950.465419] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15950.470575] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15950.477988] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14019s workers=4 idle: 32984 37 8 [15950.483307] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15935s workers=2 idle: 661 [15956.056048] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [15956.060446] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15956.069114] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [15956.085940] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15956.085954] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15956.085960] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15956.086242] RIP: 0010:native_safe_halt+0x2e/0x50 [15956.086253] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15956.086257] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15956.086262] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15956.086264] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [15956.086267] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [15956.086269] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [15956.086271] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [15956.086274] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [15956.086277] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15956.086280] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15956.086289] Call Trace: [15956.086294] [15956.086299] ? show_regs.cold.9+0x22/0x2f [15956.086310] ? watchdog_timer_fn.cold.6+0x62/0x10a [15956.086317] ? __hrtimer_run_queues+0x1c2/0x4d0 [15956.086322] ? watchdog+0x40/0x40 [15956.086326] ? hrtimer_interrupt+0x128/0x2c0 [15956.086330] ? smp_apic_timer_interrupt+0xa6/0x270 [15956.086337] ? apic_timer_interrupt+0xf/0x20 [15956.086341] [15956.086345] ? native_safe_halt+0x2e/0x50 [15956.086349] kvm_wait+0xcd/0xe0 [15956.086356] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15956.086364] do_raw_spin_lock+0x177/0x190 [15956.086369] _raw_spin_lock+0x21/0x30 [15956.086373] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15956.086587] process_one_work+0x2c8/0x700 [15956.086594] worker_thread+0x296/0x6e0 [15956.086599] ? rescuer_thread+0x570/0x570 [15956.086603] kthread+0x1d1/0x200 [15956.086608] ? set_kthread_struct+0x70/0x70 [15956.086612] ret_from_fork+0x1f/0x30 [15956.242730] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15956.282677] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15956.294138] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15956.305775] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15956.315269] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15956.338736] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15956.343057] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [15956.349235] RDX: 0000000000006d55 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [15956.357193] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [15956.363184] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [15956.367106] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15956.371473] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [15956.376213] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15956.382166] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15956.396343] Call Trace: [15956.399654] [15956.403111] ? show_regs.cold.9+0x22/0x2f [15956.406674] ? watchdog_timer_fn.cold.6+0x62/0x10a [15956.414494] ? __hrtimer_run_queues+0x1c2/0x4d0 [15956.420222] ? watchdog+0x40/0x40 [15956.422617] ? hrtimer_interrupt+0x128/0x2c0 [15956.427785] ? smp_apic_timer_interrupt+0xa6/0x270 [15956.430176] ? apic_timer_interrupt+0xf/0x20 [15956.432609] [15956.436661] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15956.442046] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15956.449258] do_raw_spin_lock+0x177/0x190 [15956.452148] _raw_spin_lock+0x21/0x30 [15956.455288] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15956.457611] obd_mod_free+0x46/0x140 [ptlrpc] [15956.459830] mdc_commit_open+0x91/0xb0 [mdc] [15956.462058] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15956.464497] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15956.469145] after_reply+0xe2c/0x1780 [ptlrpc] [15956.477104] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15956.484044] ? __next_timer_interrupt+0x160/0x160 [15956.489000] ptlrpcd+0xa82/0xe40 [ptlrpc] [15956.492044] ? do_wait_intr+0xf0/0xf0 [15956.493835] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15956.503320] kthread+0x1d1/0x200 [15956.504591] ? set_kthread_struct+0x70/0x70 [15956.506096] ret_from_fork+0x1f/0x30 [15976.052036] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [15976.054870] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15976.057022] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [15976.103453] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15976.103468] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15976.103472] RIP: 0010:native_safe_halt+0x2e/0x50 [15976.103486] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15976.103490] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [15976.103496] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15976.103499] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [15976.103501] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [15976.103504] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [15976.103506] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [15976.103510] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [15976.103513] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15976.103516] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [15976.103525] Call Trace: [15976.103532] [15976.103538] ? show_regs.cold.9+0x22/0x2f [15976.103548] ? watchdog_timer_fn.cold.6+0x62/0x10a [15976.103555] ? __hrtimer_run_queues+0x1c2/0x4d0 [15976.103559] ? watchdog+0x40/0x40 [15976.103563] ? hrtimer_interrupt+0x128/0x2c0 [15976.103567] ? smp_apic_timer_interrupt+0xa6/0x270 [15976.103573] ? apic_timer_interrupt+0xf/0x20 [15976.103577] [15976.103580] ? native_safe_halt+0x2e/0x50 [15976.103584] kvm_wait+0xcd/0xe0 [15976.103591] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15976.103598] do_raw_spin_lock+0x177/0x190 [15976.103603] _raw_spin_lock+0x21/0x30 [15976.103610] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15976.103885] ? do_raw_spin_unlock+0x75/0x190 [15976.103892] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15976.113321] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15976.113406] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15976.113414] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15976.113417] RIP: 0010:native_safe_halt+0x2e/0x50 [15976.113430] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15976.113434] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15976.113439] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15976.113442] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [15976.113444] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [15976.113447] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [15976.113449] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [15976.113453] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [15976.113456] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15976.113458] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [15976.113467] Call Trace: [15976.113473] [15976.113478] ? show_regs.cold.9+0x22/0x2f [15976.113488] ? watchdog_timer_fn.cold.6+0x62/0x10a [15976.113494] ? __hrtimer_run_queues+0x1c2/0x4d0 [15976.113498] ? watchdog+0x40/0x40 [15976.113503] ? hrtimer_interrupt+0x128/0x2c0 [15976.113506] ? smp_apic_timer_interrupt+0xa6/0x270 [15976.113514] ? apic_timer_interrupt+0xf/0x20 [15976.113517] [15976.113520] ? native_safe_halt+0x2e/0x50 [15976.113525] kvm_wait+0xcd/0xe0 [15976.113532] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15976.113540] do_raw_spin_lock+0x177/0x190 [15976.113545] _raw_spin_lock+0x21/0x30 [15976.113549] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15976.114486] ? do_raw_spin_unlock+0x75/0x190 [15976.114494] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15976.114772] ? do_wait_intr+0xf0/0xf0 [15976.114781] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15976.115330] mdc_close+0x53b/0xd70 [mdc] [15976.115372] lmv_close+0x18d/0x490 [lmv] [15976.115405] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [15976.115558] ll_close_inode_openhandle+0x493/0xea0 [lustre] [15976.115784] ll_md_real_close+0xa5/0x2a0 [lustre] [15976.115874] ll_md_close+0x274/0x800 [lustre] [15976.116027] ll_file_release+0x128/0xc50 [lustre] [15976.116250] __fput+0x10d/0x440 [15976.116259] ____fput+0x12/0x20 [15976.116263] task_work_run+0xc6/0x110 [15976.116271] exit_to_usermode_loop+0x1dd/0x1f0 [15976.116277] do_syscall_64+0x3d6/0x440 [15976.116282] entry_SYSCALL_64_after_hwframe+0x49/0xae [15976.116289] RIP: 0033:0x7f151bee3cd8 [15976.116297] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [15976.116301] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [15976.116305] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [15976.116308] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [15976.116310] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15976.116312] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15976.116314] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15976.479363] ? do_wait_intr+0xf0/0xf0 [15976.484044] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15976.489467] mdc_reint+0x2d/0x110 [mdc] [15976.490815] mdc_unlink+0x658/0x870 [mdc] [15976.494386] lmv_unlink+0x34d/0xbd0 [lmv] [15976.500801] ll_unlink+0x24d/0x7f0 [lustre] [15976.506632] vfs_unlink+0x202/0x330 [15976.510390] do_unlinkat+0x2fa/0x460 [15976.514187] __x64_sys_unlink+0x37/0x50 [15976.516193] do_syscall_64+0xc1/0x440 [15976.521907] entry_SYSCALL_64_after_hwframe+0x49/0xae [15976.526721] RIP: 0033:0x7f151bdee10b [15976.530426] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [15976.541722] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [15976.545189] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [15976.548741] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [15976.555645] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [15976.559191] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [15976.563371] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [15980.512691] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14049s! [15980.516132] Showing busy workqueues and worker pools: [15980.519300] workqueue events: flags=0x0 [15980.522523] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15980.527465] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [15980.544466] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15980.550011] pending: kfree_rcu_monitor [15980.552572] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15980.558346] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [15980.563697] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15980.567079] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [15980.570602] workqueue events_long: flags=0x0 [15980.576343] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15980.581259] pending: lu_objects_flush [obdclass] [15980.588463] workqueue events_unbound: flags=0x2 [15980.593290] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [15980.598048] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [15980.604759] workqueue events_power_efficient: flags=0x80 [15980.607325] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15980.612278] pending: do_cache_clean, neigh_periodic_work, check_lifetime [15980.618316] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15980.622948] pending: neigh_periodic_work [15980.629990] workqueue mm_percpu_wq: flags=0x8 [15980.638761] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15980.645590] pending: vmstat_update, lru_add_drain_per_cpu [15980.652414] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15980.662939] pending: vmstat_update [15980.666153] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15980.672748] pending: lru_add_drain_per_cpu [15980.675392] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [15980.685315] pending: lru_add_drain_per_cpu BAR(48) [15980.689008] workqueue writeback: flags=0x4a [15980.690857] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [15980.703710] pending: wb_workfn [15980.706636] workqueue xprtiod: flags=0xa [15980.711311] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [15980.719013] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [15980.723954] workqueue ipv6_addrconf: flags=0x40008 [15980.733409] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15980.742844] pending: addrconf_verify_work [15980.745448] workqueue knbd0-recv: flags=0x1a [15980.747843] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15980.751493] in-flight: 658:recv_work [15980.756575] workqueue ptlrpc_pinger: flags=0x2 [15980.760837] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15980.765911] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [15980.769555] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14050s workers=4 idle: 32984 37 8 [15980.777011] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15965s workers=2 idle: 661 [15984.056076] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [15984.056557] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [15984.061764] Modules linked in: [15984.065095] Modules linked in: [15984.065097] lustre(O) [15984.067452] lustre(O) [15984.070295] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15984.070371] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15984.070378] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15984.070384] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15984.070773] RIP: 0010:native_safe_halt+0x2e/0x50 [15984.070783] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15984.070787] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15984.070791] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [15984.070793] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [15984.070800] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [15984.070802] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [15984.070805] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [15984.070808] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [15984.070811] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15984.070813] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15984.070820] Call Trace: [15984.070826] [15984.070831] ? show_regs.cold.9+0x22/0x2f [15984.070841] ? watchdog_timer_fn.cold.6+0x62/0x10a [15984.070847] ? __hrtimer_run_queues+0x1c2/0x4d0 [15984.070852] ? watchdog+0x40/0x40 [15984.070861] ? hrtimer_interrupt+0x128/0x2c0 [15984.070866] ? smp_apic_timer_interrupt+0xa6/0x270 [15984.070872] ? apic_timer_interrupt+0xf/0x20 [15984.070876] [15984.070879] ? native_safe_halt+0x2e/0x50 [15984.070883] kvm_wait+0xcd/0xe0 [15984.070890] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15984.070898] do_raw_spin_lock+0x177/0x190 [15984.070902] _raw_spin_lock+0x21/0x30 [15984.070907] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15984.071239] process_one_work+0x2c8/0x700 [15984.071248] worker_thread+0x296/0x6e0 [15984.071252] ? rescuer_thread+0x570/0x570 [15984.071261] kthread+0x1d1/0x200 [15984.071265] ? set_kthread_struct+0x70/0x70 [15984.071269] ret_from_fork+0x1f/0x30 [15984.293254] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15984.335915] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15984.344806] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15984.357675] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15984.364666] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15984.385652] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15984.389876] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [15984.393789] RDX: 0000000000002610 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [15984.397800] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [15984.405519] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [15984.407868] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15984.413644] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [15984.418838] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15984.420722] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [15984.424059] Call Trace: [15984.424907] [15984.428989] ? show_regs.cold.9+0x22/0x2f [15984.431697] ? watchdog_timer_fn.cold.6+0x62/0x10a [15984.437619] ? __hrtimer_run_queues+0x1c2/0x4d0 [15984.439587] ? watchdog+0x40/0x40 [15984.440649] ? hrtimer_interrupt+0x128/0x2c0 [15984.442038] ? smp_apic_timer_interrupt+0xa6/0x270 [15984.444843] ? apic_timer_interrupt+0xf/0x20 [15984.458879] [15984.459599] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15984.468377] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15984.473864] do_raw_spin_lock+0x177/0x190 [15984.479179] _raw_spin_lock+0x21/0x30 [15984.480331] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15984.489547] obd_mod_free+0x46/0x140 [ptlrpc] [15984.493250] mdc_commit_open+0x91/0xb0 [mdc] [15984.502181] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15984.504741] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15984.507410] after_reply+0xe2c/0x1780 [ptlrpc] [15984.510804] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15984.513440] ? __next_timer_interrupt+0x160/0x160 [15984.515270] ptlrpcd+0xa82/0xe40 [ptlrpc] [15984.517165] ? do_wait_intr+0xf0/0xf0 [15984.518462] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15984.520520] kthread+0x1d1/0x200 [15984.522352] ? set_kthread_struct+0x70/0x70 [15984.523799] ret_from_fork+0x1f/0x30 [16004.057307] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [16004.060501] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16004.081729] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16004.086612] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16004.098940] RIP: 0010:native_safe_halt+0x2e/0x50 [16004.100616] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16004.116213] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16004.117983] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16004.120473] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [16004.127723] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [16004.135785] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [16004.139405] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [16004.144389] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [16004.149668] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16004.152991] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [16004.156451] Call Trace: [16004.157300] [16004.157753] ? show_regs.cold.9+0x22/0x2f [16004.163120] ? watchdog_timer_fn.cold.6+0x62/0x10a [16004.164618] ? __hrtimer_run_queues+0x1c2/0x4d0 [16004.166550] ? watchdog+0x40/0x40 [16004.168641] ? hrtimer_interrupt+0x128/0x2c0 [16004.172759] ? smp_apic_timer_interrupt+0xa6/0x270 [16004.175493] ? apic_timer_interrupt+0xf/0x20 [16004.178153] [16004.178734] ? native_safe_halt+0x2e/0x50 [16004.180105] kvm_wait+0xcd/0xe0 [16004.182172] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16004.184968] do_raw_spin_lock+0x177/0x190 [16004.185915] _raw_spin_lock+0x21/0x30 [16004.186849] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16004.191333] ? do_raw_spin_unlock+0x75/0x190 [16004.192401] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16004.195494] ? do_wait_intr+0xf0/0xf0 [16004.196762] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16004.200456] mdc_close+0x53b/0xd70 [mdc] [16004.201689] lmv_close+0x18d/0x490 [lmv] [16004.203851] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [16004.206764] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16004.214391] ll_md_real_close+0xa5/0x2a0 [lustre] [16004.219352] ll_md_close+0x274/0x800 [lustre] [16004.222086] ll_file_release+0x128/0xc50 [lustre] [16004.223641] __fput+0x10d/0x440 [16004.227047] ____fput+0x12/0x20 [16004.229710] task_work_run+0xc6/0x110 [16004.230653] exit_to_usermode_loop+0x1dd/0x1f0 [16004.231729] do_syscall_64+0x3d6/0x440 [16004.232658] entry_SYSCALL_64_after_hwframe+0x49/0xae [16004.235630] RIP: 0033:0x7f151bee3cd8 [16004.236777] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16004.247182] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16004.250036] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [16004.252653] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [16004.255443] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16004.258151] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16004.260724] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16008.053138] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [16008.061530] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16008.078490] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16008.082983] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16008.086285] RIP: 0010:native_safe_halt+0x2e/0x50 [16008.088333] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16008.095400] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16008.098325] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16008.100994] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [16008.103532] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [16008.106287] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [16008.109061] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [16008.111630] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [16008.114536] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16008.116601] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [16008.119101] Call Trace: [16008.119986] [16008.120645] ? show_regs.cold.9+0x22/0x2f [16008.122115] ? watchdog_timer_fn.cold.6+0x62/0x10a [16008.124022] ? __hrtimer_run_queues+0x1c2/0x4d0 [16008.125657] ? watchdog+0x40/0x40 [16008.127082] ? hrtimer_interrupt+0x128/0x2c0 [16008.128637] ? smp_apic_timer_interrupt+0xa6/0x270 [16008.131121] ? apic_timer_interrupt+0xf/0x20 [16008.132648] [16008.133428] ? native_safe_halt+0x2e/0x50 [16008.135353] kvm_wait+0xcd/0xe0 [16008.136475] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16008.138741] do_raw_spin_lock+0x177/0x190 [16008.140275] _raw_spin_lock+0x21/0x30 [16008.141735] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16008.144570] ? do_raw_spin_unlock+0x75/0x190 [16008.146437] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16008.149361] ? do_wait_intr+0xf0/0xf0 [16008.150711] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16008.153136] mdc_reint+0x2d/0x110 [mdc] [16008.155045] mdc_unlink+0x658/0x870 [mdc] [16008.156579] lmv_unlink+0x34d/0xbd0 [lmv] [16008.158308] ll_unlink+0x24d/0x7f0 [lustre] [16008.160021] vfs_unlink+0x202/0x330 [16008.162082] do_unlinkat+0x2fa/0x460 [16008.164035] __x64_sys_unlink+0x37/0x50 [16008.165535] do_syscall_64+0xc1/0x440 [16008.167210] entry_SYSCALL_64_after_hwframe+0x49/0xae [16008.169279] RIP: 0033:0x7f151bdee10b [16008.170683] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [16008.178417] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [16008.181509] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [16008.184683] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [16008.187734] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16008.191100] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16008.194734] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16011.234037] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14080s! [16011.236816] Showing busy workqueues and worker pools: [16011.238656] workqueue events: flags=0x0 [16011.240062] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16011.242425] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [16011.245967] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16011.248537] pending: kfree_rcu_monitor [16011.251379] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16011.255502] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [16011.266471] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16011.272027] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [16011.275349] workqueue events_long: flags=0x0 [16011.277365] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16011.279693] pending: lu_objects_flush [obdclass] [16011.282034] workqueue events_unbound: flags=0x2 [16011.284238] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [16011.287068] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16011.293166] workqueue events_power_efficient: flags=0x80 [16011.301819] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16011.311138] pending: do_cache_clean, neigh_periodic_work, check_lifetime [16011.315530] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16011.321987] pending: neigh_periodic_work [16011.324459] workqueue mm_percpu_wq: flags=0x8 [16011.326708] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16011.329940] pending: vmstat_update, lru_add_drain_per_cpu [16011.332870] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16011.336478] pending: vmstat_update [16011.338368] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16011.341968] pending: lru_add_drain_per_cpu [16011.344677] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [16011.347966] pending: lru_add_drain_per_cpu BAR(48) [16011.352505] workqueue writeback: flags=0x4a [16011.354428] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [16011.357520] pending: wb_workfn [16011.359779] workqueue xprtiod: flags=0xa [16011.362279] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16011.369262] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [16011.375014] workqueue ipv6_addrconf: flags=0x40008 [16011.376363] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16011.382812] pending: addrconf_verify_work [16011.389619] workqueue knbd0-recv: flags=0x1a [16011.392092] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16011.394274] in-flight: 658:recv_work [16011.395475] workqueue ptlrpc_pinger: flags=0x2 [16011.397610] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16011.399401] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [16011.402458] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14080s workers=4 idle: 32984 37 8 [16011.405361] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15996s workers=2 idle: 661 [16012.055010] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [16012.056024] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [16012.061013] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16012.061094] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16012.061100] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16012.061103] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16012.061118] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16012.061123] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16012.061127] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [16012.061129] RDX: 000000000000046a RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [16012.061131] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [16012.061133] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [16012.061135] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16012.061138] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [16012.061141] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16012.061143] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16012.061151] Call Trace: [16012.061156] [16012.061161] ? show_regs.cold.9+0x22/0x2f [16012.061168] ? watchdog_timer_fn.cold.6+0x62/0x10a [16012.061174] ? __hrtimer_run_queues+0x1c2/0x4d0 [16012.061178] ? watchdog+0x40/0x40 [16012.061182] ? hrtimer_interrupt+0x128/0x2c0 [16012.061186] ? smp_apic_timer_interrupt+0xa6/0x270 [16012.061195] ? apic_timer_interrupt+0xf/0x20 [16012.061200] [16012.061209] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16012.061215] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16012.061220] do_raw_spin_lock+0x177/0x190 [16012.061225] _raw_spin_lock+0x21/0x30 [16012.061231] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16012.066504] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) [16012.084386] obd_mod_free+0x46/0x140 [ptlrpc] [16012.090352] rpcsec_gss_krb5 [16012.094186] mdc_commit_open+0x91/0xb0 [mdc] [16012.097838] auth_rpcgss [16012.103663] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16012.107232] nfsv4 dns_resolver [16012.109663] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16012.113980] intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16012.114022] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16012.114029] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16012.114036] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16012.114324] RIP: 0010:native_safe_halt+0x2e/0x50 [16012.114334] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16012.114338] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16012.114343] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16012.114345] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [16012.114348] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [16012.114350] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [16012.114352] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [16012.114355] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [16012.114358] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16012.114360] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16012.114369] Call Trace: [16012.114374] [16012.114379] ? show_regs.cold.9+0x22/0x2f [16012.114389] ? watchdog_timer_fn.cold.6+0x62/0x10a [16012.114395] ? __hrtimer_run_queues+0x1c2/0x4d0 [16012.114399] ? watchdog+0x40/0x40 [16012.114402] ? hrtimer_interrupt+0x128/0x2c0 [16012.114407] ? smp_apic_timer_interrupt+0xa6/0x270 [16012.114414] ? apic_timer_interrupt+0xf/0x20 [16012.114419] [16012.114424] ? native_safe_halt+0x2e/0x50 [16012.114428] kvm_wait+0xcd/0xe0 [16012.114521] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16012.114530] do_raw_spin_lock+0x177/0x190 [16012.114534] _raw_spin_lock+0x21/0x30 [16012.114539] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16012.115291] process_one_work+0x2c8/0x700 [16012.115298] worker_thread+0x296/0x6e0 [16012.115303] ? rescuer_thread+0x570/0x570 [16012.115307] kthread+0x1d1/0x200 [16012.115311] ? set_kthread_struct+0x70/0x70 [16012.115315] ret_from_fork+0x1f/0x30 [16012.279333] after_reply+0xe2c/0x1780 [ptlrpc] [16012.281186] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16012.284461] ? __next_timer_interrupt+0x160/0x160 [16012.287938] ptlrpcd+0xa82/0xe40 [ptlrpc] [16012.291872] ? do_wait_intr+0xf0/0xf0 [16012.294517] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16012.297449] kthread+0x1d1/0x200 [16012.299941] ? set_kthread_struct+0x70/0x70 [16012.303479] ret_from_fork+0x1f/0x30 [16024.942169] rcu: INFO: rcu_sched self-detected stall on CPU [16024.944905] rcu: 1-...!: (8437132 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [16024.951359] (t=14100295 jiffies g=249657 q=6674) [16024.954403] rcu: rcu_sched kthread starved for 14095247 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [16024.959260] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [16024.962346] rcu: RCU grace-period kthread stack dump: [16024.964312] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [16024.967543] Call Trace: [16024.968335] __schedule+0x351/0xcb0 [16024.969864] schedule+0xc0/0x180 [16024.971830] schedule_timeout+0xb4/0x190 [16024.974361] ? do_raw_spin_unlock+0x75/0x190 [16024.976236] ? __next_timer_interrupt+0x160/0x160 [16024.978497] ? prepare_to_swait_event+0xf3/0x140 [16024.980947] rcu_gp_kthread+0x8dd/0x15b0 [16024.983393] ? force_qs_rnp+0x2c0/0x2c0 [16024.985276] kthread+0x1d1/0x200 [16024.986670] ? set_kthread_struct+0x70/0x70 [16024.988696] ret_from_fork+0x1f/0x30 [16024.990685] rcu: Stack dump where RCU GP kthread last ran: [16024.993384] Sending NMI from CPU 1 to CPUs 0: [16024.995588] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [16024.999386] NMI backtrace for cpu 1 [16025.001255] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16025.008124] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16025.012986] Call Trace: [16025.014072] [16025.014874] ? dump_stack+0xbb/0x10e [16025.017065] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [16025.019315] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [16025.021556] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [16025.023735] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [16025.026388] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [16025.029291] ? rcu_dump_cpu_stacks+0x11b/0x171 [16025.031368] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [16025.033660] ? update_process_times+0x6d/0xc0 [16025.035722] ? tick_sched_handle.isra.18+0x27/0x90 [16025.037900] ? tick_sched_timer+0x53/0xb0 [16025.040100] ? __hrtimer_run_queues+0x1c2/0x4d0 [16025.042214] ? tick_sched_handle.isra.18+0x90/0x90 [16025.044803] ? hrtimer_interrupt+0x128/0x2c0 [16025.046562] ? smp_apic_timer_interrupt+0xa6/0x270 [16025.048865] ? apic_timer_interrupt+0xf/0x20 [16025.050835] [16025.051873] ? native_safe_halt+0x2e/0x50 [16025.054586] ? kvm_wait+0xcd/0xe0 [16025.056266] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16025.058992] ? do_raw_spin_lock+0x177/0x190 [16025.061446] ? _raw_spin_lock+0x21/0x30 [16025.063381] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16025.065425] ? obd_mod_free+0x46/0x140 [ptlrpc] [16025.067539] ? mdc_commit_open+0x91/0xb0 [mdc] [16025.068909] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16025.071513] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16025.074466] ? after_reply+0xe2c/0x1780 [ptlrpc] [16025.077275] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16025.080239] ? __next_timer_interrupt+0x160/0x160 [16025.082376] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [16025.084629] ? do_wait_intr+0xf0/0xf0 [16025.085996] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16025.088804] ? kthread+0x1d1/0x200 [16025.090567] ? set_kthread_struct+0x70/0x70 [16025.092632] ? ret_from_fork+0x1f/0x30 [16032.057030] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [16032.060607] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16032.092087] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16032.098289] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16032.102181] RIP: 0010:native_safe_halt+0x2e/0x50 [16032.104231] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16032.119427] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16032.124201] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16032.127679] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [16032.131246] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [16032.134537] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [16032.138453] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [16032.141859] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [16032.151680] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16032.157252] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [16032.165383] Call Trace: [16032.170318] [16032.172313] ? show_regs.cold.9+0x22/0x2f [16032.176244] ? watchdog_timer_fn.cold.6+0x62/0x10a [16032.180319] ? __hrtimer_run_queues+0x1c2/0x4d0 [16032.186597] ? watchdog+0x40/0x40 [16032.188378] ? hrtimer_interrupt+0x128/0x2c0 [16032.193037] ? smp_apic_timer_interrupt+0xa6/0x270 [16032.194832] ? apic_timer_interrupt+0xf/0x20 [16032.198024] [16032.198497] ? native_safe_halt+0x2e/0x50 [16032.199563] kvm_wait+0xcd/0xe0 [16032.204147] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16032.210130] do_raw_spin_lock+0x177/0x190 [16032.212354] _raw_spin_lock+0x21/0x30 [16032.214225] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16032.218640] ? do_raw_spin_unlock+0x75/0x190 [16032.224198] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16032.226868] ? do_wait_intr+0xf0/0xf0 [16032.233856] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16032.237100] mdc_close+0x53b/0xd70 [mdc] [16032.247068] lmv_close+0x18d/0x490 [lmv] [16032.254407] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [16032.258368] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16032.262264] ll_md_real_close+0xa5/0x2a0 [lustre] [16032.270248] ll_md_close+0x274/0x800 [lustre] [16032.274462] ll_file_release+0x128/0xc50 [lustre] [16032.277847] __fput+0x10d/0x440 [16032.283183] ____fput+0x12/0x20 [16032.286356] task_work_run+0xc6/0x110 [16032.291298] exit_to_usermode_loop+0x1dd/0x1f0 [16032.294303] do_syscall_64+0x3d6/0x440 [16032.299568] entry_SYSCALL_64_after_hwframe+0x49/0xae [16032.307850] RIP: 0033:0x7f151bee3cd8 [16032.312056] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16032.331619] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16032.337126] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [16032.341432] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [16032.346140] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16032.349605] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16032.354248] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16040.056429] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [16040.063895] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16040.127790] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16040.151021] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16040.162950] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16040.166818] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16040.174047] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16040.201750] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16040.210355] RAX: 0000000000007d8d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16040.215117] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [16040.224212] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [16040.236528] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [16040.239201] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [16040.253785] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [16040.265213] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16040.275854] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16040.286812] Call Trace: [16040.293076] [16040.295796] ? show_regs.cold.9+0x22/0x2f [16040.298034] ? watchdog_timer_fn.cold.6+0x62/0x10a [16040.310122] ? __hrtimer_run_queues+0x1c2/0x4d0 [16040.315908] ? watchdog+0x40/0x40 [16040.317851] ? hrtimer_interrupt+0x128/0x2c0 [16040.321432] ? smp_apic_timer_interrupt+0xa6/0x270 [16040.325384] ? apic_timer_interrupt+0xf/0x20 [16040.328523] [16040.329590] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16040.333048] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16040.335353] do_raw_spin_lock+0x177/0x190 [16040.336959] _raw_spin_lock+0x21/0x30 [16040.338265] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16040.342434] process_one_work+0x2c8/0x700 [16040.344164] worker_thread+0x296/0x6e0 [16040.347140] ? rescuer_thread+0x570/0x570 [16040.350081] kthread+0x1d1/0x200 [16040.351899] ? set_kthread_struct+0x70/0x70 [16040.355018] ret_from_fork+0x1f/0x30 [16041.440909] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14110s! [16041.458529] Showing busy workqueues and worker pools: [16041.463225] workqueue events: flags=0x0 [16041.471700] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16041.478292] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [16041.489126] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16041.496217] pending: kfree_rcu_monitor [16041.501343] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16041.507391] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [16041.517428] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16041.528640] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [16041.536762] workqueue events_long: flags=0x0 [16041.538557] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16041.546739] pending: lu_objects_flush [obdclass] [16041.550558] workqueue events_unbound: flags=0x2 [16041.559092] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [16041.565491] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16041.576666] workqueue events_power_efficient: flags=0x80 [16041.578907] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16041.593813] pending: do_cache_clean, neigh_periodic_work, check_lifetime [16041.600427] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16041.607469] pending: neigh_periodic_work [16041.610151] workqueue mm_percpu_wq: flags=0x8 [16041.611803] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16041.614332] pending: vmstat_update, lru_add_drain_per_cpu [16041.617272] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16041.621590] pending: vmstat_update [16041.623469] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16041.627458] pending: lru_add_drain_per_cpu [16041.629828] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [16041.633414] pending: lru_add_drain_per_cpu BAR(48) [16041.636805] workqueue writeback: flags=0x4a [16041.639543] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [16041.643225] pending: wb_workfn [16041.644697] workqueue xprtiod: flags=0xa [16041.647182] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16041.650339] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [16041.655536] workqueue ipv6_addrconf: flags=0x40008 [16041.658508] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16041.661819] pending: addrconf_verify_work [16041.663885] workqueue knbd0-recv: flags=0x1a [16041.665994] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16041.669801] in-flight: 658:recv_work [16041.672444] workqueue ptlrpc_pinger: flags=0x2 [16041.675103] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16041.678821] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [16041.682125] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14111s workers=4 idle: 32984 37 8 [16041.686311] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16026s workers=2 idle: 661 [16052.055026] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [16052.057635] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16052.079694] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16052.086160] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16052.092832] RIP: 0010:native_safe_halt+0x2e/0x50 [16052.096760] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16052.113130] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16052.120365] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [16052.124285] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [16052.127249] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [16052.130736] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [16052.134052] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16052.138596] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [16052.141819] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16052.146774] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16052.152940] Call Trace: [16052.154969] [16052.156313] ? show_regs.cold.9+0x22/0x2f [16052.159846] ? watchdog_timer_fn.cold.6+0x62/0x10a [16052.163511] ? __hrtimer_run_queues+0x1c2/0x4d0 [16052.166585] ? watchdog+0x40/0x40 [16052.168834] ? hrtimer_interrupt+0x128/0x2c0 [16052.172224] ? smp_apic_timer_interrupt+0xa6/0x270 [16052.174959] ? apic_timer_interrupt+0xf/0x20 [16052.178619] [16052.179137] ? native_safe_halt+0x2e/0x50 [16052.181656] kvm_wait+0xcd/0xe0 [16052.183534] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16052.187423] do_raw_spin_lock+0x177/0x190 [16052.189993] _raw_spin_lock+0x21/0x30 [16052.192427] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16052.195846] obd_mod_free+0x46/0x140 [ptlrpc] [16052.199161] mdc_commit_open+0x91/0xb0 [mdc] [16052.201592] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16052.204702] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16052.208836] after_reply+0xe2c/0x1780 [ptlrpc] [16052.212593] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16052.217686] ? __next_timer_interrupt+0x160/0x160 [16052.219933] ptlrpcd+0xa82/0xe40 [ptlrpc] [16052.222771] ? do_wait_intr+0xf0/0xf0 [16052.225026] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16052.229219] kthread+0x1d1/0x200 [16052.231608] ? set_kthread_struct+0x70/0x70 [16052.234803] ret_from_fork+0x1f/0x30 [16060.063159] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [16060.071918] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16060.134144] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16060.150902] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16060.157361] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16060.170732] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16060.223631] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16060.234196] RAX: 000000000000350a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16060.244793] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [16060.254648] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [16060.270047] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [16060.280127] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [16060.288432] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [16060.309110] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16060.321743] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [16060.339835] Call Trace: [16060.344393] [16060.347858] ? show_regs.cold.9+0x22/0x2f [16060.354809] ? watchdog_timer_fn.cold.6+0x62/0x10a [16060.359020] ? __hrtimer_run_queues+0x1c2/0x4d0 [16060.365071] ? watchdog+0x40/0x40 [16060.372076] ? hrtimer_interrupt+0x128/0x2c0 [16060.375945] ? smp_apic_timer_interrupt+0xa6/0x270 [16060.382014] ? apic_timer_interrupt+0xf/0x20 [16060.384949] [16060.386727] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16060.391584] do_raw_spin_lock+0x177/0x190 [16060.395697] _raw_spin_lock+0x21/0x30 [16060.400830] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16060.406831] ? do_raw_spin_unlock+0x75/0x190 [16060.412380] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16060.422818] ? do_wait_intr+0xf0/0xf0 [16060.428921] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16060.435680] mdc_close+0x53b/0xd70 [mdc] [16060.443647] lmv_close+0x18d/0x490 [lmv] [16060.448129] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [16060.459041] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16060.464961] ll_md_real_close+0xa5/0x2a0 [lustre] [16060.477098] ll_md_close+0x274/0x800 [lustre] [16060.483118] ll_file_release+0x128/0xc50 [lustre] [16060.491017] __fput+0x10d/0x440 [16060.494232] ____fput+0x12/0x20 [16060.500979] task_work_run+0xc6/0x110 [16060.507016] exit_to_usermode_loop+0x1dd/0x1f0 [16060.513317] do_syscall_64+0x3d6/0x440 [16060.516227] entry_SYSCALL_64_after_hwframe+0x49/0xae [16060.526917] RIP: 0033:0x7f151bee3cd8 [16060.532846] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16060.566168] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16060.579511] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [16060.587867] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [16060.595365] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16060.605108] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16060.613847] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16068.056025] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [16068.058412] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16068.060617] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [16068.069960] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16068.069973] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16068.069979] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16068.070262] RIP: 0010:native_safe_halt+0x2e/0x50 [16068.070269] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16068.070272] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16068.070275] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16068.070277] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [16068.070279] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [16068.070280] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [16068.070282] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [16068.070284] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [16068.070286] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16068.070287] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16068.070295] Call Trace: [16068.070300] [16068.070306] ? show_regs.cold.9+0x22/0x2f [16068.070324] ? watchdog_timer_fn.cold.6+0x62/0x10a [16068.070330] ? __hrtimer_run_queues+0x1c2/0x4d0 [16068.070335] ? watchdog+0x40/0x40 [16068.070339] ? hrtimer_interrupt+0x128/0x2c0 [16068.070343] ? smp_apic_timer_interrupt+0xa6/0x270 [16068.070350] ? apic_timer_interrupt+0xf/0x20 [16068.070355] [16068.070358] ? native_safe_halt+0x2e/0x50 [16068.070362] kvm_wait+0xcd/0xe0 [16068.070371] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16068.070378] do_raw_spin_lock+0x177/0x190 [16068.070382] _raw_spin_lock+0x21/0x30 [16068.070385] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16068.070517] process_one_work+0x2c8/0x700 [16068.070522] worker_thread+0x296/0x6e0 [16068.070525] ? rescuer_thread+0x570/0x570 [16068.070527] kthread+0x1d1/0x200 [16068.070530] ? set_kthread_struct+0x70/0x70 [16068.070533] ret_from_fork+0x1f/0x30 [16068.320132] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16068.374916] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16068.386127] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16068.393101] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16068.403705] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16068.426000] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16068.434258] RAX: 0000000000003d3d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16068.443090] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [16068.451926] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [16068.461556] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [16068.471215] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [16068.480922] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [16068.487339] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16068.494074] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [16068.502405] Call Trace: [16068.503466] [16068.504236] ? show_regs.cold.9+0x22/0x2f [16068.509832] ? watchdog_timer_fn.cold.6+0x62/0x10a [16068.513735] ? __hrtimer_run_queues+0x1c2/0x4d0 [16068.515551] ? watchdog+0x40/0x40 [16068.521983] ? hrtimer_interrupt+0x128/0x2c0 [16068.531884] ? smp_apic_timer_interrupt+0xa6/0x270 [16068.536376] ? apic_timer_interrupt+0xf/0x20 [16068.541266] [16068.543424] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16068.547946] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16068.554334] do_raw_spin_lock+0x177/0x190 [16068.560328] _raw_spin_lock+0x21/0x30 [16068.564784] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16068.572329] ? do_raw_spin_unlock+0x75/0x190 [16068.575296] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16068.578427] ? do_wait_intr+0xf0/0xf0 [16068.580360] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16068.584101] mdc_reint+0x2d/0x110 [mdc] [16068.586528] mdc_unlink+0x658/0x870 [mdc] [16068.589248] lmv_unlink+0x34d/0xbd0 [lmv] [16068.591996] ll_unlink+0x24d/0x7f0 [lustre] [16068.595043] vfs_unlink+0x202/0x330 [16068.596893] do_unlinkat+0x2fa/0x460 [16068.598990] __x64_sys_unlink+0x37/0x50 [16068.600778] do_syscall_64+0xc1/0x440 [16068.602567] entry_SYSCALL_64_after_hwframe+0x49/0xae [16068.605443] RIP: 0033:0x7f151bdee10b [16068.607808] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [16068.618387] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [16068.621821] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [16068.626019] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [16068.629779] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16068.634024] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16068.642425] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16072.162915] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14141s! [16072.167149] Showing busy workqueues and worker pools: [16072.172416] workqueue events: flags=0x0 [16072.175243] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16072.179005] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [16072.187518] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16072.192893] pending: kfree_rcu_monitor [16072.195125] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16072.199238] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [16072.212752] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16072.217245] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [16072.222852] workqueue events_long: flags=0x0 [16072.225708] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16072.232116] pending: lu_objects_flush [obdclass] [16072.238984] workqueue events_unbound: flags=0x2 [16072.246403] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [16072.253767] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16072.263892] workqueue events_power_efficient: flags=0x80 [16072.271872] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16072.276422] pending: do_cache_clean, neigh_periodic_work, check_lifetime [16072.293019] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16072.297809] pending: neigh_periodic_work [16072.301847] workqueue mm_percpu_wq: flags=0x8 [16072.305410] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16072.309074] pending: vmstat_update, lru_add_drain_per_cpu [16072.314870] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16072.317025] pending: vmstat_update [16072.325126] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16072.330033] pending: lru_add_drain_per_cpu [16072.332460] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [16072.337093] pending: lru_add_drain_per_cpu BAR(48) [16072.348478] workqueue writeback: flags=0x4a [16072.353523] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [16072.364458] pending: wb_workfn [16072.366771] workqueue xprtiod: flags=0xa [16072.372608] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16072.384831] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [16072.390019] workqueue ipv6_addrconf: flags=0x40008 [16072.393464] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16072.397990] pending: addrconf_verify_work [16072.400056] workqueue knbd0-recv: flags=0x1a [16072.404970] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16072.410485] in-flight: 658:recv_work [16072.415469] workqueue ptlrpc_pinger: flags=0x2 [16072.418904] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16072.422871] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [16072.425239] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14141s workers=4 idle: 32984 37 8 [16072.440518] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16057s workers=2 idle: 661 [16080.059441] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [16080.063988] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16080.092790] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16080.096769] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16080.100259] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16080.102220] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16080.110547] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16080.113603] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [16080.115710] RDX: 0000000000005446 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [16080.119079] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [16080.121800] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [16080.124815] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16080.127901] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [16080.131310] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16080.133586] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16080.136473] Call Trace: [16080.137481] [16080.138371] ? show_regs.cold.9+0x22/0x2f [16080.139897] ? watchdog_timer_fn.cold.6+0x62/0x10a [16080.141883] ? __hrtimer_run_queues+0x1c2/0x4d0 [16080.144236] ? watchdog+0x40/0x40 [16080.145545] ? hrtimer_interrupt+0x128/0x2c0 [16080.147561] ? smp_apic_timer_interrupt+0xa6/0x270 [16080.152883] ? apic_timer_interrupt+0xf/0x20 [16080.155941] [16080.159576] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16080.163054] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16080.169235] do_raw_spin_lock+0x177/0x190 [16080.173298] _raw_spin_lock+0x21/0x30 [16080.177831] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16080.185545] obd_mod_free+0x46/0x140 [ptlrpc] [16080.191307] mdc_commit_open+0x91/0xb0 [mdc] [16080.196486] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16080.201630] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16080.208105] after_reply+0xe2c/0x1780 [ptlrpc] [16080.214621] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16080.226632] ? __next_timer_interrupt+0x160/0x160 [16080.236425] ptlrpcd+0xa82/0xe40 [ptlrpc] [16080.247426] ? do_wait_intr+0xf0/0xf0 [16080.251935] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16080.259655] kthread+0x1d1/0x200 [16080.265497] ? set_kthread_struct+0x70/0x70 [16080.270376] ret_from_fork+0x1f/0x30 [16088.057011] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [16088.059439] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16088.074407] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16088.078694] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16088.081767] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16088.083638] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16088.090180] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16088.092715] RAX: 000000000000015b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16088.095379] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [16088.097654] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [16088.100794] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [16088.103307] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [16088.106229] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [16088.109234] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16088.111180] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [16088.113580] Call Trace: [16088.114465] [16088.115292] ? show_regs.cold.9+0x22/0x2f [16088.116568] ? watchdog_timer_fn.cold.6+0x62/0x10a [16088.118065] ? __hrtimer_run_queues+0x1c2/0x4d0 [16088.119769] ? watchdog+0x40/0x40 [16088.120946] ? hrtimer_interrupt+0x128/0x2c0 [16088.122403] ? smp_apic_timer_interrupt+0xa6/0x270 [16088.123919] ? apic_timer_interrupt+0xf/0x20 [16088.125475] [16088.126236] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16088.128490] do_raw_spin_lock+0x177/0x190 [16088.129974] _raw_spin_lock+0x21/0x30 [16088.131490] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16088.133523] ? do_raw_spin_unlock+0x75/0x190 [16088.135356] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16088.137719] ? do_wait_intr+0xf0/0xf0 [16088.139059] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16088.141463] mdc_close+0x53b/0xd70 [mdc] [16088.142933] lmv_close+0x18d/0x490 [lmv] [16088.144627] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [16088.146957] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16088.149283] ll_md_real_close+0xa5/0x2a0 [lustre] [16088.151184] ll_md_close+0x274/0x800 [lustre] [16088.152874] ll_file_release+0x128/0xc50 [lustre] [16088.154730] __fput+0x10d/0x440 [16088.155867] ____fput+0x12/0x20 [16088.157055] task_work_run+0xc6/0x110 [16088.158399] exit_to_usermode_loop+0x1dd/0x1f0 [16088.159782] do_syscall_64+0x3d6/0x440 [16088.161198] entry_SYSCALL_64_after_hwframe+0x49/0xae [16088.162625] RIP: 0033:0x7f151bee3cd8 [16088.164044] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16088.170060] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16088.172402] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [16088.174489] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [16088.176703] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16088.179215] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16088.181741] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16096.056025] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [16096.060019] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16096.094510] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16096.107291] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16096.111702] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16096.114845] RIP: 0010:native_safe_halt+0x2e/0x50 [16096.116595] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16096.131684] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16096.136813] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16096.141778] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [16096.146193] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [16096.149119] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [16096.154767] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [16096.158705] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [16096.168586] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16096.172519] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16096.177786] Call Trace: [16096.180613] [16096.187578] ? show_regs.cold.9+0x22/0x2f [16096.193745] ? watchdog_timer_fn.cold.6+0x62/0x10a [16096.196277] ? __hrtimer_run_queues+0x1c2/0x4d0 [16096.199252] ? watchdog+0x40/0x40 [16096.200860] ? hrtimer_interrupt+0x128/0x2c0 [16096.206511] ? smp_apic_timer_interrupt+0xa6/0x270 [16096.209369] ? apic_timer_interrupt+0xf/0x20 [16096.210867] [16096.211570] ? native_safe_halt+0x2e/0x50 [16096.214019] kvm_wait+0xcd/0xe0 [16096.215527] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16096.219418] do_raw_spin_lock+0x177/0x190 [16096.222346] _raw_spin_lock+0x21/0x30 [16096.225461] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16096.231241] process_one_work+0x2c8/0x700 [16096.236246] worker_thread+0x296/0x6e0 [16096.240880] ? rescuer_thread+0x570/0x570 [16096.242877] kthread+0x1d1/0x200 [16096.244638] ? set_kthread_struct+0x70/0x70 [16096.251288] ret_from_fork+0x1f/0x30 [16100.053200] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [16100.057492] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16100.085311] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16100.090576] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16100.095115] RIP: 0010:native_safe_halt+0x2e/0x50 [16100.097612] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16100.108827] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16100.113640] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16100.116656] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [16100.119566] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [16100.122320] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [16100.124935] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [16100.127704] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [16100.130857] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16100.133278] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [16100.136247] Call Trace: [16100.137276] [16100.138106] ? show_regs.cold.9+0x22/0x2f [16100.139599] ? watchdog_timer_fn.cold.6+0x62/0x10a [16100.141579] ? __hrtimer_run_queues+0x1c2/0x4d0 [16100.143519] ? watchdog+0x40/0x40 [16100.144842] ? hrtimer_interrupt+0x128/0x2c0 [16100.146580] ? smp_apic_timer_interrupt+0xa6/0x270 [16100.148632] ? apic_timer_interrupt+0xf/0x20 [16100.150491] [16100.151391] ? native_safe_halt+0x2e/0x50 [16100.153065] kvm_wait+0xcd/0xe0 [16100.154333] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16100.156538] do_raw_spin_lock+0x177/0x190 [16100.158214] _raw_spin_lock+0x21/0x30 [16100.159584] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16100.161800] ? do_raw_spin_unlock+0x75/0x190 [16100.165430] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16100.170205] ? do_wait_intr+0xf0/0xf0 [16100.174416] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16100.178300] mdc_reint+0x2d/0x110 [mdc] [16100.180348] mdc_unlink+0x658/0x870 [mdc] [16100.182544] lmv_unlink+0x34d/0xbd0 [lmv] [16100.184574] ll_unlink+0x24d/0x7f0 [lustre] [16100.186636] vfs_unlink+0x202/0x330 [16100.191244] do_unlinkat+0x2fa/0x460 [16100.196247] __x64_sys_unlink+0x37/0x50 [16100.200855] do_syscall_64+0xc1/0x440 [16100.207336] entry_SYSCALL_64_after_hwframe+0x49/0xae [16100.212582] RIP: 0033:0x7f151bdee10b [16100.214686] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [16100.225330] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [16100.231565] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [16100.238273] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [16100.244470] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16100.250689] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16100.254604] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16102.880129] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14172s! [16102.883437] Showing busy workqueues and worker pools: [16102.886587] workqueue events: flags=0x0 [16102.888829] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16102.892574] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [16102.896885] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16102.899213] pending: kfree_rcu_monitor [16102.901673] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16102.905439] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [16102.911207] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16102.914820] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [16102.919094] workqueue events_long: flags=0x0 [16102.921650] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16102.926143] pending: lu_objects_flush [obdclass] [16102.929536] workqueue events_unbound: flags=0x2 [16102.931289] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [16102.933542] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16102.937742] workqueue events_power_efficient: flags=0x80 [16102.940270] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16102.943307] pending: do_cache_clean, neigh_periodic_work, check_lifetime [16102.946231] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16102.948631] pending: neigh_periodic_work [16102.950213] workqueue mm_percpu_wq: flags=0x8 [16102.953557] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16102.961641] pending: vmstat_update, lru_add_drain_per_cpu [16102.963718] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16102.966155] pending: vmstat_update [16102.968089] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16102.970374] pending: lru_add_drain_per_cpu [16102.971950] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [16102.973647] pending: lru_add_drain_per_cpu BAR(48) [16102.975191] workqueue writeback: flags=0x4a [16102.976298] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [16102.977950] pending: wb_workfn [16102.979123] workqueue xprtiod: flags=0xa [16102.980272] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16102.981942] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [16102.984188] workqueue ipv6_addrconf: flags=0x40008 [16102.985450] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16102.987228] pending: addrconf_verify_work [16102.988562] workqueue knbd0-recv: flags=0x1a [16102.990206] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16102.992775] in-flight: 658:recv_work [16102.995241] workqueue ptlrpc_pinger: flags=0x2 [16102.996821] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16102.999575] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [16103.002281] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14172s workers=4 idle: 32984 37 8 [16103.005219] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16087s workers=2 idle: 661 [16108.055026] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [16108.057629] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16108.078401] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16108.083316] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16108.087937] RIP: 0010:native_safe_halt+0x2e/0x50 [16108.090337] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16108.098603] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16108.102252] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [16108.105298] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [16108.108835] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [16108.116086] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [16108.124320] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16108.127888] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [16108.136382] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16108.142404] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16108.151080] Call Trace: [16108.152703] [16108.155488] ? show_regs.cold.9+0x22/0x2f [16108.157888] ? watchdog_timer_fn.cold.6+0x62/0x10a [16108.161021] ? __hrtimer_run_queues+0x1c2/0x4d0 [16108.163638] ? watchdog+0x40/0x40 [16108.173246] ? hrtimer_interrupt+0x128/0x2c0 [16108.179789] ? smp_apic_timer_interrupt+0xa6/0x270 [16108.183377] ? apic_timer_interrupt+0xf/0x20 [16108.191092] [16108.194425] ? native_safe_halt+0x2e/0x50 [16108.197132] kvm_wait+0xcd/0xe0 [16108.198258] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16108.211390] do_raw_spin_lock+0x177/0x190 [16108.212981] _raw_spin_lock+0x21/0x30 [16108.221692] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16108.229308] obd_mod_free+0x46/0x140 [ptlrpc] [16108.231419] mdc_commit_open+0x91/0xb0 [mdc] [16108.234670] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16108.242371] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16108.246491] after_reply+0xe2c/0x1780 [ptlrpc] [16108.249572] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16108.252458] ? __next_timer_interrupt+0x160/0x160 [16108.256182] ptlrpcd+0xa82/0xe40 [ptlrpc] [16108.259011] ? do_wait_intr+0xf0/0xf0 [16108.260706] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16108.263240] kthread+0x1d1/0x200 [16108.264461] ? set_kthread_struct+0x70/0x70 [16108.268526] ret_from_fork+0x1f/0x30 [16116.058068] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [16116.064281] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16116.125883] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16116.139674] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16116.150207] RIP: 0010:native_safe_halt+0x2e/0x50 [16116.153132] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16116.192656] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16116.201807] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16116.204821] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [16116.210545] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [16116.214599] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [16116.220507] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [16116.222867] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [16116.233430] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16116.238147] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [16116.242096] Call Trace: [16116.245609] [16116.248330] ? show_regs.cold.9+0x22/0x2f [16116.250281] ? watchdog_timer_fn.cold.6+0x62/0x10a [16116.251956] ? __hrtimer_run_queues+0x1c2/0x4d0 [16116.256539] ? watchdog+0x40/0x40 [16116.258456] ? hrtimer_interrupt+0x128/0x2c0 [16116.260598] ? smp_apic_timer_interrupt+0xa6/0x270 [16116.266361] ? apic_timer_interrupt+0xf/0x20 [16116.269910] [16116.271440] ? native_safe_halt+0x2e/0x50 [16116.273616] kvm_wait+0xcd/0xe0 [16116.275307] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16116.282031] do_raw_spin_lock+0x177/0x190 [16116.287379] _raw_spin_lock+0x21/0x30 [16116.290173] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16116.293965] ? do_raw_spin_unlock+0x75/0x190 [16116.301273] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16116.311089] ? do_wait_intr+0xf0/0xf0 [16116.317526] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16116.322849] mdc_close+0x53b/0xd70 [mdc] [16116.324424] lmv_close+0x18d/0x490 [lmv] [16116.330648] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [16116.333792] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16116.336939] ll_md_real_close+0xa5/0x2a0 [lustre] [16116.340147] ll_md_close+0x274/0x800 [lustre] [16116.342414] ll_file_release+0x128/0xc50 [lustre] [16116.344589] __fput+0x10d/0x440 [16116.346982] ____fput+0x12/0x20 [16116.348873] task_work_run+0xc6/0x110 [16116.350619] exit_to_usermode_loop+0x1dd/0x1f0 [16116.353171] do_syscall_64+0x3d6/0x440 [16116.359469] entry_SYSCALL_64_after_hwframe+0x49/0xae [16116.363981] RIP: 0033:0x7f151bee3cd8 [16116.366442] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16116.382416] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16116.387681] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [16116.398607] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [16116.406116] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16116.415461] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16116.424111] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16124.056067] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [16124.058630] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16124.083026] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16124.089944] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16124.093991] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16124.098466] RIP: 0010:native_safe_halt+0x2e/0x50 [16124.101211] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16124.110411] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16124.113867] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16124.116912] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [16124.119054] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [16124.126283] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [16124.129139] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [16124.138384] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [16124.142893] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16124.146314] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16124.150801] Call Trace: [16124.151983] [16124.152989] ? show_regs.cold.9+0x22/0x2f [16124.155033] ? watchdog_timer_fn.cold.6+0x62/0x10a [16124.157883] ? __hrtimer_run_queues+0x1c2/0x4d0 [16124.160158] ? watchdog+0x40/0x40 [16124.162026] ? hrtimer_interrupt+0x128/0x2c0 [16124.164008] ? smp_apic_timer_interrupt+0xa6/0x270 [16124.166513] ? apic_timer_interrupt+0xf/0x20 [16124.168745] [16124.170034] ? native_safe_halt+0x2e/0x50 [16124.172861] kvm_wait+0xcd/0xe0 [16124.174500] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16124.177378] do_raw_spin_lock+0x177/0x190 [16124.179925] _raw_spin_lock+0x21/0x30 [16124.182519] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16124.185337] process_one_work+0x2c8/0x700 [16124.187406] worker_thread+0x296/0x6e0 [16124.189427] ? rescuer_thread+0x570/0x570 [16124.191513] kthread+0x1d1/0x200 [16124.193563] ? set_kthread_struct+0x70/0x70 [16124.196321] ret_from_fork+0x1f/0x30 [16128.052081] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [16128.054263] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16128.085043] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16128.095565] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16128.107452] RIP: 0010:native_safe_halt+0x2e/0x50 [16128.110459] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16128.122210] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16128.128120] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16128.132004] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [16128.136742] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [16128.141451] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [16128.147866] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [16128.151752] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [16128.160398] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16128.164114] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [16128.172440] Call Trace: [16128.174029] [16128.175121] ? show_regs.cold.9+0x22/0x2f [16128.177587] ? watchdog_timer_fn.cold.6+0x62/0x10a [16128.179887] ? __hrtimer_run_queues+0x1c2/0x4d0 [16128.181915] ? watchdog+0x40/0x40 [16128.186160] ? hrtimer_interrupt+0x128/0x2c0 [16128.188281] ? smp_apic_timer_interrupt+0xa6/0x270 [16128.192001] ? apic_timer_interrupt+0xf/0x20 [16128.194382] [16128.195369] ? native_safe_halt+0x2e/0x50 [16128.197013] kvm_wait+0xcd/0xe0 [16128.200360] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16128.204373] do_raw_spin_lock+0x177/0x190 [16128.206759] _raw_spin_lock+0x21/0x30 [16128.208569] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16128.211203] ? do_raw_spin_unlock+0x75/0x190 [16128.213960] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16128.216989] ? do_wait_intr+0xf0/0xf0 [16128.220540] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16128.223565] mdc_reint+0x2d/0x110 [mdc] [16128.227216] mdc_unlink+0x658/0x870 [mdc] [16128.228689] lmv_unlink+0x34d/0xbd0 [lmv] [16128.230092] ll_unlink+0x24d/0x7f0 [lustre] [16128.231696] vfs_unlink+0x202/0x330 [16128.234045] do_unlinkat+0x2fa/0x460 [16128.234930] __x64_sys_unlink+0x37/0x50 [16128.236182] do_syscall_64+0xc1/0x440 [16128.239886] entry_SYSCALL_64_after_hwframe+0x49/0xae [16128.241541] RIP: 0033:0x7f151bdee10b [16128.245850] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [16128.254877] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [16128.257193] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [16128.259902] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [16128.267092] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16128.271861] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16128.278739] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16133.088039] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14202s! [16133.092582] Showing busy workqueues and worker pools: [16133.094904] workqueue events: flags=0x0 [16133.096646] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16133.100292] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [16133.106941] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16133.108739] pending: kfree_rcu_monitor [16133.109897] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16133.111768] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [16133.115665] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16133.122455] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [16133.126093] workqueue events_long: flags=0x0 [16133.128609] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16133.131507] pending: lu_objects_flush [obdclass] [16133.134658] workqueue events_unbound: flags=0x2 [16133.136663] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [16133.141897] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16133.147825] workqueue events_power_efficient: flags=0x80 [16133.152462] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16133.156651] pending: do_cache_clean, neigh_periodic_work, check_lifetime [16133.163965] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16133.168774] pending: neigh_periodic_work [16133.171775] workqueue mm_percpu_wq: flags=0x8 [16133.175233] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16133.178050] pending: vmstat_update, lru_add_drain_per_cpu [16133.182242] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16133.192734] pending: vmstat_update [16133.194460] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16133.197945] pending: lru_add_drain_per_cpu [16133.199349] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [16133.202691] pending: lru_add_drain_per_cpu BAR(48) [16133.205126] workqueue writeback: flags=0x4a [16133.209915] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [16133.213546] pending: wb_workfn [16133.216215] workqueue xprtiod: flags=0xa [16133.217734] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16133.220173] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [16133.224210] workqueue ipv6_addrconf: flags=0x40008 [16133.229369] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16133.236288] pending: addrconf_verify_work [16133.238413] workqueue knbd0-recv: flags=0x1a [16133.241059] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16133.245208] in-flight: 658:recv_work [16133.248134] workqueue ptlrpc_pinger: flags=0x2 [16133.255259] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16133.261541] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [16133.264970] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14202s workers=4 idle: 32984 37 8 [16133.270721] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16118s workers=2 idle: 661 [16136.057081] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [16136.061324] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16136.089215] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16136.093879] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16136.099310] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16136.101665] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16136.109391] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16136.112707] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [16136.115796] RDX: 0000000000000f3d RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [16136.122667] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [16136.142428] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [16136.154589] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16136.160499] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [16136.168420] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16136.174422] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16136.186093] Call Trace: [16136.191915] [16136.192619] ? show_regs.cold.9+0x22/0x2f [16136.196486] ? watchdog_timer_fn.cold.6+0x62/0x10a [16136.200864] ? __hrtimer_run_queues+0x1c2/0x4d0 [16136.205494] ? watchdog+0x40/0x40 [16136.214281] ? hrtimer_interrupt+0x128/0x2c0 [16136.220544] ? smp_apic_timer_interrupt+0xa6/0x270 [16136.224320] ? apic_timer_interrupt+0xf/0x20 [16136.231980] [16136.232999] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16136.236397] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16136.238797] do_raw_spin_lock+0x177/0x190 [16136.241992] _raw_spin_lock+0x21/0x30 [16136.243419] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16136.246726] obd_mod_free+0x46/0x140 [ptlrpc] [16136.249349] mdc_commit_open+0x91/0xb0 [mdc] [16136.251226] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16136.253750] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16136.256173] after_reply+0xe2c/0x1780 [ptlrpc] [16136.258374] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16136.262119] ? __next_timer_interrupt+0x160/0x160 [16136.263621] ptlrpcd+0xa82/0xe40 [ptlrpc] [16136.266018] ? do_wait_intr+0xf0/0xf0 [16136.268108] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16136.270945] kthread+0x1d1/0x200 [16136.272121] ? set_kthread_struct+0x70/0x70 [16136.274842] ret_from_fork+0x1f/0x30 [16144.057775] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [16144.066773] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16144.116710] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16144.127473] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16144.138517] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16144.144521] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16144.165821] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16144.174757] RAX: 0000000000006a88 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16144.180268] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [16144.185423] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [16144.189161] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [16144.194944] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [16144.200950] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [16144.206381] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16144.212280] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [16144.218238] Call Trace: [16144.219285] [16144.224041] ? show_regs.cold.9+0x22/0x2f [16144.226051] ? watchdog_timer_fn.cold.6+0x62/0x10a [16144.232286] ? __hrtimer_run_queues+0x1c2/0x4d0 [16144.237000] ? watchdog+0x40/0x40 [16144.239316] ? hrtimer_interrupt+0x128/0x2c0 [16144.244637] ? smp_apic_timer_interrupt+0xa6/0x270 [16144.246829] ? apic_timer_interrupt+0xf/0x20 [16144.252926] [16144.253921] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16144.257189] do_raw_spin_lock+0x177/0x190 [16144.262347] _raw_spin_lock+0x21/0x30 [16144.266232] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16144.270784] ? do_raw_spin_unlock+0x75/0x190 [16144.277712] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16144.287248] ? do_wait_intr+0xf0/0xf0 [16144.291156] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16144.296139] mdc_close+0x53b/0xd70 [mdc] [16144.300090] lmv_close+0x18d/0x490 [lmv] [16144.304347] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [16144.314963] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16144.330501] ll_md_real_close+0xa5/0x2a0 [lustre] [16144.338677] ll_md_close+0x274/0x800 [lustre] [16144.341231] ll_file_release+0x128/0xc50 [lustre] [16144.352029] __fput+0x10d/0x440 [16144.355357] ____fput+0x12/0x20 [16144.360401] task_work_run+0xc6/0x110 [16144.366440] exit_to_usermode_loop+0x1dd/0x1f0 [16144.369174] do_syscall_64+0x3d6/0x440 [16144.374618] entry_SYSCALL_64_after_hwframe+0x49/0xae [16144.377243] RIP: 0033:0x7f151bee3cd8 [16144.379563] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16144.396820] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16144.411373] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [16144.422332] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [16144.429198] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16144.438588] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16144.455183] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16152.056050] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [16152.059578] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16152.081471] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16152.086719] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16152.091503] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16152.095423] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16152.098648] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16152.109125] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16152.112941] RAX: 0000000000007224 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16152.116517] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [16152.119918] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [16152.123264] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [16152.126931] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [16152.130118] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [16152.133785] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16152.136372] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16152.140026] Call Trace: [16152.141393] [16152.142185] ? show_regs.cold.9+0x22/0x2f [16152.144103] ? watchdog_timer_fn.cold.6+0x62/0x10a [16152.146132] ? __hrtimer_run_queues+0x1c2/0x4d0 [16152.148017] ? watchdog+0x40/0x40 [16152.150335] ? hrtimer_interrupt+0x128/0x2c0 [16152.153077] ? smp_apic_timer_interrupt+0xa6/0x270 [16152.155560] ? apic_timer_interrupt+0xf/0x20 [16152.158470] [16152.159355] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16152.162085] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16152.164459] do_raw_spin_lock+0x177/0x190 [16152.165939] _raw_spin_lock+0x21/0x30 [16152.167649] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16152.170333] process_one_work+0x2c8/0x700 [16152.172212] worker_thread+0x296/0x6e0 [16152.173646] ? rescuer_thread+0x570/0x570 [16152.174733] kthread+0x1d1/0x200 [16152.175904] ? set_kthread_struct+0x70/0x70 [16152.177324] ret_from_fork+0x1f/0x30 [16160.052456] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [16160.057488] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16160.084662] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16160.090491] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16160.094312] RIP: 0010:native_safe_halt+0x2e/0x50 [16160.096688] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16160.108680] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16160.113719] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16160.117646] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [16160.121098] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [16160.123205] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [16160.127231] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [16160.131665] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [16160.135549] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16160.138199] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [16160.142131] Call Trace: [16160.142958] [16160.143734] ? show_regs.cold.9+0x22/0x2f [16160.145156] ? watchdog_timer_fn.cold.6+0x62/0x10a [16160.146594] ? __hrtimer_run_queues+0x1c2/0x4d0 [16160.148292] ? watchdog+0x40/0x40 [16160.149205] ? hrtimer_interrupt+0x128/0x2c0 [16160.150826] ? smp_apic_timer_interrupt+0xa6/0x270 [16160.152242] ? apic_timer_interrupt+0xf/0x20 [16160.153592] [16160.154258] ? native_safe_halt+0x2e/0x50 [16160.155313] kvm_wait+0xcd/0xe0 [16160.156732] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16160.158714] do_raw_spin_lock+0x177/0x190 [16160.160301] _raw_spin_lock+0x21/0x30 [16160.161991] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16160.164165] ? do_raw_spin_unlock+0x75/0x190 [16160.165755] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16160.168177] ? do_wait_intr+0xf0/0xf0 [16160.169531] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16160.171816] mdc_reint+0x2d/0x110 [mdc] [16160.173240] mdc_unlink+0x658/0x870 [mdc] [16160.174815] lmv_unlink+0x34d/0xbd0 [lmv] [16160.176527] ll_unlink+0x24d/0x7f0 [lustre] [16160.178252] vfs_unlink+0x202/0x330 [16160.179695] do_unlinkat+0x2fa/0x460 [16160.181001] __x64_sys_unlink+0x37/0x50 [16160.182261] do_syscall_64+0xc1/0x440 [16160.183675] entry_SYSCALL_64_after_hwframe+0x49/0xae [16160.185327] RIP: 0033:0x7f151bdee10b [16160.186636] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [16160.194034] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [16160.197047] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [16160.200156] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [16160.203040] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16160.205932] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16160.208263] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16163.298293] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14232s! [16163.308850] Showing busy workqueues and worker pools: [16163.313895] workqueue events: flags=0x0 [16163.315220] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16163.320615] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [16163.327211] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16163.330914] pending: kfree_rcu_monitor [16163.332991] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16163.335817] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [16163.340000] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16163.343624] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [16163.348049] workqueue events_long: flags=0x0 [16163.349733] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16163.352912] pending: lu_objects_flush [obdclass] [16163.355595] workqueue events_unbound: flags=0x2 [16163.357544] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [16163.361122] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16163.364983] workqueue events_power_efficient: flags=0x80 [16163.367695] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16163.370755] pending: do_cache_clean, neigh_periodic_work, check_lifetime [16163.373672] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16163.376240] pending: neigh_periodic_work [16163.377803] workqueue mm_percpu_wq: flags=0x8 [16163.379169] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16163.381502] pending: vmstat_update, lru_add_drain_per_cpu [16163.383554] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16163.386020] pending: vmstat_update [16163.387314] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16163.389687] pending: lru_add_drain_per_cpu [16163.394970] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [16163.397594] pending: lru_add_drain_per_cpu BAR(48) [16163.405574] workqueue writeback: flags=0x4a [16163.407902] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [16163.413620] pending: wb_workfn [16163.415190] workqueue xprtiod: flags=0xa [16163.416566] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16163.418546] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [16163.421956] workqueue ipv6_addrconf: flags=0x40008 [16163.424465] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16163.427671] pending: addrconf_verify_work [16163.429631] workqueue knbd0-recv: flags=0x1a [16163.431670] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16163.434759] in-flight: 658:recv_work [16163.436508] workqueue ptlrpc_pinger: flags=0x2 [16163.438607] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16163.441731] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [16163.444964] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14232s workers=4 idle: 32984 37 8 [16163.448367] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16148s workers=2 idle: 661 [16164.055041] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [16164.060028] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16164.089635] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16164.096670] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16164.102146] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16164.106365] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16164.121288] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16164.126297] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [16164.132113] RDX: 000000000000491c RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [16164.137323] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [16164.141991] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [16164.147109] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16164.152179] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [16164.157211] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16164.161980] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16164.167224] Call Trace: [16164.168983] [16164.171241] ? show_regs.cold.9+0x22/0x2f [16164.174559] ? watchdog_timer_fn.cold.6+0x62/0x10a [16164.178847] ? __hrtimer_run_queues+0x1c2/0x4d0 [16164.182475] ? watchdog+0x40/0x40 [16164.185312] ? hrtimer_interrupt+0x128/0x2c0 [16164.188674] ? smp_apic_timer_interrupt+0xa6/0x270 [16164.192178] ? apic_timer_interrupt+0xf/0x20 [16164.195172] [16164.197143] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16164.201722] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16164.205778] do_raw_spin_lock+0x177/0x190 [16164.209158] _raw_spin_lock+0x21/0x30 [16164.211572] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16164.214985] obd_mod_free+0x46/0x140 [ptlrpc] [16164.217196] mdc_commit_open+0x91/0xb0 [mdc] [16164.222059] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16164.224931] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16164.232081] after_reply+0xe2c/0x1780 [ptlrpc] [16164.238744] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16164.243901] ? __next_timer_interrupt+0x160/0x160 [16164.251227] ptlrpcd+0xa82/0xe40 [ptlrpc] [16164.259909] ? do_wait_intr+0xf0/0xf0 [16164.264081] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16164.267677] kthread+0x1d1/0x200 [16164.272437] ? set_kthread_struct+0x70/0x70 [16164.274368] ret_from_fork+0x1f/0x30 [16172.059545] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [16172.062010] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16172.112550] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16172.124094] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16172.132939] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16172.148969] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16172.185952] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16172.190927] RAX: 0000000000005233 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16172.212932] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [16172.222133] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [16172.235427] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [16172.251388] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [16172.263843] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [16172.277392] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16172.291278] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [16172.293569] Call Trace: [16172.301270] [16172.302991] ? show_regs.cold.9+0x22/0x2f [16172.314338] ? watchdog_timer_fn.cold.6+0x62/0x10a [16172.324030] ? __hrtimer_run_queues+0x1c2/0x4d0 [16172.330105] ? watchdog+0x40/0x40 [16172.335685] ? hrtimer_interrupt+0x128/0x2c0 [16172.336981] ? smp_apic_timer_interrupt+0xa6/0x270 [16172.353765] ? apic_timer_interrupt+0xf/0x20 [16172.355222] [16172.355990] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16172.368515] do_raw_spin_lock+0x177/0x190 [16172.372975] _raw_spin_lock+0x21/0x30 [16172.379443] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16172.388479] ? do_raw_spin_unlock+0x75/0x190 [16172.394826] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16172.406158] ? do_wait_intr+0xf0/0xf0 [16172.410371] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16172.423760] mdc_close+0x53b/0xd70 [mdc] [16172.425327] lmv_close+0x18d/0x490 [lmv] [16172.434007] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [16172.437789] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16172.441209] ll_md_real_close+0xa5/0x2a0 [lustre] [16172.444346] ll_md_close+0x274/0x800 [lustre] [16172.451503] ll_file_release+0x128/0xc50 [lustre] [16172.455677] __fput+0x10d/0x440 [16172.458690] ____fput+0x12/0x20 [16172.462485] task_work_run+0xc6/0x110 [16172.465071] exit_to_usermode_loop+0x1dd/0x1f0 [16172.471983] do_syscall_64+0x3d6/0x440 [16172.475308] entry_SYSCALL_64_after_hwframe+0x49/0xae [16172.479687] RIP: 0033:0x7f151bee3cd8 [16172.481573] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16172.500986] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16172.508864] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [16172.513609] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [16172.516121] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16172.533675] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16172.543086] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16180.057507] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [16180.068332] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16180.122685] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16180.149686] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16180.162383] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16180.168542] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16180.177051] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16180.197482] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16180.206315] RAX: 0000000000007f85 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16180.212951] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [16180.216427] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [16180.230682] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [16180.242799] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [16180.248650] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [16180.253690] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16180.258039] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16180.266994] Call Trace: [16180.268703] [16180.269721] ? show_regs.cold.9+0x22/0x2f [16180.272824] ? watchdog_timer_fn.cold.6+0x62/0x10a [16180.277604] ? __hrtimer_run_queues+0x1c2/0x4d0 [16180.280995] ? watchdog+0x40/0x40 [16180.283021] ? hrtimer_interrupt+0x128/0x2c0 [16180.286598] ? smp_apic_timer_interrupt+0xa6/0x270 [16180.288956] ? apic_timer_interrupt+0xf/0x20 [16180.291111] [16180.292168] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16180.296363] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16180.298932] do_raw_spin_lock+0x177/0x190 [16180.302896] _raw_spin_lock+0x21/0x30 [16180.304874] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16180.314039] process_one_work+0x2c8/0x700 [16180.315916] worker_thread+0x296/0x6e0 [16180.319557] ? rescuer_thread+0x570/0x570 [16180.322533] kthread+0x1d1/0x200 [16180.326018] ? set_kthread_struct+0x70/0x70 [16180.328450] ret_from_fork+0x1f/0x30 [16188.055129] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [16188.059111] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16188.088517] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16188.104456] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16188.123383] RIP: 0010:native_safe_halt+0x2e/0x50 [16188.128550] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16188.172360] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16188.180681] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16188.187599] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [16188.201981] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [16188.210948] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [16188.215907] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [16188.219971] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [16188.223976] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16188.226868] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [16188.230434] Call Trace: [16188.231554] [16188.232574] ? show_regs.cold.9+0x22/0x2f [16188.234533] ? watchdog_timer_fn.cold.6+0x62/0x10a [16188.237102] ? __hrtimer_run_queues+0x1c2/0x4d0 [16188.239635] ? watchdog+0x40/0x40 [16188.241569] ? hrtimer_interrupt+0x128/0x2c0 [16188.244572] ? smp_apic_timer_interrupt+0xa6/0x270 [16188.247654] ? apic_timer_interrupt+0xf/0x20 [16188.250068] [16188.251531] ? native_safe_halt+0x2e/0x50 [16188.254003] kvm_wait+0xcd/0xe0 [16188.255499] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16188.258635] do_raw_spin_lock+0x177/0x190 [16188.260591] _raw_spin_lock+0x21/0x30 [16188.262542] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16188.266017] ? do_raw_spin_unlock+0x75/0x190 [16188.268486] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16188.271391] ? do_wait_intr+0xf0/0xf0 [16188.273444] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16188.277266] mdc_reint+0x2d/0x110 [mdc] [16188.279445] mdc_unlink+0x658/0x870 [mdc] [16188.282184] lmv_unlink+0x34d/0xbd0 [lmv] [16188.286791] ll_unlink+0x24d/0x7f0 [lustre] [16188.289543] vfs_unlink+0x202/0x330 [16188.293598] do_unlinkat+0x2fa/0x460 [16188.295592] __x64_sys_unlink+0x37/0x50 [16188.298322] do_syscall_64+0xc1/0x440 [16188.303383] entry_SYSCALL_64_after_hwframe+0x49/0xae [16188.306094] RIP: 0033:0x7f151bdee10b [16188.310236] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [16188.322159] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [16188.326216] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [16188.329742] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [16188.333205] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16188.338675] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16188.342783] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16192.055021] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [16192.060040] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16192.086751] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16192.092242] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16192.096297] RIP: 0010:native_safe_halt+0x2e/0x50 [16192.098200] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16192.105476] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16192.108534] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [16192.111599] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [16192.114583] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [16192.117468] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [16192.120368] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16192.123334] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [16192.126525] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16192.129264] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16192.132223] Call Trace: [16192.133224] [16192.134151] ? show_regs.cold.9+0x22/0x2f [16192.135920] ? watchdog_timer_fn.cold.6+0x62/0x10a [16192.137940] ? __hrtimer_run_queues+0x1c2/0x4d0 [16192.140023] ? watchdog+0x40/0x40 [16192.141420] ? hrtimer_interrupt+0x128/0x2c0 [16192.143285] ? smp_apic_timer_interrupt+0xa6/0x270 [16192.145341] ? apic_timer_interrupt+0xf/0x20 [16192.147130] [16192.147912] ? native_safe_halt+0x2e/0x50 [16192.149766] kvm_wait+0xcd/0xe0 [16192.151117] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16192.153334] do_raw_spin_lock+0x177/0x190 [16192.155300] _raw_spin_lock+0x21/0x30 [16192.157039] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16192.159809] obd_mod_free+0x46/0x140 [ptlrpc] [16192.162193] mdc_commit_open+0x91/0xb0 [mdc] [16192.164351] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16192.167539] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16192.169544] after_reply+0xe2c/0x1780 [ptlrpc] [16192.171841] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16192.174794] ? __next_timer_interrupt+0x160/0x160 [16192.176633] ptlrpcd+0xa82/0xe40 [ptlrpc] [16192.178859] ? do_wait_intr+0xf0/0xf0 [16192.180493] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16192.183191] kthread+0x1d1/0x200 [16192.184766] ? set_kthread_struct+0x70/0x70 [16192.186595] ret_from_fork+0x1f/0x30 [16193.506346] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14262s! [16193.510514] Showing busy workqueues and worker pools: [16193.515261] workqueue events: flags=0x0 [16193.518079] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16193.520682] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [16193.525803] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16193.530447] pending: kfree_rcu_monitor [16193.532436] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16193.536653] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [16193.543273] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16193.545798] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [16193.553087] workqueue events_long: flags=0x0 [16193.554722] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16193.559963] pending: lu_objects_flush [obdclass] [16193.563086] workqueue events_unbound: flags=0x2 [16193.564721] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [16193.569646] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16193.576278] workqueue events_power_efficient: flags=0x80 [16193.578877] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16193.585714] pending: do_cache_clean, neigh_periodic_work, check_lifetime [16193.590600] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16193.594256] pending: neigh_periodic_work [16193.596119] workqueue mm_percpu_wq: flags=0x8 [16193.598475] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16193.602870] pending: vmstat_update, lru_add_drain_per_cpu [16193.605653] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16193.610420] pending: vmstat_update [16193.613263] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16193.616383] pending: lru_add_drain_per_cpu [16193.618351] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [16193.621720] pending: lru_add_drain_per_cpu BAR(48) [16193.623987] workqueue writeback: flags=0x4a [16193.626257] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [16193.629983] pending: wb_workfn [16193.631662] workqueue xprtiod: flags=0xa [16193.633614] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16193.637409] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [16193.644671] workqueue ipv6_addrconf: flags=0x40008 [16193.647327] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16193.650833] pending: addrconf_verify_work [16193.653390] workqueue knbd0-recv: flags=0x1a [16193.655382] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16193.658584] in-flight: 658:recv_work [16193.664428] workqueue ptlrpc_pinger: flags=0x2 [16193.667533] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16193.670730] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [16193.674733] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14263s workers=4 idle: 32984 37 8 [16193.679153] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16178s workers=2 idle: 661 [16200.057023] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [16200.059620] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16200.083252] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16200.089662] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16200.099234] RIP: 0010:native_safe_halt+0x2e/0x50 [16200.101256] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16200.113289] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16200.118415] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16200.121104] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [16200.129780] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [16200.132544] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [16200.136816] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [16200.140327] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [16200.144113] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16200.146555] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [16200.149408] Call Trace: [16200.150587] [16200.151322] ? show_regs.cold.9+0x22/0x2f [16200.154448] ? watchdog_timer_fn.cold.6+0x62/0x10a [16200.162409] ? __hrtimer_run_queues+0x1c2/0x4d0 [16200.167239] ? watchdog+0x40/0x40 [16200.169615] ? hrtimer_interrupt+0x128/0x2c0 [16200.173726] ? smp_apic_timer_interrupt+0xa6/0x270 [16200.176362] ? apic_timer_interrupt+0xf/0x20 [16200.180874] [16200.182612] ? native_safe_halt+0x2e/0x50 [16200.184749] kvm_wait+0xcd/0xe0 [16200.189291] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16200.194583] do_raw_spin_lock+0x177/0x190 [16200.197725] _raw_spin_lock+0x21/0x30 [16200.198872] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16200.202137] ? do_raw_spin_unlock+0x75/0x190 [16200.204054] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16200.205783] ? do_wait_intr+0xf0/0xf0 [16200.209876] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16200.212260] mdc_close+0x53b/0xd70 [mdc] [16200.213633] lmv_close+0x18d/0x490 [lmv] [16200.215256] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [16200.217493] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16200.222296] ll_md_real_close+0xa5/0x2a0 [lustre] [16200.223732] ll_md_close+0x274/0x800 [lustre] [16200.232318] ll_file_release+0x128/0xc50 [lustre] [16200.235616] __fput+0x10d/0x440 [16200.239659] ____fput+0x12/0x20 [16200.245785] task_work_run+0xc6/0x110 [16200.250157] exit_to_usermode_loop+0x1dd/0x1f0 [16200.259198] do_syscall_64+0x3d6/0x440 [16200.261979] entry_SYSCALL_64_after_hwframe+0x49/0xae [16200.266510] RIP: 0033:0x7f151bee3cd8 [16200.267543] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16200.280992] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16200.286337] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [16200.290417] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [16200.293631] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16200.296382] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16200.298342] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16204.945804] rcu: INFO: rcu_sched self-detected stall on CPU [16204.950394] rcu: 1-...!: (8553237 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [16204.956042] (t=14280298 jiffies g=249657 q=6753) [16204.958274] rcu: rcu_sched kthread starved for 14275250 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [16204.974797] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [16204.989446] rcu: RCU grace-period kthread stack dump: [16204.994814] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [16205.001357] Call Trace: [16205.002296] __schedule+0x351/0xcb0 [16205.004828] schedule+0xc0/0x180 [16205.007926] schedule_timeout+0xb4/0x190 [16205.011830] ? do_raw_spin_unlock+0x75/0x190 [16205.015047] ? __next_timer_interrupt+0x160/0x160 [16205.019011] ? prepare_to_swait_event+0xf3/0x140 [16205.021439] rcu_gp_kthread+0x8dd/0x15b0 [16205.023277] ? force_qs_rnp+0x2c0/0x2c0 [16205.025194] kthread+0x1d1/0x200 [16205.026837] ? set_kthread_struct+0x70/0x70 [16205.028962] ret_from_fork+0x1f/0x30 [16205.030796] rcu: Stack dump where RCU GP kthread last ran: [16205.033605] Sending NMI from CPU 1 to CPUs 2: [16205.036635] NMI backtrace for cpu 2 skipped: idling at native_safe_halt+0x2e/0x50 [16205.039968] NMI backtrace for cpu 1 [16205.041747] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16205.048189] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16205.052686] Call Trace: [16205.053912] [16205.055198] ? dump_stack+0xbb/0x10e [16205.057079] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [16205.059678] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [16205.062063] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [16205.063971] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [16205.067678] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [16205.072283] ? rcu_dump_cpu_stacks+0x11b/0x171 [16205.076047] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [16205.079411] ? update_process_times+0x6d/0xc0 [16205.081311] ? tick_sched_handle.isra.18+0x27/0x90 [16205.083816] ? tick_sched_timer+0x53/0xb0 [16205.086089] ? __hrtimer_run_queues+0x1c2/0x4d0 [16205.088542] ? tick_sched_handle.isra.18+0x90/0x90 [16205.091639] ? hrtimer_interrupt+0x128/0x2c0 [16205.094303] ? smp_apic_timer_interrupt+0xa6/0x270 [16205.096779] ? apic_timer_interrupt+0xf/0x20 [16205.099277] [16205.100449] ? native_safe_halt+0x2e/0x50 [16205.102574] ? kvm_wait+0xcd/0xe0 [16205.104491] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16205.107456] ? do_raw_spin_lock+0x177/0x190 [16205.109651] ? _raw_spin_lock+0x21/0x30 [16205.111838] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16205.114942] ? obd_mod_free+0x46/0x140 [ptlrpc] [16205.118316] ? mdc_commit_open+0x91/0xb0 [mdc] [16205.120965] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16205.124102] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16205.127607] ? after_reply+0xe2c/0x1780 [ptlrpc] [16205.130709] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16205.132995] ? __next_timer_interrupt+0x160/0x160 [16205.135561] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [16205.138490] ? do_wait_intr+0xf0/0xf0 [16205.140579] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16205.145426] ? kthread+0x1d1/0x200 [16205.148164] ? set_kthread_struct+0x70/0x70 [16205.149722] ? ret_from_fork+0x1f/0x30 [16220.052020] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [16220.055337] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16220.072318] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16220.075496] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16220.078290] RIP: 0010:native_safe_halt+0x2e/0x50 [16220.080125] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16220.088121] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16220.091937] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16220.094832] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [16220.098268] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [16220.100961] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [16220.104065] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [16220.106540] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [16220.109693] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16220.116349] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [16220.123967] Call Trace: [16220.125521] [16220.126606] ? show_regs.cold.9+0x22/0x2f [16220.129135] ? watchdog_timer_fn.cold.6+0x62/0x10a [16220.132094] ? __hrtimer_run_queues+0x1c2/0x4d0 [16220.135531] ? watchdog+0x40/0x40 [16220.137295] ? hrtimer_interrupt+0x128/0x2c0 [16220.139978] ? smp_apic_timer_interrupt+0xa6/0x270 [16220.142673] ? apic_timer_interrupt+0xf/0x20 [16220.145658] [16220.147259] ? native_safe_halt+0x2e/0x50 [16220.150438] kvm_wait+0xcd/0xe0 [16220.159228] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16220.160582] do_raw_spin_lock+0x177/0x190 [16220.161633] _raw_spin_lock+0x21/0x30 [16220.164329] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16220.168392] ? do_raw_spin_unlock+0x75/0x190 [16220.172409] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16220.176294] ? do_wait_intr+0xf0/0xf0 [16220.178315] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16220.183588] mdc_reint+0x2d/0x110 [mdc] [16220.189621] mdc_unlink+0x658/0x870 [mdc] [16220.191348] lmv_unlink+0x34d/0xbd0 [lmv] [16220.197456] ll_unlink+0x24d/0x7f0 [lustre] [16220.200527] vfs_unlink+0x202/0x330 [16220.202465] do_unlinkat+0x2fa/0x460 [16220.204974] __x64_sys_unlink+0x37/0x50 [16220.207367] do_syscall_64+0xc1/0x440 [16220.211841] entry_SYSCALL_64_after_hwframe+0x49/0xae [16220.215361] RIP: 0033:0x7f151bdee10b [16220.219245] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [16220.241914] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [16220.249769] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [16220.259484] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [16220.275724] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16220.288588] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16220.299716] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16223.717406] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14293s! [16223.728475] Showing busy workqueues and worker pools: [16223.739281] workqueue events: flags=0x0 [16223.747912] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16223.761763] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [16223.770895] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16223.781863] pending: kfree_rcu_monitor [16223.786619] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16223.792509] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [16223.798238] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16223.803269] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [16223.808206] workqueue events_long: flags=0x0 [16223.810644] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16223.814599] pending: lu_objects_flush [obdclass] [16223.817554] workqueue events_unbound: flags=0x2 [16223.820259] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [16223.823603] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16223.836469] workqueue events_power_efficient: flags=0x80 [16223.845291] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16223.847862] pending: do_cache_clean, neigh_periodic_work, check_lifetime [16223.853406] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16223.858607] pending: neigh_periodic_work [16223.860635] workqueue mm_percpu_wq: flags=0x8 [16223.864384] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16223.869764] pending: vmstat_update, lru_add_drain_per_cpu [16223.875384] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16223.887321] pending: vmstat_update [16223.891401] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16223.901444] pending: lru_add_drain_per_cpu [16223.904763] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [16223.910414] pending: lru_add_drain_per_cpu BAR(48) [16223.913429] workqueue writeback: flags=0x4a [16223.918671] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [16223.922648] pending: wb_workfn [16223.924637] workqueue xprtiod: flags=0xa [16223.928632] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16223.936740] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [16223.944981] workqueue ipv6_addrconf: flags=0x40008 [16223.949281] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16223.956711] pending: addrconf_verify_work [16223.959890] workqueue knbd0-recv: flags=0x1a [16223.964298] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16223.970445] in-flight: 658:recv_work [16223.977460] workqueue ptlrpc_pinger: flags=0x2 [16223.983514] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16223.993580] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [16224.004782] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14293s workers=4 idle: 32984 37 8 [16224.007536] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16208s workers=2 idle: 661 [16228.057219] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [16228.063046] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16228.084730] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16228.088965] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16228.092332] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16228.095487] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16228.110913] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16228.115085] RAX: 0000000000007c8d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16228.119435] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [16228.122411] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [16228.128755] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [16228.135129] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [16228.142853] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [16228.149257] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16228.154241] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [16228.160345] Call Trace: [16228.162367] [16228.163541] ? show_regs.cold.9+0x22/0x2f [16228.167126] ? watchdog_timer_fn.cold.6+0x62/0x10a [16228.170692] ? __hrtimer_run_queues+0x1c2/0x4d0 [16228.174103] ? watchdog+0x40/0x40 [16228.175099] ? hrtimer_interrupt+0x128/0x2c0 [16228.178303] ? smp_apic_timer_interrupt+0xa6/0x270 [16228.181437] ? apic_timer_interrupt+0xf/0x20 [16228.184983] [16228.185631] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16228.192212] do_raw_spin_lock+0x177/0x190 [16228.193659] _raw_spin_lock+0x21/0x30 [16228.198806] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16228.200375] ? do_raw_spin_unlock+0x75/0x190 [16228.202256] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16228.203812] ? do_wait_intr+0xf0/0xf0 [16228.205005] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16228.212099] mdc_close+0x53b/0xd70 [mdc] [16228.217381] lmv_close+0x18d/0x490 [lmv] [16228.219741] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [16228.224190] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16228.227264] ll_md_real_close+0xa5/0x2a0 [lustre] [16228.232954] ll_md_close+0x274/0x800 [lustre] [16228.235676] ll_file_release+0x128/0xc50 [lustre] [16228.240165] __fput+0x10d/0x440 [16228.241334] ____fput+0x12/0x20 [16228.242394] task_work_run+0xc6/0x110 [16228.244544] exit_to_usermode_loop+0x1dd/0x1f0 [16228.247256] do_syscall_64+0x3d6/0x440 [16228.248410] entry_SYSCALL_64_after_hwframe+0x49/0xae [16228.255503] RIP: 0033:0x7f151bee3cd8 [16228.258251] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16228.276482] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16228.278835] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [16228.280596] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [16228.282376] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16228.284501] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16228.286723] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16232.056239] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [16232.057330] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [16232.070238] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16232.070709] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16232.070716] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16232.070728] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16232.070744] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16232.070748] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16232.070752] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [16232.070754] RDX: 0000000000006803 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [16232.070756] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [16232.070758] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [16232.070760] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16232.070763] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [16232.070766] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16232.070768] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16232.070778] Call Trace: [16232.070784] [16232.070789] ? show_regs.cold.9+0x22/0x2f [16232.070799] ? watchdog_timer_fn.cold.6+0x62/0x10a [16232.070804] ? __hrtimer_run_queues+0x1c2/0x4d0 [16232.070808] ? watchdog+0x40/0x40 [16232.070812] ? hrtimer_interrupt+0x128/0x2c0 [16232.070816] ? smp_apic_timer_interrupt+0xa6/0x270 [16232.070826] ? apic_timer_interrupt+0xf/0x20 [16232.070830] [16232.070833] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16232.070838] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16232.070842] do_raw_spin_lock+0x177/0x190 [16232.070847] _raw_spin_lock+0x21/0x30 [16232.070853] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16232.072933] obd_mod_free+0x46/0x140 [ptlrpc] [16232.075736] mdc_commit_open+0x91/0xb0 [mdc] [16232.075774] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16232.076824] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16232.083377] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16232.083457] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16232.083463] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16232.083469] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16232.087797] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16232.087814] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16232.087833] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16232.087837] RAX: 000000000000743a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16232.087839] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [16232.087840] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [16232.087842] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [16232.087844] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [16232.087847] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [16232.087849] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16232.087850] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16232.087857] Call Trace: [16232.087862] [16232.088746] ? show_regs.cold.9+0x22/0x2f [16232.088765] ? watchdog_timer_fn.cold.6+0x62/0x10a [16232.088772] ? __hrtimer_run_queues+0x1c2/0x4d0 [16232.088777] ? watchdog+0x40/0x40 [16232.088781] ? hrtimer_interrupt+0x128/0x2c0 [16232.088785] ? smp_apic_timer_interrupt+0xa6/0x270 [16232.088796] ? apic_timer_interrupt+0xf/0x20 [16232.088801] [16232.088805] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16232.088811] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16232.089164] do_raw_spin_lock+0x177/0x190 [16232.089175] _raw_spin_lock+0x21/0x30 [16232.089183] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16232.090674] process_one_work+0x2c8/0x700 [16232.090684] worker_thread+0x296/0x6e0 [16232.090689] ? rescuer_thread+0x570/0x570 [16232.090693] kthread+0x1d1/0x200 [16232.090698] ? set_kthread_struct+0x70/0x70 [16232.090702] ret_from_fork+0x1f/0x30 [16232.680328] after_reply+0xe2c/0x1780 [ptlrpc] [16232.686491] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16232.699776] ? __next_timer_interrupt+0x160/0x160 [16232.709556] ptlrpcd+0xa82/0xe40 [ptlrpc] [16232.718708] ? do_wait_intr+0xf0/0xf0 [16232.722628] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16232.728677] kthread+0x1d1/0x200 [16232.736379] ? set_kthread_struct+0x70/0x70 [16232.744612] ret_from_fork+0x1f/0x30 [16248.054311] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [16248.060873] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16248.105664] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16248.128270] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16248.141581] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16248.144794] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16248.154129] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16248.158081] RAX: 0000000000001f2d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16248.161763] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [16248.165414] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [16248.169138] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [16248.173395] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [16248.177504] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [16248.182491] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16248.185758] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [16248.190272] Call Trace: [16248.192089] [16248.193762] ? show_regs.cold.9+0x22/0x2f [16248.196022] ? watchdog_timer_fn.cold.6+0x62/0x10a [16248.198880] ? __hrtimer_run_queues+0x1c2/0x4d0 [16248.201781] ? watchdog+0x40/0x40 [16248.203710] ? hrtimer_interrupt+0x128/0x2c0 [16248.205982] ? smp_apic_timer_interrupt+0xa6/0x270 [16248.208578] ? apic_timer_interrupt+0xf/0x20 [16248.210853] [16248.212020] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16248.214993] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16248.217973] do_raw_spin_lock+0x177/0x190 [16248.220248] _raw_spin_lock+0x21/0x30 [16248.222235] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16248.225297] ? do_raw_spin_unlock+0x75/0x190 [16248.227538] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16248.230508] ? do_wait_intr+0xf0/0xf0 [16248.232390] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16248.235392] mdc_reint+0x2d/0x110 [mdc] [16248.237270] mdc_unlink+0x658/0x870 [mdc] [16248.239296] lmv_unlink+0x34d/0xbd0 [lmv] [16248.241374] ll_unlink+0x24d/0x7f0 [lustre] [16248.244402] vfs_unlink+0x202/0x330 [16248.246128] do_unlinkat+0x2fa/0x460 [16248.247965] __x64_sys_unlink+0x37/0x50 [16248.249968] do_syscall_64+0xc1/0x440 [16248.251899] entry_SYSCALL_64_after_hwframe+0x49/0xae [16248.254408] RIP: 0033:0x7f151bdee10b [16248.256292] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [16248.270757] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [16248.277904] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [16248.284581] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [16248.290539] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16248.299952] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16248.306303] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16254.432606] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14323s! [16254.439224] Showing busy workqueues and worker pools: [16254.444919] workqueue events: flags=0x0 [16254.448150] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16254.451631] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [16254.461781] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16254.466108] pending: kfree_rcu_monitor [16254.470250] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16254.476019] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [16254.480181] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16254.488170] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [16254.494721] workqueue events_long: flags=0x0 [16254.497800] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16254.500337] pending: lu_objects_flush [obdclass] [16254.502694] workqueue events_unbound: flags=0x2 [16254.504988] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [16254.506704] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16254.510744] workqueue events_power_efficient: flags=0x80 [16254.515279] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16254.519701] pending: do_cache_clean, neigh_periodic_work, check_lifetime [16254.523106] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16254.526158] pending: neigh_periodic_work [16254.530455] workqueue mm_percpu_wq: flags=0x8 [16254.536718] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16254.540020] pending: vmstat_update, lru_add_drain_per_cpu [16254.543102] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16254.548284] pending: vmstat_update [16254.551874] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16254.561325] pending: lru_add_drain_per_cpu [16254.566848] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [16254.572894] pending: lru_add_drain_per_cpu BAR(48) [16254.575325] workqueue writeback: flags=0x4a [16254.576967] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [16254.580488] pending: wb_workfn [16254.581813] workqueue xprtiod: flags=0xa [16254.585176] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16254.590853] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [16254.597531] workqueue ipv6_addrconf: flags=0x40008 [16254.599297] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16254.602540] pending: addrconf_verify_work [16254.604231] workqueue knbd0-recv: flags=0x1a [16254.607494] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16254.610964] in-flight: 658:recv_work [16254.612665] workqueue ptlrpc_pinger: flags=0x2 [16254.616481] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16254.619103] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [16254.621783] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14324s workers=4 idle: 32984 37 8 [16254.625064] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16239s workers=2 idle: 661 [16256.057105] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [16256.066645] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16256.105692] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16256.118541] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16256.123072] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16256.129306] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16256.152659] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16256.160141] RAX: 0000000000006c09 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16256.166623] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [16256.170331] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [16256.180456] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [16256.184732] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [16256.188847] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [16256.192952] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16256.201438] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [16256.206731] Call Trace: [16256.208050] [16256.210009] ? show_regs.cold.9+0x22/0x2f [16256.222118] ? watchdog_timer_fn.cold.6+0x62/0x10a [16256.223822] ? __hrtimer_run_queues+0x1c2/0x4d0 [16256.230420] ? watchdog+0x40/0x40 [16256.233625] ? hrtimer_interrupt+0x128/0x2c0 [16256.235476] ? smp_apic_timer_interrupt+0xa6/0x270 [16256.241844] ? apic_timer_interrupt+0xf/0x20 [16256.244479] [16256.246780] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16256.253569] do_raw_spin_lock+0x177/0x190 [16256.256513] _raw_spin_lock+0x21/0x30 [16256.259831] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16256.262557] ? do_raw_spin_unlock+0x75/0x190 [16256.266778] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16256.269847] ? do_wait_intr+0xf0/0xf0 [16256.272892] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16256.277101] mdc_close+0x53b/0xd70 [mdc] [16256.279291] lmv_close+0x18d/0x490 [lmv] [16256.281325] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [16256.287306] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16256.289469] ll_md_real_close+0xa5/0x2a0 [lustre] [16256.299194] ll_md_close+0x274/0x800 [lustre] [16256.303568] ll_file_release+0x128/0xc50 [lustre] [16256.307681] __fput+0x10d/0x440 [16256.309105] ____fput+0x12/0x20 [16256.310963] task_work_run+0xc6/0x110 [16256.312822] exit_to_usermode_loop+0x1dd/0x1f0 [16256.317550] do_syscall_64+0x3d6/0x440 [16256.320149] entry_SYSCALL_64_after_hwframe+0x49/0xae [16256.322746] RIP: 0033:0x7f151bee3cd8 [16256.326890] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16256.344508] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16256.350822] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [16256.355516] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [16256.361001] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16256.365666] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16256.372097] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16260.056013] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [16260.058108] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [16260.062984] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16260.063068] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16260.063076] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16260.063082] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16260.071629] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16260.071703] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16260.071710] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16260.071713] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16260.071729] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16260.071733] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16260.071738] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [16260.071740] RDX: 0000000000007338 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [16260.071743] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [16260.071745] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [16260.071748] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16260.071751] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [16260.071754] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16260.071756] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16260.071764] Call Trace: [16260.071770] [16260.071775] ? show_regs.cold.9+0x22/0x2f [16260.071785] ? watchdog_timer_fn.cold.6+0x62/0x10a [16260.071793] ? __hrtimer_run_queues+0x1c2/0x4d0 [16260.071797] ? watchdog+0x40/0x40 [16260.071802] ? hrtimer_interrupt+0x128/0x2c0 [16260.071818] ? smp_apic_timer_interrupt+0xa6/0x270 [16260.071825] ? apic_timer_interrupt+0xf/0x20 [16260.071830] [16260.071833] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16260.071839] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16260.097502] [16260.097512] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16260.097531] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16260.097535] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16260.097540] RAX: 0000000000005adb RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16260.097543] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [16260.097545] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [16260.097547] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [16260.097549] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [16260.097552] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [16260.097555] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16260.097557] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16260.097566] Call Trace: [16260.097571] [16260.097577] ? show_regs.cold.9+0x22/0x2f [16260.097588] ? watchdog_timer_fn.cold.6+0x62/0x10a [16260.097594] ? __hrtimer_run_queues+0x1c2/0x4d0 [16260.097598] ? watchdog+0x40/0x40 [16260.097603] ? hrtimer_interrupt+0x128/0x2c0 [16260.097606] ? smp_apic_timer_interrupt+0xa6/0x270 [16260.097615] ? apic_timer_interrupt+0xf/0x20 [16260.097619] [16260.097622] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16260.097627] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16260.097634] do_raw_spin_lock+0x177/0x190 [16260.097640] _raw_spin_lock+0x21/0x30 [16260.097644] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16260.097899] process_one_work+0x2c8/0x700 [16260.097904] worker_thread+0x296/0x6e0 [16260.097907] ? rescuer_thread+0x570/0x570 [16260.097909] kthread+0x1d1/0x200 [16260.097913] ? set_kthread_struct+0x70/0x70 [16260.097915] ret_from_fork+0x1f/0x30 [16260.404457] do_raw_spin_lock+0x177/0x190 [16260.406883] _raw_spin_lock+0x21/0x30 [16260.407917] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16260.413486] obd_mod_free+0x46/0x140 [ptlrpc] [16260.416107] mdc_commit_open+0x91/0xb0 [mdc] [16260.419179] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16260.422163] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16260.425222] after_reply+0xe2c/0x1780 [ptlrpc] [16260.427652] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16260.430031] ? __next_timer_interrupt+0x160/0x160 [16260.432131] ptlrpcd+0xa82/0xe40 [ptlrpc] [16260.434197] ? do_wait_intr+0xf0/0xf0 [16260.435886] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16260.438334] kthread+0x1d1/0x200 [16260.439802] ? set_kthread_struct+0x70/0x70 [16260.441671] ret_from_fork+0x1f/0x30 [16280.053147] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [16280.057700] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16280.132471] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16280.139857] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16280.158262] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16280.166202] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16280.199749] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16280.204044] RAX: 0000000000007b84 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16280.219703] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [16280.234538] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [16280.249463] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [16280.257109] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [16280.261963] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [16280.271729] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16280.274542] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [16280.280406] Call Trace: [16280.281792] [16280.286842] ? show_regs.cold.9+0x22/0x2f [16280.291223] ? watchdog_timer_fn.cold.6+0x62/0x10a [16280.299351] ? __hrtimer_run_queues+0x1c2/0x4d0 [16280.308335] ? watchdog+0x40/0x40 [16280.309473] ? hrtimer_interrupt+0x128/0x2c0 [16280.310921] ? smp_apic_timer_interrupt+0xa6/0x270 [16280.323766] ? apic_timer_interrupt+0xf/0x20 [16280.334468] [16280.335230] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16280.337078] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16280.345821] do_raw_spin_lock+0x177/0x190 [16280.348511] _raw_spin_lock+0x21/0x30 [16280.352475] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16280.360358] ? do_raw_spin_unlock+0x75/0x190 [16280.365749] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16280.371513] ? do_wait_intr+0xf0/0xf0 [16280.376629] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16280.380577] mdc_reint+0x2d/0x110 [mdc] [16280.383388] mdc_unlink+0x658/0x870 [mdc] [16280.387211] lmv_unlink+0x34d/0xbd0 [lmv] [16280.392945] ll_unlink+0x24d/0x7f0 [lustre] [16280.399610] vfs_unlink+0x202/0x330 [16280.401830] do_unlinkat+0x2fa/0x460 [16280.405947] __x64_sys_unlink+0x37/0x50 [16280.407355] do_syscall_64+0xc1/0x440 [16280.418441] entry_SYSCALL_64_after_hwframe+0x49/0xae [16280.421056] RIP: 0033:0x7f151bdee10b [16280.422922] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [16280.456641] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [16280.466491] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [16280.473261] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [16280.480637] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16280.487181] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16280.494547] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16284.057261] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [16284.063376] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16284.084374] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16284.088730] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16284.091953] RIP: 0010:native_safe_halt+0x2e/0x50 [16284.093657] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16284.100767] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16284.103601] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16284.106453] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [16284.108987] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [16284.111795] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [16284.114279] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [16284.116780] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [16284.119740] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16284.121838] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [16284.124367] Call Trace: [16284.125442] [16284.126229] ? show_regs.cold.9+0x22/0x2f [16284.127880] ? watchdog_timer_fn.cold.6+0x62/0x10a [16284.130044] ? __hrtimer_run_queues+0x1c2/0x4d0 [16284.131896] ? watchdog+0x40/0x40 [16284.133104] ? hrtimer_interrupt+0x128/0x2c0 [16284.134895] ? smp_apic_timer_interrupt+0xa6/0x270 [16284.136710] ? apic_timer_interrupt+0xf/0x20 [16284.138293] [16284.138930] ? native_safe_halt+0x2e/0x50 [16284.140588] kvm_wait+0xcd/0xe0 [16284.141833] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16284.143932] do_raw_spin_lock+0x177/0x190 [16284.145699] _raw_spin_lock+0x21/0x30 [16284.146999] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16284.148974] ? do_raw_spin_unlock+0x75/0x190 [16284.150731] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16284.153122] ? do_wait_intr+0xf0/0xf0 [16284.154555] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16284.156931] mdc_close+0x53b/0xd70 [mdc] [16284.158598] lmv_close+0x18d/0x490 [lmv] [16284.166353] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [16284.171230] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16284.177741] ll_md_real_close+0xa5/0x2a0 [lustre] [16284.184008] ll_md_close+0x274/0x800 [lustre] [16284.186579] ll_file_release+0x128/0xc50 [lustre] [16284.188879] __fput+0x10d/0x440 [16284.190335] ____fput+0x12/0x20 [16284.192024] task_work_run+0xc6/0x110 [16284.193837] exit_to_usermode_loop+0x1dd/0x1f0 [16284.195945] do_syscall_64+0x3d6/0x440 [16284.197262] entry_SYSCALL_64_after_hwframe+0x49/0xae [16284.199354] RIP: 0033:0x7f151bee3cd8 [16284.200986] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16284.209823] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16284.213015] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [16284.215806] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [16284.218651] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16284.221547] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16284.224203] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16284.641019] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14354s! [16284.644376] Showing busy workqueues and worker pools: [16284.646802] workqueue events: flags=0x0 [16284.648852] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16284.651979] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [16284.657743] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16284.661087] pending: kfree_rcu_monitor [16284.663413] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16284.666530] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [16284.671583] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16284.674647] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [16284.678037] workqueue events_long: flags=0x0 [16284.679961] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16284.682892] pending: lu_objects_flush [obdclass] [16284.685255] workqueue events_unbound: flags=0x2 [16284.687628] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [16284.690561] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16284.694790] workqueue events_power_efficient: flags=0x80 [16284.697017] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16284.700066] pending: do_cache_clean, neigh_periodic_work, check_lifetime [16284.703664] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16284.706646] pending: neigh_periodic_work [16284.708393] workqueue mm_percpu_wq: flags=0x8 [16284.709921] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16284.712732] pending: vmstat_update, lru_add_drain_per_cpu [16284.715032] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16284.717809] pending: vmstat_update [16284.719723] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16284.722930] pending: lru_add_drain_per_cpu [16284.724891] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [16284.727557] pending: lru_add_drain_per_cpu BAR(48) [16284.729196] workqueue writeback: flags=0x4a [16284.730903] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [16284.733075] pending: wb_workfn [16284.734444] workqueue xprtiod: flags=0xa [16284.735804] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16284.738092] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [16284.741169] workqueue ipv6_addrconf: flags=0x40008 [16284.743267] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16284.745822] pending: addrconf_verify_work [16284.747606] workqueue knbd0-recv: flags=0x1a [16284.749407] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16284.751887] in-flight: 658:recv_work [16284.753808] workqueue ptlrpc_pinger: flags=0x2 [16284.756318] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16284.758969] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [16284.762053] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14354s workers=4 idle: 32984 37 8 [16284.766388] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16269s workers=2 idle: 661 [16288.055636] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [16288.056681] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [16288.059895] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16288.059973] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16288.059981] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16288.059984] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16288.060004] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16288.060009] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16288.060013] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [16288.060016] RDX: 0000000000003048 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [16288.060018] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [16288.060020] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [16288.060023] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16288.060026] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [16288.060033] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16288.060036] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16288.060044] Call Trace: [16288.060049] [16288.060055] ? show_regs.cold.9+0x22/0x2f [16288.060065] ? watchdog_timer_fn.cold.6+0x62/0x10a [16288.060071] ? __hrtimer_run_queues+0x1c2/0x4d0 [16288.060076] ? watchdog+0x40/0x40 [16288.060080] ? hrtimer_interrupt+0x128/0x2c0 [16288.060085] ? smp_apic_timer_interrupt+0xa6/0x270 [16288.060100] ? apic_timer_interrupt+0xf/0x20 [16288.060109] [16288.060111] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16288.060116] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16288.060121] do_raw_spin_lock+0x177/0x190 [16288.060127] _raw_spin_lock+0x21/0x30 [16288.060134] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16288.060497] obd_mod_free+0x46/0x140 [ptlrpc] [16288.060769] mdc_commit_open+0x91/0xb0 [mdc] [16288.060813] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16288.061052] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16288.061295] after_reply+0xe2c/0x1780 [ptlrpc] [16288.061523] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16288.061871] ? __next_timer_interrupt+0x160/0x160 [16288.061886] ptlrpcd+0xa82/0xe40 [ptlrpc] [16288.062143] ? do_wait_intr+0xf0/0xf0 [16288.062153] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16288.062752] kthread+0x1d1/0x200 [16288.062759] ? set_kthread_struct+0x70/0x70 [16288.062762] ret_from_fork+0x1f/0x30 [16288.307460] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16288.356614] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16288.367492] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16288.376202] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16288.383425] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16288.387362] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16288.419933] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [16288.430362] RAX: 0000000000007dfe RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16288.438393] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [16288.450588] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [16288.457968] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [16288.462011] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [16288.467345] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [16288.480605] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16288.482700] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16288.493912] Call Trace: [16288.496975] [16288.498397] ? show_regs.cold.9+0x22/0x2f [16288.500420] ? watchdog_timer_fn.cold.6+0x62/0x10a [16288.504298] ? __hrtimer_run_queues+0x1c2/0x4d0 [16288.507436] ? watchdog+0x40/0x40 [16288.509554] ? hrtimer_interrupt+0x128/0x2c0 [16288.512521] ? smp_apic_timer_interrupt+0xa6/0x270 [16288.521469] ? apic_timer_interrupt+0xf/0x20 [16288.523230] [16288.523991] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16288.531357] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16288.535437] do_raw_spin_lock+0x177/0x190 [16288.536814] _raw_spin_lock+0x21/0x30 [16288.553383] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16288.559678] process_one_work+0x2c8/0x700 [16288.561685] worker_thread+0x296/0x6e0 [16288.563066] ? rescuer_thread+0x570/0x570 [16288.568764] kthread+0x1d1/0x200 [16288.571182] ? set_kthread_struct+0x70/0x70 [16288.573147] ret_from_fork+0x1f/0x30 [16312.052046] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [16312.055477] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16312.057055] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [16312.072749] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16312.072762] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16312.072765] RIP: 0010:native_safe_halt+0x2e/0x50 [16312.072861] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16312.082516] Modules linked in: [16312.088007] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16312.088021] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16312.088024] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [16312.088028] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [16312.088030] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [16312.088033] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [16312.088037] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [16312.088040] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16312.088042] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [16312.088051] Call Trace: [16312.093227] lustre(O) [16312.095487] [16312.107850] osp(O) [16312.109162] ? show_regs.cold.9+0x22/0x2f [16312.109178] ? watchdog_timer_fn.cold.6+0x62/0x10a [16312.109185] ? __hrtimer_run_queues+0x1c2/0x4d0 [16312.109189] ? watchdog+0x40/0x40 [16312.109194] ? hrtimer_interrupt+0x128/0x2c0 [16312.109198] ? smp_apic_timer_interrupt+0xa6/0x270 [16312.109207] ? apic_timer_interrupt+0xf/0x20 [16312.109212] [16312.109217] ? native_safe_halt+0x2e/0x50 [16312.109222] kvm_wait+0xcd/0xe0 [16312.109234] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16312.109241] do_raw_spin_lock+0x177/0x190 [16312.109246] _raw_spin_lock+0x21/0x30 [16312.109252] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16312.109515] ? do_raw_spin_unlock+0x75/0x190 [16312.109521] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16312.109919] ? do_wait_intr+0xf0/0xf0 [16312.109926] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16312.110137] mdc_reint+0x2d/0x110 [mdc] [16312.110177] mdc_unlink+0x658/0x870 [mdc] [16312.110219] lmv_unlink+0x34d/0xbd0 [lmv] [16312.110250] ll_unlink+0x24d/0x7f0 [lustre] [16312.110370] vfs_unlink+0x202/0x330 [16312.110378] do_unlinkat+0x2fa/0x460 [16312.110382] __x64_sys_unlink+0x37/0x50 [16312.110386] do_syscall_64+0xc1/0x440 [16312.110393] entry_SYSCALL_64_after_hwframe+0x49/0xae [16312.110399] RIP: 0033:0x7f151bdee10b [16312.110411] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [16312.110415] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [16312.110421] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [16312.110423] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [16312.110425] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16312.110428] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16312.110430] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16312.313091] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16312.334959] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16312.341727] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16312.346318] RIP: 0010:native_safe_halt+0x2e/0x50 [16312.349483] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16312.360783] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16312.365593] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16312.370192] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [16312.376980] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [16312.381883] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [16312.388026] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [16312.395331] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [16312.399473] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16312.403287] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [16312.406325] Call Trace: [16312.407771] [16312.408990] ? show_regs.cold.9+0x22/0x2f [16312.411400] ? watchdog_timer_fn.cold.6+0x62/0x10a [16312.414102] ? __hrtimer_run_queues+0x1c2/0x4d0 [16312.416642] ? watchdog+0x40/0x40 [16312.418449] ? hrtimer_interrupt+0x128/0x2c0 [16312.420428] ? smp_apic_timer_interrupt+0xa6/0x270 [16312.422737] ? apic_timer_interrupt+0xf/0x20 [16312.427849] [16312.429137] ? native_safe_halt+0x2e/0x50 [16312.432219] kvm_wait+0xcd/0xe0 [16312.436662] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16312.439419] do_raw_spin_lock+0x177/0x190 [16312.441419] _raw_spin_lock+0x21/0x30 [16312.445306] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16312.450975] ? do_raw_spin_unlock+0x75/0x190 [16312.453766] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16312.458123] ? do_wait_intr+0xf0/0xf0 [16312.460808] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16312.464408] mdc_close+0x53b/0xd70 [mdc] [16312.466600] lmv_close+0x18d/0x490 [lmv] [16312.468676] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [16312.473492] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16312.476873] ll_md_real_close+0xa5/0x2a0 [lustre] [16312.479783] ll_md_close+0x274/0x800 [lustre] [16312.482525] ll_file_release+0x128/0xc50 [lustre] [16312.485542] __fput+0x10d/0x440 [16312.487499] ____fput+0x12/0x20 [16312.489590] task_work_run+0xc6/0x110 [16312.491801] exit_to_usermode_loop+0x1dd/0x1f0 [16312.494339] do_syscall_64+0x3d6/0x440 [16312.495797] entry_SYSCALL_64_after_hwframe+0x49/0xae [16312.497760] RIP: 0033:0x7f151bee3cd8 [16312.499769] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16312.510573] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16312.513779] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [16312.517493] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [16312.529462] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16312.533893] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16312.538272] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16314.860132] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14384s! [16314.864882] Showing busy workqueues and worker pools: [16314.867770] workqueue events: flags=0x0 [16314.869945] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16314.873951] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [16314.879691] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16314.884091] pending: kfree_rcu_monitor [16314.886517] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16314.889936] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [16314.895306] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16314.899572] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [16314.903714] workqueue events_long: flags=0x0 [16314.906538] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16314.911207] pending: lu_objects_flush [obdclass] [16314.914542] workqueue events_unbound: flags=0x2 [16314.917674] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [16314.921339] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16314.927847] workqueue events_power_efficient: flags=0x80 [16314.931418] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16314.935330] pending: do_cache_clean, neigh_periodic_work, check_lifetime [16314.939099] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16314.943040] pending: neigh_periodic_work [16314.945651] workqueue mm_percpu_wq: flags=0x8 [16314.947960] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16314.951711] pending: vmstat_update, lru_add_drain_per_cpu [16314.955028] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16314.959088] pending: vmstat_update [16314.961070] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16314.964496] pending: lru_add_drain_per_cpu [16314.966844] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [16314.971086] pending: lru_add_drain_per_cpu BAR(48) [16314.973869] workqueue writeback: flags=0x4a [16314.976591] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [16314.979858] pending: wb_workfn [16314.981583] workqueue xprtiod: flags=0xa [16314.983544] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16314.986904] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [16314.991567] workqueue ipv6_addrconf: flags=0x40008 [16314.994681] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16314.997026] pending: addrconf_verify_work [16314.999071] workqueue knbd0-recv: flags=0x1a [16315.001133] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16315.005331] in-flight: 658:recv_work [16315.007729] workqueue ptlrpc_pinger: flags=0x2 [16315.010071] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16315.013636] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [16315.017247] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14384s workers=4 idle: 32984 37 8 [16315.021330] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16299s workers=2 idle: 661 [16316.056074] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [16316.061404] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16316.062325] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [16316.089149] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16316.089163] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16316.089170] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16316.089659] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16316.089670] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16316.089674] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16316.089679] RAX: 0000000000007409 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16316.089682] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [16316.089684] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [16316.089686] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [16316.089688] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [16316.089691] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [16316.089694] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16316.089696] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16316.089705] Call Trace: [16316.089710] [16316.089715] ? show_regs.cold.9+0x22/0x2f [16316.089725] ? watchdog_timer_fn.cold.6+0x62/0x10a [16316.089732] ? __hrtimer_run_queues+0x1c2/0x4d0 [16316.089736] ? watchdog+0x40/0x40 [16316.089741] ? hrtimer_interrupt+0x128/0x2c0 [16316.089745] ? smp_apic_timer_interrupt+0xa6/0x270 [16316.089754] ? apic_timer_interrupt+0xf/0x20 [16316.089759] [16316.089762] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16316.090048] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16316.090057] do_raw_spin_lock+0x177/0x190 [16316.090062] _raw_spin_lock+0x21/0x30 [16316.090067] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16316.090288] process_one_work+0x2c8/0x700 [16316.090295] worker_thread+0x296/0x6e0 [16316.090299] ? rescuer_thread+0x570/0x570 [16316.090303] kthread+0x1d1/0x200 [16316.090317] ? set_kthread_struct+0x70/0x70 [16316.090322] ret_from_fork+0x1f/0x30 [16316.294696] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16316.327454] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16316.344649] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16316.351249] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16316.353491] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16316.378263] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16316.384344] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [16316.391351] RDX: 00000000000002be RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [16316.394429] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [16316.399281] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [16316.403655] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16316.407971] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [16316.418360] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16316.428614] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16316.435508] Call Trace: [16316.438473] [16316.442182] ? show_regs.cold.9+0x22/0x2f [16316.444342] ? watchdog_timer_fn.cold.6+0x62/0x10a [16316.450211] ? __hrtimer_run_queues+0x1c2/0x4d0 [16316.454472] ? watchdog+0x40/0x40 [16316.455575] ? hrtimer_interrupt+0x128/0x2c0 [16316.462557] ? smp_apic_timer_interrupt+0xa6/0x270 [16316.468545] ? apic_timer_interrupt+0xf/0x20 [16316.471115] [16316.471886] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16316.478311] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16316.484288] do_raw_spin_lock+0x177/0x190 [16316.488868] _raw_spin_lock+0x21/0x30 [16316.491474] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16316.495600] obd_mod_free+0x46/0x140 [ptlrpc] [16316.501592] mdc_commit_open+0x91/0xb0 [mdc] [16316.504799] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16316.506842] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16316.523296] after_reply+0xe2c/0x1780 [ptlrpc] [16316.529477] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16316.537468] ? __next_timer_interrupt+0x160/0x160 [16316.540755] ptlrpcd+0xa82/0xe40 [ptlrpc] [16316.546378] ? do_wait_intr+0xf0/0xf0 [16316.550795] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16316.564337] kthread+0x1d1/0x200 [16316.572475] ? set_kthread_struct+0x70/0x70 [16316.581486] ret_from_fork+0x1f/0x30 [16340.054731] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [16340.057308] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [16340.060447] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16340.060522] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16340.060529] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16340.060531] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16340.060544] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16340.060548] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16340.060552] RAX: 0000000000001754 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16340.060555] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [16340.060557] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [16340.060559] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [16340.060561] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [16340.060564] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [16340.064643] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16340.064782] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16340.064790] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16340.064792] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [16340.064806] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [16340.064810] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16340.064815] RAX: 0000000000004db5 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16340.064817] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [16340.064819] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [16340.064822] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [16340.064824] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [16340.064828] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [16340.064831] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16340.064834] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [16340.064842] Call Trace: [16340.064848] [16340.064853] ? show_regs.cold.9+0x22/0x2f [16340.100175] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16340.100187] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [16340.100196] Call Trace: [16340.100203] [16340.100210] ? show_regs.cold.9+0x22/0x2f [16340.100223] ? watchdog_timer_fn.cold.6+0x62/0x10a [16340.100229] ? __hrtimer_run_queues+0x1c2/0x4d0 [16340.100234] ? watchdog+0x40/0x40 [16340.100239] ? hrtimer_interrupt+0x128/0x2c0 [16340.100244] ? smp_apic_timer_interrupt+0xa6/0x270 [16340.100253] ? apic_timer_interrupt+0xf/0x20 [16340.100258] [16340.100261] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16340.100268] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16340.100273] do_raw_spin_lock+0x177/0x190 [16340.100278] _raw_spin_lock+0x21/0x30 [16340.100285] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16340.101066] ? do_raw_spin_unlock+0x75/0x190 [16340.101074] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16340.101290] ? do_wait_intr+0xf0/0xf0 [16340.101299] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16340.119663] ? watchdog_timer_fn.cold.6+0x62/0x10a [16340.128383] mdc_reint+0x2d/0x110 [mdc] [16340.128441] mdc_unlink+0x658/0x870 [mdc] [16340.128466] lmv_unlink+0x34d/0xbd0 [lmv] [16340.128493] ll_unlink+0x24d/0x7f0 [lustre] [16340.128605] vfs_unlink+0x202/0x330 [16340.129408] do_unlinkat+0x2fa/0x460 [16340.129414] __x64_sys_unlink+0x37/0x50 [16340.129417] do_syscall_64+0xc1/0x440 [16340.140904] ? __hrtimer_run_queues+0x1c2/0x4d0 [16340.140924] ? watchdog+0x40/0x40 [16340.140930] ? hrtimer_interrupt+0x128/0x2c0 [16340.140934] ? smp_apic_timer_interrupt+0xa6/0x270 [16340.140941] ? apic_timer_interrupt+0xf/0x20 [16340.140946] [16340.140951] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [16340.140963] do_raw_spin_lock+0x177/0x190 [16340.140969] _raw_spin_lock+0x21/0x30 [16340.140974] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16340.141382] ? do_raw_spin_unlock+0x75/0x190 [16340.141388] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16340.141602] ? do_wait_intr+0xf0/0xf0 [16340.141611] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16340.141818] mdc_close+0x53b/0xd70 [mdc] [16340.141856] lmv_close+0x18d/0x490 [lmv] [16340.141885] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [16340.142038] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16340.142145] ll_md_real_close+0xa5/0x2a0 [lustre] [16340.158716] entry_SYSCALL_64_after_hwframe+0x49/0xae [16340.158738] RIP: 0033:0x7f151bdee10b [16340.158745] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [16340.158749] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [16340.158754] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [16340.158756] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [16340.158758] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16340.158760] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16340.158763] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16340.604769] ll_md_close+0x274/0x800 [lustre] [16340.606511] ll_file_release+0x128/0xc50 [lustre] [16340.611790] __fput+0x10d/0x440 [16340.613842] ____fput+0x12/0x20 [16340.615646] task_work_run+0xc6/0x110 [16340.617878] exit_to_usermode_loop+0x1dd/0x1f0 [16340.620882] do_syscall_64+0x3d6/0x440 [16340.623098] entry_SYSCALL_64_after_hwframe+0x49/0xae [16340.625860] RIP: 0033:0x7f151bee3cd8 [16340.627611] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16340.649541] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16340.667813] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [16340.675805] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [16340.680032] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16340.686293] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16340.693200] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16344.056209] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [16344.057916] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [16344.067190] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16344.067281] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16344.067347] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16344.067354] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16344.067675] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16344.067686] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16344.067691] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16344.067695] RAX: 000000000000666a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16344.067697] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [16344.067699] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [16344.067701] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [16344.067703] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [16344.067706] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [16344.067710] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16344.067712] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16344.067721] Call Trace: [16344.067726] [16344.067731] ? show_regs.cold.9+0x22/0x2f [16344.067740] ? watchdog_timer_fn.cold.6+0x62/0x10a [16344.067752] ? __hrtimer_run_queues+0x1c2/0x4d0 [16344.067756] ? watchdog+0x40/0x40 [16344.067760] ? hrtimer_interrupt+0x128/0x2c0 [16344.067764] ? smp_apic_timer_interrupt+0xa6/0x270 [16344.067773] ? apic_timer_interrupt+0xf/0x20 [16344.067777] [16344.067781] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16344.067785] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16344.067790] do_raw_spin_lock+0x177/0x190 [16344.067795] _raw_spin_lock+0x21/0x30 [16344.067800] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16344.068019] process_one_work+0x2c8/0x700 [16344.068027] worker_thread+0x296/0x6e0 [16344.068032] ? rescuer_thread+0x570/0x570 [16344.068036] kthread+0x1d1/0x200 [16344.068042] ? set_kthread_struct+0x70/0x70 [16344.068045] ret_from_fork+0x1f/0x30 [16344.328296] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16344.372664] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16344.394328] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16344.400208] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16344.409994] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16344.428479] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16344.434245] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [16344.437354] RDX: 0000000000007bda RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [16344.440973] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [16344.444131] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [16344.447053] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16344.450680] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [16344.460359] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16344.462280] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16344.477894] Call Trace: [16344.479916] [16344.484266] ? show_regs.cold.9+0x22/0x2f [16344.489421] ? watchdog_timer_fn.cold.6+0x62/0x10a [16344.497191] ? __hrtimer_run_queues+0x1c2/0x4d0 [16344.502121] ? watchdog+0x40/0x40 [16344.503111] ? hrtimer_interrupt+0x128/0x2c0 [16344.506231] ? smp_apic_timer_interrupt+0xa6/0x270 [16344.514832] ? apic_timer_interrupt+0xf/0x20 [16344.522267] [16344.522993] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16344.526029] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16344.528127] do_raw_spin_lock+0x177/0x190 [16344.530191] _raw_spin_lock+0x21/0x30 [16344.532238] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16344.539288] obd_mod_free+0x46/0x140 [ptlrpc] [16344.548103] mdc_commit_open+0x91/0xb0 [mdc] [16344.552232] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16344.555311] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16344.559000] after_reply+0xe2c/0x1780 [ptlrpc] [16344.561177] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16344.567468] ? __next_timer_interrupt+0x160/0x160 [16344.571086] ptlrpcd+0xa82/0xe40 [ptlrpc] [16344.572741] ? do_wait_intr+0xf0/0xf0 [16344.579316] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16344.594313] kthread+0x1d1/0x200 [16344.596993] ? set_kthread_struct+0x70/0x70 [16344.600213] ret_from_fork+0x1f/0x30 [16345.056209] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14414s! [16345.069100] Showing busy workqueues and worker pools: [16345.079041] workqueue events: flags=0x0 [16345.083153] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16345.095461] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [16345.113655] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16345.122050] pending: kfree_rcu_monitor [16345.127743] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16345.136706] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [16345.153424] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16345.168181] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [16345.186565] workqueue events_long: flags=0x0 [16345.191361] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16345.199149] pending: lu_objects_flush [obdclass] [16345.206306] workqueue events_unbound: flags=0x2 [16345.208030] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [16345.216200] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16345.228304] workqueue events_power_efficient: flags=0x80 [16345.233762] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16345.247530] pending: do_cache_clean, neigh_periodic_work, check_lifetime [16345.249539] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16345.261067] pending: neigh_periodic_work [16345.265101] workqueue mm_percpu_wq: flags=0x8 [16345.270143] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16345.280231] pending: vmstat_update, lru_add_drain_per_cpu [16345.286870] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16345.294158] pending: vmstat_update [16345.298837] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16345.306906] pending: lru_add_drain_per_cpu [16345.311593] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [16345.319270] pending: lru_add_drain_per_cpu BAR(48) [16345.324240] workqueue writeback: flags=0x4a [16345.328086] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [16345.330244] pending: wb_workfn [16345.334652] workqueue xprtiod: flags=0xa [16345.336881] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16345.345893] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [16345.352268] workqueue ipv6_addrconf: flags=0x40008 [16345.361387] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16345.367226] pending: addrconf_verify_work [16345.370596] workqueue knbd0-recv: flags=0x1a [16345.373819] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16345.377330] in-flight: 658:recv_work [16345.382976] workqueue ptlrpc_pinger: flags=0x2 [16345.386345] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16345.393998] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [16345.400495] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14414s workers=4 idle: 32984 37 8 [16345.407925] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16330s workers=2 idle: 661 [16368.057603] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [16368.061435] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16368.124296] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16368.142091] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16368.146351] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16368.153055] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16368.175729] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16368.189280] RAX: 0000000000007b3b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16368.193661] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [16368.210138] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [16368.234383] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [16368.243730] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [16368.252815] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [16368.260363] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16368.271147] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [16368.287278] Call Trace: [16368.288292] [16368.289134] ? show_regs.cold.9+0x22/0x2f [16368.295115] ? watchdog_timer_fn.cold.6+0x62/0x10a [16368.300631] ? __hrtimer_run_queues+0x1c2/0x4d0 [16368.306829] ? watchdog+0x40/0x40 [16368.308671] ? hrtimer_interrupt+0x128/0x2c0 [16368.311481] ? smp_apic_timer_interrupt+0xa6/0x270 [16368.320485] ? apic_timer_interrupt+0xf/0x20 [16368.322076] [16368.322866] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16368.330457] do_raw_spin_lock+0x177/0x190 [16368.338467] _raw_spin_lock+0x21/0x30 [16368.339791] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16368.347720] ? do_raw_spin_unlock+0x75/0x190 [16368.349279] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16368.359647] ? do_wait_intr+0xf0/0xf0 [16368.366267] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16368.370992] mdc_close+0x53b/0xd70 [mdc] [16368.372493] lmv_close+0x18d/0x490 [lmv] [16368.380627] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [16368.387622] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16368.395970] ll_md_real_close+0xa5/0x2a0 [lustre] [16368.407120] ll_md_close+0x274/0x800 [lustre] [16368.408882] ll_file_release+0x128/0xc50 [lustre] [16368.418601] __fput+0x10d/0x440 [16368.423238] ____fput+0x12/0x20 [16368.426224] task_work_run+0xc6/0x110 [16368.428852] exit_to_usermode_loop+0x1dd/0x1f0 [16368.431103] do_syscall_64+0x3d6/0x440 [16368.434138] entry_SYSCALL_64_after_hwframe+0x49/0xae [16368.442406] RIP: 0033:0x7f151bee3cd8 [16368.445231] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16368.460099] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16368.469136] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [16368.473330] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [16368.482650] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16368.487097] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16368.494529] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16372.055911] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [16372.057038] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [16372.057661] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [16372.057668] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16372.057736] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16372.057741] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16372.057743] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16372.057757] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16372.057762] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16372.057766] RAX: 0000000000004070 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16372.057768] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [16372.057771] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [16372.057773] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [16372.057775] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [16372.057778] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [16372.058426] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16372.058428] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [16372.058436] Call Trace: [16372.058442] [16372.058448] ? show_regs.cold.9+0x22/0x2f [16372.058458] ? watchdog_timer_fn.cold.6+0x62/0x10a [16372.058465] ? __hrtimer_run_queues+0x1c2/0x4d0 [16372.058469] ? watchdog+0x40/0x40 [16372.058474] ? hrtimer_interrupt+0x128/0x2c0 [16372.058478] ? smp_apic_timer_interrupt+0xa6/0x270 [16372.058487] ? apic_timer_interrupt+0xf/0x20 [16372.058491] [16372.058494] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16372.058498] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16372.058502] do_raw_spin_lock+0x177/0x190 [16372.058507] _raw_spin_lock+0x21/0x30 [16372.058512] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16372.058771] ? do_raw_spin_unlock+0x75/0x190 [16372.058777] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16372.061804] ? do_wait_intr+0xf0/0xf0 [16372.061816] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16372.062116] mdc_reint+0x2d/0x110 [mdc] [16372.062150] mdc_unlink+0x658/0x870 [mdc] [16372.062189] lmv_unlink+0x34d/0xbd0 [lmv] [16372.062221] ll_unlink+0x24d/0x7f0 [lustre] [16372.062356] vfs_unlink+0x202/0x330 [16372.062376] do_unlinkat+0x2fa/0x460 [16372.062381] __x64_sys_unlink+0x37/0x50 [16372.062385] do_syscall_64+0xc1/0x440 [16372.062392] entry_SYSCALL_64_after_hwframe+0x49/0xae [16372.062401] RIP: 0033:0x7f151bdee10b [16372.062408] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [16372.062413] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [16372.062418] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [16372.062420] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [16372.062422] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16372.062424] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16372.062427] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16372.065041] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16372.065128] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16372.065136] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16372.065139] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16372.065153] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16372.065158] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16372.065163] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [16372.065165] RDX: 0000000000005197 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [16372.065168] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [16372.065171] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [16372.065173] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16372.065177] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [16372.065180] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16372.065183] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16372.065193] Call Trace: [16372.065198] [16372.065203] ? show_regs.cold.9+0x22/0x2f [16372.065211] ? watchdog_timer_fn.cold.6+0x62/0x10a [16372.065217] ? __hrtimer_run_queues+0x1c2/0x4d0 [16372.065221] ? watchdog+0x40/0x40 [16372.065226] ? hrtimer_interrupt+0x128/0x2c0 [16372.065231] ? smp_apic_timer_interrupt+0xa6/0x270 [16372.065238] ? apic_timer_interrupt+0xf/0x20 [16372.065243] [16372.065247] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16372.065251] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16372.065256] do_raw_spin_lock+0x177/0x190 [16372.065261] _raw_spin_lock+0x21/0x30 [16372.065266] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16372.065587] obd_mod_free+0x46/0x140 [ptlrpc] [16372.065782] mdc_commit_open+0x91/0xb0 [mdc] [16372.065824] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16372.066170] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16372.066485] after_reply+0xe2c/0x1780 [ptlrpc] [16372.067089] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16372.067655] ? __next_timer_interrupt+0x160/0x160 [16372.067665] ptlrpcd+0xa82/0xe40 [ptlrpc] [16372.067912] ? do_wait_intr+0xf0/0xf0 [16372.067921] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16372.068127] kthread+0x1d1/0x200 [16372.068133] ? set_kthread_struct+0x70/0x70 [16372.068137] ret_from_fork+0x1f/0x30 [16372.772558] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16372.825549] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16372.836476] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16372.852055] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16372.865539] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16372.874838] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16372.903850] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16372.924152] RAX: 0000000000005486 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16372.933751] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [16372.940723] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [16372.950200] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [16372.960271] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [16372.971961] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [16372.985216] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16372.998207] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16373.016112] Call Trace: [16373.017886] [16373.023958] ? show_regs.cold.9+0x22/0x2f [16373.025559] ? watchdog_timer_fn.cold.6+0x62/0x10a [16373.033016] ? __hrtimer_run_queues+0x1c2/0x4d0 [16373.034595] ? watchdog+0x40/0x40 [16373.042054] ? hrtimer_interrupt+0x128/0x2c0 [16373.047096] ? smp_apic_timer_interrupt+0xa6/0x270 [16373.057164] ? apic_timer_interrupt+0xf/0x20 [16373.058666] [16373.066698] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16373.075156] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16373.082107] do_raw_spin_lock+0x177/0x190 [16373.095803] _raw_spin_lock+0x21/0x30 [16373.097180] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16373.111958] process_one_work+0x2c8/0x700 [16373.118010] worker_thread+0x296/0x6e0 [16373.119569] ? rescuer_thread+0x570/0x570 [16373.121009] kthread+0x1d1/0x200 [16373.131704] ? set_kthread_struct+0x70/0x70 [16373.135589] ret_from_fork+0x1f/0x30 [16375.778315] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14445s! [16375.791468] Showing busy workqueues and worker pools: [16375.795062] workqueue events: flags=0x0 [16375.799703] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16375.806012] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [16375.817868] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16375.821798] pending: kfree_rcu_monitor [16375.825631] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16375.828119] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [16375.842905] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16375.845269] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [16375.862053] workqueue events_long: flags=0x0 [16375.869206] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16375.884883] pending: lu_objects_flush [obdclass] [16375.892916] workqueue events_unbound: flags=0x2 [16375.904602] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [16375.906584] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16375.923689] workqueue events_power_efficient: flags=0x80 [16375.930385] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16375.938136] pending: do_cache_clean, neigh_periodic_work, check_lifetime [16375.950888] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16375.957913] pending: neigh_periodic_work [16375.962432] workqueue mm_percpu_wq: flags=0x8 [16375.967319] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16375.970821] pending: vmstat_update, lru_add_drain_per_cpu [16375.976340] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16375.992105] pending: vmstat_update [16375.999934] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16376.018070] pending: lru_add_drain_per_cpu [16376.023975] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [16376.030023] pending: lru_add_drain_per_cpu BAR(48) [16376.035127] workqueue writeback: flags=0x4a [16376.041109] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [16376.054009] pending: wb_workfn [16376.062908] workqueue xprtiod: flags=0xa [16376.067159] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16376.077376] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [16376.093447] workqueue ipv6_addrconf: flags=0x40008 [16376.101028] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16376.103188] pending: addrconf_verify_work [16376.123490] workqueue knbd0-recv: flags=0x1a [16376.131417] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16376.133734] in-flight: 658:recv_work [16376.148767] workqueue ptlrpc_pinger: flags=0x2 [16376.161945] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16376.173801] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [16376.182465] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14445s workers=4 idle: 32984 37 8 [16376.193894] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16360s workers=2 idle: 661 [16384.949446] rcu: INFO: rcu_sched self-detected stall on CPU [16384.951904] rcu: 1-...!: (8659098 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [16384.957088] (t=14460302 jiffies g=249657 q=6812) [16384.961447] rcu: rcu_sched kthread starved for 14455254 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [16384.968459] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [16384.974819] rcu: RCU grace-period kthread stack dump: [16384.977386] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [16384.985552] Call Trace: [16384.987586] __schedule+0x351/0xcb0 [16384.990448] schedule+0xc0/0x180 [16384.992207] schedule_timeout+0xb4/0x190 [16384.997892] ? do_raw_spin_unlock+0x75/0x190 [16385.009283] ? __next_timer_interrupt+0x160/0x160 [16385.021769] ? prepare_to_swait_event+0xf3/0x140 [16385.027730] rcu_gp_kthread+0x8dd/0x15b0 [16385.036625] ? force_qs_rnp+0x2c0/0x2c0 [16385.038079] kthread+0x1d1/0x200 [16385.039005] ? set_kthread_struct+0x70/0x70 [16385.040202] ret_from_fork+0x1f/0x30 [16385.048335] rcu: Stack dump where RCU GP kthread last ran: [16385.059120] NMI backtrace for cpu 1 [16385.066595] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16385.083263] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16385.098065] Call Trace: [16385.108548] [16385.109298] ? dump_stack+0xbb/0x10e [16385.110570] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [16385.134426] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [16385.144005] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [16385.156209] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [16385.169328] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [16385.184617] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [16385.198001] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [16385.214958] ? update_process_times+0x6d/0xc0 [16385.237887] ? tick_sched_handle.isra.18+0x27/0x90 [16385.249700] ? tick_sched_timer+0x53/0xb0 [16385.259253] ? __hrtimer_run_queues+0x1c2/0x4d0 [16385.260871] ? tick_sched_handle.isra.18+0x90/0x90 [16385.276823] ? hrtimer_interrupt+0x128/0x2c0 [16385.278380] ? smp_apic_timer_interrupt+0xa6/0x270 [16385.280151] ? apic_timer_interrupt+0xf/0x20 [16385.298553] [16385.303437] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16385.315387] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16385.318886] ? do_raw_spin_lock+0x177/0x190 [16385.330704] ? _raw_spin_lock+0x21/0x30 [16385.337792] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16385.351983] ? obd_mod_free+0x46/0x140 [ptlrpc] [16385.366998] ? mdc_commit_open+0x91/0xb0 [mdc] [16385.378616] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16385.380278] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16385.390792] ? after_reply+0xe2c/0x1780 [ptlrpc] [16385.394695] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16385.400468] ? __next_timer_interrupt+0x160/0x160 [16385.404711] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [16385.406939] ? do_wait_intr+0xf0/0xf0 [16385.410031] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16385.415479] ? kthread+0x1d1/0x200 [16385.416314] ? set_kthread_struct+0x70/0x70 [16385.420328] ? ret_from_fork+0x1f/0x30 [16385.421246] NMI backtrace for cpu 1 [16385.422863] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16385.427834] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16385.432772] Call Trace: [16385.433735] [16385.434302] ? dump_stack+0xbb/0x10e [16385.435202] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [16385.437943] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [16385.440065] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [16385.441293] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [16385.443873] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [16385.446586] ? rcu_dump_cpu_stacks+0x11b/0x171 [16385.450115] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [16385.451968] ? update_process_times+0x6d/0xc0 [16385.454300] ? tick_sched_handle.isra.18+0x27/0x90 [16385.456561] ? tick_sched_timer+0x53/0xb0 [16385.457603] ? __hrtimer_run_queues+0x1c2/0x4d0 [16385.459029] ? tick_sched_handle.isra.18+0x90/0x90 [16385.461644] ? hrtimer_interrupt+0x128/0x2c0 [16385.463894] ? smp_apic_timer_interrupt+0xa6/0x270 [16385.466709] ? apic_timer_interrupt+0xf/0x20 [16385.468700] [16385.469970] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16385.477445] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16385.483906] ? do_raw_spin_lock+0x177/0x190 [16385.485697] ? _raw_spin_lock+0x21/0x30 [16385.486740] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16385.491461] ? obd_mod_free+0x46/0x140 [ptlrpc] [16385.492779] ? mdc_commit_open+0x91/0xb0 [mdc] [16385.493908] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16385.498280] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16385.502350] ? after_reply+0xe2c/0x1780 [ptlrpc] [16385.504528] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16385.507910] ? __next_timer_interrupt+0x160/0x160 [16385.510476] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [16385.511850] ? do_wait_intr+0xf0/0xf0 [16385.512762] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16385.516961] ? kthread+0x1d1/0x200 [16385.524875] ? set_kthread_struct+0x70/0x70 [16385.528482] ? ret_from_fork+0x1f/0x30 [16396.057687] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [16396.063391] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16396.093362] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16396.104479] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16396.111788] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16396.119808] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16396.135959] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16396.141738] RAX: 0000000000007dc9 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16396.145478] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [16396.149837] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [16396.153800] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [16396.157542] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [16396.162438] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [16396.166824] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16396.169758] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [16396.173620] Call Trace: [16396.174856] [16396.176194] ? show_regs.cold.9+0x22/0x2f [16396.178499] ? watchdog_timer_fn.cold.6+0x62/0x10a [16396.181407] ? __hrtimer_run_queues+0x1c2/0x4d0 [16396.183801] ? watchdog+0x40/0x40 [16396.186402] ? hrtimer_interrupt+0x128/0x2c0 [16396.188745] ? smp_apic_timer_interrupt+0xa6/0x270 [16396.191369] ? apic_timer_interrupt+0xf/0x20 [16396.195226] [16396.196119] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16396.198487] do_raw_spin_lock+0x177/0x190 [16396.200021] _raw_spin_lock+0x21/0x30 [16396.203786] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16396.212139] ? do_raw_spin_unlock+0x75/0x190 [16396.214785] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16396.221446] ? do_wait_intr+0xf0/0xf0 [16396.223897] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16396.228917] mdc_close+0x53b/0xd70 [mdc] [16396.230842] lmv_close+0x18d/0x490 [lmv] [16396.232881] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [16396.235329] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16396.238199] ll_md_real_close+0xa5/0x2a0 [lustre] [16396.247837] ll_md_close+0x274/0x800 [lustre] [16396.254115] ll_file_release+0x128/0xc50 [lustre] [16396.256528] __fput+0x10d/0x440 [16396.259185] ____fput+0x12/0x20 [16396.264875] task_work_run+0xc6/0x110 [16396.267046] exit_to_usermode_loop+0x1dd/0x1f0 [16396.269294] do_syscall_64+0x3d6/0x440 [16396.270898] entry_SYSCALL_64_after_hwframe+0x49/0xae [16396.276912] RIP: 0033:0x7f151bee3cd8 [16396.278591] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16396.303274] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16396.311297] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [16396.320813] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [16396.336473] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16396.346363] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16396.355904] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16400.056228] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [16400.059264] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16400.082124] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16400.087628] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16400.097504] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16400.103070] RIP: 0010:native_safe_halt+0x2e/0x50 [16400.107305] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16400.116664] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16400.121338] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16400.124483] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [16400.127814] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [16400.129692] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [16400.134240] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [16400.135806] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [16400.142686] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16400.146717] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16400.151340] Call Trace: [16400.152459] [16400.154421] ? show_regs.cold.9+0x22/0x2f [16400.157975] ? watchdog_timer_fn.cold.6+0x62/0x10a [16400.159711] ? __hrtimer_run_queues+0x1c2/0x4d0 [16400.163798] ? watchdog+0x40/0x40 [16400.165642] ? hrtimer_interrupt+0x128/0x2c0 [16400.166896] ? smp_apic_timer_interrupt+0xa6/0x270 [16400.169763] ? apic_timer_interrupt+0xf/0x20 [16400.173418] [16400.175762] ? native_safe_halt+0x2e/0x50 [16400.178854] kvm_wait+0xcd/0xe0 [16400.184423] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16400.185874] do_raw_spin_lock+0x177/0x190 [16400.190496] _raw_spin_lock+0x21/0x30 [16400.192160] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16400.194530] process_one_work+0x2c8/0x700 [16400.196895] worker_thread+0x296/0x6e0 [16400.201708] ? rescuer_thread+0x570/0x570 [16400.203867] kthread+0x1d1/0x200 [16400.205982] ? set_kthread_struct+0x70/0x70 [16400.209485] ret_from_fork+0x1f/0x30 [16404.052084] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [16404.061696] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16404.096748] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16404.105155] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16404.109226] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16404.113494] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16404.124699] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16404.128851] RAX: 0000000000007f59 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16404.133834] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [16404.136867] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [16404.139293] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [16404.145121] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [16404.149695] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [16404.154385] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16404.158692] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [16404.163501] Call Trace: [16404.165439] [16404.166246] ? show_regs.cold.9+0x22/0x2f [16404.167506] ? watchdog_timer_fn.cold.6+0x62/0x10a [16404.169521] ? __hrtimer_run_queues+0x1c2/0x4d0 [16404.171182] ? watchdog+0x40/0x40 [16404.174374] ? hrtimer_interrupt+0x128/0x2c0 [16404.176641] ? smp_apic_timer_interrupt+0xa6/0x270 [16404.179480] ? apic_timer_interrupt+0xf/0x20 [16404.182562] [16404.183777] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16404.186456] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16404.188741] do_raw_spin_lock+0x177/0x190 [16404.191327] _raw_spin_lock+0x21/0x30 [16404.193563] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16404.195481] ? do_raw_spin_unlock+0x75/0x190 [16404.196933] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16404.198950] ? do_wait_intr+0xf0/0xf0 [16404.200182] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16404.203093] mdc_reint+0x2d/0x110 [mdc] [16404.204677] mdc_unlink+0x658/0x870 [mdc] [16404.206491] lmv_unlink+0x34d/0xbd0 [lmv] [16404.208514] ll_unlink+0x24d/0x7f0 [lustre] [16404.215812] vfs_unlink+0x202/0x330 [16404.221503] do_unlinkat+0x2fa/0x460 [16404.223504] __x64_sys_unlink+0x37/0x50 [16404.224897] do_syscall_64+0xc1/0x440 [16404.226446] entry_SYSCALL_64_after_hwframe+0x49/0xae [16404.231065] RIP: 0033:0x7f151bdee10b [16404.232218] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [16404.245528] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [16404.249076] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [16404.255833] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [16404.266848] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16404.269928] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16404.278399] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16406.498113] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14475s! [16406.501633] Showing busy workqueues and worker pools: [16406.504060] workqueue events: flags=0x0 [16406.505955] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16406.509096] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [16406.513414] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16406.516391] pending: kfree_rcu_monitor [16406.518647] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16406.522039] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [16406.527144] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16406.531708] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [16406.535790] workqueue events_long: flags=0x0 [16406.538062] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16406.541195] pending: lu_objects_flush [obdclass] [16406.543621] workqueue events_unbound: flags=0x2 [16406.545801] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [16406.548761] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16406.553533] workqueue events_power_efficient: flags=0x80 [16406.556203] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16406.558820] pending: do_cache_clean, neigh_periodic_work, check_lifetime [16406.561979] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16406.564886] pending: neigh_periodic_work [16406.566642] workqueue mm_percpu_wq: flags=0x8 [16406.568311] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16406.571713] pending: vmstat_update, lru_add_drain_per_cpu [16406.574413] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16406.578015] pending: vmstat_update [16406.579409] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16406.581567] pending: lru_add_drain_per_cpu [16406.583223] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [16406.585231] pending: lru_add_drain_per_cpu BAR(48) [16406.587105] workqueue writeback: flags=0x4a [16406.588440] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [16406.590609] pending: wb_workfn [16406.591645] workqueue xprtiod: flags=0xa [16406.592664] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16406.594821] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [16406.597444] workqueue ipv6_addrconf: flags=0x40008 [16406.600185] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16406.602367] pending: addrconf_verify_work [16406.603664] workqueue knbd0-recv: flags=0x1a [16406.605390] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16406.608090] in-flight: 658:recv_work [16406.609355] workqueue ptlrpc_pinger: flags=0x2 [16406.610534] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16406.612384] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [16406.614324] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14476s workers=4 idle: 32984 37 8 [16406.616604] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16391s workers=2 idle: 661 [16412.055020] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [16412.057646] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16412.093945] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16412.102297] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16412.111497] RIP: 0010:native_safe_halt+0x2e/0x50 [16412.114472] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16412.132538] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16412.150456] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [16412.164670] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [16412.184172] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [16412.189596] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [16412.197563] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16412.203513] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [16412.211565] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16412.214630] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16412.219437] Call Trace: [16412.222423] [16412.224277] ? show_regs.cold.9+0x22/0x2f [16412.227756] ? watchdog_timer_fn.cold.6+0x62/0x10a [16412.231556] ? __hrtimer_run_queues+0x1c2/0x4d0 [16412.233787] ? watchdog+0x40/0x40 [16412.235309] ? hrtimer_interrupt+0x128/0x2c0 [16412.237125] ? smp_apic_timer_interrupt+0xa6/0x270 [16412.239443] ? apic_timer_interrupt+0xf/0x20 [16412.241412] [16412.242345] ? native_safe_halt+0x2e/0x50 [16412.243793] kvm_wait+0xcd/0xe0 [16412.245307] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16412.247096] do_raw_spin_lock+0x177/0x190 [16412.248432] _raw_spin_lock+0x21/0x30 [16412.249948] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16412.252715] obd_mod_free+0x46/0x140 [ptlrpc] [16412.255380] mdc_commit_open+0x91/0xb0 [mdc] [16412.257094] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16412.259122] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16412.260818] after_reply+0xe2c/0x1780 [ptlrpc] [16412.263009] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16412.264604] ? __next_timer_interrupt+0x160/0x160 [16412.268492] ptlrpcd+0xa82/0xe40 [ptlrpc] [16412.270189] ? do_wait_intr+0xf0/0xf0 [16412.274387] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16412.276441] kthread+0x1d1/0x200 [16412.279651] ? set_kthread_struct+0x70/0x70 [16412.284504] ret_from_fork+0x1f/0x30 [16424.057021] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [16424.063703] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16424.101236] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16424.108009] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16424.114506] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16424.118571] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16424.133109] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16424.136737] RAX: 0000000000002bc1 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16424.140001] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [16424.143506] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [16424.147259] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [16424.150688] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [16424.153919] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [16424.157999] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16424.160697] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [16424.164381] Call Trace: [16424.165544] [16424.166223] ? show_regs.cold.9+0x22/0x2f [16424.167451] ? watchdog_timer_fn.cold.6+0x62/0x10a [16424.169354] ? __hrtimer_run_queues+0x1c2/0x4d0 [16424.171394] ? watchdog+0x40/0x40 [16424.173223] ? hrtimer_interrupt+0x128/0x2c0 [16424.175013] ? smp_apic_timer_interrupt+0xa6/0x270 [16424.176742] ? apic_timer_interrupt+0xf/0x20 [16424.178432] [16424.179185] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16424.180632] do_raw_spin_lock+0x177/0x190 [16424.182412] _raw_spin_lock+0x21/0x30 [16424.183568] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16424.186363] ? do_raw_spin_unlock+0x75/0x190 [16424.188258] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16424.189975] ? do_wait_intr+0xf0/0xf0 [16424.191008] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16424.194126] mdc_close+0x53b/0xd70 [mdc] [16424.195320] lmv_close+0x18d/0x490 [lmv] [16424.197048] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [16424.198671] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16424.200426] ll_md_real_close+0xa5/0x2a0 [lustre] [16424.202212] ll_md_close+0x274/0x800 [lustre] [16424.204027] ll_file_release+0x128/0xc50 [lustre] [16424.205590] __fput+0x10d/0x440 [16424.207009] ____fput+0x12/0x20 [16424.208386] task_work_run+0xc6/0x110 [16424.209713] exit_to_usermode_loop+0x1dd/0x1f0 [16424.211231] do_syscall_64+0x3d6/0x440 [16424.212360] entry_SYSCALL_64_after_hwframe+0x49/0xae [16424.214040] RIP: 0033:0x7f151bee3cd8 [16424.215395] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16424.221294] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16424.223502] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [16424.225960] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [16424.228537] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16424.232045] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16424.236335] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16428.056154] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [16428.061051] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16428.104503] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16428.113476] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16428.124036] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16428.128545] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16428.133874] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16428.159924] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16428.170049] RAX: 0000000000004e0f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16428.183862] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [16428.190847] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [16428.198502] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [16428.214903] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [16428.221573] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [16428.225952] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16428.231337] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16428.234791] Call Trace: [16428.235909] [16428.236928] ? show_regs.cold.9+0x22/0x2f [16428.238976] ? watchdog_timer_fn.cold.6+0x62/0x10a [16428.241904] ? __hrtimer_run_queues+0x1c2/0x4d0 [16428.244318] ? watchdog+0x40/0x40 [16428.245849] ? hrtimer_interrupt+0x128/0x2c0 [16428.248676] ? smp_apic_timer_interrupt+0xa6/0x270 [16428.250550] ? apic_timer_interrupt+0xf/0x20 [16428.252669] [16428.253660] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16428.255614] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16428.257956] do_raw_spin_lock+0x177/0x190 [16428.259794] _raw_spin_lock+0x21/0x30 [16428.261271] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16428.266191] process_one_work+0x2c8/0x700 [16428.269893] worker_thread+0x296/0x6e0 [16428.274843] ? rescuer_thread+0x570/0x570 [16428.277420] kthread+0x1d1/0x200 [16428.280876] ? set_kthread_struct+0x70/0x70 [16428.284786] ret_from_fork+0x1f/0x30 [16432.052055] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [16432.055239] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16432.082845] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16432.087072] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16432.090992] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16432.093967] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16432.102402] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16432.105505] RAX: 0000000000003b2a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16432.107590] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [16432.109923] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [16432.113264] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [16432.116879] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [16432.120064] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [16432.124040] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16432.126562] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [16432.129975] Call Trace: [16432.131418] [16432.132623] ? show_regs.cold.9+0x22/0x2f [16432.134336] ? watchdog_timer_fn.cold.6+0x62/0x10a [16432.136585] ? __hrtimer_run_queues+0x1c2/0x4d0 [16432.138728] ? watchdog+0x40/0x40 [16432.140152] ? hrtimer_interrupt+0x128/0x2c0 [16432.142195] ? smp_apic_timer_interrupt+0xa6/0x270 [16432.144212] ? apic_timer_interrupt+0xf/0x20 [16432.146222] [16432.147189] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16432.149725] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16432.152240] do_raw_spin_lock+0x177/0x190 [16432.154082] _raw_spin_lock+0x21/0x30 [16432.155840] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16432.158209] ? do_raw_spin_unlock+0x75/0x190 [16432.160109] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16432.162603] ? do_wait_intr+0xf0/0xf0 [16432.164188] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16432.166815] mdc_reint+0x2d/0x110 [mdc] [16432.168844] mdc_unlink+0x658/0x870 [mdc] [16432.170775] lmv_unlink+0x34d/0xbd0 [lmv] [16432.172743] ll_unlink+0x24d/0x7f0 [lustre] [16432.174897] vfs_unlink+0x202/0x330 [16432.176382] do_unlinkat+0x2fa/0x460 [16432.178195] __x64_sys_unlink+0x37/0x50 [16432.179852] do_syscall_64+0xc1/0x440 [16432.181572] entry_SYSCALL_64_after_hwframe+0x49/0xae [16432.183893] RIP: 0033:0x7f151bdee10b [16432.185479] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [16432.193363] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [16432.197179] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [16432.200282] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [16432.203638] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16432.206885] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16432.211843] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16436.704114] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14506s! [16436.710374] Showing busy workqueues and worker pools: [16436.715306] workqueue events: flags=0x0 [16436.716536] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16436.724385] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [16436.731120] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16436.738137] pending: kfree_rcu_monitor [16436.741323] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16436.748337] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [16436.754318] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16436.757519] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [16436.768131] workqueue events_long: flags=0x0 [16436.769754] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16436.772552] pending: lu_objects_flush [obdclass] [16436.774893] workqueue events_unbound: flags=0x2 [16436.776712] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [16436.779010] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16436.783152] workqueue events_power_efficient: flags=0x80 [16436.785247] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16436.787762] pending: do_cache_clean, neigh_periodic_work, check_lifetime [16436.790581] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16436.793328] pending: neigh_periodic_work [16436.795117] workqueue mm_percpu_wq: flags=0x8 [16436.796939] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16436.799577] pending: vmstat_update, lru_add_drain_per_cpu [16436.801351] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16436.803492] pending: vmstat_update [16436.804763] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16436.806863] pending: lru_add_drain_per_cpu [16436.808331] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [16436.810680] pending: lru_add_drain_per_cpu BAR(48) [16436.812743] workqueue writeback: flags=0x4a [16436.814429] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [16436.816801] pending: wb_workfn [16436.818025] workqueue xprtiod: flags=0xa [16436.819464] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16436.821954] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [16436.825066] workqueue ipv6_addrconf: flags=0x40008 [16436.826898] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16436.829185] pending: addrconf_verify_work [16436.830952] workqueue knbd0-recv: flags=0x1a [16436.832726] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16436.835212] in-flight: 658:recv_work [16436.836882] workqueue ptlrpc_pinger: flags=0x2 [16436.838511] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16436.841034] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [16436.843473] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14506s workers=4 idle: 32984 37 8 [16436.846317] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16421s workers=2 idle: 661 [16440.055102] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [16440.066171] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16440.117623] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16440.125326] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16440.132190] RIP: 0010:native_safe_halt+0x2e/0x50 [16440.134177] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16440.141840] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16440.145627] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [16440.150506] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [16440.153722] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [16440.157196] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [16440.159966] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16440.163051] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [16440.166355] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16440.168898] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16440.172287] Call Trace: [16440.173315] [16440.174687] ? show_regs.cold.9+0x22/0x2f [16440.176233] ? watchdog_timer_fn.cold.6+0x62/0x10a [16440.178241] ? __hrtimer_run_queues+0x1c2/0x4d0 [16440.180105] ? watchdog+0x40/0x40 [16440.181332] ? hrtimer_interrupt+0x128/0x2c0 [16440.183147] ? smp_apic_timer_interrupt+0xa6/0x270 [16440.185243] ? apic_timer_interrupt+0xf/0x20 [16440.187071] [16440.188003] ? native_safe_halt+0x2e/0x50 [16440.189519] kvm_wait+0xcd/0xe0 [16440.191044] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16440.193264] do_raw_spin_lock+0x177/0x190 [16440.195153] _raw_spin_lock+0x21/0x30 [16440.197114] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16440.199637] obd_mod_free+0x46/0x140 [ptlrpc] [16440.201880] mdc_commit_open+0x91/0xb0 [mdc] [16440.203828] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16440.206243] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16440.208974] after_reply+0xe2c/0x1780 [ptlrpc] [16440.211294] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16440.213909] ? __next_timer_interrupt+0x160/0x160 [16440.215864] ptlrpcd+0xa82/0xe40 [ptlrpc] [16440.217857] ? do_wait_intr+0xf0/0xf0 [16440.219227] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16440.221799] kthread+0x1d1/0x200 [16440.223227] ? set_kthread_struct+0x70/0x70 [16440.224948] ret_from_fork+0x1f/0x30 [16452.057245] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [16452.064676] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16452.092294] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16452.104116] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16452.109352] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16452.114627] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16452.125232] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16452.133197] RAX: 0000000000006f0b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16452.142666] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [16452.148658] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [16452.159559] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [16452.168562] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [16452.173566] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [16452.180204] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16452.185194] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [16452.194992] Call Trace: [16452.200046] [16452.202889] ? show_regs.cold.9+0x22/0x2f [16452.206337] ? watchdog_timer_fn.cold.6+0x62/0x10a [16452.212838] ? __hrtimer_run_queues+0x1c2/0x4d0 [16452.216614] ? watchdog+0x40/0x40 [16452.220636] ? hrtimer_interrupt+0x128/0x2c0 [16452.224064] ? smp_apic_timer_interrupt+0xa6/0x270 [16452.229165] ? apic_timer_interrupt+0xf/0x20 [16452.234110] [16452.236865] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16452.242670] do_raw_spin_lock+0x177/0x190 [16452.248126] _raw_spin_lock+0x21/0x30 [16452.250171] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16452.254419] ? do_raw_spin_unlock+0x75/0x190 [16452.257920] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16452.264432] ? do_wait_intr+0xf0/0xf0 [16452.267008] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16452.271679] mdc_close+0x53b/0xd70 [mdc] [16452.274723] lmv_close+0x18d/0x490 [lmv] [16452.284284] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [16452.286651] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16452.289235] ll_md_real_close+0xa5/0x2a0 [lustre] [16452.295267] ll_md_close+0x274/0x800 [lustre] [16452.299725] ll_file_release+0x128/0xc50 [lustre] [16452.305228] __fput+0x10d/0x440 [16452.309691] ____fput+0x12/0x20 [16452.311533] task_work_run+0xc6/0x110 [16452.315686] exit_to_usermode_loop+0x1dd/0x1f0 [16452.321922] do_syscall_64+0x3d6/0x440 [16452.328669] entry_SYSCALL_64_after_hwframe+0x49/0xae [16452.336354] RIP: 0033:0x7f151bee3cd8 [16452.337498] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16452.346363] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16452.349000] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [16452.354663] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [16452.357561] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16452.360739] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16452.363663] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16456.056028] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [16456.059599] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16456.079193] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16456.083958] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16456.087946] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16456.091150] RIP: 0010:native_safe_halt+0x2e/0x50 [16456.093224] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16456.100880] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16456.104086] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16456.107644] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [16456.110520] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [16456.113637] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [16456.116822] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [16456.119838] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [16456.123189] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16456.125305] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16456.128084] Call Trace: [16456.128931] [16456.129374] ? show_regs.cold.9+0x22/0x2f [16456.130681] ? watchdog_timer_fn.cold.6+0x62/0x10a [16456.132101] ? __hrtimer_run_queues+0x1c2/0x4d0 [16456.133616] ? watchdog+0x40/0x40 [16456.134736] ? hrtimer_interrupt+0x128/0x2c0 [16456.136011] ? smp_apic_timer_interrupt+0xa6/0x270 [16456.137867] ? apic_timer_interrupt+0xf/0x20 [16456.139326] [16456.140002] ? native_safe_halt+0x2e/0x50 [16456.141753] kvm_wait+0xcd/0xe0 [16456.142993] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16456.144910] do_raw_spin_lock+0x177/0x190 [16456.146584] _raw_spin_lock+0x21/0x30 [16456.148090] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16456.150687] process_one_work+0x2c8/0x700 [16456.152219] worker_thread+0x296/0x6e0 [16456.154021] ? rescuer_thread+0x570/0x570 [16456.155657] kthread+0x1d1/0x200 [16456.156871] ? set_kthread_struct+0x70/0x70 [16456.158488] ret_from_fork+0x1f/0x30 [16464.052026] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [16464.056449] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16464.079214] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16464.085680] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16464.089843] RIP: 0010:native_safe_halt+0x2e/0x50 [16464.092039] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16464.101153] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16464.105020] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16464.109133] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [16464.112807] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [16464.116256] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [16464.120063] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [16464.123989] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [16464.128215] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16464.131017] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [16464.134720] Call Trace: [16464.135994] [16464.136997] ? show_regs.cold.9+0x22/0x2f [16464.139048] ? watchdog_timer_fn.cold.6+0x62/0x10a [16464.141943] ? __hrtimer_run_queues+0x1c2/0x4d0 [16464.143927] ? watchdog+0x40/0x40 [16464.145442] ? hrtimer_interrupt+0x128/0x2c0 [16464.147970] ? smp_apic_timer_interrupt+0xa6/0x270 [16464.150132] ? apic_timer_interrupt+0xf/0x20 [16464.152206] [16464.153315] ? native_safe_halt+0x2e/0x50 [16464.155435] kvm_wait+0xcd/0xe0 [16464.157452] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16464.160899] do_raw_spin_lock+0x177/0x190 [16464.162265] _raw_spin_lock+0x21/0x30 [16464.164317] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16464.167028] ? do_raw_spin_unlock+0x75/0x190 [16464.169718] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16464.172705] ? do_wait_intr+0xf0/0xf0 [16464.174720] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16464.177193] mdc_reint+0x2d/0x110 [mdc] [16464.181281] mdc_unlink+0x658/0x870 [mdc] [16464.186686] lmv_unlink+0x34d/0xbd0 [lmv] [16464.189091] ll_unlink+0x24d/0x7f0 [lustre] [16464.191120] vfs_unlink+0x202/0x330 [16464.192822] do_unlinkat+0x2fa/0x460 [16464.196059] __x64_sys_unlink+0x37/0x50 [16464.198126] do_syscall_64+0xc1/0x440 [16464.200104] entry_SYSCALL_64_after_hwframe+0x49/0xae [16464.204428] RIP: 0033:0x7f151bdee10b [16464.205933] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [16464.216432] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [16464.220416] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [16464.225456] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [16464.233674] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16464.237179] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16464.243798] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16466.913047] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14536s! [16466.921986] Showing busy workqueues and worker pools: [16466.927045] workqueue events: flags=0x0 [16466.935818] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16466.941201] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [16466.951902] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16466.957989] pending: kfree_rcu_monitor [16466.959850] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16466.963838] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [16466.967829] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16466.975480] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [16466.984654] workqueue events_long: flags=0x0 [16466.990289] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16466.996501] pending: lu_objects_flush [obdclass] [16466.998878] workqueue events_unbound: flags=0x2 [16467.002625] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [16467.004615] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16467.010782] workqueue events_power_efficient: flags=0x80 [16467.013540] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16467.016225] pending: do_cache_clean, neigh_periodic_work, check_lifetime [16467.019148] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16467.022113] pending: neigh_periodic_work [16467.024037] workqueue mm_percpu_wq: flags=0x8 [16467.026028] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16467.028965] pending: vmstat_update, lru_add_drain_per_cpu [16467.031237] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16467.040668] pending: vmstat_update [16467.047553] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16467.053483] pending: lru_add_drain_per_cpu [16467.055780] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [16467.059829] pending: lru_add_drain_per_cpu BAR(48) [16467.062085] workqueue writeback: flags=0x4a [16467.064103] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [16467.069135] pending: wb_workfn [16467.074873] workqueue xprtiod: flags=0xa [16467.082864] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16467.085632] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [16467.094089] workqueue ipv6_addrconf: flags=0x40008 [16467.100921] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16467.106960] pending: addrconf_verify_work [16467.109211] workqueue knbd0-recv: flags=0x1a [16467.111745] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16467.115452] in-flight: 658:recv_work [16467.117526] workqueue ptlrpc_pinger: flags=0x2 [16467.120504] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16467.129169] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [16467.135063] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14536s workers=4 idle: 32984 37 8 [16467.141042] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16451s workers=2 idle: 661 [16468.055114] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [16468.058128] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16468.080088] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16468.084730] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16468.088035] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16468.090035] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16468.097067] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16468.099985] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [16468.102471] RDX: 0000000000003637 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [16468.105674] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [16468.108103] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [16468.110911] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16468.113702] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [16468.116747] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16468.118622] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16468.120423] Call Trace: [16468.121104] [16468.121904] ? show_regs.cold.9+0x22/0x2f [16468.123669] ? watchdog_timer_fn.cold.6+0x62/0x10a [16468.125272] ? __hrtimer_run_queues+0x1c2/0x4d0 [16468.127176] ? watchdog+0x40/0x40 [16468.128270] ? hrtimer_interrupt+0x128/0x2c0 [16468.129718] ? smp_apic_timer_interrupt+0xa6/0x270 [16468.132850] ? apic_timer_interrupt+0xf/0x20 [16468.133880] [16468.134383] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16468.138917] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16468.142865] do_raw_spin_lock+0x177/0x190 [16468.144896] _raw_spin_lock+0x21/0x30 [16468.146747] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16468.148297] obd_mod_free+0x46/0x140 [ptlrpc] [16468.150900] mdc_commit_open+0x91/0xb0 [mdc] [16468.152238] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16468.158677] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16468.162339] after_reply+0xe2c/0x1780 [ptlrpc] [16468.164140] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16468.168685] ? __next_timer_interrupt+0x160/0x160 [16468.173843] ptlrpcd+0xa82/0xe40 [ptlrpc] [16468.181171] ? do_wait_intr+0xf0/0xf0 [16468.183188] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16468.185732] kthread+0x1d1/0x200 [16468.188013] ? set_kthread_struct+0x70/0x70 [16468.190042] ret_from_fork+0x1f/0x30 [16480.057030] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [16480.059696] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16480.085152] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16480.089833] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16480.094079] RIP: 0010:native_safe_halt+0x2e/0x50 [16480.101902] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16480.110229] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16480.113999] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16480.123002] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [16480.128009] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [16480.138265] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [16480.141711] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [16480.145846] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [16480.157340] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16480.159743] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [16480.163247] Call Trace: [16480.164774] [16480.165650] ? show_regs.cold.9+0x22/0x2f [16480.168147] ? watchdog_timer_fn.cold.6+0x62/0x10a [16480.170518] ? __hrtimer_run_queues+0x1c2/0x4d0 [16480.173804] ? watchdog+0x40/0x40 [16480.175666] ? hrtimer_interrupt+0x128/0x2c0 [16480.177808] ? smp_apic_timer_interrupt+0xa6/0x270 [16480.180225] ? apic_timer_interrupt+0xf/0x20 [16480.181884] [16480.182695] ? native_safe_halt+0x2e/0x50 [16480.184846] kvm_wait+0xcd/0xe0 [16480.186628] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16480.189080] do_raw_spin_lock+0x177/0x190 [16480.191025] _raw_spin_lock+0x21/0x30 [16480.192254] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16480.196225] ? do_raw_spin_unlock+0x75/0x190 [16480.199748] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16480.202278] ? do_wait_intr+0xf0/0xf0 [16480.204355] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16480.207110] mdc_close+0x53b/0xd70 [mdc] [16480.208826] lmv_close+0x18d/0x490 [lmv] [16480.210116] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [16480.213983] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16480.217522] ll_md_real_close+0xa5/0x2a0 [lustre] [16480.220702] ll_md_close+0x274/0x800 [lustre] [16480.224113] ll_file_release+0x128/0xc50 [lustre] [16480.226666] __fput+0x10d/0x440 [16480.228347] ____fput+0x12/0x20 [16480.229788] task_work_run+0xc6/0x110 [16480.232190] exit_to_usermode_loop+0x1dd/0x1f0 [16480.234241] do_syscall_64+0x3d6/0x440 [16480.235957] entry_SYSCALL_64_after_hwframe+0x49/0xae [16480.238486] RIP: 0033:0x7f151bee3cd8 [16480.240957] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16480.260250] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16480.264839] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [16480.267461] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [16480.269989] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16480.273509] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16480.284472] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16484.056097] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [16484.059591] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16484.102006] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16484.109973] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16484.116042] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16484.121926] RIP: 0010:native_safe_halt+0x2e/0x50 [16484.128069] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16484.138381] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16484.143591] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16484.148310] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [16484.153971] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [16484.158931] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [16484.163047] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [16484.169133] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [16484.175783] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16484.179943] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16484.185077] Call Trace: [16484.187050] [16484.188731] ? show_regs.cold.9+0x22/0x2f [16484.190373] ? watchdog_timer_fn.cold.6+0x62/0x10a [16484.192463] ? __hrtimer_run_queues+0x1c2/0x4d0 [16484.195997] ? watchdog+0x40/0x40 [16484.198855] ? hrtimer_interrupt+0x128/0x2c0 [16484.201917] ? smp_apic_timer_interrupt+0xa6/0x270 [16484.203723] ? apic_timer_interrupt+0xf/0x20 [16484.207251] [16484.208760] ? native_safe_halt+0x2e/0x50 [16484.211776] kvm_wait+0xcd/0xe0 [16484.214137] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16484.218422] do_raw_spin_lock+0x177/0x190 [16484.221434] _raw_spin_lock+0x21/0x30 [16484.224417] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16484.229087] process_one_work+0x2c8/0x700 [16484.232539] worker_thread+0x296/0x6e0 [16484.235082] ? rescuer_thread+0x570/0x570 [16484.236946] kthread+0x1d1/0x200 [16484.238419] ? set_kthread_struct+0x70/0x70 [16484.240365] ret_from_fork+0x1f/0x30 [16492.054540] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [16492.060794] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16492.106522] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16492.112633] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16492.115686] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16492.117857] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16492.135544] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16492.150371] RAX: 00000000000011d2 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16492.154832] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [16492.163057] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [16492.171007] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [16492.182458] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [16492.184284] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [16492.193559] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16492.198989] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [16492.207752] Call Trace: [16492.208741] [16492.209363] ? show_regs.cold.9+0x22/0x2f [16492.212977] ? watchdog_timer_fn.cold.6+0x62/0x10a [16492.221163] ? __hrtimer_run_queues+0x1c2/0x4d0 [16492.225440] ? watchdog+0x40/0x40 [16492.231426] ? hrtimer_interrupt+0x128/0x2c0 [16492.232640] ? smp_apic_timer_interrupt+0xa6/0x270 [16492.234081] ? apic_timer_interrupt+0xf/0x20 [16492.241008] [16492.243350] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16492.250522] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16492.252217] do_raw_spin_lock+0x177/0x190 [16492.257560] _raw_spin_lock+0x21/0x30 [16492.263025] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16492.270370] ? do_raw_spin_unlock+0x75/0x190 [16492.272362] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16492.275223] ? do_wait_intr+0xf0/0xf0 [16492.276855] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16492.286800] mdc_reint+0x2d/0x110 [mdc] [16492.288653] mdc_unlink+0x658/0x870 [mdc] [16492.293526] lmv_unlink+0x34d/0xbd0 [lmv] [16492.296862] ll_unlink+0x24d/0x7f0 [lustre] [16492.298832] vfs_unlink+0x202/0x330 [16492.300544] do_unlinkat+0x2fa/0x460 [16492.301874] __x64_sys_unlink+0x37/0x50 [16492.303873] do_syscall_64+0xc1/0x440 [16492.304857] entry_SYSCALL_64_after_hwframe+0x49/0xae [16492.306173] RIP: 0033:0x7f151bdee10b [16492.309887] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [16492.323129] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [16492.328222] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [16492.333280] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [16492.342378] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16492.349002] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16492.355173] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16496.055284] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [16496.060448] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16496.081107] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16496.086178] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16496.090137] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16496.093103] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16496.101104] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16496.104551] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [16496.107735] RDX: 00000000000041f2 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [16496.110817] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [16496.113903] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [16496.117529] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16496.121124] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [16496.124606] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16496.126997] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16496.130377] Call Trace: [16496.131440] [16496.132305] ? show_regs.cold.9+0x22/0x2f [16496.134023] ? watchdog_timer_fn.cold.6+0x62/0x10a [16496.136430] ? __hrtimer_run_queues+0x1c2/0x4d0 [16496.138466] ? watchdog+0x40/0x40 [16496.140020] ? hrtimer_interrupt+0x128/0x2c0 [16496.141675] ? smp_apic_timer_interrupt+0xa6/0x270 [16496.143851] ? apic_timer_interrupt+0xf/0x20 [16496.145700] [16496.146735] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16496.149310] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16496.151456] do_raw_spin_lock+0x177/0x190 [16496.153257] _raw_spin_lock+0x21/0x30 [16496.154905] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16496.157393] obd_mod_free+0x46/0x140 [ptlrpc] [16496.159627] mdc_commit_open+0x91/0xb0 [mdc] [16496.161379] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16496.163786] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16496.166564] after_reply+0xe2c/0x1780 [ptlrpc] [16496.169337] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16496.171924] ? __next_timer_interrupt+0x160/0x160 [16496.174235] ptlrpcd+0xa82/0xe40 [ptlrpc] [16496.176442] ? do_wait_intr+0xf0/0xf0 [16496.177976] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16496.180202] kthread+0x1d1/0x200 [16496.181440] ? set_kthread_struct+0x70/0x70 [16496.183419] ret_from_fork+0x1f/0x30 [16497.636040] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14567s! [16497.640510] Showing busy workqueues and worker pools: [16497.642232] workqueue events: flags=0x0 [16497.643384] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16497.645968] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [16497.650466] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16497.653870] pending: kfree_rcu_monitor [16497.655921] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16497.657930] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [16497.663663] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16497.667158] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [16497.670875] workqueue events_long: flags=0x0 [16497.673161] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16497.676956] pending: lu_objects_flush [obdclass] [16497.680141] workqueue events_unbound: flags=0x2 [16497.682137] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [16497.685741] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16497.690975] workqueue events_power_efficient: flags=0x80 [16497.694209] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16497.697729] pending: do_cache_clean, neigh_periodic_work, check_lifetime [16497.702207] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16497.705769] pending: neigh_periodic_work [16497.707459] workqueue mm_percpu_wq: flags=0x8 [16497.708871] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16497.710978] pending: vmstat_update, lru_add_drain_per_cpu [16497.713596] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16497.717326] pending: vmstat_update [16497.719239] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16497.722294] pending: lru_add_drain_per_cpu [16497.724574] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [16497.727555] pending: lru_add_drain_per_cpu BAR(48) [16497.730271] workqueue writeback: flags=0x4a [16497.732605] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [16497.734826] pending: wb_workfn [16497.736523] workqueue xprtiod: flags=0xa [16497.738308] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16497.740471] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [16497.742828] workqueue ipv6_addrconf: flags=0x40008 [16497.744278] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16497.746130] pending: addrconf_verify_work [16497.747516] workqueue knbd0-recv: flags=0x1a [16497.748849] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16497.750822] in-flight: 658:recv_work [16497.752300] workqueue ptlrpc_pinger: flags=0x2 [16497.753559] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16497.755598] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [16497.757442] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14567s workers=4 idle: 32984 37 8 [16497.759656] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16482s workers=2 idle: 661 [16508.057709] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [16508.060936] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16508.084882] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16508.088592] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16508.097820] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16508.102384] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16508.108808] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16508.112123] RAX: 0000000000002e0f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16508.114294] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [16508.116733] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [16508.120539] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [16508.124061] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [16508.127728] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [16508.138152] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16508.146420] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [16508.155346] Call Trace: [16508.157368] [16508.160171] ? show_regs.cold.9+0x22/0x2f [16508.166435] ? watchdog_timer_fn.cold.6+0x62/0x10a [16508.168563] ? __hrtimer_run_queues+0x1c2/0x4d0 [16508.174715] ? watchdog+0x40/0x40 [16508.179513] ? hrtimer_interrupt+0x128/0x2c0 [16508.182173] ? smp_apic_timer_interrupt+0xa6/0x270 [16508.184336] ? apic_timer_interrupt+0xf/0x20 [16508.186641] [16508.187613] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16508.195516] do_raw_spin_lock+0x177/0x190 [16508.197575] _raw_spin_lock+0x21/0x30 [16508.199224] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16508.201773] ? do_raw_spin_unlock+0x75/0x190 [16508.203694] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16508.206653] ? do_wait_intr+0xf0/0xf0 [16508.208237] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16508.211100] mdc_close+0x53b/0xd70 [mdc] [16508.212781] lmv_close+0x18d/0x490 [lmv] [16508.214503] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [16508.217500] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16508.220173] ll_md_real_close+0xa5/0x2a0 [lustre] [16508.222444] ll_md_close+0x274/0x800 [lustre] [16508.224158] ll_file_release+0x128/0xc50 [lustre] [16508.226437] __fput+0x10d/0x440 [16508.228035] ____fput+0x12/0x20 [16508.229514] task_work_run+0xc6/0x110 [16508.231257] exit_to_usermode_loop+0x1dd/0x1f0 [16508.232917] do_syscall_64+0x3d6/0x440 [16508.234394] entry_SYSCALL_64_after_hwframe+0x49/0xae [16508.236571] RIP: 0033:0x7f151bee3cd8 [16508.243139] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16508.261639] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16508.266513] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [16508.272293] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [16508.276558] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16508.279878] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16508.286366] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16512.056383] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [16512.059328] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16512.112758] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16512.123747] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16512.129518] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16512.133642] RIP: 0010:native_safe_halt+0x2e/0x50 [16512.136042] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16512.151440] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16512.161197] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16512.165718] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [16512.169293] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [16512.172225] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [16512.177615] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [16512.186699] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [16512.189676] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16512.192761] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16512.197680] Call Trace: [16512.198826] [16512.200203] ? show_regs.cold.9+0x22/0x2f [16512.202445] ? watchdog_timer_fn.cold.6+0x62/0x10a [16512.205147] ? __hrtimer_run_queues+0x1c2/0x4d0 [16512.207574] ? watchdog+0x40/0x40 [16512.209261] ? hrtimer_interrupt+0x128/0x2c0 [16512.216219] ? smp_apic_timer_interrupt+0xa6/0x270 [16512.217912] ? apic_timer_interrupt+0xf/0x20 [16512.221476] [16512.222731] ? native_safe_halt+0x2e/0x50 [16512.227563] kvm_wait+0xcd/0xe0 [16512.231335] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16512.234268] do_raw_spin_lock+0x177/0x190 [16512.236518] _raw_spin_lock+0x21/0x30 [16512.240480] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16512.243770] process_one_work+0x2c8/0x700 [16512.246490] worker_thread+0x296/0x6e0 [16512.249408] ? rescuer_thread+0x570/0x570 [16512.253343] kthread+0x1d1/0x200 [16512.254575] ? set_kthread_struct+0x70/0x70 [16512.260590] ret_from_fork+0x1f/0x30 [16524.055409] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [16524.061379] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16524.064032] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [16524.142572] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16524.142586] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16524.142589] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16524.142603] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16524.142607] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16524.142613] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [16524.142615] RDX: 0000000000007936 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [16524.142618] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [16524.142620] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [16524.142623] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16524.142627] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [16524.142629] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16524.142631] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16524.142641] Call Trace: [16524.142646] [16524.142652] ? show_regs.cold.9+0x22/0x2f [16524.142662] ? watchdog_timer_fn.cold.6+0x62/0x10a [16524.142669] ? __hrtimer_run_queues+0x1c2/0x4d0 [16524.142674] ? watchdog+0x40/0x40 [16524.142678] ? hrtimer_interrupt+0x128/0x2c0 [16524.142682] ? smp_apic_timer_interrupt+0xa6/0x270 [16524.142690] ? apic_timer_interrupt+0xf/0x20 [16524.142694] [16524.142697] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16524.142701] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16524.142706] do_raw_spin_lock+0x177/0x190 [16524.142710] _raw_spin_lock+0x21/0x30 [16524.142717] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16524.143911] obd_mod_free+0x46/0x140 [ptlrpc] [16524.144901] mdc_commit_open+0x91/0xb0 [mdc] [16524.144942] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16524.145660] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16524.145969] after_reply+0xe2c/0x1780 [ptlrpc] [16524.147095] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16524.148256] ? __next_timer_interrupt+0x160/0x160 [16524.148267] ptlrpcd+0xa82/0xe40 [ptlrpc] [16524.152730] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16524.152804] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16524.152811] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16524.152814] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16524.152828] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16524.152832] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [16524.152837] RAX: 0000000000007c88 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16524.152839] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [16524.152842] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [16524.152843] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [16524.152846] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [16524.152849] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [16524.152852] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16524.152854] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [16524.152862] Call Trace: [16524.152867] [16524.152873] ? show_regs.cold.9+0x22/0x2f [16524.152881] ? watchdog_timer_fn.cold.6+0x62/0x10a [16524.174349] ? do_wait_intr+0xf0/0xf0 [16524.174371] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16524.189230] ? __hrtimer_run_queues+0x1c2/0x4d0 [16524.189248] ? watchdog+0x40/0x40 [16524.189254] ? hrtimer_interrupt+0x128/0x2c0 [16524.189258] ? smp_apic_timer_interrupt+0xa6/0x270 [16524.189267] ? apic_timer_interrupt+0xf/0x20 [16524.189271] [16524.189275] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16524.189281] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16524.189285] do_raw_spin_lock+0x177/0x190 [16524.189291] _raw_spin_lock+0x21/0x30 [16524.189298] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16524.189556] ? do_raw_spin_unlock+0x75/0x190 [16524.189563] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16524.189750] ? do_wait_intr+0xf0/0xf0 [16524.189760] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16524.189969] mdc_reint+0x2d/0x110 [mdc] [16524.190005] mdc_unlink+0x658/0x870 [mdc] [16524.190032] lmv_unlink+0x34d/0xbd0 [lmv] [16524.190064] ll_unlink+0x24d/0x7f0 [lustre] [16524.190171] vfs_unlink+0x202/0x330 [16524.190179] do_unlinkat+0x2fa/0x460 [16524.190184] __x64_sys_unlink+0x37/0x50 [16524.190187] do_syscall_64+0xc1/0x440 [16524.190193] entry_SYSCALL_64_after_hwframe+0x49/0xae [16524.190200] RIP: 0033:0x7f151bdee10b [16524.190207] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [16524.190211] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [16524.190217] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [16524.190220] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [16524.190223] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16524.190225] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16524.190227] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16525.074735] kthread+0x1d1/0x200 [16525.079999] ? set_kthread_struct+0x70/0x70 [16525.083757] ret_from_fork+0x1f/0x30 [16527.851823] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14597s! [16527.869163] Showing busy workqueues and worker pools: [16527.877476] workqueue events: flags=0x0 [16527.883244] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16527.890605] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [16527.901698] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16527.910571] pending: kfree_rcu_monitor [16527.920990] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16527.923534] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [16527.952799] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16527.979254] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [16528.006501] workqueue events_long: flags=0x0 [16528.008188] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16528.016426] pending: lu_objects_flush [obdclass] [16528.029353] workqueue events_unbound: flags=0x2 [16528.035649] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [16528.037689] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16528.061989] workqueue events_power_efficient: flags=0x80 [16528.063970] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16528.075208] pending: do_cache_clean, neigh_periodic_work, check_lifetime [16528.094466] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16528.115895] pending: neigh_periodic_work [16528.117487] workqueue mm_percpu_wq: flags=0x8 [16528.121551] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16528.133925] pending: vmstat_update, lru_add_drain_per_cpu [16528.152018] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16528.158260] pending: vmstat_update [16528.163272] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16528.172644] pending: lru_add_drain_per_cpu [16528.179414] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [16528.187416] pending: lru_add_drain_per_cpu BAR(48) [16528.202345] workqueue writeback: flags=0x4a [16528.206431] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [16528.217593] pending: wb_workfn [16528.226396] workqueue xprtiod: flags=0xa [16528.233316] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16528.239665] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [16528.257545] workqueue ipv6_addrconf: flags=0x40008 [16528.259037] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16528.273129] pending: addrconf_verify_work [16528.278289] workqueue knbd0-recv: flags=0x1a [16528.283550] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16528.289603] in-flight: 658:recv_work [16528.293606] workqueue ptlrpc_pinger: flags=0x2 [16528.301907] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16528.306376] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [16528.317639] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14597s workers=4 idle: 32984 37 8 [16528.339689] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16513s workers=2 idle: 661 [16536.058573] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [16536.070227] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16536.174309] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16536.184723] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16536.196764] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16536.202561] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16536.241699] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16536.257817] RAX: 00000000000059f3 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16536.266087] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [16536.278378] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [16536.285603] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [16536.301914] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [16536.317343] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [16536.333809] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16536.339216] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [16536.354264] Call Trace: [16536.359259] [16536.366133] ? show_regs.cold.9+0x22/0x2f [16536.369984] ? watchdog_timer_fn.cold.6+0x62/0x10a [16536.382206] ? __hrtimer_run_queues+0x1c2/0x4d0 [16536.390792] ? watchdog+0x40/0x40 [16536.396278] ? hrtimer_interrupt+0x128/0x2c0 [16536.400267] ? smp_apic_timer_interrupt+0xa6/0x270 [16536.404152] ? apic_timer_interrupt+0xf/0x20 [16536.406968] [16536.408206] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16536.419498] do_raw_spin_lock+0x177/0x190 [16536.424090] _raw_spin_lock+0x21/0x30 [16536.435144] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16536.445820] ? do_raw_spin_unlock+0x75/0x190 [16536.456100] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16536.471912] ? do_wait_intr+0xf0/0xf0 [16536.477232] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16536.485016] mdc_close+0x53b/0xd70 [mdc] [16536.495345] lmv_close+0x18d/0x490 [lmv] [16536.499384] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [16536.510748] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16536.522177] ll_md_real_close+0xa5/0x2a0 [lustre] [16536.526112] ll_md_close+0x274/0x800 [lustre] [16536.532747] ll_file_release+0x128/0xc50 [lustre] [16536.534384] __fput+0x10d/0x440 [16536.539612] ____fput+0x12/0x20 [16536.545501] task_work_run+0xc6/0x110 [16536.554111] exit_to_usermode_loop+0x1dd/0x1f0 [16536.569592] do_syscall_64+0x3d6/0x440 [16536.572268] entry_SYSCALL_64_after_hwframe+0x49/0xae [16536.581059] RIP: 0033:0x7f151bee3cd8 [16536.591738] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16536.622460] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16536.632715] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [16536.638095] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [16536.649782] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16536.656879] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16536.662513] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16540.058305] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [16540.074365] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16540.149250] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16540.162675] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16540.171452] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16540.181079] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16540.187654] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16540.210504] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16540.219149] RAX: 00000000000077b1 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16540.228750] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [16540.236611] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [16540.258197] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [16540.270177] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [16540.279896] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [16540.288471] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16540.295515] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16540.304921] Call Trace: [16540.307773] [16540.309078] ? show_regs.cold.9+0x22/0x2f [16540.315549] ? watchdog_timer_fn.cold.6+0x62/0x10a [16540.320282] ? __hrtimer_run_queues+0x1c2/0x4d0 [16540.323924] ? watchdog+0x40/0x40 [16540.326416] ? hrtimer_interrupt+0x128/0x2c0 [16540.328566] ? smp_apic_timer_interrupt+0xa6/0x270 [16540.332430] ? apic_timer_interrupt+0xf/0x20 [16540.339384] [16540.344404] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16540.347560] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16540.353583] do_raw_spin_lock+0x177/0x190 [16540.355578] _raw_spin_lock+0x21/0x30 [16540.357353] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16540.362443] process_one_work+0x2c8/0x700 [16540.365352] worker_thread+0x296/0x6e0 [16540.368828] ? rescuer_thread+0x570/0x570 [16540.374916] kthread+0x1d1/0x200 [16540.381214] ? set_kthread_struct+0x70/0x70 [16540.384386] ret_from_fork+0x1f/0x30 [16552.057513] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [16552.065521] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16552.105737] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16552.115058] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16552.132289] RIP: 0010:native_safe_halt+0x2e/0x50 [16552.136634] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16552.159251] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16552.170136] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [16552.184154] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [16552.192521] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [16552.199976] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [16552.210098] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16552.219969] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [16552.239291] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16552.252951] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16552.273681] Call Trace: [16552.280247] [16552.283658] ? show_regs.cold.9+0x22/0x2f [16552.295355] ? watchdog_timer_fn.cold.6+0x62/0x10a [16552.301198] ? __hrtimer_run_queues+0x1c2/0x4d0 [16552.311973] ? watchdog+0x40/0x40 [16552.314296] ? hrtimer_interrupt+0x128/0x2c0 [16552.330031] ? smp_apic_timer_interrupt+0xa6/0x270 [16552.347194] ? apic_timer_interrupt+0xf/0x20 [16552.350856] [16552.351644] ? native_safe_halt+0x2e/0x50 [16552.360047] kvm_wait+0xcd/0xe0 [16552.361349] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16552.370917] do_raw_spin_lock+0x177/0x190 [16552.375711] _raw_spin_lock+0x21/0x30 [16552.380189] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16552.386811] obd_mod_free+0x46/0x140 [ptlrpc] [16552.392192] mdc_commit_open+0x91/0xb0 [mdc] [16552.393677] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16552.398436] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16552.400655] after_reply+0xe2c/0x1780 [ptlrpc] [16552.406985] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16552.417768] ? __next_timer_interrupt+0x160/0x160 [16552.422758] ptlrpcd+0xa82/0xe40 [ptlrpc] [16552.429691] ? do_wait_intr+0xf0/0xf0 [16552.431047] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16552.440966] kthread+0x1d1/0x200 [16552.450410] ? set_kthread_struct+0x70/0x70 [16552.452006] ret_from_fork+0x1f/0x30 [16556.059442] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [16556.063742] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16556.141795] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16556.157200] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16556.168598] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16556.177127] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16556.204718] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16556.222065] RAX: 0000000000005d54 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16556.236252] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [16556.245978] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [16556.255660] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [16556.264179] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [16556.277757] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [16556.284945] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16556.297119] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [16556.307500] Call Trace: [16556.320902] [16556.321565] ? show_regs.cold.9+0x22/0x2f [16556.330650] ? watchdog_timer_fn.cold.6+0x62/0x10a [16556.339630] ? __hrtimer_run_queues+0x1c2/0x4d0 [16556.343414] ? watchdog+0x40/0x40 [16556.345649] ? hrtimer_interrupt+0x128/0x2c0 [16556.347843] ? smp_apic_timer_interrupt+0xa6/0x270 [16556.350114] ? apic_timer_interrupt+0xf/0x20 [16556.352156] [16556.353258] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16556.358383] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16556.366129] do_raw_spin_lock+0x177/0x190 [16556.369671] _raw_spin_lock+0x21/0x30 [16556.373936] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16556.381686] ? do_raw_spin_unlock+0x75/0x190 [16556.390672] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16556.395083] ? do_wait_intr+0xf0/0xf0 [16556.402563] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16556.411386] mdc_reint+0x2d/0x110 [mdc] [16556.417345] mdc_unlink+0x658/0x870 [mdc] [16556.422412] lmv_unlink+0x34d/0xbd0 [lmv] [16556.430235] ll_unlink+0x24d/0x7f0 [lustre] [16556.438023] vfs_unlink+0x202/0x330 [16556.442177] do_unlinkat+0x2fa/0x460 [16556.444608] __x64_sys_unlink+0x37/0x50 [16556.447613] do_syscall_64+0xc1/0x440 [16556.449571] entry_SYSCALL_64_after_hwframe+0x49/0xae [16556.452299] RIP: 0033:0x7f151bdee10b [16556.454518] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [16556.472136] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [16556.480015] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [16556.487763] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [16556.501164] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16556.507357] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16556.512174] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16558.573391] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14627s! [16558.582267] Showing busy workqueues and worker pools: [16558.591663] workqueue events: flags=0x0 [16558.594112] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16558.602356] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [16558.611229] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16558.618354] pending: kfree_rcu_monitor [16558.628478] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16558.637003] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [16558.643401] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16558.658310] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [16558.668610] workqueue events_long: flags=0x0 [16558.674686] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16558.686050] pending: lu_objects_flush [obdclass] [16558.691702] workqueue events_unbound: flags=0x2 [16558.695546] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [16558.701673] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16558.718924] workqueue events_power_efficient: flags=0x80 [16558.727702] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16558.747083] pending: do_cache_clean, neigh_periodic_work, check_lifetime [16558.757991] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16558.773957] pending: neigh_periodic_work [16558.780242] workqueue mm_percpu_wq: flags=0x8 [16558.786621] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16558.797574] pending: vmstat_update, lru_add_drain_per_cpu [16558.800661] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16558.807278] pending: vmstat_update [16558.811080] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16558.815663] pending: lru_add_drain_per_cpu [16558.818264] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [16558.821721] pending: lru_add_drain_per_cpu BAR(48) [16558.824323] workqueue writeback: flags=0x4a [16558.826377] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [16558.829582] pending: wb_workfn [16558.831196] workqueue xprtiod: flags=0xa [16558.832984] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16558.835749] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [16558.838623] workqueue ipv6_addrconf: flags=0x40008 [16558.840206] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16558.843310] pending: addrconf_verify_work [16558.845155] workqueue knbd0-recv: flags=0x1a [16558.847041] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16558.857043] in-flight: 658:recv_work [16558.863224] workqueue ptlrpc_pinger: flags=0x2 [16558.865167] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16558.869980] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [16558.877771] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14628s workers=4 idle: 32984 37 8 [16558.888585] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16543s workers=2 idle: 661 [16564.057125] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [16564.060201] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16564.079846] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16564.085919] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16564.089749] RIP: 0010:native_safe_halt+0x2e/0x50 [16564.091180] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16564.122943] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16564.135220] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16564.142362] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [16564.148153] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [16564.153823] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [16564.157702] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [16564.162993] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [16564.169771] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16564.176741] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [16564.184220] Call Trace: [16564.184873] [16564.185388] ? show_regs.cold.9+0x22/0x2f [16564.186470] ? watchdog_timer_fn.cold.6+0x62/0x10a [16564.192138] ? __hrtimer_run_queues+0x1c2/0x4d0 [16564.196808] ? watchdog+0x40/0x40 [16564.200665] ? hrtimer_interrupt+0x128/0x2c0 [16564.204993] ? smp_apic_timer_interrupt+0xa6/0x270 [16564.207957] ? apic_timer_interrupt+0xf/0x20 [16564.213174] [16564.216169] ? native_safe_halt+0x2e/0x50 [16564.219754] kvm_wait+0xcd/0xe0 [16564.221914] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16564.225349] do_raw_spin_lock+0x177/0x190 [16564.227956] _raw_spin_lock+0x21/0x30 [16564.230987] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16564.235589] ? do_raw_spin_unlock+0x75/0x190 [16564.242088] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16564.247759] ? do_wait_intr+0xf0/0xf0 [16564.249302] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16564.253253] mdc_close+0x53b/0xd70 [mdc] [16564.256858] lmv_close+0x18d/0x490 [lmv] [16564.260738] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [16564.264563] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16564.269129] ll_md_real_close+0xa5/0x2a0 [lustre] [16564.270554] ll_md_close+0x274/0x800 [lustre] [16564.273320] ll_file_release+0x128/0xc50 [lustre] [16564.276988] __fput+0x10d/0x440 [16564.279289] ____fput+0x12/0x20 [16564.281732] task_work_run+0xc6/0x110 [16564.284698] exit_to_usermode_loop+0x1dd/0x1f0 [16564.290311] do_syscall_64+0x3d6/0x440 [16564.291859] entry_SYSCALL_64_after_hwframe+0x49/0xae [16564.295406] RIP: 0033:0x7f151bee3cd8 [16564.298315] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16564.315927] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16564.325102] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [16564.343095] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [16564.364094] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16564.375374] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16564.392148] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16564.952240] rcu: INFO: rcu_sched self-detected stall on CPU [16564.955022] rcu: 1-...!: (8781715 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [16564.963821] (t=14640305 jiffies g=249657 q=6866) [16564.971503] rcu: rcu_sched kthread starved for 14635257 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [16564.987269] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [16565.006666] rcu: RCU grace-period kthread stack dump: [16565.017081] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [16565.019919] Call Trace: [16565.029400] __schedule+0x351/0xcb0 [16565.030915] schedule+0xc0/0x180 [16565.035753] schedule_timeout+0xb4/0x190 [16565.043288] ? do_raw_spin_unlock+0x75/0x190 [16565.054405] ? __next_timer_interrupt+0x160/0x160 [16565.062894] ? prepare_to_swait_event+0xf3/0x140 [16565.065769] rcu_gp_kthread+0x8dd/0x15b0 [16565.075135] ? force_qs_rnp+0x2c0/0x2c0 [16565.083263] kthread+0x1d1/0x200 [16565.084341] ? set_kthread_struct+0x70/0x70 [16565.092057] ret_from_fork+0x1f/0x30 [16565.093213] rcu: Stack dump where RCU GP kthread last ran: [16565.098335] Sending NMI from CPU 1 to CPUs 0: [16565.106167] NMI backtrace for cpu 0 [16565.106176] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16565.106179] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16565.106180] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30a/0x6d0 [16565.106182] Code: 03 00 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 [16565.106184] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 [16565.106186] RAX: 0000000000004747 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16565.106188] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [16565.106189] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [16565.106190] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [16565.106192] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [16565.106193] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [16565.106195] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16565.106196] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [16565.106197] Call Trace: [16565.106198] [16565.106199] ? show_regs.cold.9+0x22/0x2f [16565.106200] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [16565.106201] ? nmi_cpu_backtrace_handler+0x15/0x40 [16565.106202] ? nmi_handle+0xbd/0x210 [16565.106203] ? default_do_nmi+0x4b/0x1b0 [16565.106204] ? do_nmi+0x280/0x380 [16565.106205] ? end_repeat_nmi+0x16/0x1f [16565.106206] ? __pv_queued_spin_lock_slowpath+0x30a/0x6d0 [16565.106208] ? __pv_queued_spin_lock_slowpath+0x30a/0x6d0 [16565.106209] ? __pv_queued_spin_lock_slowpath+0x30a/0x6d0 [16565.106210] [16565.106211] do_raw_spin_lock+0x177/0x190 [16565.106212] _raw_spin_lock+0x21/0x30 [16565.106213] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16565.106214] ? do_raw_spin_unlock+0x75/0x190 [16565.106215] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16565.106216] ? do_wait_intr+0xf0/0xf0 [16565.106217] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16565.106219] mdc_reint+0x2d/0x110 [mdc] [16565.106220] mdc_unlink+0x658/0x870 [mdc] [16565.106221] lmv_unlink+0x34d/0xbd0 [lmv] [16565.106222] ll_unlink+0x24d/0x7f0 [lustre] [16565.106233] vfs_unlink+0x202/0x330 [16565.106235] do_unlinkat+0x2fa/0x460 [16565.106236] __x64_sys_unlink+0x37/0x50 [16565.106237] do_syscall_64+0xc1/0x440 [16565.106238] entry_SYSCALL_64_after_hwframe+0x49/0xae [16565.106239] RIP: 0033:0x7f151bdee10b [16565.106241] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [16565.106247] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [16565.106250] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [16565.106251] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [16565.106253] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16565.106254] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16565.106256] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16565.108533] NMI backtrace for cpu 1 [16565.406460] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16565.410851] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16565.414020] Call Trace: [16565.414978] [16565.415757] ? dump_stack+0xbb/0x10e [16565.417093] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [16565.418895] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [16565.420441] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [16565.422054] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [16565.423878] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [16565.425801] ? rcu_dump_cpu_stacks+0x11b/0x171 [16565.427691] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [16565.429220] ? update_process_times+0x6d/0xc0 [16565.431718] ? tick_sched_handle.isra.18+0x27/0x90 [16565.433578] ? tick_sched_timer+0x53/0xb0 [16565.434891] ? __hrtimer_run_queues+0x1c2/0x4d0 [16565.436252] ? tick_sched_handle.isra.18+0x90/0x90 [16565.437839] ? hrtimer_interrupt+0x128/0x2c0 [16565.439300] ? smp_apic_timer_interrupt+0xa6/0x270 [16565.440984] ? apic_timer_interrupt+0xf/0x20 [16565.442405] [16565.443194] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16565.445169] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16565.448154] ? do_raw_spin_lock+0x177/0x190 [16565.449611] ? _raw_spin_lock+0x21/0x30 [16565.451172] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16565.453699] ? obd_mod_free+0x46/0x140 [ptlrpc] [16565.455848] ? mdc_commit_open+0x91/0xb0 [mdc] [16565.457758] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16565.460264] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16565.463189] ? after_reply+0xe2c/0x1780 [ptlrpc] [16565.465049] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16565.467284] ? __next_timer_interrupt+0x160/0x160 [16565.469083] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [16565.470917] ? do_wait_intr+0xf0/0xf0 [16565.472334] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16565.474349] ? kthread+0x1d1/0x200 [16565.476285] ? set_kthread_struct+0x70/0x70 [16565.477924] ? ret_from_fork+0x1f/0x30 [16568.057612] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [16568.062320] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16568.096173] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16568.103957] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16568.108877] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16568.112916] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16568.116871] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16568.129800] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16568.134649] RAX: 0000000000003b83 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16568.138542] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [16568.143122] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [16568.147446] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [16568.151121] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [16568.155498] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [16568.160839] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16568.168640] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16568.175291] Call Trace: [16568.178814] [16568.183916] ? show_regs.cold.9+0x22/0x2f [16568.187963] ? watchdog_timer_fn.cold.6+0x62/0x10a [16568.192317] ? __hrtimer_run_queues+0x1c2/0x4d0 [16568.198269] ? watchdog+0x40/0x40 [16568.203851] ? hrtimer_interrupt+0x128/0x2c0 [16568.209184] ? smp_apic_timer_interrupt+0xa6/0x270 [16568.212986] ? apic_timer_interrupt+0xf/0x20 [16568.215206] [16568.216096] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16568.221017] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16568.227390] do_raw_spin_lock+0x177/0x190 [16568.228614] _raw_spin_lock+0x21/0x30 [16568.238645] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16568.252915] process_one_work+0x2c8/0x700 [16568.254940] worker_thread+0x296/0x6e0 [16568.261143] ? rescuer_thread+0x570/0x570 [16568.264731] kthread+0x1d1/0x200 [16568.266651] ? set_kthread_struct+0x70/0x70 [16568.268877] ret_from_fork+0x1f/0x30 [16589.286110] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14658s! [16589.290350] Showing busy workqueues and worker pools: [16589.293300] workqueue events: flags=0x0 [16589.295201] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16589.298014] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [16589.301889] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16589.304706] pending: kfree_rcu_monitor [16589.308325] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16589.311463] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [16589.316960] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16589.319990] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [16589.323050] workqueue events_long: flags=0x0 [16589.325020] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16589.328000] pending: lu_objects_flush [obdclass] [16589.330286] workqueue events_unbound: flags=0x2 [16589.331951] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [16589.334812] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16589.339298] workqueue events_power_efficient: flags=0x80 [16589.341558] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16589.344241] pending: do_cache_clean, neigh_periodic_work, check_lifetime [16589.347561] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16589.352297] pending: neigh_periodic_work [16589.354440] workqueue mm_percpu_wq: flags=0x8 [16589.356963] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16589.361349] pending: vmstat_update, lru_add_drain_per_cpu [16589.363711] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16589.367281] pending: vmstat_update [16589.368948] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16589.372242] pending: lru_add_drain_per_cpu [16589.374529] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [16589.377396] pending: lru_add_drain_per_cpu BAR(48) [16589.379409] workqueue writeback: flags=0x4a [16589.381060] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [16589.384126] pending: wb_workfn [16589.385801] workqueue xprtiod: flags=0xa [16589.387788] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16589.390654] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [16589.394112] workqueue ipv6_addrconf: flags=0x40008 [16589.396295] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16589.399313] pending: addrconf_verify_work [16589.401042] workqueue knbd0-recv: flags=0x1a [16589.402912] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16589.405708] in-flight: 658:recv_work [16589.407661] workqueue ptlrpc_pinger: flags=0x2 [16589.409622] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16589.412014] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [16589.415641] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14658s workers=4 idle: 32984 37 8 [16589.419466] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16574s workers=2 idle: 661 [16592.057446] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [16592.057662] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [16592.063300] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16592.063380] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16592.063388] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16592.063391] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16592.063407] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16592.063411] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16592.063416] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [16592.063419] RDX: 0000000000007b35 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [16592.063421] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [16592.063424] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [16592.063426] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16592.063429] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [16592.063434] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16592.063436] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16592.063445] Call Trace: [16592.063451] [16592.067555] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16592.067634] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16592.067641] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16592.067644] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16592.067659] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16592.067663] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [16592.067668] RAX: 000000000000215b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16592.067670] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [16592.067672] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [16592.067675] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [16592.067678] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [16592.067681] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [16592.067685] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16592.067687] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [16592.067696] Call Trace: [16592.067702] [16592.067709] ? show_regs.cold.9+0x22/0x2f [16592.067719] ? watchdog_timer_fn.cold.6+0x62/0x10a [16592.067725] ? __hrtimer_run_queues+0x1c2/0x4d0 [16592.067730] ? watchdog+0x40/0x40 [16592.067734] ? hrtimer_interrupt+0x128/0x2c0 [16592.067739] ? smp_apic_timer_interrupt+0xa6/0x270 [16592.067749] ? apic_timer_interrupt+0xf/0x20 [16592.067754] [16592.067757] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16592.067764] do_raw_spin_lock+0x177/0x190 [16592.067769] _raw_spin_lock+0x21/0x30 [16592.067774] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16592.124104] ? show_regs.cold.9+0x22/0x2f [16592.124124] ? watchdog_timer_fn.cold.6+0x62/0x10a [16592.124131] ? __hrtimer_run_queues+0x1c2/0x4d0 [16592.124135] ? watchdog+0x40/0x40 [16592.124139] ? hrtimer_interrupt+0x128/0x2c0 [16592.124143] ? smp_apic_timer_interrupt+0xa6/0x270 [16592.124150] ? apic_timer_interrupt+0xf/0x20 [16592.124154] [16592.124159] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16592.124167] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16592.124172] do_raw_spin_lock+0x177/0x190 [16592.124178] _raw_spin_lock+0x21/0x30 [16592.124186] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16592.131263] ? do_raw_spin_unlock+0x75/0x190 [16592.131285] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16592.131842] ? do_wait_intr+0xf0/0xf0 [16592.131850] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16592.132086] mdc_close+0x53b/0xd70 [mdc] [16592.132122] lmv_close+0x18d/0x490 [lmv] [16592.132155] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [16592.132344] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16592.132456] ll_md_real_close+0xa5/0x2a0 [lustre] [16592.132540] ll_md_close+0x274/0x800 [lustre] [16592.132627] ll_file_release+0x128/0xc50 [lustre] [16592.132717] __fput+0x10d/0x440 [16592.132727] ____fput+0x12/0x20 [16592.132731] task_work_run+0xc6/0x110 [16592.132738] exit_to_usermode_loop+0x1dd/0x1f0 [16592.132745] do_syscall_64+0x3d6/0x440 [16592.132749] entry_SYSCALL_64_after_hwframe+0x49/0xae [16592.132758] RIP: 0033:0x7f151bee3cd8 [16592.132765] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16592.132769] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16592.132774] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [16592.132776] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [16592.132778] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16592.132781] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16592.132783] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16592.430578] obd_mod_free+0x46/0x140 [ptlrpc] [16592.432012] mdc_commit_open+0x91/0xb0 [mdc] [16592.433293] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16592.435278] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16592.436973] after_reply+0xe2c/0x1780 [ptlrpc] [16592.438698] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16592.440600] ? __next_timer_interrupt+0x160/0x160 [16592.441860] ptlrpcd+0xa82/0xe40 [ptlrpc] [16592.443306] ? do_wait_intr+0xf0/0xf0 [16592.444442] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16592.446036] kthread+0x1d1/0x200 [16592.447802] ? set_kthread_struct+0x70/0x70 [16592.449601] ret_from_fork+0x1f/0x30 [16596.056116] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [16596.058767] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16596.079370] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16596.085959] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16596.090237] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16596.094026] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16596.096627] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16596.105768] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16596.109243] RAX: 0000000000004b12 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16596.112993] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [16596.116769] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [16596.120703] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [16596.123956] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [16596.127803] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [16596.131484] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16596.134203] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16596.137624] Call Trace: [16596.138931] [16596.139814] ? show_regs.cold.9+0x22/0x2f [16596.141858] ? watchdog_timer_fn.cold.6+0x62/0x10a [16596.144267] ? __hrtimer_run_queues+0x1c2/0x4d0 [16596.146612] ? watchdog+0x40/0x40 [16596.151112] ? hrtimer_interrupt+0x128/0x2c0 [16596.153648] ? smp_apic_timer_interrupt+0xa6/0x270 [16596.158308] ? apic_timer_interrupt+0xf/0x20 [16596.161041] [16596.169558] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16596.173681] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16596.175955] do_raw_spin_lock+0x177/0x190 [16596.177957] _raw_spin_lock+0x21/0x30 [16596.185710] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16596.190548] process_one_work+0x2c8/0x700 [16596.195719] worker_thread+0x296/0x6e0 [16596.197819] ? rescuer_thread+0x570/0x570 [16596.206139] kthread+0x1d1/0x200 [16596.207665] ? set_kthread_struct+0x70/0x70 [16596.215128] ret_from_fork+0x1f/0x30 [16616.052021] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [16616.058518] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16616.093593] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16616.103423] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16616.112048] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16616.120524] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16616.141304] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16616.145580] RAX: 00000000000067c3 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16616.151667] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [16616.159537] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [16616.169413] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [16616.177620] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [16616.183051] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [16616.193533] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16616.196763] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [16616.202600] Call Trace: [16616.204043] [16616.205811] ? show_regs.cold.9+0x22/0x2f [16616.208452] ? watchdog_timer_fn.cold.6+0x62/0x10a [16616.209720] ? __hrtimer_run_queues+0x1c2/0x4d0 [16616.214304] ? watchdog+0x40/0x40 [16616.219966] ? hrtimer_interrupt+0x128/0x2c0 [16616.222395] ? smp_apic_timer_interrupt+0xa6/0x270 [16616.224682] ? apic_timer_interrupt+0xf/0x20 [16616.228445] [16616.229329] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16616.231765] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16616.234312] do_raw_spin_lock+0x177/0x190 [16616.236387] _raw_spin_lock+0x21/0x30 [16616.238062] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16616.240938] ? do_raw_spin_unlock+0x75/0x190 [16616.242827] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16616.245555] ? do_wait_intr+0xf0/0xf0 [16616.247161] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16616.249630] mdc_reint+0x2d/0x110 [mdc] [16616.251401] mdc_unlink+0x658/0x870 [mdc] [16616.253168] lmv_unlink+0x34d/0xbd0 [lmv] [16616.255034] ll_unlink+0x24d/0x7f0 [lustre] [16616.257058] vfs_unlink+0x202/0x330 [16616.258519] do_unlinkat+0x2fa/0x460 [16616.260409] __x64_sys_unlink+0x37/0x50 [16616.262367] do_syscall_64+0xc1/0x440 [16616.263841] entry_SYSCALL_64_after_hwframe+0x49/0xae [16616.266126] RIP: 0033:0x7f151bdee10b [16616.267665] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [16616.276004] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [16616.279202] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [16616.282355] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [16616.285448] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16616.288181] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16616.291169] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16619.488932] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14688s! [16619.500314] Showing busy workqueues and worker pools: [16619.506099] workqueue events: flags=0x0 [16619.508600] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16619.512666] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [16619.516601] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16619.520737] pending: kfree_rcu_monitor [16619.523398] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16619.530116] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [16619.538785] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16619.545321] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [16619.554658] workqueue events_long: flags=0x0 [16619.565123] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16619.570295] pending: lu_objects_flush [obdclass] [16619.572271] workqueue events_unbound: flags=0x2 [16619.575505] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [16619.581935] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16619.586818] workqueue events_power_efficient: flags=0x80 [16619.589731] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16619.592934] pending: do_cache_clean, neigh_periodic_work, check_lifetime [16619.596480] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16619.599949] pending: neigh_periodic_work [16619.602068] workqueue mm_percpu_wq: flags=0x8 [16619.604296] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16619.607500] pending: vmstat_update, lru_add_drain_per_cpu [16619.609980] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16619.613458] pending: vmstat_update [16619.615618] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16619.619099] pending: lru_add_drain_per_cpu [16619.621460] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [16619.624863] pending: lru_add_drain_per_cpu BAR(48) [16619.627814] workqueue writeback: flags=0x4a [16619.629663] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [16619.633272] pending: wb_workfn [16619.635031] workqueue xprtiod: flags=0xa [16619.637519] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16619.640635] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [16619.644660] workqueue ipv6_addrconf: flags=0x40008 [16619.647337] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16619.650548] pending: addrconf_verify_work [16619.652789] workqueue knbd0-recv: flags=0x1a [16619.656193] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16619.663143] in-flight: 658:recv_work [16619.665619] workqueue ptlrpc_pinger: flags=0x2 [16619.670753] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16619.675286] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [16619.679726] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14689s workers=4 idle: 32984 37 8 [16619.684348] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16604s workers=2 idle: 661 [16620.055066] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [16620.057023] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [16620.059969] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) [16620.062560] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16620.062638] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16620.062645] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16620.062648] RIP: 0010:native_safe_halt+0x2e/0x50 [16620.062662] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16620.062666] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16620.062671] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16620.062673] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [16620.062675] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [16620.062677] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [16620.062679] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [16620.062681] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [16620.062684] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16620.062685] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [16620.062692] Call Trace: [16620.062697] [16620.062713] ? show_regs.cold.9+0x22/0x2f [16620.062723] ? watchdog_timer_fn.cold.6+0x62/0x10a [16620.062729] ? __hrtimer_run_queues+0x1c2/0x4d0 [16620.062731] ? watchdog+0x40/0x40 [16620.062734] ? hrtimer_interrupt+0x128/0x2c0 [16620.062737] ? smp_apic_timer_interrupt+0xa6/0x270 [16620.062742] ? apic_timer_interrupt+0xf/0x20 [16620.062745] [16620.062747] ? native_safe_halt+0x2e/0x50 [16620.062749] kvm_wait+0xcd/0xe0 [16620.062755] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16620.062761] do_raw_spin_lock+0x177/0x190 [16620.062764] _raw_spin_lock+0x21/0x30 [16620.062767] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16620.063434] ? do_raw_spin_unlock+0x75/0x190 [16620.063441] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16620.063732] ? do_wait_intr+0xf0/0xf0 [16620.063740] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16620.071670] mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16620.071733] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16620.071741] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16620.071744] RIP: 0010:native_safe_halt+0x2e/0x50 [16620.071757] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16620.071762] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16620.071767] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [16620.071770] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [16620.071772] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [16620.071774] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [16620.071777] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16620.071780] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [16620.071783] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16620.071785] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16620.071794] Call Trace: [16620.071800] [16620.071807] ? show_regs.cold.9+0x22/0x2f [16620.071818] ? watchdog_timer_fn.cold.6+0x62/0x10a [16620.071838] ? __hrtimer_run_queues+0x1c2/0x4d0 [16620.071842] ? watchdog+0x40/0x40 [16620.071846] ? hrtimer_interrupt+0x128/0x2c0 [16620.071850] ? smp_apic_timer_interrupt+0xa6/0x270 [16620.071857] ? apic_timer_interrupt+0xf/0x20 [16620.071861] [16620.071865] ? native_safe_halt+0x2e/0x50 [16620.071868] kvm_wait+0xcd/0xe0 [16620.071876] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16620.071883] do_raw_spin_lock+0x177/0x190 [16620.071887] _raw_spin_lock+0x21/0x30 [16620.071893] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16620.072184] obd_mod_free+0x46/0x140 [ptlrpc] [16620.072382] mdc_commit_open+0x91/0xb0 [mdc] [16620.072424] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16620.072619] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16620.072813] after_reply+0xe2c/0x1780 [ptlrpc] [16620.073029] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16620.073223] ? __next_timer_interrupt+0x160/0x160 [16620.073233] ptlrpcd+0xa82/0xe40 [ptlrpc] [16620.073435] ? do_wait_intr+0xf0/0xf0 [16620.073445] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16620.073640] kthread+0x1d1/0x200 [16620.073647] ? set_kthread_struct+0x70/0x70 [16620.073650] ret_from_fork+0x1f/0x30 [16620.426106] mdc_close+0x53b/0xd70 [mdc] [16620.427587] lmv_close+0x18d/0x490 [lmv] [16620.428812] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [16620.432717] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16620.435566] ll_md_real_close+0xa5/0x2a0 [lustre] [16620.437697] ll_md_close+0x274/0x800 [lustre] [16620.439568] ll_file_release+0x128/0xc50 [lustre] [16620.444093] __fput+0x10d/0x440 [16620.445511] ____fput+0x12/0x20 [16620.446920] task_work_run+0xc6/0x110 [16620.449173] exit_to_usermode_loop+0x1dd/0x1f0 [16620.451574] do_syscall_64+0x3d6/0x440 [16620.453239] entry_SYSCALL_64_after_hwframe+0x49/0xae [16620.455482] RIP: 0033:0x7f151bee3cd8 [16620.459138] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16620.468585] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16620.473489] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [16620.485990] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [16620.495821] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16620.500994] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16620.504423] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16624.056526] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [16624.061435] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16624.090914] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16624.099159] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16624.103642] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16624.112206] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16624.117273] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16624.128614] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16624.132679] RAX: 000000000000488b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16624.136266] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [16624.141053] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [16624.146128] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [16624.150093] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [16624.153895] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [16624.157970] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16624.161504] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16624.165693] Call Trace: [16624.166962] [16624.170233] ? show_regs.cold.9+0x22/0x2f [16624.181704] ? watchdog_timer_fn.cold.6+0x62/0x10a [16624.192955] ? __hrtimer_run_queues+0x1c2/0x4d0 [16624.194483] ? watchdog+0x40/0x40 [16624.195638] ? hrtimer_interrupt+0x128/0x2c0 [16624.208768] ? smp_apic_timer_interrupt+0xa6/0x270 [16624.210289] ? apic_timer_interrupt+0xf/0x20 [16624.225160] [16624.229448] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16624.235712] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16624.240587] do_raw_spin_lock+0x177/0x190 [16624.246236] _raw_spin_lock+0x21/0x30 [16624.249557] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16624.255639] process_one_work+0x2c8/0x700 [16624.262611] worker_thread+0x296/0x6e0 [16624.273271] ? rescuer_thread+0x570/0x570 [16624.278158] kthread+0x1d1/0x200 [16624.285368] ? set_kthread_struct+0x70/0x70 [16624.286331] ret_from_fork+0x1f/0x30 [16644.052139] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [16644.056709] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16644.080569] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16644.085881] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16644.090451] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16644.093392] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16644.101438] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16644.105005] RAX: 000000000000672c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16644.108537] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [16644.111752] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [16644.115687] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [16644.119376] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [16644.126905] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [16644.131773] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16644.134417] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [16644.137842] Call Trace: [16644.138962] [16644.139897] ? show_regs.cold.9+0x22/0x2f [16644.141650] ? watchdog_timer_fn.cold.6+0x62/0x10a [16644.143766] ? __hrtimer_run_queues+0x1c2/0x4d0 [16644.145630] ? watchdog+0x40/0x40 [16644.147003] ? hrtimer_interrupt+0x128/0x2c0 [16644.148869] ? smp_apic_timer_interrupt+0xa6/0x270 [16644.150774] ? apic_timer_interrupt+0xf/0x20 [16644.152616] [16644.156947] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16644.159887] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16644.165617] do_raw_spin_lock+0x177/0x190 [16644.167409] _raw_spin_lock+0x21/0x30 [16644.169090] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16644.172169] ? do_raw_spin_unlock+0x75/0x190 [16644.174118] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16644.176847] ? do_wait_intr+0xf0/0xf0 [16644.178464] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16644.180808] mdc_reint+0x2d/0x110 [mdc] [16644.182695] mdc_unlink+0x658/0x870 [mdc] [16644.184373] lmv_unlink+0x34d/0xbd0 [lmv] [16644.186421] ll_unlink+0x24d/0x7f0 [lustre] [16644.188564] vfs_unlink+0x202/0x330 [16644.190032] do_unlinkat+0x2fa/0x460 [16644.191516] __x64_sys_unlink+0x37/0x50 [16644.193307] do_syscall_64+0xc1/0x440 [16644.195133] entry_SYSCALL_64_after_hwframe+0x49/0xae [16644.197398] RIP: 0033:0x7f151bdee10b [16644.199176] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [16644.207277] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [16644.210878] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [16644.214128] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [16644.217669] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16644.220851] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16644.223878] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16648.055044] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [16648.058667] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16648.060308] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [16648.076817] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16648.076831] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16648.076834] RIP: 0010:native_safe_halt+0x2e/0x50 [16648.076847] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16648.076851] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16648.076857] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [16648.076859] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [16648.076861] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [16648.076864] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [16648.076866] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16648.076869] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [16648.076872] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16648.076874] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16648.079533] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16648.079676] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16648.079693] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16648.079696] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16648.079709] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16648.079713] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16648.079717] RAX: 0000000000007408 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16648.079719] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [16648.079722] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [16648.079724] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [16648.079726] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [16648.079729] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [16648.079732] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16648.079734] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [16648.079741] Call Trace: [16648.079746] [16648.079751] ? show_regs.cold.9+0x22/0x2f [16648.079759] ? watchdog_timer_fn.cold.6+0x62/0x10a [16648.079765] ? __hrtimer_run_queues+0x1c2/0x4d0 [16648.079769] ? watchdog+0x40/0x40 [16648.079773] ? hrtimer_interrupt+0x128/0x2c0 [16648.079777] ? smp_apic_timer_interrupt+0xa6/0x270 [16648.079784] ? apic_timer_interrupt+0xf/0x20 [16648.079788] [16648.079790] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16648.079797] do_raw_spin_lock+0x177/0x190 [16648.079801] _raw_spin_lock+0x21/0x30 [16648.079805] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16648.080081] ? do_raw_spin_unlock+0x75/0x190 [16648.080087] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16648.080279] ? do_wait_intr+0xf0/0xf0 [16648.080286] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16648.080479] mdc_close+0x53b/0xd70 [mdc] [16648.080517] lmv_close+0x18d/0x490 [lmv] [16648.080546] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [16648.080952] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16648.081112] ll_md_real_close+0xa5/0x2a0 [lustre] [16648.081193] ll_md_close+0x274/0x800 [lustre] [16648.081326] ll_file_release+0x128/0xc50 [lustre] [16648.081406] __fput+0x10d/0x440 [16648.081415] ____fput+0x12/0x20 [16648.081419] task_work_run+0xc6/0x110 [16648.081426] exit_to_usermode_loop+0x1dd/0x1f0 [16648.081433] do_syscall_64+0x3d6/0x440 [16648.081438] entry_SYSCALL_64_after_hwframe+0x49/0xae [16648.081445] RIP: 0033:0x7f151bee3cd8 [16648.081452] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16648.081456] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16648.081460] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [16648.081462] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [16648.081465] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16648.081467] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16648.081469] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16648.476408] Call Trace: [16648.479985] [16648.481033] ? show_regs.cold.9+0x22/0x2f [16648.482543] ? watchdog_timer_fn.cold.6+0x62/0x10a [16648.487045] ? __hrtimer_run_queues+0x1c2/0x4d0 [16648.489444] ? watchdog+0x40/0x40 [16648.492497] ? hrtimer_interrupt+0x128/0x2c0 [16648.496524] ? smp_apic_timer_interrupt+0xa6/0x270 [16648.500839] ? apic_timer_interrupt+0xf/0x20 [16648.502133] [16648.502627] ? native_safe_halt+0x2e/0x50 [16648.506557] kvm_wait+0xcd/0xe0 [16648.510022] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16648.514019] do_raw_spin_lock+0x177/0x190 [16648.515905] _raw_spin_lock+0x21/0x30 [16648.517494] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16648.523123] obd_mod_free+0x46/0x140 [ptlrpc] [16648.536897] mdc_commit_open+0x91/0xb0 [mdc] [16648.542690] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16648.545063] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16648.549252] after_reply+0xe2c/0x1780 [ptlrpc] [16648.553762] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16648.564002] ? __next_timer_interrupt+0x160/0x160 [16648.570115] ptlrpcd+0xa82/0xe40 [ptlrpc] [16648.572373] ? do_wait_intr+0xf0/0xf0 [16648.573952] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16648.581471] kthread+0x1d1/0x200 [16648.582447] ? set_kthread_struct+0x70/0x70 [16648.583671] ret_from_fork+0x1f/0x30 [16649.696887] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14719s! [16649.715516] Showing busy workqueues and worker pools: [16649.717189] workqueue events: flags=0x0 [16649.718499] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16649.736027] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [16649.750356] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16649.752545] pending: kfree_rcu_monitor [16649.761334] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16649.768818] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [16649.782152] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16649.791091] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [16649.798439] workqueue events_long: flags=0x0 [16649.800335] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16649.816059] pending: lu_objects_flush [obdclass] [16649.827824] workqueue events_unbound: flags=0x2 [16649.834865] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [16649.836936] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16649.861374] workqueue events_power_efficient: flags=0x80 [16649.870509] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16649.885047] pending: do_cache_clean, neigh_periodic_work, check_lifetime [16649.899956] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16649.910423] pending: neigh_periodic_work [16649.920088] workqueue mm_percpu_wq: flags=0x8 [16649.924933] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16649.929411] pending: vmstat_update, lru_add_drain_per_cpu [16649.932506] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16649.937340] pending: vmstat_update [16649.946274] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16649.953872] pending: lru_add_drain_per_cpu [16649.959735] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [16649.972328] pending: lru_add_drain_per_cpu BAR(48) [16649.985811] workqueue writeback: flags=0x4a [16649.987135] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [16649.996676] pending: wb_workfn [16649.997710] workqueue xprtiod: flags=0xa [16650.002808] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16650.008415] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [16650.014822] workqueue ipv6_addrconf: flags=0x40008 [16650.018386] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16650.026377] pending: addrconf_verify_work [16650.030875] workqueue knbd0-recv: flags=0x1a [16650.037300] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16650.039411] in-flight: 658:recv_work [16650.049798] workqueue ptlrpc_pinger: flags=0x2 [16650.055367] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16650.061447] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [16650.069176] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14719s workers=4 idle: 32984 37 8 [16650.077252] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16634s workers=2 idle: 661 [16652.056018] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [16652.058700] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16652.099101] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16652.105964] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16652.113262] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16652.120315] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16652.127209] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16652.142708] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16652.146481] RAX: 0000000000005794 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16652.153627] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [16652.163449] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [16652.175022] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [16652.181490] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [16652.188611] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [16652.191203] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16652.194335] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16652.197317] Call Trace: [16652.198258] [16652.199112] ? show_regs.cold.9+0x22/0x2f [16652.200718] ? watchdog_timer_fn.cold.6+0x62/0x10a [16652.202567] ? __hrtimer_run_queues+0x1c2/0x4d0 [16652.204441] ? watchdog+0x40/0x40 [16652.208748] ? hrtimer_interrupt+0x128/0x2c0 [16652.210960] ? smp_apic_timer_interrupt+0xa6/0x270 [16652.216476] ? apic_timer_interrupt+0xf/0x20 [16652.227063] [16652.230319] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16652.235271] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16652.243041] do_raw_spin_lock+0x177/0x190 [16652.248287] _raw_spin_lock+0x21/0x30 [16652.251490] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16652.257206] process_one_work+0x2c8/0x700 [16652.261110] worker_thread+0x296/0x6e0 [16652.265031] ? rescuer_thread+0x570/0x570 [16652.266490] kthread+0x1d1/0x200 [16652.267573] ? set_kthread_struct+0x70/0x70 [16652.269457] ret_from_fork+0x1f/0x30 [16676.052295] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [16676.055023] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [16676.057421] Modules linked in: lustre(O) [16676.060088] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16676.060176] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16676.060184] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16676.060187] RIP: 0010:native_safe_halt+0x2e/0x50 [16676.060201] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16676.060205] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16676.060210] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [16676.060213] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [16676.060215] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [16676.060217] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [16676.060220] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16676.060226] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [16676.060229] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16676.060231] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16676.060238] Call Trace: [16676.060248] [16676.060254] ? show_regs.cold.9+0x22/0x2f [16676.060264] ? watchdog_timer_fn.cold.6+0x62/0x10a [16676.060274] ? __hrtimer_run_queues+0x1c2/0x4d0 [16676.060278] ? watchdog+0x40/0x40 [16676.060283] ? hrtimer_interrupt+0x128/0x2c0 [16676.060289] ? smp_apic_timer_interrupt+0xa6/0x270 [16676.060296] ? apic_timer_interrupt+0xf/0x20 [16676.060301] [16676.060304] ? native_safe_halt+0x2e/0x50 [16676.060308] kvm_wait+0xcd/0xe0 [16676.060315] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16676.060326] do_raw_spin_lock+0x177/0x190 [16676.060331] _raw_spin_lock+0x21/0x30 [16676.060336] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16676.060736] obd_mod_free+0x46/0x140 [ptlrpc] [16676.060955] mdc_commit_open+0x91/0xb0 [mdc] [16676.060995] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16676.061229] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16676.061455] after_reply+0xe2c/0x1780 [ptlrpc] [16676.061797] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16676.062002] ? __next_timer_interrupt+0x160/0x160 [16676.062010] ptlrpcd+0xa82/0xe40 [ptlrpc] [16676.062210] ? do_wait_intr+0xf0/0xf0 [16676.062218] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16676.062425] kthread+0x1d1/0x200 [16676.062434] ? set_kthread_struct+0x70/0x70 [16676.062438] ret_from_fork+0x1f/0x30 [16676.062616] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [16676.062641] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16676.062729] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16676.062734] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16676.062736] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16676.062860] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16676.062864] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16676.062869] RAX: 0000000000006e89 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16676.062876] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [16676.062947] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [16676.062950] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [16676.062956] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [16676.062960] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [16676.062962] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16676.062964] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [16676.062971] Call Trace: [16676.062975] [16676.062982] ? show_regs.cold.9+0x22/0x2f [16676.062993] ? watchdog_timer_fn.cold.6+0x62/0x10a [16676.062999] ? __hrtimer_run_queues+0x1c2/0x4d0 [16676.063003] ? watchdog+0x40/0x40 [16676.063009] ? hrtimer_interrupt+0x128/0x2c0 [16676.063014] ? smp_apic_timer_interrupt+0xa6/0x270 [16676.063023] ? apic_timer_interrupt+0xf/0x20 [16676.063032] [16676.063035] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16676.063041] do_raw_spin_lock+0x177/0x190 [16676.063050] _raw_spin_lock+0x21/0x30 [16676.063055] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16676.063442] ? do_raw_spin_unlock+0x75/0x190 [16676.063455] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16676.067299] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16676.068314] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16676.068323] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16676.068463] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16676.068480] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16676.068484] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16676.068489] RAX: 0000000000004408 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16676.068502] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [16676.068505] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [16676.068507] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [16676.068510] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [16676.068513] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [16676.068516] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16676.068517] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [16676.068525] Call Trace: [16676.068531] [16676.068537] ? show_regs.cold.9+0x22/0x2f [16676.068547] ? watchdog_timer_fn.cold.6+0x62/0x10a [16676.068553] ? __hrtimer_run_queues+0x1c2/0x4d0 [16676.068557] ? watchdog+0x40/0x40 [16676.068562] ? hrtimer_interrupt+0x128/0x2c0 [16676.068565] ? smp_apic_timer_interrupt+0xa6/0x270 [16676.068573] ? apic_timer_interrupt+0xf/0x20 [16676.068579] [16676.068581] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16676.068585] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16676.068589] do_raw_spin_lock+0x177/0x190 [16676.068594] _raw_spin_lock+0x21/0x30 [16676.068599] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16676.098832] ? do_wait_intr+0xf0/0xf0 [16676.098852] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16676.099129] mdc_close+0x53b/0xd70 [mdc] [16676.099168] lmv_close+0x18d/0x490 [lmv] [16676.099197] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [16676.099368] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16676.112257] ? do_raw_spin_unlock+0x75/0x190 [16676.112280] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16676.112599] ? do_wait_intr+0xf0/0xf0 [16676.112606] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16676.112804] mdc_reint+0x2d/0x110 [mdc] [16676.112841] mdc_unlink+0x658/0x870 [mdc] [16676.112869] lmv_unlink+0x34d/0xbd0 [lmv] [16676.112902] ll_unlink+0x24d/0x7f0 [lustre] [16676.113016] vfs_unlink+0x202/0x330 [16676.113027] do_unlinkat+0x2fa/0x460 [16676.113031] __x64_sys_unlink+0x37/0x50 [16676.113035] do_syscall_64+0xc1/0x440 [16676.113042] entry_SYSCALL_64_after_hwframe+0x49/0xae [16676.113052] RIP: 0033:0x7f151bdee10b [16676.113061] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [16676.113065] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [16676.113070] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [16676.113072] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [16676.113074] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16676.113076] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16676.113079] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16677.018637] ll_md_real_close+0xa5/0x2a0 [lustre] [16677.024768] ll_md_close+0x274/0x800 [lustre] [16677.030091] ll_file_release+0x128/0xc50 [lustre] [16677.035205] __fput+0x10d/0x440 [16677.043676] ____fput+0x12/0x20 [16677.044964] task_work_run+0xc6/0x110 [16677.052779] exit_to_usermode_loop+0x1dd/0x1f0 [16677.058244] do_syscall_64+0x3d6/0x440 [16677.061918] entry_SYSCALL_64_after_hwframe+0x49/0xae [16677.065130] RIP: 0033:0x7f151bee3cd8 [16677.067335] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16677.078698] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16677.081878] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [16677.084804] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [16677.087733] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16677.090534] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16677.093320] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16680.056024] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [16680.059527] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16680.084900] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16680.090818] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16680.095753] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16680.099212] RIP: 0010:native_safe_halt+0x2e/0x50 [16680.101710] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16680.110980] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16680.114952] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16680.118901] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [16680.122250] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [16680.126254] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [16680.130600] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [16680.134019] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [16680.138069] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16680.140780] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16680.144197] Call Trace: [16680.145703] [16680.147451] ? show_regs.cold.9+0x22/0x2f [16680.149517] ? watchdog_timer_fn.cold.6+0x62/0x10a [16680.151954] ? __hrtimer_run_queues+0x1c2/0x4d0 [16680.154411] ? watchdog+0x40/0x40 [16680.155978] ? hrtimer_interrupt+0x128/0x2c0 [16680.158108] ? smp_apic_timer_interrupt+0xa6/0x270 [16680.160828] ? apic_timer_interrupt+0xf/0x20 [16680.162950] [16680.163979] ? native_safe_halt+0x2e/0x50 [16680.166056] kvm_wait+0xcd/0xe0 [16680.167648] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16680.170435] do_raw_spin_lock+0x177/0x190 [16680.172991] _raw_spin_lock+0x21/0x30 [16680.174835] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16680.178293] process_one_work+0x2c8/0x700 [16680.180214] worker_thread+0x296/0x6e0 [16680.182155] ? rescuer_thread+0x570/0x570 [16680.184130] kthread+0x1d1/0x200 [16680.185902] ? set_kthread_struct+0x70/0x70 [16680.188000] ret_from_fork+0x1f/0x30 [16680.421782] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14749s! [16680.428117] Showing busy workqueues and worker pools: [16680.433721] workqueue events: flags=0x0 [16680.437093] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16680.445556] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [16680.462716] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16680.470505] pending: kfree_rcu_monitor [16680.473071] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16680.479244] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [16680.485041] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16680.491741] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [16680.497450] workqueue events_long: flags=0x0 [16680.500831] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16680.504164] pending: lu_objects_flush [obdclass] [16680.512538] workqueue events_unbound: flags=0x2 [16680.517737] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [16680.527919] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16680.546113] workqueue events_power_efficient: flags=0x80 [16680.552508] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16680.570334] pending: do_cache_clean, neigh_periodic_work, check_lifetime [16680.580952] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16680.587500] pending: neigh_periodic_work [16680.595151] workqueue mm_percpu_wq: flags=0x8 [16680.613949] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16680.627996] pending: vmstat_update, lru_add_drain_per_cpu [16680.640427] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16680.652508] pending: vmstat_update [16680.662512] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16680.670859] pending: lru_add_drain_per_cpu [16680.673558] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [16680.694782] pending: lru_add_drain_per_cpu BAR(48) [16680.707558] workqueue writeback: flags=0x4a [16680.715599] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [16680.725753] pending: wb_workfn [16680.734027] workqueue xprtiod: flags=0xa [16680.743234] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16680.756947] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [16680.768760] workqueue ipv6_addrconf: flags=0x40008 [16680.773654] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16680.784815] pending: addrconf_verify_work [16680.789052] workqueue knbd0-recv: flags=0x1a [16680.795320] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16680.806105] in-flight: 658:recv_work [16680.810671] workqueue ptlrpc_pinger: flags=0x2 [16680.812239] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16680.825982] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [16680.834200] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14750s workers=4 idle: 32984 37 8 [16680.842026] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16665s workers=2 idle: 661 [16704.055027] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [16704.057104] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [16704.058308] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16704.058516] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16704.058524] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16704.058527] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16704.058542] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16704.058546] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16704.058551] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [16704.058553] RDX: 0000000000006ca6 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [16704.058556] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [16704.058558] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [16704.058560] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16704.058563] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [16704.058566] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16704.058568] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16704.058577] Call Trace: [16704.058582] [16704.058592] ? show_regs.cold.9+0x22/0x2f [16704.058600] ? watchdog_timer_fn.cold.6+0x62/0x10a [16704.058606] ? __hrtimer_run_queues+0x1c2/0x4d0 [16704.058610] ? watchdog+0x40/0x40 [16704.058614] ? hrtimer_interrupt+0x128/0x2c0 [16704.058617] ? smp_apic_timer_interrupt+0xa6/0x270 [16704.058625] ? apic_timer_interrupt+0xf/0x20 [16704.058630] [16704.058633] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16704.058638] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16704.058643] do_raw_spin_lock+0x177/0x190 [16704.058648] _raw_spin_lock+0x21/0x30 [16704.058653] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16704.061898] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16704.061977] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16704.061986] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16704.061990] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16704.062013] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16704.062018] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16704.062023] RAX: 000000000000790d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16704.062025] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [16704.062028] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [16704.062030] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [16704.062032] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [16704.062035] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [16704.062038] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16704.062040] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [16704.062047] Call Trace: [16704.062053] [16704.062058] ? show_regs.cold.9+0x22/0x2f [16704.062068] ? watchdog_timer_fn.cold.6+0x62/0x10a [16704.062075] ? __hrtimer_run_queues+0x1c2/0x4d0 [16704.062079] ? watchdog+0x40/0x40 [16704.062083] ? hrtimer_interrupt+0x128/0x2c0 [16704.062087] ? smp_apic_timer_interrupt+0xa6/0x270 [16704.062094] ? apic_timer_interrupt+0xf/0x20 [16704.062098] [16704.062101] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16704.062107] do_raw_spin_lock+0x177/0x190 [16704.062112] _raw_spin_lock+0x21/0x30 [16704.062116] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16704.062459] ? do_raw_spin_unlock+0x75/0x190 [16704.062465] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16704.062664] ? do_wait_intr+0xf0/0xf0 [16704.062672] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16704.062868] mdc_close+0x53b/0xd70 [mdc] [16704.062905] lmv_close+0x18d/0x490 [lmv] [16704.062933] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [16704.063254] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16704.086582] obd_mod_free+0x46/0x140 [ptlrpc] [16704.086924] mdc_commit_open+0x91/0xb0 [mdc] [16704.086964] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16704.087193] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16704.087550] after_reply+0xe2c/0x1780 [ptlrpc] [16704.087784] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16704.088001] ? __next_timer_interrupt+0x160/0x160 [16704.088084] ptlrpcd+0xa82/0xe40 [ptlrpc] [16704.088454] ? do_wait_intr+0xf0/0xf0 [16704.088464] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16704.088678] kthread+0x1d1/0x200 [16704.088749] ? set_kthread_struct+0x70/0x70 [16704.088753] ret_from_fork+0x1f/0x30 [16704.410825] ll_md_real_close+0xa5/0x2a0 [lustre] [16704.415239] ll_md_close+0x274/0x800 [lustre] [16704.417669] ll_file_release+0x128/0xc50 [lustre] [16704.422494] __fput+0x10d/0x440 [16704.423431] ____fput+0x12/0x20 [16704.424549] task_work_run+0xc6/0x110 [16704.429239] exit_to_usermode_loop+0x1dd/0x1f0 [16704.430444] do_syscall_64+0x3d6/0x440 [16704.431480] entry_SYSCALL_64_after_hwframe+0x49/0xae [16704.434907] RIP: 0033:0x7f151bee3cd8 [16704.436966] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16704.450724] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16704.456389] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [16704.464558] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [16704.469601] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16704.474267] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16704.478801] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16708.052652] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [16708.056046] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [16708.057119] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16708.057316] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16708.063456] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16708.063539] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16708.063547] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16708.063555] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16708.064181] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16708.064193] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16708.064198] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16708.064202] RAX: 0000000000003f23 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16708.064205] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [16708.064207] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [16708.064209] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [16708.064211] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [16708.064215] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [16708.064217] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16708.064219] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16708.064228] Call Trace: [16708.064235] [16708.064239] ? show_regs.cold.9+0x22/0x2f [16708.064320] ? watchdog_timer_fn.cold.6+0x62/0x10a [16708.064618] ? __hrtimer_run_queues+0x1c2/0x4d0 [16708.064622] ? watchdog+0x40/0x40 [16708.064626] ? hrtimer_interrupt+0x128/0x2c0 [16708.064630] ? smp_apic_timer_interrupt+0xa6/0x270 [16708.064640] ? apic_timer_interrupt+0xf/0x20 [16708.064644] [16708.064648] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16708.064652] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16708.064658] do_raw_spin_lock+0x177/0x190 [16708.064663] _raw_spin_lock+0x21/0x30 [16708.064668] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16708.094050] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16708.094062] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16708.094079] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16708.094084] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16708.094089] RAX: 00000000000063ca RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16708.116980] process_one_work+0x2c8/0x700 [16708.117001] worker_thread+0x296/0x6e0 [16708.117006] ? rescuer_thread+0x570/0x570 [16708.117010] kthread+0x1d1/0x200 [16708.117077] ? set_kthread_struct+0x70/0x70 [16708.117085] ret_from_fork+0x1f/0x30 [16708.330622] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [16708.335401] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [16708.337738] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [16708.342368] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [16708.348592] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [16708.350825] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16708.352894] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [16708.355119] Call Trace: [16708.355764] [16708.357550] ? show_regs.cold.9+0x22/0x2f [16708.358731] ? watchdog_timer_fn.cold.6+0x62/0x10a [16708.362482] ? __hrtimer_run_queues+0x1c2/0x4d0 [16708.367514] ? watchdog+0x40/0x40 [16708.369780] ? hrtimer_interrupt+0x128/0x2c0 [16708.370974] ? smp_apic_timer_interrupt+0xa6/0x270 [16708.376688] ? apic_timer_interrupt+0xf/0x20 [16708.378508] [16708.382486] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16708.386247] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16708.389562] do_raw_spin_lock+0x177/0x190 [16708.392050] _raw_spin_lock+0x21/0x30 [16708.393627] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16708.408505] ? do_raw_spin_unlock+0x75/0x190 [16708.411404] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16708.418909] ? do_wait_intr+0xf0/0xf0 [16708.423817] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16708.426468] mdc_reint+0x2d/0x110 [mdc] [16708.430069] mdc_unlink+0x658/0x870 [mdc] [16708.432925] lmv_unlink+0x34d/0xbd0 [lmv] [16708.433926] ll_unlink+0x24d/0x7f0 [lustre] [16708.435176] vfs_unlink+0x202/0x330 [16708.437286] do_unlinkat+0x2fa/0x460 [16708.438726] __x64_sys_unlink+0x37/0x50 [16708.439684] do_syscall_64+0xc1/0x440 [16708.440784] entry_SYSCALL_64_after_hwframe+0x49/0xae [16708.442744] RIP: 0033:0x7f151bdee10b [16708.444092] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [16708.460240] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [16708.463028] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [16708.465831] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [16708.471687] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16708.477245] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16708.481661] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16711.136714] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14780s! [16711.144152] Showing busy workqueues and worker pools: [16711.147222] workqueue events: flags=0x0 [16711.151503] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16711.165683] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [16711.174459] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16711.183372] pending: kfree_rcu_monitor [16711.185452] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16711.194496] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [16711.210247] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16711.219480] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [16711.229319] workqueue events_long: flags=0x0 [16711.232367] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16711.239836] pending: lu_objects_flush [obdclass] [16711.243359] workqueue events_unbound: flags=0x2 [16711.245929] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [16711.253953] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16711.262925] workqueue events_power_efficient: flags=0x80 [16711.277422] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16711.284829] pending: do_cache_clean, neigh_periodic_work, check_lifetime [16711.290461] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16711.297711] pending: neigh_periodic_work [16711.301308] workqueue mm_percpu_wq: flags=0x8 [16711.303748] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16711.313205] pending: vmstat_update, lru_add_drain_per_cpu [16711.315356] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16711.320535] pending: vmstat_update [16711.325862] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16711.331177] pending: lru_add_drain_per_cpu [16711.335711] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [16711.339040] pending: lru_add_drain_per_cpu BAR(48) [16711.340914] workqueue writeback: flags=0x4a [16711.349706] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [16711.356321] pending: wb_workfn [16711.358744] workqueue xprtiod: flags=0xa [16711.363354] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16711.370200] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [16711.381589] workqueue ipv6_addrconf: flags=0x40008 [16711.389133] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16711.399186] pending: addrconf_verify_work [16711.404435] workqueue knbd0-recv: flags=0x1a [16711.409920] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16711.420628] in-flight: 658:recv_work [16711.424458] workqueue ptlrpc_pinger: flags=0x2 [16711.429257] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16711.439832] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [16711.446545] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14780s workers=4 idle: 32984 37 8 [16711.455417] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16696s workers=2 idle: 661 [16732.055055] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [16732.058373] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [16732.058515] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16732.058586] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16732.058592] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16732.058596] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16732.058610] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16732.058615] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16732.058619] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [16732.058622] RDX: 0000000000005b75 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [16732.058624] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [16732.058626] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [16732.058629] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16732.058632] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [16732.058635] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16732.058637] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16732.058647] Call Trace: [16732.058652] [16732.058659] ? show_regs.cold.9+0x22/0x2f [16732.058668] ? watchdog_timer_fn.cold.6+0x62/0x10a [16732.058674] ? __hrtimer_run_queues+0x1c2/0x4d0 [16732.058678] ? watchdog+0x40/0x40 [16732.058681] ? hrtimer_interrupt+0x128/0x2c0 [16732.058685] ? smp_apic_timer_interrupt+0xa6/0x270 [16732.058693] ? apic_timer_interrupt+0xf/0x20 [16732.058698] [16732.058701] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16732.058706] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16732.058712] do_raw_spin_lock+0x177/0x190 [16732.058717] _raw_spin_lock+0x21/0x30 [16732.058723] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16732.059020] obd_mod_free+0x46/0x140 [ptlrpc] [16732.059215] mdc_commit_open+0x91/0xb0 [mdc] [16732.059247] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16732.059447] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16732.059641] after_reply+0xe2c/0x1780 [ptlrpc] [16732.059834] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16732.060639] ? __next_timer_interrupt+0x160/0x160 [16732.060649] ptlrpcd+0xa82/0xe40 [ptlrpc] [16732.061686] ? do_wait_intr+0xf0/0xf0 [16732.061697] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16732.062372] kthread+0x1d1/0x200 [16732.062379] ? set_kthread_struct+0x70/0x70 [16732.062383] ret_from_fork+0x1f/0x30 [16732.257942] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16732.281295] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16732.286353] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16732.290450] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16732.293175] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16732.304337] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16732.322576] RAX: 0000000000007497 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16732.325506] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [16732.329034] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [16732.332379] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [16732.335535] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [16732.339356] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [16732.343291] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16732.348239] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [16732.352083] Call Trace: [16732.356471] [16732.357244] ? show_regs.cold.9+0x22/0x2f [16732.364224] ? watchdog_timer_fn.cold.6+0x62/0x10a [16732.368826] ? __hrtimer_run_queues+0x1c2/0x4d0 [16732.372650] ? watchdog+0x40/0x40 [16732.374430] ? hrtimer_interrupt+0x128/0x2c0 [16732.376466] ? smp_apic_timer_interrupt+0xa6/0x270 [16732.380540] ? apic_timer_interrupt+0xf/0x20 [16732.382435] [16732.383369] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16732.389184] do_raw_spin_lock+0x177/0x190 [16732.391479] _raw_spin_lock+0x21/0x30 [16732.393462] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16732.396535] ? do_raw_spin_unlock+0x75/0x190 [16732.407411] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16732.411261] ? do_wait_intr+0xf0/0xf0 [16732.422095] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16732.429323] mdc_close+0x53b/0xd70 [mdc] [16732.435904] lmv_close+0x18d/0x490 [lmv] [16732.439362] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [16732.450783] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16732.456620] ll_md_real_close+0xa5/0x2a0 [lustre] [16732.463480] ll_md_close+0x274/0x800 [lustre] [16732.477183] ll_file_release+0x128/0xc50 [lustre] [16732.478996] __fput+0x10d/0x440 [16732.488368] ____fput+0x12/0x20 [16732.489386] task_work_run+0xc6/0x110 [16732.495141] exit_to_usermode_loop+0x1dd/0x1f0 [16732.500181] do_syscall_64+0x3d6/0x440 [16732.501268] entry_SYSCALL_64_after_hwframe+0x49/0xae [16732.506515] RIP: 0033:0x7f151bee3cd8 [16732.507600] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16732.526602] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16732.532897] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [16732.538314] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [16732.541627] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16732.548097] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16732.563513] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16736.052613] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [16736.056519] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [16736.075106] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16736.075179] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16736.075186] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16736.075188] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16736.075201] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16736.075204] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [16736.075208] RAX: 0000000000007476 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16736.075211] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [16736.075212] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [16736.075215] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [16736.075217] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [16736.075221] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [16736.075224] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16736.075226] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [16736.075235] Call Trace: [16736.075240] [16736.075246] ? show_regs.cold.9+0x22/0x2f [16736.075257] ? watchdog_timer_fn.cold.6+0x62/0x10a [16736.075264] ? __hrtimer_run_queues+0x1c2/0x4d0 [16736.075268] ? watchdog+0x40/0x40 [16736.075272] ? hrtimer_interrupt+0x128/0x2c0 [16736.075277] ? smp_apic_timer_interrupt+0xa6/0x270 [16736.075284] ? apic_timer_interrupt+0xf/0x20 [16736.075289] [16736.075292] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16736.075297] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16736.075302] do_raw_spin_lock+0x177/0x190 [16736.075306] _raw_spin_lock+0x21/0x30 [16736.075312] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16736.081448] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16736.081528] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16736.081536] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16736.081543] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16736.082028] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16736.082041] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16736.082045] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16736.082050] RAX: 0000000000007333 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16736.082052] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [16736.082054] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [16736.082057] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [16736.082059] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [16736.082063] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [16736.082066] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16736.082068] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16736.082078] Call Trace: [16736.082084] [16736.082090] ? show_regs.cold.9+0x22/0x2f [16736.082101] ? watchdog_timer_fn.cold.6+0x62/0x10a [16736.082108] ? __hrtimer_run_queues+0x1c2/0x4d0 [16736.082113] ? watchdog+0x40/0x40 [16736.082117] ? hrtimer_interrupt+0x128/0x2c0 [16736.082121] ? smp_apic_timer_interrupt+0xa6/0x270 [16736.082129] ? apic_timer_interrupt+0xf/0x20 [16736.082134] [16736.082138] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16736.082143] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16736.082148] do_raw_spin_lock+0x177/0x190 [16736.082153] _raw_spin_lock+0x21/0x30 [16736.082159] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16736.082389] process_one_work+0x2c8/0x700 [16736.082397] worker_thread+0x296/0x6e0 [16736.082401] ? rescuer_thread+0x570/0x570 [16736.082405] kthread+0x1d1/0x200 [16736.082409] ? set_kthread_struct+0x70/0x70 [16736.082413] ret_from_fork+0x1f/0x30 [16736.264990] ? do_raw_spin_unlock+0x75/0x190 [16736.266339] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16736.268336] ? do_wait_intr+0xf0/0xf0 [16736.269434] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16736.271380] mdc_reint+0x2d/0x110 [mdc] [16736.272660] mdc_unlink+0x658/0x870 [mdc] [16736.274157] lmv_unlink+0x34d/0xbd0 [lmv] [16736.275449] ll_unlink+0x24d/0x7f0 [lustre] [16736.277135] vfs_unlink+0x202/0x330 [16736.278324] do_unlinkat+0x2fa/0x460 [16736.279490] __x64_sys_unlink+0x37/0x50 [16736.281044] do_syscall_64+0xc1/0x440 [16736.282255] entry_SYSCALL_64_after_hwframe+0x49/0xae [16736.284016] RIP: 0033:0x7f151bdee10b [16736.285222] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [16736.291486] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [16736.294067] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [16736.297171] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [16736.299196] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16736.305619] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16736.311066] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16741.857263] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14811s! [16741.870567] Showing busy workqueues and worker pools: [16741.876608] workqueue events: flags=0x0 [16741.883157] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16741.893350] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [16741.905547] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16741.914169] pending: kfree_rcu_monitor [16741.915676] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16741.925854] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [16741.936491] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16741.945890] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [16741.962155] workqueue events_long: flags=0x0 [16741.969079] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16741.978397] pending: lu_objects_flush [obdclass] [16741.982878] workqueue events_unbound: flags=0x2 [16741.987735] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [16742.000238] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16742.010080] workqueue events_power_efficient: flags=0x80 [16742.020783] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16742.032315] pending: do_cache_clean, neigh_periodic_work, check_lifetime [16742.045801] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16742.048179] pending: neigh_periodic_work [16742.059762] workqueue mm_percpu_wq: flags=0x8 [16742.067454] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16742.077809] pending: vmstat_update, lru_add_drain_per_cpu [16742.086564] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16742.095844] pending: vmstat_update [16742.100824] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16742.106836] pending: lru_add_drain_per_cpu [16742.116146] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [16742.127818] pending: lru_add_drain_per_cpu BAR(48) [16742.140574] workqueue writeback: flags=0x4a [16742.145416] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [16742.157958] pending: wb_workfn [16742.167414] workqueue xprtiod: flags=0xa [16742.171441] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16742.178121] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [16742.192588] workqueue ipv6_addrconf: flags=0x40008 [16742.204592] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16742.211408] pending: addrconf_verify_work [16742.222926] workqueue knbd0-recv: flags=0x1a [16742.230399] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16742.235352] in-flight: 658:recv_work [16742.244585] workqueue ptlrpc_pinger: flags=0x2 [16742.253732] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16742.264205] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [16742.272930] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14811s workers=4 idle: 32984 37 8 [16742.285478] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16727s workers=2 idle: 661 [16744.955062] rcu: INFO: rcu_sched self-detected stall on CPU [16744.957488] rcu: 1-...!: (8902189 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [16744.961458] (t=14820308 jiffies g=249657 q=6930) [16744.964344] rcu: rcu_sched kthread starved for 14815260 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [16744.981757] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [16744.991323] rcu: RCU grace-period kthread stack dump: [16744.999319] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [16745.008720] Call Trace: [16745.009963] __schedule+0x351/0xcb0 [16745.013186] schedule+0xc0/0x180 [16745.017935] schedule_timeout+0xb4/0x190 [16745.021012] ? do_raw_spin_unlock+0x75/0x190 [16745.024303] ? __next_timer_interrupt+0x160/0x160 [16745.029525] ? prepare_to_swait_event+0xf3/0x140 [16745.035247] rcu_gp_kthread+0x8dd/0x15b0 [16745.044274] ? force_qs_rnp+0x2c0/0x2c0 [16745.049370] kthread+0x1d1/0x200 [16745.052605] ? set_kthread_struct+0x70/0x70 [16745.062499] ret_from_fork+0x1f/0x30 [16745.071832] rcu: Stack dump where RCU GP kthread last ran: [16745.076705] Sending NMI from CPU 1 to CPUs 2: [16745.086618] NMI backtrace for cpu 2 skipped: idling at native_safe_halt+0x2e/0x50 [16745.088960] NMI backtrace for cpu 1 [16745.104507] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16745.123155] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16745.135020] Call Trace: [16745.140027] [16745.144985] ? dump_stack+0xbb/0x10e [16745.150497] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [16745.156080] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [16745.159034] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [16745.161693] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [16745.164566] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [16745.168197] ? rcu_dump_cpu_stacks+0x11b/0x171 [16745.170225] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [16745.173148] ? update_process_times+0x6d/0xc0 [16745.174913] ? tick_sched_handle.isra.18+0x27/0x90 [16745.178035] ? tick_sched_timer+0x53/0xb0 [16745.179244] ? __hrtimer_run_queues+0x1c2/0x4d0 [16745.181378] ? tick_sched_handle.isra.18+0x90/0x90 [16745.183469] ? hrtimer_interrupt+0x128/0x2c0 [16745.185301] ? smp_apic_timer_interrupt+0xa6/0x270 [16745.186954] ? apic_timer_interrupt+0xf/0x20 [16745.189554] [16745.190365] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16745.196412] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16745.199848] ? do_raw_spin_lock+0x177/0x190 [16745.205446] ? _raw_spin_lock+0x21/0x30 [16745.208146] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16745.214915] ? obd_mod_free+0x46/0x140 [ptlrpc] [16745.217090] ? mdc_commit_open+0x91/0xb0 [mdc] [16745.219342] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16745.225398] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16745.229461] ? after_reply+0xe2c/0x1780 [ptlrpc] [16745.233939] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16745.242653] ? __next_timer_interrupt+0x160/0x160 [16745.244926] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [16745.255110] ? do_wait_intr+0xf0/0xf0 [16745.258113] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16745.265403] ? kthread+0x1d1/0x200 [16745.269473] ? set_kthread_struct+0x70/0x70 [16745.271410] ? ret_from_fork+0x1f/0x30 [16760.057061] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [16760.060674] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16760.085930] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16760.090197] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16760.095928] RIP: 0010:native_safe_halt+0x2e/0x50 [16760.100255] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16760.113058] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16760.116359] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16760.119671] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [16760.123659] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [16760.128815] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [16760.133627] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [16760.138739] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [16760.150618] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16760.153325] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [16760.156299] Call Trace: [16760.157955] [16760.158516] ? show_regs.cold.9+0x22/0x2f [16760.160042] ? watchdog_timer_fn.cold.6+0x62/0x10a [16760.161383] ? __hrtimer_run_queues+0x1c2/0x4d0 [16760.163386] ? watchdog+0x40/0x40 [16760.164720] ? hrtimer_interrupt+0x128/0x2c0 [16760.166676] ? smp_apic_timer_interrupt+0xa6/0x270 [16760.170897] ? apic_timer_interrupt+0xf/0x20 [16760.172872] [16760.175273] ? native_safe_halt+0x2e/0x50 [16760.183151] kvm_wait+0xcd/0xe0 [16760.185079] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16760.187717] do_raw_spin_lock+0x177/0x190 [16760.189780] _raw_spin_lock+0x21/0x30 [16760.192461] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16760.194549] ? do_raw_spin_unlock+0x75/0x190 [16760.197496] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16760.199213] ? do_wait_intr+0xf0/0xf0 [16760.208156] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16760.213284] mdc_close+0x53b/0xd70 [mdc] [16760.215788] lmv_close+0x18d/0x490 [lmv] [16760.217889] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [16760.219819] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16760.223152] ll_md_real_close+0xa5/0x2a0 [lustre] [16760.225217] ll_md_close+0x274/0x800 [lustre] [16760.227178] ll_file_release+0x128/0xc50 [lustre] [16760.231090] __fput+0x10d/0x440 [16760.232735] ____fput+0x12/0x20 [16760.233838] task_work_run+0xc6/0x110 [16760.235234] exit_to_usermode_loop+0x1dd/0x1f0 [16760.236819] do_syscall_64+0x3d6/0x440 [16760.241207] entry_SYSCALL_64_after_hwframe+0x49/0xae [16760.244160] RIP: 0033:0x7f151bee3cd8 [16760.246638] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16760.259417] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16760.263819] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [16760.273160] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [16760.282360] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16760.287860] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16760.294142] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16768.052031] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [16768.058267] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16768.106886] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16768.120712] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16768.127835] RIP: 0010:native_safe_halt+0x2e/0x50 [16768.132020] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16768.159572] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16768.169803] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16768.174703] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [16768.185987] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [16768.190997] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [16768.205145] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [16768.221553] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [16768.227229] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16768.229234] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [16768.244815] Call Trace: [16768.248392] [16768.249280] ? show_regs.cold.9+0x22/0x2f [16768.255861] ? watchdog_timer_fn.cold.6+0x62/0x10a [16768.257821] ? __hrtimer_run_queues+0x1c2/0x4d0 [16768.260070] ? watchdog+0x40/0x40 [16768.264258] ? hrtimer_interrupt+0x128/0x2c0 [16768.265759] ? smp_apic_timer_interrupt+0xa6/0x270 [16768.267051] ? apic_timer_interrupt+0xf/0x20 [16768.277376] [16768.278202] ? native_safe_halt+0x2e/0x50 [16768.280296] kvm_wait+0xcd/0xe0 [16768.286835] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16768.289573] do_raw_spin_lock+0x177/0x190 [16768.294374] _raw_spin_lock+0x21/0x30 [16768.297028] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16768.305909] ? do_raw_spin_unlock+0x75/0x190 [16768.309106] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16768.314593] ? do_wait_intr+0xf0/0xf0 [16768.315941] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16768.323836] mdc_reint+0x2d/0x110 [mdc] [16768.324998] mdc_unlink+0x658/0x870 [mdc] [16768.329604] lmv_unlink+0x34d/0xbd0 [lmv] [16768.335407] ll_unlink+0x24d/0x7f0 [lustre] [16768.342074] vfs_unlink+0x202/0x330 [16768.344317] do_unlinkat+0x2fa/0x460 [16768.348145] __x64_sys_unlink+0x37/0x50 [16768.351647] do_syscall_64+0xc1/0x440 [16768.357611] entry_SYSCALL_64_after_hwframe+0x49/0xae [16768.360208] RIP: 0033:0x7f151bdee10b [16768.361965] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [16768.381642] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [16768.384485] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [16768.391221] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [16768.394831] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16768.396744] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16768.400193] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16772.056500] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [16772.056513] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [16772.060282] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16772.061598] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16772.061609] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16772.061611] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16772.061627] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16772.061632] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16772.061636] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [16772.061639] RDX: 0000000000004d89 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [16772.061641] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [16772.061643] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [16772.061646] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16772.061649] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [16772.061652] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16772.061654] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16772.061663] Call Trace: [16772.061669] [16772.061677] ? show_regs.cold.9+0x22/0x2f [16772.061687] ? watchdog_timer_fn.cold.6+0x62/0x10a [16772.061694] ? __hrtimer_run_queues+0x1c2/0x4d0 [16772.061698] ? watchdog+0x40/0x40 [16772.061703] ? hrtimer_interrupt+0x128/0x2c0 [16772.061707] ? smp_apic_timer_interrupt+0xa6/0x270 [16772.061716] ? apic_timer_interrupt+0xf/0x20 [16772.061775] [16772.061778] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16772.061784] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16772.061788] do_raw_spin_lock+0x177/0x190 [16772.061793] _raw_spin_lock+0x21/0x30 [16772.061801] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16772.062074] obd_mod_free+0x46/0x140 [ptlrpc] [16772.062269] mdc_commit_open+0x91/0xb0 [mdc] [16772.062311] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16772.067290] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16772.067751] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16772.067764] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16772.067771] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16772.068183] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16772.068194] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16772.068198] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 [16772.068203] RAX: 0000000000003200 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16772.068205] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [16772.068208] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [16772.068210] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [16772.068213] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [16772.068216] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [16772.068219] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16772.068221] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16772.068231] Call Trace: [16772.068235] [16772.068241] ? show_regs.cold.9+0x22/0x2f [16772.068250] ? watchdog_timer_fn.cold.6+0x62/0x10a [16772.068257] ? __hrtimer_run_queues+0x1c2/0x4d0 [16772.068262] ? watchdog+0x40/0x40 [16772.068267] ? hrtimer_interrupt+0x128/0x2c0 [16772.068271] ? smp_apic_timer_interrupt+0xa6/0x270 [16772.068279] ? apic_timer_interrupt+0xf/0x20 [16772.068284] [16772.068288] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16772.068292] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16772.068298] do_raw_spin_lock+0x177/0x190 [16772.068304] _raw_spin_lock+0x21/0x30 [16772.068309] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16772.107914] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16772.108203] after_reply+0xe2c/0x1780 [ptlrpc] [16772.117045] process_one_work+0x2c8/0x700 [16772.117069] worker_thread+0x296/0x6e0 [16772.117074] ? rescuer_thread+0x570/0x570 [16772.117078] kthread+0x1d1/0x200 [16772.117083] ? set_kthread_struct+0x70/0x70 [16772.117087] ret_from_fork+0x1f/0x30 [16772.429177] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16772.436451] ? __next_timer_interrupt+0x160/0x160 [16772.439631] ptlrpcd+0xa82/0xe40 [ptlrpc] [16772.442283] ? do_wait_intr+0xf0/0xf0 [16772.444298] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16772.447066] kthread+0x1d1/0x200 [16772.448505] ? set_kthread_struct+0x70/0x70 [16772.452372] ret_from_fork+0x1f/0x30 [16772.577166] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14841s! [16772.588513] Showing busy workqueues and worker pools: [16772.591011] workqueue events: flags=0x0 [16772.594518] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16772.603925] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [16772.614243] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16772.619742] pending: kfree_rcu_monitor [16772.626810] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16772.634504] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [16772.644272] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16772.654242] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [16772.665282] workqueue events_long: flags=0x0 [16772.676155] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16772.687766] pending: lu_objects_flush [obdclass] [16772.692174] workqueue events_unbound: flags=0x2 [16772.695232] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [16772.699978] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16772.706293] workqueue events_power_efficient: flags=0x80 [16772.710277] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16772.713813] pending: do_cache_clean, neigh_periodic_work, check_lifetime [16772.722207] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16772.732940] pending: neigh_periodic_work [16772.739721] workqueue mm_percpu_wq: flags=0x8 [16772.745877] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16772.754897] pending: vmstat_update, lru_add_drain_per_cpu [16772.763025] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16772.773215] pending: vmstat_update [16772.775881] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16772.784668] pending: lru_add_drain_per_cpu [16772.787566] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [16772.794834] pending: lru_add_drain_per_cpu BAR(48) [16772.801091] workqueue writeback: flags=0x4a [16772.804280] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [16772.810163] pending: wb_workfn [16772.814461] workqueue xprtiod: flags=0xa [16772.817321] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16772.821245] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [16772.830561] workqueue ipv6_addrconf: flags=0x40008 [16772.835077] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16772.844353] pending: addrconf_verify_work [16772.848914] workqueue knbd0-recv: flags=0x1a [16772.853698] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16772.857758] in-flight: 658:recv_work [16772.861114] workqueue ptlrpc_pinger: flags=0x2 [16772.871947] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16772.876679] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [16772.880710] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14842s workers=4 idle: 32984 37 8 [16772.886757] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16757s workers=2 idle: 661 [16788.067575] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [16788.078001] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16788.117986] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16788.130006] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16788.137033] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16788.147797] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16788.170211] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16788.182177] RAX: 0000000000001f86 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16788.186931] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [16788.191322] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [16788.193518] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [16788.204807] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [16788.213290] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [16788.218925] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16788.230505] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [16788.241152] Call Trace: [16788.243711] [16788.244435] ? show_regs.cold.9+0x22/0x2f [16788.246865] ? watchdog_timer_fn.cold.6+0x62/0x10a [16788.250287] ? __hrtimer_run_queues+0x1c2/0x4d0 [16788.253188] ? watchdog+0x40/0x40 [16788.255306] ? hrtimer_interrupt+0x128/0x2c0 [16788.258078] ? smp_apic_timer_interrupt+0xa6/0x270 [16788.261087] ? apic_timer_interrupt+0xf/0x20 [16788.263551] [16788.266540] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16788.273139] do_raw_spin_lock+0x177/0x190 [16788.276389] _raw_spin_lock+0x21/0x30 [16788.280968] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16788.285730] ? do_raw_spin_unlock+0x75/0x190 [16788.287933] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16788.291567] ? do_wait_intr+0xf0/0xf0 [16788.294151] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16788.301788] mdc_close+0x53b/0xd70 [mdc] [16788.315859] lmv_close+0x18d/0x490 [lmv] [16788.319336] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [16788.323083] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16788.330779] ll_md_real_close+0xa5/0x2a0 [lustre] [16788.336356] ll_md_close+0x274/0x800 [lustre] [16788.342115] ll_file_release+0x128/0xc50 [lustre] [16788.343677] __fput+0x10d/0x440 [16788.350389] ____fput+0x12/0x20 [16788.357800] task_work_run+0xc6/0x110 [16788.365043] exit_to_usermode_loop+0x1dd/0x1f0 [16788.366628] do_syscall_64+0x3d6/0x440 [16788.373962] entry_SYSCALL_64_after_hwframe+0x49/0xae [16788.379915] RIP: 0033:0x7f151bee3cd8 [16788.387940] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16788.411921] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16788.426152] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [16788.432365] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [16788.455625] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16788.464614] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16788.477150] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16800.052015] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [16800.056154] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16800.058322] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [16800.062571] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [16800.062589] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16800.062665] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16800.062672] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16800.062679] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16800.063014] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16800.063027] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16800.063031] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16800.063035] RAX: 0000000000000591 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16800.063037] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [16800.063039] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [16800.063041] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [16800.063044] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [16800.063047] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [16800.063050] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16800.063053] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16800.063062] Call Trace: [16800.063068] [16800.063074] ? show_regs.cold.9+0x22/0x2f [16800.063083] ? watchdog_timer_fn.cold.6+0x62/0x10a [16800.063089] ? __hrtimer_run_queues+0x1c2/0x4d0 [16800.063093] ? watchdog+0x40/0x40 [16800.063098] ? hrtimer_interrupt+0x128/0x2c0 [16800.063102] ? smp_apic_timer_interrupt+0xa6/0x270 [16800.063110] ? apic_timer_interrupt+0xf/0x20 [16800.063115] [16800.063118] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16800.063122] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16800.063128] do_raw_spin_lock+0x177/0x190 [16800.063132] _raw_spin_lock+0x21/0x30 [16800.063136] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16800.063346] process_one_work+0x2c8/0x700 [16800.063353] worker_thread+0x296/0x6e0 [16800.063357] ? rescuer_thread+0x570/0x570 [16800.063361] kthread+0x1d1/0x200 [16800.063365] ? set_kthread_struct+0x70/0x70 [16800.063368] ret_from_fork+0x1f/0x30 [16800.096464] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16800.096483] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16800.096486] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16800.096502] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16800.096507] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16800.096513] RAX: 0000000000000822 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16800.096516] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [16800.096518] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [16800.096520] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [16800.096522] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [16800.096526] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [16800.096529] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16800.096531] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [16800.096540] Call Trace: [16800.096546] [16800.096552] ? show_regs.cold.9+0x22/0x2f [16800.096565] ? watchdog_timer_fn.cold.6+0x62/0x10a [16800.096572] ? __hrtimer_run_queues+0x1c2/0x4d0 [16800.096576] ? watchdog+0x40/0x40 [16800.096580] ? hrtimer_interrupt+0x128/0x2c0 [16800.096585] ? smp_apic_timer_interrupt+0xa6/0x270 [16800.096594] ? apic_timer_interrupt+0xf/0x20 [16800.096599] [16800.096602] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16800.096607] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16800.096613] do_raw_spin_lock+0x177/0x190 [16800.096617] _raw_spin_lock+0x21/0x30 [16800.096625] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16800.096898] ? do_raw_spin_unlock+0x75/0x190 [16800.096904] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16800.097102] ? do_wait_intr+0xf0/0xf0 [16800.097110] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16800.097316] mdc_reint+0x2d/0x110 [mdc] [16800.097361] mdc_unlink+0x658/0x870 [mdc] [16800.097388] lmv_unlink+0x34d/0xbd0 [lmv] [16800.097421] ll_unlink+0x24d/0x7f0 [lustre] [16800.097539] vfs_unlink+0x202/0x330 [16800.097547] do_unlinkat+0x2fa/0x460 [16800.097553] __x64_sys_unlink+0x37/0x50 [16800.097557] do_syscall_64+0xc1/0x440 [16800.097563] entry_SYSCALL_64_after_hwframe+0x49/0xae [16800.097570] RIP: 0033:0x7f151bdee10b [16800.097577] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [16800.097581] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [16800.097585] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [16800.097588] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [16800.097590] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16800.097593] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16800.097595] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16800.494334] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16800.524642] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16800.529342] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16800.533023] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16800.535289] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16800.543947] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16800.547177] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [16800.550405] RDX: 0000000000002079 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [16800.553160] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [16800.555795] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [16800.558615] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16800.561516] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [16800.564374] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16800.566493] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16800.568748] Call Trace: [16800.569632] [16800.570328] ? show_regs.cold.9+0x22/0x2f [16800.571598] ? watchdog_timer_fn.cold.6+0x62/0x10a [16800.573272] ? __hrtimer_run_queues+0x1c2/0x4d0 [16800.574621] ? watchdog+0x40/0x40 [16800.575555] ? hrtimer_interrupt+0x128/0x2c0 [16800.576803] ? smp_apic_timer_interrupt+0xa6/0x270 [16800.578489] ? apic_timer_interrupt+0xf/0x20 [16800.580390] [16800.581317] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16800.583690] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16800.585359] do_raw_spin_lock+0x177/0x190 [16800.586719] _raw_spin_lock+0x21/0x30 [16800.587895] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16800.590192] obd_mod_free+0x46/0x140 [ptlrpc] [16800.592522] mdc_commit_open+0x91/0xb0 [mdc] [16800.594050] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16800.596401] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16800.598818] after_reply+0xe2c/0x1780 [ptlrpc] [16800.601758] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16800.604487] ? __next_timer_interrupt+0x160/0x160 [16800.606619] ptlrpcd+0xa82/0xe40 [ptlrpc] [16800.609189] ? do_wait_intr+0xf0/0xf0 [16800.610798] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16800.613731] kthread+0x1d1/0x200 [16800.614767] ? set_kthread_struct+0x70/0x70 [16800.616833] ret_from_fork+0x1f/0x30 [16803.298655] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14872s! [16803.308454] Showing busy workqueues and worker pools: [16803.311783] workqueue events: flags=0x0 [16803.313238] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16803.319358] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [16803.329823] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16803.338925] pending: kfree_rcu_monitor [16803.348116] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16803.355019] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [16803.368267] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16803.378931] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [16803.385920] workqueue events_long: flags=0x0 [16803.391959] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16803.398830] pending: lu_objects_flush [obdclass] [16803.405981] workqueue events_unbound: flags=0x2 [16803.414775] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [16803.422221] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16803.433711] workqueue events_power_efficient: flags=0x80 [16803.442215] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16803.449801] pending: do_cache_clean, neigh_periodic_work, check_lifetime [16803.456261] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16803.460850] pending: neigh_periodic_work [16803.462979] workqueue mm_percpu_wq: flags=0x8 [16803.464949] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16803.468915] pending: vmstat_update, lru_add_drain_per_cpu [16803.471258] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16803.473792] pending: vmstat_update [16803.475474] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16803.478255] pending: lru_add_drain_per_cpu [16803.480173] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [16803.483096] pending: lru_add_drain_per_cpu BAR(48) [16803.485446] workqueue writeback: flags=0x4a [16803.487009] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [16803.489383] pending: wb_workfn [16803.490566] workqueue xprtiod: flags=0xa [16803.491851] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16803.495800] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [16803.505689] workqueue ipv6_addrconf: flags=0x40008 [16803.507896] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16803.512705] pending: addrconf_verify_work [16803.514925] workqueue knbd0-recv: flags=0x1a [16803.517310] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16803.524045] in-flight: 658:recv_work [16803.532908] workqueue ptlrpc_pinger: flags=0x2 [16803.534862] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16803.544513] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [16803.553024] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14872s workers=4 idle: 32984 37 8 [16803.565963] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16788s workers=2 idle: 661 [16816.057030] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [16816.060692] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16816.086492] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16816.095162] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16816.101824] RIP: 0010:native_safe_halt+0x2e/0x50 [16816.109476] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16816.126077] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [16816.129545] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16816.136254] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [16816.144628] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [16816.149668] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [16816.153136] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [16816.156489] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [16816.160134] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16816.163136] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [16816.165701] Call Trace: [16816.167192] [16816.168187] ? show_regs.cold.9+0x22/0x2f [16816.170201] ? watchdog_timer_fn.cold.6+0x62/0x10a [16816.173599] ? __hrtimer_run_queues+0x1c2/0x4d0 [16816.175690] ? watchdog+0x40/0x40 [16816.177422] ? hrtimer_interrupt+0x128/0x2c0 [16816.179641] ? smp_apic_timer_interrupt+0xa6/0x270 [16816.183436] ? apic_timer_interrupt+0xf/0x20 [16816.185283] [16816.186244] ? native_safe_halt+0x2e/0x50 [16816.188417] kvm_wait+0xcd/0xe0 [16816.190038] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16816.192625] do_raw_spin_lock+0x177/0x190 [16816.195626] _raw_spin_lock+0x21/0x30 [16816.197285] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16816.199728] ? do_raw_spin_unlock+0x75/0x190 [16816.203666] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16816.209191] ? do_wait_intr+0xf0/0xf0 [16816.212329] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16816.220198] mdc_close+0x53b/0xd70 [mdc] [16816.225135] lmv_close+0x18d/0x490 [lmv] [16816.229252] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [16816.233460] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16816.236717] ll_md_real_close+0xa5/0x2a0 [lustre] [16816.239387] ll_md_close+0x274/0x800 [lustre] [16816.242405] ll_file_release+0x128/0xc50 [lustre] [16816.245486] __fput+0x10d/0x440 [16816.247096] ____fput+0x12/0x20 [16816.248934] task_work_run+0xc6/0x110 [16816.250620] exit_to_usermode_loop+0x1dd/0x1f0 [16816.253106] do_syscall_64+0x3d6/0x440 [16816.255122] entry_SYSCALL_64_after_hwframe+0x49/0xae [16816.259085] RIP: 0033:0x7f151bee3cd8 [16816.261012] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16816.269656] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16816.273752] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [16816.277743] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [16816.281044] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16816.285424] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16816.289670] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16828.052271] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [16828.056557] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [16828.056690] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [16828.056697] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16828.056770] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16828.056777] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16828.056784] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16828.057305] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16828.057317] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16828.057322] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16828.057326] RAX: 0000000000007f85 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16828.057329] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [16828.057331] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [16828.057334] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [16828.057336] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [16828.057339] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [16828.057342] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16828.057345] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16828.057354] Call Trace: [16828.057361] [16828.057366] ? show_regs.cold.9+0x22/0x2f [16828.057375] ? watchdog_timer_fn.cold.6+0x62/0x10a [16828.057382] ? __hrtimer_run_queues+0x1c2/0x4d0 [16828.057386] ? watchdog+0x40/0x40 [16828.057390] ? hrtimer_interrupt+0x128/0x2c0 [16828.057393] ? smp_apic_timer_interrupt+0xa6/0x270 [16828.057401] ? apic_timer_interrupt+0xf/0x20 [16828.057407] [16828.057409] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16828.057414] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16828.057418] do_raw_spin_lock+0x177/0x190 [16828.057423] _raw_spin_lock+0x21/0x30 [16828.057428] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16828.057637] process_one_work+0x2c8/0x700 [16828.057646] worker_thread+0x296/0x6e0 [16828.057651] ? rescuer_thread+0x570/0x570 [16828.057654] kthread+0x1d1/0x200 [16828.057658] ? set_kthread_struct+0x70/0x70 [16828.057662] ret_from_fork+0x1f/0x30 [16828.061042] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16828.061199] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16828.061218] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16828.061222] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16828.061240] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16828.061245] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16828.061250] RAX: 0000000000005ab5 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16828.061252] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [16828.061254] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [16828.061257] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [16828.061259] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [16828.061368] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [16828.061372] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16828.061374] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [16828.061383] Call Trace: [16828.061390] [16828.061398] ? show_regs.cold.9+0x22/0x2f [16828.061481] ? watchdog_timer_fn.cold.6+0x62/0x10a [16828.061492] ? __hrtimer_run_queues+0x1c2/0x4d0 [16828.061497] ? watchdog+0x40/0x40 [16828.061502] ? hrtimer_interrupt+0x128/0x2c0 [16828.061507] ? smp_apic_timer_interrupt+0xa6/0x270 [16828.061518] ? apic_timer_interrupt+0xf/0x20 [16828.061591] [16828.061595] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16828.061602] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16828.061608] do_raw_spin_lock+0x177/0x190 [16828.061613] _raw_spin_lock+0x21/0x30 [16828.061620] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16828.068994] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16828.069267] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16828.069274] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16828.069278] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16828.069291] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16828.069295] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16828.069300] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [16828.069302] RDX: 0000000000003115 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [16828.069305] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [16828.069308] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [16828.069310] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16828.069313] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [16828.069316] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16828.069319] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16828.069326] Call Trace: [16828.069331] [16828.069336] ? show_regs.cold.9+0x22/0x2f [16828.069345] ? watchdog_timer_fn.cold.6+0x62/0x10a [16828.069352] ? __hrtimer_run_queues+0x1c2/0x4d0 [16828.069356] ? watchdog+0x40/0x40 [16828.069360] ? hrtimer_interrupt+0x128/0x2c0 [16828.069364] ? smp_apic_timer_interrupt+0xa6/0x270 [16828.069372] ? apic_timer_interrupt+0xf/0x20 [16828.069376] [16828.069379] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16828.069383] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16828.069388] do_raw_spin_lock+0x177/0x190 [16828.069392] _raw_spin_lock+0x21/0x30 [16828.069397] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16828.069772] obd_mod_free+0x46/0x140 [ptlrpc] [16828.070195] mdc_commit_open+0x91/0xb0 [mdc] [16828.070253] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16828.071405] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16828.071594] after_reply+0xe2c/0x1780 [ptlrpc] [16828.071786] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16828.082182] ? do_raw_spin_unlock+0x75/0x190 [16828.121852] ? __next_timer_interrupt+0x160/0x160 [16828.121874] ptlrpcd+0xa82/0xe40 [ptlrpc] [16828.122357] ? do_wait_intr+0xf0/0xf0 [16828.122367] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16828.122574] kthread+0x1d1/0x200 [16828.122581] ? set_kthread_struct+0x70/0x70 [16828.122586] ret_from_fork+0x1f/0x30 [16828.628353] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16828.630569] ? do_wait_intr+0xf0/0xf0 [16828.631703] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16828.633738] mdc_reint+0x2d/0x110 [mdc] [16828.635573] mdc_unlink+0x658/0x870 [mdc] [16828.637207] lmv_unlink+0x34d/0xbd0 [lmv] [16828.638409] ll_unlink+0x24d/0x7f0 [lustre] [16828.639885] vfs_unlink+0x202/0x330 [16828.641125] do_unlinkat+0x2fa/0x460 [16828.642456] __x64_sys_unlink+0x37/0x50 [16828.643918] do_syscall_64+0xc1/0x440 [16828.645154] entry_SYSCALL_64_after_hwframe+0x49/0xae [16828.646458] RIP: 0033:0x7f151bdee10b [16828.647615] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [16828.653992] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [16828.656315] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [16828.658630] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [16828.661413] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16828.663450] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16828.665781] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16834.016801] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14903s! [16834.022865] Showing busy workqueues and worker pools: [16834.025044] workqueue events: flags=0x0 [16834.027019] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16834.033225] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [16834.042112] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16834.047592] pending: kfree_rcu_monitor [16834.052894] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16834.060868] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [16834.071373] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16834.076055] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [16834.083019] workqueue events_long: flags=0x0 [16834.088220] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16834.093396] pending: lu_objects_flush [obdclass] [16834.097893] workqueue events_unbound: flags=0x2 [16834.103086] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [16834.108545] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16834.117646] workqueue events_power_efficient: flags=0x80 [16834.123487] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16834.128163] pending: do_cache_clean, neigh_periodic_work, check_lifetime [16834.134964] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16834.141836] pending: neigh_periodic_work [16834.146048] workqueue mm_percpu_wq: flags=0x8 [16834.150420] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16834.154386] pending: vmstat_update, lru_add_drain_per_cpu [16834.156907] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16834.160130] pending: vmstat_update [16834.161681] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16834.164795] pending: lru_add_drain_per_cpu [16834.166716] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [16834.169690] pending: lru_add_drain_per_cpu BAR(48) [16834.171834] workqueue writeback: flags=0x4a [16834.174275] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [16834.177381] pending: wb_workfn [16834.179467] workqueue xprtiod: flags=0xa [16834.181087] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16834.184462] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [16834.188433] workqueue ipv6_addrconf: flags=0x40008 [16834.193499] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16834.200953] pending: addrconf_verify_work [16834.203464] workqueue knbd0-recv: flags=0x1a [16834.205250] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16834.208153] in-flight: 658:recv_work [16834.213123] workqueue ptlrpc_pinger: flags=0x2 [16834.220603] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16834.227474] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [16834.233289] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14903s workers=4 idle: 32984 37 8 [16834.242274] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16818s workers=2 idle: 661 [16844.063101] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [16844.071584] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16844.124614] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16844.130350] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16844.134612] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16844.136636] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16844.153628] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16844.163550] RAX: 0000000000000079 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16844.175189] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [16844.181617] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [16844.189823] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [16844.193428] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [16844.198380] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [16844.205642] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16844.214090] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [16844.220501] Call Trace: [16844.222412] [16844.226336] ? show_regs.cold.9+0x22/0x2f [16844.227759] ? watchdog_timer_fn.cold.6+0x62/0x10a [16844.233526] ? __hrtimer_run_queues+0x1c2/0x4d0 [16844.241039] ? watchdog+0x40/0x40 [16844.247552] ? hrtimer_interrupt+0x128/0x2c0 [16844.252458] ? smp_apic_timer_interrupt+0xa6/0x270 [16844.262118] ? apic_timer_interrupt+0xf/0x20 [16844.270915] [16844.273459] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16844.281284] do_raw_spin_lock+0x177/0x190 [16844.284086] _raw_spin_lock+0x21/0x30 [16844.285567] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16844.288512] ? do_raw_spin_unlock+0x75/0x190 [16844.291235] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16844.298311] ? do_wait_intr+0xf0/0xf0 [16844.304662] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16844.313048] mdc_close+0x53b/0xd70 [mdc] [16844.319606] lmv_close+0x18d/0x490 [lmv] [16844.321032] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [16844.328059] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16844.331079] ll_md_real_close+0xa5/0x2a0 [lustre] [16844.333538] ll_md_close+0x274/0x800 [lustre] [16844.342192] ll_file_release+0x128/0xc50 [lustre] [16844.346439] __fput+0x10d/0x440 [16844.351311] ____fput+0x12/0x20 [16844.353347] task_work_run+0xc6/0x110 [16844.355827] exit_to_usermode_loop+0x1dd/0x1f0 [16844.360217] do_syscall_64+0x3d6/0x440 [16844.361712] entry_SYSCALL_64_after_hwframe+0x49/0xae [16844.364630] RIP: 0033:0x7f151bee3cd8 [16844.366385] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16844.391702] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16844.399328] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [16844.408200] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [16844.413739] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16844.422970] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16844.430070] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16856.055020] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [16856.056030] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [16856.059359] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) [16856.064018] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16856.064313] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16856.064320] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16856.064410] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16856.070150] lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16856.070217] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16856.070224] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16856.070226] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16856.070279] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16856.070283] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16856.070288] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [16856.070291] RDX: 0000000000002eb0 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [16856.070293] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [16856.070296] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [16856.070298] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16856.070301] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [16856.070304] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16856.070307] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16856.070316] Call Trace: [16856.070335] [16856.070341] ? show_regs.cold.9+0x22/0x2f [16856.070351] ? watchdog_timer_fn.cold.6+0x62/0x10a [16856.070358] ? __hrtimer_run_queues+0x1c2/0x4d0 [16856.070362] ? watchdog+0x40/0x40 [16856.070366] ? hrtimer_interrupt+0x128/0x2c0 [16856.070370] ? smp_apic_timer_interrupt+0xa6/0x270 [16856.070379] ? apic_timer_interrupt+0xf/0x20 [16856.070384] [16856.070387] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16856.070392] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16856.070397] do_raw_spin_lock+0x177/0x190 [16856.070402] _raw_spin_lock+0x21/0x30 [16856.070408] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16856.104820] [16856.104831] RIP: 0010:native_safe_halt+0x2e/0x50 [16856.104852] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16856.104857] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16856.104862] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16856.104864] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [16856.104866] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [16856.104868] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [16856.104870] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [16856.104873] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [16856.104876] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16856.104878] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16856.104886] Call Trace: [16856.104892] [16856.104898] ? show_regs.cold.9+0x22/0x2f [16856.104909] ? watchdog_timer_fn.cold.6+0x62/0x10a [16856.104915] ? __hrtimer_run_queues+0x1c2/0x4d0 [16856.104920] ? watchdog+0x40/0x40 [16856.104924] ? hrtimer_interrupt+0x128/0x2c0 [16856.104928] ? smp_apic_timer_interrupt+0xa6/0x270 [16856.104935] ? apic_timer_interrupt+0xf/0x20 [16856.104940] [16856.104943] ? native_safe_halt+0x2e/0x50 [16856.104947] kvm_wait+0xcd/0xe0 [16856.104955] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16856.104965] do_raw_spin_lock+0x177/0x190 [16856.104970] _raw_spin_lock+0x21/0x30 [16856.104975] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16856.105526] process_one_work+0x2c8/0x700 [16856.105735] worker_thread+0x296/0x6e0 [16856.105740] ? rescuer_thread+0x570/0x570 [16856.105745] kthread+0x1d1/0x200 [16856.105749] ? set_kthread_struct+0x70/0x70 [16856.105825] ret_from_fork+0x1f/0x30 [16856.452454] obd_mod_free+0x46/0x140 [ptlrpc] [16856.454392] mdc_commit_open+0x91/0xb0 [mdc] [16856.456058] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16856.458189] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16856.460396] after_reply+0xe2c/0x1780 [ptlrpc] [16856.462350] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16856.464481] ? __next_timer_interrupt+0x160/0x160 [16856.466214] ptlrpcd+0xa82/0xe40 [ptlrpc] [16856.468025] ? do_wait_intr+0xf0/0xf0 [16856.470383] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16856.473824] kthread+0x1d1/0x200 [16856.475389] ? set_kthread_struct+0x70/0x70 [16856.478680] ret_from_fork+0x1f/0x30 [16860.055690] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [16860.073189] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16860.116351] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16860.125962] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16860.131397] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16860.134799] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16860.147582] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16860.154547] RAX: 0000000000002c87 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16860.160484] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [16860.166560] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [16860.171286] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [16860.174638] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [16860.179279] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [16860.190362] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16860.193815] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [16860.199474] Call Trace: [16860.201124] [16860.202660] ? show_regs.cold.9+0x22/0x2f [16860.204477] ? watchdog_timer_fn.cold.6+0x62/0x10a [16860.207013] ? __hrtimer_run_queues+0x1c2/0x4d0 [16860.208959] ? watchdog+0x40/0x40 [16860.210483] ? hrtimer_interrupt+0x128/0x2c0 [16860.212475] ? smp_apic_timer_interrupt+0xa6/0x270 [16860.214517] ? apic_timer_interrupt+0xf/0x20 [16860.216155] [16860.217036] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16860.219764] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16860.222362] do_raw_spin_lock+0x177/0x190 [16860.224805] _raw_spin_lock+0x21/0x30 [16860.226094] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16860.228803] ? do_raw_spin_unlock+0x75/0x190 [16860.231010] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16860.234248] ? do_wait_intr+0xf0/0xf0 [16860.236211] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16860.239006] mdc_reint+0x2d/0x110 [mdc] [16860.240788] mdc_unlink+0x658/0x870 [mdc] [16860.242972] lmv_unlink+0x34d/0xbd0 [lmv] [16860.245056] ll_unlink+0x24d/0x7f0 [lustre] [16860.247414] vfs_unlink+0x202/0x330 [16860.249157] do_unlinkat+0x2fa/0x460 [16860.250967] __x64_sys_unlink+0x37/0x50 [16860.252964] do_syscall_64+0xc1/0x440 [16860.260147] entry_SYSCALL_64_after_hwframe+0x49/0xae [16860.263664] RIP: 0033:0x7f151bdee10b [16860.265968] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [16860.284643] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [16860.292466] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [16860.301679] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [16860.310158] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16860.315308] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16860.322324] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16864.739680] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14934s! [16864.753087] Showing busy workqueues and worker pools: [16864.756188] workqueue events: flags=0x0 [16864.758431] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16864.764854] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [16864.768888] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16864.771367] pending: kfree_rcu_monitor [16864.777327] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16864.781941] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [16864.789872] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16864.792827] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [16864.797099] workqueue events_long: flags=0x0 [16864.799368] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16864.808237] pending: lu_objects_flush [obdclass] [16864.811974] workqueue events_unbound: flags=0x2 [16864.818422] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [16864.822598] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16864.829072] workqueue events_power_efficient: flags=0x80 [16864.831885] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16864.845165] pending: do_cache_clean, neigh_periodic_work, check_lifetime [16864.852661] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16864.857219] pending: neigh_periodic_work [16864.863217] workqueue mm_percpu_wq: flags=0x8 [16864.866031] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16864.873627] pending: vmstat_update, lru_add_drain_per_cpu [16864.876968] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16864.881357] pending: vmstat_update [16864.887151] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16864.891203] pending: lru_add_drain_per_cpu [16864.894190] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [16864.896930] pending: lru_add_drain_per_cpu BAR(48) [16864.903145] workqueue writeback: flags=0x4a [16864.910805] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [16864.914096] pending: wb_workfn [16864.915327] workqueue xprtiod: flags=0xa [16864.916869] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16864.920956] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [16864.927892] workqueue ipv6_addrconf: flags=0x40008 [16864.931632] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16864.935075] pending: addrconf_verify_work [16864.941620] workqueue knbd0-recv: flags=0x1a [16864.944411] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16864.950982] in-flight: 658:recv_work [16864.953493] workqueue ptlrpc_pinger: flags=0x2 [16864.955417] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16864.959290] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [16864.965130] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14934s workers=4 idle: 32984 37 8 [16864.970922] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16849s workers=2 idle: 661 [16872.059234] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [16872.070879] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16872.090524] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16872.111236] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16872.131524] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16872.133421] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16872.167013] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16872.172814] RAX: 0000000000004641 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16872.180610] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [16872.189049] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [16872.194232] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [16872.199919] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [16872.205225] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [16872.215649] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16872.218123] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [16872.223652] Call Trace: [16872.225020] [16872.225943] ? show_regs.cold.9+0x22/0x2f [16872.226988] ? watchdog_timer_fn.cold.6+0x62/0x10a [16872.228851] ? __hrtimer_run_queues+0x1c2/0x4d0 [16872.231651] ? watchdog+0x40/0x40 [16872.234113] ? hrtimer_interrupt+0x128/0x2c0 [16872.236100] ? smp_apic_timer_interrupt+0xa6/0x270 [16872.238527] ? apic_timer_interrupt+0xf/0x20 [16872.241243] [16872.242295] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16872.244602] do_raw_spin_lock+0x177/0x190 [16872.246026] _raw_spin_lock+0x21/0x30 [16872.247694] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16872.251926] ? do_raw_spin_unlock+0x75/0x190 [16872.254125] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16872.264506] ? do_wait_intr+0xf0/0xf0 [16872.267857] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16872.270553] mdc_close+0x53b/0xd70 [mdc] [16872.278374] lmv_close+0x18d/0x490 [lmv] [16872.281287] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [16872.285529] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16872.287867] ll_md_real_close+0xa5/0x2a0 [lustre] [16872.293051] ll_md_close+0x274/0x800 [lustre] [16872.300359] ll_file_release+0x128/0xc50 [lustre] [16872.302764] __fput+0x10d/0x440 [16872.304811] ____fput+0x12/0x20 [16872.307143] task_work_run+0xc6/0x110 [16872.309617] exit_to_usermode_loop+0x1dd/0x1f0 [16872.311847] do_syscall_64+0x3d6/0x440 [16872.317582] entry_SYSCALL_64_after_hwframe+0x49/0xae [16872.320246] RIP: 0033:0x7f151bee3cd8 [16872.322971] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16872.341084] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16872.353096] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [16872.368075] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [16872.376002] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16872.382680] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16872.385277] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16884.055072] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [16884.056047] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [16884.059333] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16884.059426] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16884.059434] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16884.059437] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16884.059520] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16884.059524] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16884.059529] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [16884.059531] RDX: 0000000000000226 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [16884.059533] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [16884.059535] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [16884.059538] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16884.059541] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [16884.059544] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16884.059547] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16884.059555] Call Trace: [16884.059560] [16884.059565] ? show_regs.cold.9+0x22/0x2f [16884.059575] ? watchdog_timer_fn.cold.6+0x62/0x10a [16884.059581] ? __hrtimer_run_queues+0x1c2/0x4d0 [16884.059585] ? watchdog+0x40/0x40 [16884.059589] ? hrtimer_interrupt+0x128/0x2c0 [16884.059593] ? smp_apic_timer_interrupt+0xa6/0x270 [16884.059602] ? apic_timer_interrupt+0xf/0x20 [16884.059607] [16884.059609] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16884.059614] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16884.059619] do_raw_spin_lock+0x177/0x190 [16884.059624] _raw_spin_lock+0x21/0x30 [16884.059630] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16884.064134] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16884.064211] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16884.064219] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16884.064226] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16884.064767] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16884.064778] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16884.064782] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16884.064787] RAX: 0000000000007d92 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16884.064789] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [16884.064792] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [16884.064794] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [16884.065142] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [16884.065146] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [16884.065149] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16884.065151] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16884.065159] Call Trace: [16884.065166] [16884.065172] ? show_regs.cold.9+0x22/0x2f [16884.065182] ? watchdog_timer_fn.cold.6+0x62/0x10a [16884.065187] ? __hrtimer_run_queues+0x1c2/0x4d0 [16884.065192] ? watchdog+0x40/0x40 [16884.065197] ? hrtimer_interrupt+0x128/0x2c0 [16884.065201] ? smp_apic_timer_interrupt+0xa6/0x270 [16884.065210] ? apic_timer_interrupt+0xf/0x20 [16884.065215] [16884.065217] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16884.065222] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16884.065228] do_raw_spin_lock+0x177/0x190 [16884.065233] _raw_spin_lock+0x21/0x30 [16884.065238] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16884.065819] process_one_work+0x2c8/0x700 [16884.065828] worker_thread+0x296/0x6e0 [16884.065832] ? rescuer_thread+0x570/0x570 [16884.065836] kthread+0x1d1/0x200 [16884.065840] ? set_kthread_struct+0x70/0x70 [16884.065844] ret_from_fork+0x1f/0x30 [16884.441623] obd_mod_free+0x46/0x140 [ptlrpc] [16884.447123] mdc_commit_open+0x91/0xb0 [mdc] [16884.451211] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16884.455589] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16884.460151] after_reply+0xe2c/0x1780 [ptlrpc] [16884.462321] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16884.466610] ? __next_timer_interrupt+0x160/0x160 [16884.469516] ptlrpcd+0xa82/0xe40 [ptlrpc] [16884.471301] ? do_wait_intr+0xf0/0xf0 [16884.472600] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16884.474979] kthread+0x1d1/0x200 [16884.476233] ? set_kthread_struct+0x70/0x70 [16884.479015] ret_from_fork+0x1f/0x30 [16892.053046] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [16892.055898] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16892.105936] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16892.120823] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16892.129292] RIP: 0010:native_safe_halt+0x2e/0x50 [16892.135640] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16892.151710] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16892.160988] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16892.165841] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [16892.177108] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [16892.184552] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [16892.190711] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [16892.196437] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [16892.199378] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16892.204261] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [16892.210960] Call Trace: [16892.212602] [16892.213926] ? show_regs.cold.9+0x22/0x2f [16892.216658] ? watchdog_timer_fn.cold.6+0x62/0x10a [16892.219362] ? __hrtimer_run_queues+0x1c2/0x4d0 [16892.221573] ? watchdog+0x40/0x40 [16892.222991] ? hrtimer_interrupt+0x128/0x2c0 [16892.224824] ? smp_apic_timer_interrupt+0xa6/0x270 [16892.227349] ? apic_timer_interrupt+0xf/0x20 [16892.229232] [16892.230262] ? native_safe_halt+0x2e/0x50 [16892.231822] kvm_wait+0xcd/0xe0 [16892.233755] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16892.235765] do_raw_spin_lock+0x177/0x190 [16892.237267] _raw_spin_lock+0x21/0x30 [16892.238988] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16892.243368] ? do_raw_spin_unlock+0x75/0x190 [16892.245978] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16892.249536] ? do_wait_intr+0xf0/0xf0 [16892.253924] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16892.261330] mdc_reint+0x2d/0x110 [mdc] [16892.263530] mdc_unlink+0x658/0x870 [mdc] [16892.269110] lmv_unlink+0x34d/0xbd0 [lmv] [16892.271544] ll_unlink+0x24d/0x7f0 [lustre] [16892.273876] vfs_unlink+0x202/0x330 [16892.275961] do_unlinkat+0x2fa/0x460 [16892.277740] __x64_sys_unlink+0x37/0x50 [16892.280411] do_syscall_64+0xc1/0x440 [16892.282586] entry_SYSCALL_64_after_hwframe+0x49/0xae [16892.288407] RIP: 0033:0x7f151bdee10b [16892.291545] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [16892.306888] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [16892.313903] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [16892.323227] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [16892.334636] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16892.338689] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16892.348962] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16895.459527] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14964s! [16895.465556] Showing busy workqueues and worker pools: [16895.468271] workqueue events: flags=0x0 [16895.470615] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16895.474270] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [16895.479601] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16895.483533] pending: kfree_rcu_monitor [16895.486011] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16895.489668] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [16895.494668] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16895.497933] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [16895.501961] workqueue events_long: flags=0x0 [16895.504637] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16895.508579] pending: lu_objects_flush [obdclass] [16895.511858] workqueue events_unbound: flags=0x2 [16895.513937] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [16895.517552] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16895.525431] workqueue events_power_efficient: flags=0x80 [16895.529060] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16895.532579] pending: do_cache_clean, neigh_periodic_work, check_lifetime [16895.536001] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16895.539220] pending: neigh_periodic_work [16895.541339] workqueue mm_percpu_wq: flags=0x8 [16895.543659] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16895.546991] pending: vmstat_update, lru_add_drain_per_cpu [16895.551651] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16895.555883] pending: vmstat_update [16895.557171] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16895.559617] pending: lru_add_drain_per_cpu [16895.561136] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [16895.563699] pending: lru_add_drain_per_cpu BAR(48) [16895.565500] workqueue writeback: flags=0x4a [16895.566786] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [16895.568808] pending: wb_workfn [16895.570167] workqueue xprtiod: flags=0xa [16895.571231] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16895.573097] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [16895.576120] workqueue ipv6_addrconf: flags=0x40008 [16895.578086] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16895.580827] pending: addrconf_verify_work [16895.582472] workqueue knbd0-recv: flags=0x1a [16895.583837] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16895.586056] in-flight: 658:recv_work [16895.587638] workqueue ptlrpc_pinger: flags=0x2 [16895.589016] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16895.591463] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [16895.594272] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14964s workers=4 idle: 32984 37 8 [16895.597836] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16880s workers=2 idle: 661 [16900.058559] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [16900.064967] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16900.098973] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16900.112057] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16900.119539] RIP: 0010:native_safe_halt+0x2e/0x50 [16900.121970] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16900.138988] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16900.148986] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16900.154731] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [16900.167032] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [16900.186443] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [16900.209153] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [16900.241026] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [16900.283009] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16900.290465] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [16900.306753] Call Trace: [16900.307649] [16900.313838] ? show_regs.cold.9+0x22/0x2f [16900.320655] ? watchdog_timer_fn.cold.6+0x62/0x10a [16900.333924] ? __hrtimer_run_queues+0x1c2/0x4d0 [16900.335501] ? watchdog+0x40/0x40 [16900.357910] ? hrtimer_interrupt+0x128/0x2c0 [16900.359348] ? smp_apic_timer_interrupt+0xa6/0x270 [16900.371044] ? apic_timer_interrupt+0xf/0x20 [16900.372955] [16900.374382] ? native_safe_halt+0x2e/0x50 [16900.375850] kvm_wait+0xcd/0xe0 [16900.377222] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16900.379474] do_raw_spin_lock+0x177/0x190 [16900.381285] _raw_spin_lock+0x21/0x30 [16900.382876] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16900.385317] ? do_raw_spin_unlock+0x75/0x190 [16900.387000] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16900.389111] ? do_wait_intr+0xf0/0xf0 [16900.390562] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16900.392653] mdc_close+0x53b/0xd70 [mdc] [16900.394585] lmv_close+0x18d/0x490 [lmv] [16900.396079] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [16900.398416] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16900.415995] ll_md_real_close+0xa5/0x2a0 [lustre] [16900.417744] ll_md_close+0x274/0x800 [lustre] [16900.419890] ll_file_release+0x128/0xc50 [lustre] [16900.421845] __fput+0x10d/0x440 [16900.423078] ____fput+0x12/0x20 [16900.424563] task_work_run+0xc6/0x110 [16900.426300] exit_to_usermode_loop+0x1dd/0x1f0 [16900.428314] do_syscall_64+0x3d6/0x440 [16900.429766] entry_SYSCALL_64_after_hwframe+0x49/0xae [16900.433913] RIP: 0033:0x7f151bee3cd8 [16900.435406] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16900.443072] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16900.446281] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [16900.449257] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [16900.452258] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16900.455250] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16900.458096] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16912.057092] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [16912.065514] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [16912.073077] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16912.073454] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16912.073463] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16912.073470] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16912.073765] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16912.073776] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16912.073781] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16912.073785] RAX: 0000000000005107 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16912.073788] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [16912.073790] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [16912.073792] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [16912.073795] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [16912.073798] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [16912.073801] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16912.073804] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16912.073812] Call Trace: [16912.073818] [16912.073823] ? show_regs.cold.9+0x22/0x2f [16912.073832] ? watchdog_timer_fn.cold.6+0x62/0x10a [16912.073838] ? __hrtimer_run_queues+0x1c2/0x4d0 [16912.073842] ? watchdog+0x40/0x40 [16912.073847] ? hrtimer_interrupt+0x128/0x2c0 [16912.073851] ? smp_apic_timer_interrupt+0xa6/0x270 [16912.073859] ? apic_timer_interrupt+0xf/0x20 [16912.073863] [16912.073866] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16912.073871] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16912.073876] do_raw_spin_lock+0x177/0x190 [16912.073881] _raw_spin_lock+0x21/0x30 [16912.073886] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16912.075436] process_one_work+0x2c8/0x700 [16912.075444] worker_thread+0x296/0x6e0 [16912.075449] ? rescuer_thread+0x570/0x570 [16912.075453] kthread+0x1d1/0x200 [16912.075458] ? set_kthread_struct+0x70/0x70 [16912.075462] ret_from_fork+0x1f/0x30 [16912.421269] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16912.446430] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16912.451491] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16912.456333] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16912.459883] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16912.471831] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16912.475957] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [16912.479834] RDX: 000000000000250b RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [16912.483953] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [16912.487799] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [16912.491365] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16912.493966] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [16912.498293] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16912.500959] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16912.504032] Call Trace: [16912.505539] [16912.507532] ? show_regs.cold.9+0x22/0x2f [16912.509689] ? watchdog_timer_fn.cold.6+0x62/0x10a [16912.512425] ? __hrtimer_run_queues+0x1c2/0x4d0 [16912.514482] ? watchdog+0x40/0x40 [16912.516515] ? hrtimer_interrupt+0x128/0x2c0 [16912.518329] ? smp_apic_timer_interrupt+0xa6/0x270 [16912.521286] ? apic_timer_interrupt+0xf/0x20 [16912.523284] [16912.524483] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16912.527573] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16912.530378] do_raw_spin_lock+0x177/0x190 [16912.532913] _raw_spin_lock+0x21/0x30 [16912.534899] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16912.538038] obd_mod_free+0x46/0x140 [ptlrpc] [16912.540783] mdc_commit_open+0x91/0xb0 [mdc] [16912.542904] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16912.545929] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16912.550303] after_reply+0xe2c/0x1780 [ptlrpc] [16912.554358] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16912.557787] ? __next_timer_interrupt+0x160/0x160 [16912.559975] ptlrpcd+0xa82/0xe40 [ptlrpc] [16912.562965] ? do_wait_intr+0xf0/0xf0 [16912.564989] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16912.568760] kthread+0x1d1/0x200 [16912.570414] ? set_kthread_struct+0x70/0x70 [16912.572553] ret_from_fork+0x1f/0x30 [16920.054135] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [16920.057745] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16920.079886] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16920.090746] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16920.099034] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16920.108636] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16920.129587] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 [16920.134647] RAX: 0000000000002000 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16920.139736] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [16920.143848] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [16920.147370] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [16920.150688] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [16920.160883] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [16920.169720] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16920.175852] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [16920.182544] Call Trace: [16920.183639] [16920.184610] ? show_regs.cold.9+0x22/0x2f [16920.188195] ? watchdog_timer_fn.cold.6+0x62/0x10a [16920.191088] ? __hrtimer_run_queues+0x1c2/0x4d0 [16920.193592] ? watchdog+0x40/0x40 [16920.195751] ? hrtimer_interrupt+0x128/0x2c0 [16920.197955] ? smp_apic_timer_interrupt+0xa6/0x270 [16920.200063] ? apic_timer_interrupt+0xf/0x20 [16920.202184] [16920.203104] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16920.206116] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16920.213173] do_raw_spin_lock+0x177/0x190 [16920.216573] _raw_spin_lock+0x21/0x30 [16920.219597] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16920.223124] ? do_raw_spin_unlock+0x75/0x190 [16920.225974] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16920.229738] ? do_wait_intr+0xf0/0xf0 [16920.242609] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16920.244471] mdc_reint+0x2d/0x110 [mdc] [16920.249301] mdc_unlink+0x658/0x870 [mdc] [16920.256531] lmv_unlink+0x34d/0xbd0 [lmv] [16920.262726] ll_unlink+0x24d/0x7f0 [lustre] [16920.272131] vfs_unlink+0x202/0x330 [16920.275508] do_unlinkat+0x2fa/0x460 [16920.281815] __x64_sys_unlink+0x37/0x50 [16920.283513] do_syscall_64+0xc1/0x440 [16920.290972] entry_SYSCALL_64_after_hwframe+0x49/0xae [16920.293673] RIP: 0033:0x7f151bdee10b [16920.297698] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [16920.320945] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [16920.332422] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [16920.339129] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [16920.349489] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16920.358335] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16920.368697] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16924.958327] rcu: INFO: rcu_sched self-detected stall on CPU [16924.960652] rcu: 1-...!: (9008555 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [16924.964957] (t=15000311 jiffies g=249657 q=6996) [16924.966731] rcu: rcu_sched kthread starved for 14995263 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [16924.971581] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [16924.975606] rcu: RCU grace-period kthread stack dump: [16924.978205] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [16924.982547] Call Trace: [16924.983725] __schedule+0x351/0xcb0 [16924.985370] schedule+0xc0/0x180 [16924.986989] schedule_timeout+0xb4/0x190 [16924.988739] ? do_raw_spin_unlock+0x75/0x190 [16924.990760] ? __next_timer_interrupt+0x160/0x160 [16924.993119] ? prepare_to_swait_event+0xf3/0x140 [16924.995479] rcu_gp_kthread+0x8dd/0x15b0 [16924.997278] ? force_qs_rnp+0x2c0/0x2c0 [16924.998893] kthread+0x1d1/0x200 [16925.000467] ? set_kthread_struct+0x70/0x70 [16925.002557] ret_from_fork+0x1f/0x30 [16925.004337] rcu: Stack dump where RCU GP kthread last ran: [16925.007268] NMI backtrace for cpu 1 [16925.008761] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16925.013967] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16925.017969] Call Trace: [16925.019168] [16925.020363] ? dump_stack+0xbb/0x10e [16925.021836] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [16925.024333] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [16925.026423] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [16925.028606] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [16925.030698] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [16925.033390] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [16925.035714] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [16925.037829] ? update_process_times+0x6d/0xc0 [16925.039771] ? tick_sched_handle.isra.18+0x27/0x90 [16925.041577] ? tick_sched_timer+0x53/0xb0 [16925.043322] ? __hrtimer_run_queues+0x1c2/0x4d0 [16925.045243] ? tick_sched_handle.isra.18+0x90/0x90 [16925.046987] ? hrtimer_interrupt+0x128/0x2c0 [16925.048756] ? smp_apic_timer_interrupt+0xa6/0x270 [16925.050794] ? apic_timer_interrupt+0xf/0x20 [16925.052655] [16925.053519] ? native_safe_halt+0x2e/0x50 [16925.055357] ? kvm_wait+0xcd/0xe0 [16925.056560] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16925.058889] ? do_raw_spin_lock+0x177/0x190 [16925.060737] ? _raw_spin_lock+0x21/0x30 [16925.062569] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16925.065604] ? obd_mod_free+0x46/0x140 [ptlrpc] [16925.067899] ? mdc_commit_open+0x91/0xb0 [mdc] [16925.069657] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16925.072357] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16925.074834] ? after_reply+0xe2c/0x1780 [ptlrpc] [16925.079399] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16925.084370] ? __next_timer_interrupt+0x160/0x160 [16925.087349] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [16925.089771] ? do_wait_intr+0xf0/0xf0 [16925.091711] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16925.096194] ? kthread+0x1d1/0x200 [16925.097848] ? set_kthread_struct+0x70/0x70 [16925.099732] ? ret_from_fork+0x1f/0x30 [16925.101675] NMI backtrace for cpu 1 [16925.103511] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16925.108315] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16925.116613] Call Trace: [16925.120697] [16925.122111] ? dump_stack+0xbb/0x10e [16925.123440] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [16925.126076] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [16925.128535] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [16925.130710] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [16925.133569] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [16925.136452] ? rcu_dump_cpu_stacks+0x11b/0x171 [16925.138435] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [16925.140440] ? update_process_times+0x6d/0xc0 [16925.142291] ? tick_sched_handle.isra.18+0x27/0x90 [16925.144727] ? tick_sched_timer+0x53/0xb0 [16925.146513] ? __hrtimer_run_queues+0x1c2/0x4d0 [16925.148602] ? tick_sched_handle.isra.18+0x90/0x90 [16925.150612] ? hrtimer_interrupt+0x128/0x2c0 [16925.152383] ? smp_apic_timer_interrupt+0xa6/0x270 [16925.154484] ? apic_timer_interrupt+0xf/0x20 [16925.156237] [16925.156861] ? native_safe_halt+0x2e/0x50 [16925.159075] ? kvm_wait+0xcd/0xe0 [16925.160723] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16925.163287] ? do_raw_spin_lock+0x177/0x190 [16925.164602] ? _raw_spin_lock+0x21/0x30 [16925.165667] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16925.170111] ? obd_mod_free+0x46/0x140 [ptlrpc] [16925.173539] ? mdc_commit_open+0x91/0xb0 [mdc] [16925.174834] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16925.179263] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16925.182150] ? after_reply+0xe2c/0x1780 [ptlrpc] [16925.185384] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16925.188707] ? __next_timer_interrupt+0x160/0x160 [16925.191881] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [16925.194602] ? do_wait_intr+0xf0/0xf0 [16925.196470] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16925.199594] ? kthread+0x1d1/0x200 [16925.201914] ? set_kthread_struct+0x70/0x70 [16925.203912] ? ret_from_fork+0x1f/0x30 [16925.666143] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14995s! [16925.670737] Showing busy workqueues and worker pools: [16925.673841] workqueue events: flags=0x0 [16925.676090] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16925.679160] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [16925.684661] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16925.690672] pending: kfree_rcu_monitor [16925.696251] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16925.700237] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [16925.708127] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16925.711159] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [16925.714554] workqueue events_long: flags=0x0 [16925.723548] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16925.730364] pending: lu_objects_flush [obdclass] [16925.736409] workqueue events_unbound: flags=0x2 [16925.738360] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [16925.744916] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16925.755377] workqueue events_power_efficient: flags=0x80 [16925.761023] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16925.767082] pending: do_cache_clean, neigh_periodic_work, check_lifetime [16925.780811] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16925.784649] pending: neigh_periodic_work [16925.789569] workqueue mm_percpu_wq: flags=0x8 [16925.792444] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16925.800782] pending: vmstat_update, lru_add_drain_per_cpu [16925.804553] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16925.807470] pending: vmstat_update [16925.809361] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16925.812435] pending: lru_add_drain_per_cpu [16925.814574] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [16925.817216] pending: lru_add_drain_per_cpu BAR(48) [16925.819680] workqueue writeback: flags=0x4a [16925.821673] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [16925.824701] pending: wb_workfn [16925.828548] workqueue xprtiod: flags=0xa [16925.830539] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16925.837454] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [16925.844656] workqueue ipv6_addrconf: flags=0x40008 [16925.850048] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16925.856426] pending: addrconf_verify_work [16925.861605] workqueue knbd0-recv: flags=0x1a [16925.864536] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16925.868120] in-flight: 658:recv_work [16925.870101] workqueue ptlrpc_pinger: flags=0x2 [16925.871700] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16925.874476] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [16925.876550] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14995s workers=4 idle: 32984 37 8 [16925.880316] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16910s workers=2 idle: 661 [16928.057717] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [16928.063169] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16928.143586] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16928.158309] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16928.171273] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16928.176533] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16928.204558] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16928.223332] RAX: 000000000000760e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16928.238740] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [16928.253464] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [16928.274184] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [16928.278364] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [16928.291355] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [16928.301799] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16928.308778] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [16928.325375] Call Trace: [16928.326534] [16928.329281] ? show_regs.cold.9+0x22/0x2f [16928.336685] ? watchdog_timer_fn.cold.6+0x62/0x10a [16928.346454] ? __hrtimer_run_queues+0x1c2/0x4d0 [16928.351517] ? watchdog+0x40/0x40 [16928.357622] ? hrtimer_interrupt+0x128/0x2c0 [16928.363852] ? smp_apic_timer_interrupt+0xa6/0x270 [16928.371959] ? apic_timer_interrupt+0xf/0x20 [16928.377352] [16928.384561] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16928.394973] do_raw_spin_lock+0x177/0x190 [16928.401335] _raw_spin_lock+0x21/0x30 [16928.409262] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16928.421386] ? do_raw_spin_unlock+0x75/0x190 [16928.425692] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16928.436395] ? do_wait_intr+0xf0/0xf0 [16928.444934] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16928.449436] mdc_close+0x53b/0xd70 [mdc] [16928.462813] lmv_close+0x18d/0x490 [lmv] [16928.471006] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [16928.479654] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16928.487480] ll_md_real_close+0xa5/0x2a0 [lustre] [16928.496013] ll_md_close+0x274/0x800 [lustre] [16928.504634] ll_file_release+0x128/0xc50 [lustre] [16928.511938] __fput+0x10d/0x440 [16928.531312] ____fput+0x12/0x20 [16928.536695] task_work_run+0xc6/0x110 [16928.549861] exit_to_usermode_loop+0x1dd/0x1f0 [16928.566292] do_syscall_64+0x3d6/0x440 [16928.573168] entry_SYSCALL_64_after_hwframe+0x49/0xae [16928.580678] RIP: 0033:0x7f151bee3cd8 [16928.597458] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16928.644871] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16928.665285] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [16928.672677] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [16928.690555] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16928.701629] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16928.726584] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16940.056113] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [16940.059473] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16940.110203] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16940.117272] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16940.125866] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16940.129319] RIP: 0010:native_safe_halt+0x2e/0x50 [16940.131605] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16940.145278] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16940.156580] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16940.167150] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [16940.178038] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [16940.180784] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [16940.187334] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [16940.193082] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [16940.197843] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16940.202275] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16940.206966] Call Trace: [16940.208862] [16940.209517] ? show_regs.cold.9+0x22/0x2f [16940.213654] ? watchdog_timer_fn.cold.6+0x62/0x10a [16940.217071] ? __hrtimer_run_queues+0x1c2/0x4d0 [16940.218474] ? watchdog+0x40/0x40 [16940.219509] ? hrtimer_interrupt+0x128/0x2c0 [16940.221056] ? smp_apic_timer_interrupt+0xa6/0x270 [16940.223047] ? apic_timer_interrupt+0xf/0x20 [16940.224928] [16940.225758] ? native_safe_halt+0x2e/0x50 [16940.227282] kvm_wait+0xcd/0xe0 [16940.228259] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16940.230296] do_raw_spin_lock+0x177/0x190 [16940.231957] _raw_spin_lock+0x21/0x30 [16940.233136] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16940.235178] process_one_work+0x2c8/0x700 [16940.237156] worker_thread+0x296/0x6e0 [16940.238791] ? rescuer_thread+0x570/0x570 [16940.240363] kthread+0x1d1/0x200 [16940.241552] ? set_kthread_struct+0x70/0x70 [16940.243016] ret_from_fork+0x1f/0x30 [16952.052529] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [16952.055048] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [16952.057760] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16952.057844] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16952.057852] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16952.057855] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16952.057868] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16952.057872] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16952.057876] RAX: 0000000000006e22 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16952.057879] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [16952.057881] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [16952.057883] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [16952.057886] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [16952.057889] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [16952.057892] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16952.057894] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [16952.057908] Call Trace: [16952.057914] [16952.057920] ? show_regs.cold.9+0x22/0x2f [16952.057930] ? watchdog_timer_fn.cold.6+0x62/0x10a [16952.057937] ? __hrtimer_run_queues+0x1c2/0x4d0 [16952.057942] ? watchdog+0x40/0x40 [16952.057946] ? hrtimer_interrupt+0x128/0x2c0 [16952.057950] ? smp_apic_timer_interrupt+0xa6/0x270 [16952.057958] ? apic_timer_interrupt+0xf/0x20 [16952.057963] [16952.057967] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16952.057972] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16952.057976] do_raw_spin_lock+0x177/0x190 [16952.057981] _raw_spin_lock+0x21/0x30 [16952.057998] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16952.058555] ? do_raw_spin_unlock+0x75/0x190 [16952.058562] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16952.058782] ? do_wait_intr+0xf0/0xf0 [16952.058790] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16952.059187] mdc_reint+0x2d/0x110 [mdc] [16952.059224] mdc_unlink+0x658/0x870 [mdc] [16952.059252] lmv_unlink+0x34d/0xbd0 [lmv] [16952.059281] ll_unlink+0x24d/0x7f0 [lustre] [16952.059400] vfs_unlink+0x202/0x330 [16952.059409] do_unlinkat+0x2fa/0x460 [16952.059414] __x64_sys_unlink+0x37/0x50 [16952.059418] do_syscall_64+0xc1/0x440 [16952.059425] entry_SYSCALL_64_after_hwframe+0x49/0xae [16952.059431] RIP: 0033:0x7f151bdee10b [16952.059438] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [16952.059442] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [16952.059446] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [16952.059448] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [16952.059450] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16952.059452] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16952.059454] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16952.418385] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16952.433546] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16952.437129] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16952.440287] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16952.442342] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16952.449138] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16952.457268] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [16952.463604] RDX: 0000000000007375 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [16952.467794] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [16952.471393] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [16952.476288] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16952.482097] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [16952.486371] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16952.492682] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16952.495518] Call Trace: [16952.496880] [16952.497550] ? show_regs.cold.9+0x22/0x2f [16952.499542] ? watchdog_timer_fn.cold.6+0x62/0x10a [16952.501515] ? __hrtimer_run_queues+0x1c2/0x4d0 [16952.506118] ? watchdog+0x40/0x40 [16952.511361] ? hrtimer_interrupt+0x128/0x2c0 [16952.514383] ? smp_apic_timer_interrupt+0xa6/0x270 [16952.525062] ? apic_timer_interrupt+0xf/0x20 [16952.530430] [16952.535968] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16952.541523] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16952.547939] do_raw_spin_lock+0x177/0x190 [16952.560381] _raw_spin_lock+0x21/0x30 [16952.568598] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16952.572328] obd_mod_free+0x46/0x140 [ptlrpc] [16952.576216] mdc_commit_open+0x91/0xb0 [mdc] [16952.579304] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16952.585615] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16952.595227] after_reply+0xe2c/0x1780 [ptlrpc] [16952.602339] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16952.605660] ? __next_timer_interrupt+0x160/0x160 [16952.607826] ptlrpcd+0xa82/0xe40 [ptlrpc] [16952.610447] ? do_wait_intr+0xf0/0xf0 [16952.615535] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16952.622901] kthread+0x1d1/0x200 [16952.625196] ? set_kthread_struct+0x70/0x70 [16952.629465] ret_from_fork+0x1f/0x30 [16956.057058] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [16956.060321] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16956.095873] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16956.101749] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16956.106647] RIP: 0010:native_safe_halt+0x2e/0x50 [16956.109264] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16956.119310] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16956.122779] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16956.125995] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [16956.129822] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [16956.135799] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [16956.141823] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [16956.147921] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [16956.153891] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16956.156788] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [16956.164794] Call Trace: [16956.166104] [16956.167626] ? show_regs.cold.9+0x22/0x2f [16956.169955] ? watchdog_timer_fn.cold.6+0x62/0x10a [16956.172711] ? __hrtimer_run_queues+0x1c2/0x4d0 [16956.174907] ? watchdog+0x40/0x40 [16956.183282] ? hrtimer_interrupt+0x128/0x2c0 [16956.190055] ? smp_apic_timer_interrupt+0xa6/0x270 [16956.191846] ? apic_timer_interrupt+0xf/0x20 [16956.194200] [16956.195252] ? native_safe_halt+0x2e/0x50 [16956.197339] kvm_wait+0xcd/0xe0 [16956.199058] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16956.206100] do_raw_spin_lock+0x177/0x190 [16956.211461] _raw_spin_lock+0x21/0x30 [16956.213742] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16956.220240] ? do_raw_spin_unlock+0x75/0x190 [16956.223208] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16956.226262] ? do_wait_intr+0xf0/0xf0 [16956.232006] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16956.239011] mdc_close+0x53b/0xd70 [mdc] [16956.241951] lmv_close+0x18d/0x490 [lmv] [16956.246245] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [16956.250168] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16956.253093] ll_md_real_close+0xa5/0x2a0 [lustre] [16956.256936] ll_md_close+0x274/0x800 [lustre] [16956.260800] ll_file_release+0x128/0xc50 [lustre] [16956.266193] __fput+0x10d/0x440 [16956.269959] ____fput+0x12/0x20 [16956.270774] task_work_run+0xc6/0x110 [16956.271653] exit_to_usermode_loop+0x1dd/0x1f0 [16956.276160] do_syscall_64+0x3d6/0x440 [16956.277452] entry_SYSCALL_64_after_hwframe+0x49/0xae [16956.282235] RIP: 0033:0x7f151bee3cd8 [16956.284691] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16956.299135] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16956.307434] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [16956.317130] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [16956.321868] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16956.328152] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16956.338815] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16956.387041] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15025s! [16956.395214] Showing busy workqueues and worker pools: [16956.400546] workqueue events: flags=0x0 [16956.407993] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16956.411745] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [16956.418875] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16956.422248] pending: kfree_rcu_monitor [16956.433553] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16956.439959] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [16956.453285] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16956.462017] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [16956.468397] workqueue events_long: flags=0x0 [16956.472227] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16956.476663] pending: lu_objects_flush [obdclass] [16956.481489] workqueue events_unbound: flags=0x2 [16956.483655] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [16956.485973] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16956.490787] workqueue events_power_efficient: flags=0x80 [16956.492710] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16956.496083] pending: do_cache_clean, neigh_periodic_work, check_lifetime [16956.503241] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16956.510950] pending: neigh_periodic_work [16956.513931] workqueue mm_percpu_wq: flags=0x8 [16956.515538] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16956.522283] pending: vmstat_update, lru_add_drain_per_cpu [16956.529066] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16956.532220] pending: vmstat_update [16956.534543] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16956.538768] pending: lru_add_drain_per_cpu [16956.541314] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [16956.544758] pending: lru_add_drain_per_cpu BAR(48) [16956.547105] workqueue writeback: flags=0x4a [16956.549111] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [16956.552033] pending: wb_workfn [16956.553788] workqueue xprtiod: flags=0xa [16956.555824] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16956.558962] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [16956.562656] workqueue ipv6_addrconf: flags=0x40008 [16956.565077] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16956.568720] pending: addrconf_verify_work [16956.570736] workqueue knbd0-recv: flags=0x1a [16956.573580] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16956.580703] in-flight: 658:recv_work [16956.583319] workqueue ptlrpc_pinger: flags=0x2 [16956.589248] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16956.592999] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [16956.595521] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15025s workers=4 idle: 32984 37 8 [16956.600620] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16941s workers=2 idle: 661 [16968.056027] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [16968.060433] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16968.083137] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16968.093921] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16968.102930] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16968.109148] RIP: 0010:native_safe_halt+0x2e/0x50 [16968.111309] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16968.120980] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16968.124265] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16968.127410] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [16968.131618] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [16968.138687] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [16968.142913] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [16968.145842] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [16968.150081] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16968.154983] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16968.159826] Call Trace: [16968.161246] [16968.162305] ? show_regs.cold.9+0x22/0x2f [16968.168875] ? watchdog_timer_fn.cold.6+0x62/0x10a [16968.171324] ? __hrtimer_run_queues+0x1c2/0x4d0 [16968.173564] ? watchdog+0x40/0x40 [16968.175350] ? hrtimer_interrupt+0x128/0x2c0 [16968.178157] ? smp_apic_timer_interrupt+0xa6/0x270 [16968.184222] ? apic_timer_interrupt+0xf/0x20 [16968.190414] [16968.191889] ? native_safe_halt+0x2e/0x50 [16968.195925] kvm_wait+0xcd/0xe0 [16968.200759] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16968.204379] do_raw_spin_lock+0x177/0x190 [16968.207191] _raw_spin_lock+0x21/0x30 [16968.215032] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16968.219899] process_one_work+0x2c8/0x700 [16968.224172] worker_thread+0x296/0x6e0 [16968.236011] ? rescuer_thread+0x570/0x570 [16968.241173] kthread+0x1d1/0x200 [16968.247214] ? set_kthread_struct+0x70/0x70 [16968.254410] ret_from_fork+0x1f/0x30 [16980.055228] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [16980.059674] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16980.084889] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16980.091170] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16980.095933] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16980.098955] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16980.108522] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16980.112224] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [16980.115901] RDX: 0000000000000309 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [16980.119199] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [16980.122973] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [16980.126404] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16980.129624] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [16980.134235] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16980.137422] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16980.141138] Call Trace: [16980.143019] [16980.143755] ? show_regs.cold.9+0x22/0x2f [16980.146021] ? watchdog_timer_fn.cold.6+0x62/0x10a [16980.148754] ? __hrtimer_run_queues+0x1c2/0x4d0 [16980.151235] ? watchdog+0x40/0x40 [16980.152977] ? hrtimer_interrupt+0x128/0x2c0 [16980.155498] ? smp_apic_timer_interrupt+0xa6/0x270 [16980.158511] ? apic_timer_interrupt+0xf/0x20 [16980.160588] [16980.161888] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16980.163952] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16980.167211] do_raw_spin_lock+0x177/0x190 [16980.170025] _raw_spin_lock+0x21/0x30 [16980.171976] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16980.174953] obd_mod_free+0x46/0x140 [ptlrpc] [16980.177965] mdc_commit_open+0x91/0xb0 [mdc] [16980.180479] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16980.184859] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16980.188180] after_reply+0xe2c/0x1780 [ptlrpc] [16980.191765] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16980.195744] ? __next_timer_interrupt+0x160/0x160 [16980.198176] ptlrpcd+0xa82/0xe40 [ptlrpc] [16980.201423] ? do_wait_intr+0xf0/0xf0 [16980.203729] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16980.206797] kthread+0x1d1/0x200 [16980.208690] ? set_kthread_struct+0x70/0x70 [16980.210987] ret_from_fork+0x1f/0x30 [16984.052671] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [16984.055094] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16984.059017] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [16984.095099] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16984.095119] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16984.095195] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16984.095211] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16984.095215] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16984.095221] RAX: 000000000000248f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16984.095224] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [16984.095226] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [16984.095229] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [16984.095232] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [16984.095235] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [16984.095239] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16984.095241] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [16984.095250] Call Trace: [16984.095256] [16984.095262] ? show_regs.cold.9+0x22/0x2f [16984.095274] ? watchdog_timer_fn.cold.6+0x62/0x10a [16984.095281] ? __hrtimer_run_queues+0x1c2/0x4d0 [16984.095286] ? watchdog+0x40/0x40 [16984.095290] ? hrtimer_interrupt+0x128/0x2c0 [16984.095363] ? smp_apic_timer_interrupt+0xa6/0x270 [16984.095374] ? apic_timer_interrupt+0xf/0x20 [16984.095379] [16984.095382] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16984.095387] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16984.095392] do_raw_spin_lock+0x177/0x190 [16984.095398] _raw_spin_lock+0x21/0x30 [16984.095406] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16984.095825] ? do_raw_spin_unlock+0x75/0x190 [16984.095831] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16984.096250] ? do_wait_intr+0xf0/0xf0 [16984.096259] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16984.096619] mdc_reint+0x2d/0x110 [mdc] [16984.096786] mdc_unlink+0x658/0x870 [mdc] [16984.096818] lmv_unlink+0x34d/0xbd0 [lmv] [16984.096856] ll_unlink+0x24d/0x7f0 [lustre] [16984.096987] vfs_unlink+0x202/0x330 [16984.096996] do_unlinkat+0x2fa/0x460 [16984.097001] __x64_sys_unlink+0x37/0x50 [16984.097005] do_syscall_64+0xc1/0x440 [16984.097012] entry_SYSCALL_64_after_hwframe+0x49/0xae [16984.097019] RIP: 0033:0x7f151bdee10b [16984.097027] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [16984.097032] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [16984.097037] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [16984.097040] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [16984.097042] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16984.097044] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16984.097046] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16984.322064] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16984.366608] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16984.380393] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16984.387535] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16984.394801] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16984.411718] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [16984.417576] RAX: 0000000000007584 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16984.427100] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [16984.439918] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [16984.444125] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [16984.451556] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [16984.466584] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [16984.475620] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16984.478208] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [16984.483072] Call Trace: [16984.484482] [16984.486097] ? show_regs.cold.9+0x22/0x2f [16984.489141] ? watchdog_timer_fn.cold.6+0x62/0x10a [16984.495270] ? __hrtimer_run_queues+0x1c2/0x4d0 [16984.498651] ? watchdog+0x40/0x40 [16984.503126] ? hrtimer_interrupt+0x128/0x2c0 [16984.505892] ? smp_apic_timer_interrupt+0xa6/0x270 [16984.509450] ? apic_timer_interrupt+0xf/0x20 [16984.512076] [16984.514075] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16984.521715] do_raw_spin_lock+0x177/0x190 [16984.526970] _raw_spin_lock+0x21/0x30 [16984.533399] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16984.540220] ? do_raw_spin_unlock+0x75/0x190 [16984.544999] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16984.551461] ? do_wait_intr+0xf0/0xf0 [16984.555539] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16984.560450] mdc_close+0x53b/0xd70 [mdc] [16984.565451] lmv_close+0x18d/0x490 [lmv] [16984.566774] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [16984.577778] ll_close_inode_openhandle+0x493/0xea0 [lustre] [16984.581724] ll_md_real_close+0xa5/0x2a0 [lustre] [16984.585467] ll_md_close+0x274/0x800 [lustre] [16984.594982] ll_file_release+0x128/0xc50 [lustre] [16984.603756] __fput+0x10d/0x440 [16984.610181] ____fput+0x12/0x20 [16984.617519] task_work_run+0xc6/0x110 [16984.632354] exit_to_usermode_loop+0x1dd/0x1f0 [16984.637876] do_syscall_64+0x3d6/0x440 [16984.646178] entry_SYSCALL_64_after_hwframe+0x49/0xae [16984.658165] RIP: 0033:0x7f151bee3cd8 [16984.659369] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [16984.678402] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [16984.686528] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [16984.694102] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [16984.704395] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [16984.710337] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [16984.714391] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [16987.105122] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15056s! [16987.114207] Showing busy workqueues and worker pools: [16987.119013] workqueue events: flags=0x0 [16987.121189] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16987.126492] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [16987.134522] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16987.141175] pending: kfree_rcu_monitor [16987.144755] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16987.150482] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [16987.158814] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16987.162733] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [16987.174541] workqueue events_long: flags=0x0 [16987.178090] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16987.185495] pending: lu_objects_flush [obdclass] [16987.190511] workqueue events_unbound: flags=0x2 [16987.198620] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [16987.206630] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [16987.219192] workqueue events_power_efficient: flags=0x80 [16987.222169] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16987.228450] pending: do_cache_clean, neigh_periodic_work, check_lifetime [16987.232260] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16987.237069] pending: neigh_periodic_work [16987.241884] workqueue mm_percpu_wq: flags=0x8 [16987.246866] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16987.254203] pending: vmstat_update, lru_add_drain_per_cpu [16987.260085] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16987.267536] pending: vmstat_update [16987.271175] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16987.278460] pending: lru_add_drain_per_cpu [16987.283329] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [16987.293144] pending: lru_add_drain_per_cpu BAR(48) [16987.299232] workqueue writeback: flags=0x4a [16987.302264] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [16987.306467] pending: wb_workfn [16987.311879] workqueue xprtiod: flags=0xa [16987.320108] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [16987.329627] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [16987.337623] workqueue ipv6_addrconf: flags=0x40008 [16987.341507] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16987.349335] pending: addrconf_verify_work [16987.354552] workqueue knbd0-recv: flags=0x1a [16987.357420] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16987.366217] in-flight: 658:recv_work [16987.369536] workqueue ptlrpc_pinger: flags=0x2 [16987.374228] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16987.379273] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [16987.387213] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15056s workers=4 idle: 32984 37 8 [16987.393936] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16972s workers=2 idle: 661 [16996.056572] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [16996.061043] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16996.086056] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16996.091933] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16996.096308] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16996.099840] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16996.102845] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16996.112995] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16996.116875] RAX: 0000000000004e1c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [16996.120457] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [16996.124555] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [16996.127921] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [16996.131568] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [16996.135543] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [16996.139923] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16996.142566] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [16996.146577] Call Trace: [16996.147850] [16996.149022] ? show_regs.cold.9+0x22/0x2f [16996.151153] ? watchdog_timer_fn.cold.6+0x62/0x10a [16996.153301] ? __hrtimer_run_queues+0x1c2/0x4d0 [16996.155719] ? watchdog+0x40/0x40 [16996.157366] ? hrtimer_interrupt+0x128/0x2c0 [16996.159581] ? smp_apic_timer_interrupt+0xa6/0x270 [16996.162211] ? apic_timer_interrupt+0xf/0x20 [16996.164319] [16996.165558] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16996.168017] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16996.170739] do_raw_spin_lock+0x177/0x190 [16996.173143] _raw_spin_lock+0x21/0x30 [16996.174718] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16996.177273] process_one_work+0x2c8/0x700 [16996.179147] worker_thread+0x296/0x6e0 [16996.181333] ? rescuer_thread+0x570/0x570 [16996.183009] kthread+0x1d1/0x200 [16996.184642] ? set_kthread_struct+0x70/0x70 [16996.186859] ret_from_fork+0x1f/0x30 [17008.055038] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [17008.060942] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17008.089744] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17008.097383] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17008.101435] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17008.105429] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17008.113625] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17008.119056] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [17008.120859] RDX: 00000000000022cd RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [17008.124840] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [17008.128699] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [17008.132618] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17008.135984] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [17008.140308] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17008.142780] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17008.147354] Call Trace: [17008.150455] [17008.151444] ? show_regs.cold.9+0x22/0x2f [17008.153953] ? watchdog_timer_fn.cold.6+0x62/0x10a [17008.157892] ? __hrtimer_run_queues+0x1c2/0x4d0 [17008.159883] ? watchdog+0x40/0x40 [17008.163267] ? hrtimer_interrupt+0x128/0x2c0 [17008.164805] ? smp_apic_timer_interrupt+0xa6/0x270 [17008.166743] ? apic_timer_interrupt+0xf/0x20 [17008.171770] [17008.172816] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17008.175914] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17008.177340] do_raw_spin_lock+0x177/0x190 [17008.179885] _raw_spin_lock+0x21/0x30 [17008.182413] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17008.185279] obd_mod_free+0x46/0x140 [ptlrpc] [17008.187026] mdc_commit_open+0x91/0xb0 [mdc] [17008.189742] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17008.193896] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17008.197953] after_reply+0xe2c/0x1780 [ptlrpc] [17008.201272] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17008.203118] ? __next_timer_interrupt+0x160/0x160 [17008.204651] ptlrpcd+0xa82/0xe40 [ptlrpc] [17008.206353] ? do_wait_intr+0xf0/0xf0 [17008.207601] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17008.209629] kthread+0x1d1/0x200 [17008.211141] ? set_kthread_struct+0x70/0x70 [17008.212943] ret_from_fork+0x1f/0x30 [17012.052066] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [17012.055595] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17012.057268] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [17012.076605] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17012.089243] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17012.089322] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17012.089330] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17012.089333] RIP: 0010:native_safe_halt+0x2e/0x50 [17012.089346] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17012.089350] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [17012.089355] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17012.089357] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [17012.089359] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [17012.089361] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [17012.089364] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [17012.089367] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [17012.089370] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17012.089373] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [17012.089382] Call Trace: [17012.089387] [17012.089393] ? show_regs.cold.9+0x22/0x2f [17012.089567] ? watchdog_timer_fn.cold.6+0x62/0x10a [17012.089575] ? __hrtimer_run_queues+0x1c2/0x4d0 [17012.089579] ? watchdog+0x40/0x40 [17012.089583] ? hrtimer_interrupt+0x128/0x2c0 [17012.089588] ? smp_apic_timer_interrupt+0xa6/0x270 [17012.089595] ? apic_timer_interrupt+0xf/0x20 [17012.089599] [17012.089603] ? native_safe_halt+0x2e/0x50 [17012.089607] kvm_wait+0xcd/0xe0 [17012.089614] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17012.089623] do_raw_spin_lock+0x177/0x190 [17012.089627] _raw_spin_lock+0x21/0x30 [17012.089632] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17012.090607] ? do_raw_spin_unlock+0x75/0x190 [17012.090617] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17012.091235] ? do_wait_intr+0xf0/0xf0 [17012.091619] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17012.092363] mdc_close+0x53b/0xd70 [mdc] [17012.092973] lmv_close+0x18d/0x490 [lmv] [17012.093135] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [17012.093377] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17012.093863] ll_md_real_close+0xa5/0x2a0 [lustre] [17012.093953] ll_md_close+0x274/0x800 [lustre] [17012.104877] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17012.104889] RIP: 0010:native_safe_halt+0x2e/0x50 [17012.105076] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17012.105080] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17012.105087] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17012.105089] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [17012.105091] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [17012.105093] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [17012.105096] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [17012.105099] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [17012.105106] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17012.105108] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [17012.105116] Call Trace: [17012.105122] [17012.105129] ? show_regs.cold.9+0x22/0x2f [17012.105139] ? watchdog_timer_fn.cold.6+0x62/0x10a [17012.105145] ? __hrtimer_run_queues+0x1c2/0x4d0 [17012.105149] ? watchdog+0x40/0x40 [17012.105153] ? hrtimer_interrupt+0x128/0x2c0 [17012.105168] ? smp_apic_timer_interrupt+0xa6/0x270 [17012.105175] ? apic_timer_interrupt+0xf/0x20 [17012.105179] [17012.105182] ? native_safe_halt+0x2e/0x50 [17012.105187] kvm_wait+0xcd/0xe0 [17012.105194] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17012.105201] do_raw_spin_lock+0x177/0x190 [17012.105205] _raw_spin_lock+0x21/0x30 [17012.105212] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17012.105757] ? do_raw_spin_unlock+0x75/0x190 [17012.105763] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17012.106140] ? do_wait_intr+0xf0/0xf0 [17012.106149] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17012.106670] mdc_reint+0x2d/0x110 [mdc] [17012.106712] mdc_unlink+0x658/0x870 [mdc] [17012.106738] lmv_unlink+0x34d/0xbd0 [lmv] [17012.106776] ll_unlink+0x24d/0x7f0 [lustre] [17012.106964] vfs_unlink+0x202/0x330 [17012.106972] do_unlinkat+0x2fa/0x460 [17012.106977] __x64_sys_unlink+0x37/0x50 [17012.106981] do_syscall_64+0xc1/0x440 [17012.107171] entry_SYSCALL_64_after_hwframe+0x49/0xae [17012.107178] RIP: 0033:0x7f151bdee10b [17012.107185] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [17012.107188] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [17012.107192] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [17012.107194] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [17012.107196] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17012.107198] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17012.107200] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17012.469942] ll_file_release+0x128/0xc50 [lustre] [17012.477688] __fput+0x10d/0x440 [17012.479637] ____fput+0x12/0x20 [17012.481086] task_work_run+0xc6/0x110 [17012.483079] exit_to_usermode_loop+0x1dd/0x1f0 [17012.485142] do_syscall_64+0x3d6/0x440 [17012.486957] entry_SYSCALL_64_after_hwframe+0x49/0xae [17012.488775] RIP: 0033:0x7f151bee3cd8 [17012.489956] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17012.496672] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17012.505162] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [17012.508488] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [17012.512124] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17012.515118] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17012.517722] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17017.826093] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15087s! [17017.831303] Showing busy workqueues and worker pools: [17017.834754] workqueue events: flags=0x0 [17017.837706] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17017.842135] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [17017.861808] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17017.871797] pending: kfree_rcu_monitor [17017.873392] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17017.887116] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [17017.903591] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17017.916485] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [17017.922244] workqueue events_long: flags=0x0 [17017.934441] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17017.941461] pending: lu_objects_flush [obdclass] [17017.951361] workqueue events_unbound: flags=0x2 [17017.958877] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [17017.969777] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17017.981907] workqueue events_power_efficient: flags=0x80 [17017.984822] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17017.989194] pending: do_cache_clean, neigh_periodic_work, check_lifetime [17017.994810] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17018.003457] pending: neigh_periodic_work [17018.007616] workqueue mm_percpu_wq: flags=0x8 [17018.010727] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17018.019863] pending: vmstat_update, lru_add_drain_per_cpu [17018.022078] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17018.033524] pending: vmstat_update [17018.034826] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17018.038408] pending: lru_add_drain_per_cpu [17018.040667] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [17018.045204] pending: lru_add_drain_per_cpu BAR(48) [17018.047724] workqueue writeback: flags=0x4a [17018.049714] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [17018.052378] pending: wb_workfn [17018.053882] workqueue xprtiod: flags=0xa [17018.055830] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17018.058843] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [17018.062640] workqueue ipv6_addrconf: flags=0x40008 [17018.064816] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17018.070167] pending: addrconf_verify_work [17018.071794] workqueue knbd0-recv: flags=0x1a [17018.082439] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17018.091589] in-flight: 658:recv_work [17018.093912] workqueue ptlrpc_pinger: flags=0x2 [17018.096767] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17018.102770] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [17018.106882] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15087s workers=4 idle: 32984 37 8 [17018.111676] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17002s workers=2 idle: 661 [17024.056103] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [17024.061523] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17024.091789] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17024.097300] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17024.104240] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17024.107358] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17024.109661] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17024.122312] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17024.130774] RAX: 0000000000005652 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17024.136751] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [17024.143321] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [17024.151014] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [17024.153024] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [17024.158386] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [17024.162669] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17024.167017] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17024.175438] Call Trace: [17024.176789] [17024.177536] ? show_regs.cold.9+0x22/0x2f [17024.181431] ? watchdog_timer_fn.cold.6+0x62/0x10a [17024.187676] ? __hrtimer_run_queues+0x1c2/0x4d0 [17024.190401] ? watchdog+0x40/0x40 [17024.191640] ? hrtimer_interrupt+0x128/0x2c0 [17024.193419] ? smp_apic_timer_interrupt+0xa6/0x270 [17024.195714] ? apic_timer_interrupt+0xf/0x20 [17024.203491] [17024.204396] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17024.209676] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17024.211867] do_raw_spin_lock+0x177/0x190 [17024.214149] _raw_spin_lock+0x21/0x30 [17024.215383] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17024.220744] process_one_work+0x2c8/0x700 [17024.222341] worker_thread+0x296/0x6e0 [17024.224030] ? rescuer_thread+0x570/0x570 [17024.226934] kthread+0x1d1/0x200 [17024.228380] ? set_kthread_struct+0x70/0x70 [17024.230417] ret_from_fork+0x1f/0x30 [17036.055018] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [17036.059151] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17036.086613] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17036.093709] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17036.099992] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17036.103087] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17036.125502] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17036.138345] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [17036.149003] RDX: 0000000000003ba6 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [17036.153355] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [17036.157672] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [17036.162179] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17036.166264] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [17036.171140] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17036.174561] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17036.180313] Call Trace: [17036.181728] [17036.183275] ? show_regs.cold.9+0x22/0x2f [17036.185193] ? watchdog_timer_fn.cold.6+0x62/0x10a [17036.187902] ? __hrtimer_run_queues+0x1c2/0x4d0 [17036.190651] ? watchdog+0x40/0x40 [17036.192572] ? hrtimer_interrupt+0x128/0x2c0 [17036.195561] ? smp_apic_timer_interrupt+0xa6/0x270 [17036.199023] ? apic_timer_interrupt+0xf/0x20 [17036.201457] [17036.203025] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17036.206940] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17036.210061] do_raw_spin_lock+0x177/0x190 [17036.212297] _raw_spin_lock+0x21/0x30 [17036.214463] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17036.218315] obd_mod_free+0x46/0x140 [ptlrpc] [17036.220966] mdc_commit_open+0x91/0xb0 [mdc] [17036.223157] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17036.227533] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17036.231258] after_reply+0xe2c/0x1780 [ptlrpc] [17036.234220] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17036.237749] ? __next_timer_interrupt+0x160/0x160 [17036.240879] ptlrpcd+0xa82/0xe40 [ptlrpc] [17036.249345] ? do_wait_intr+0xf0/0xf0 [17036.253973] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17036.260339] kthread+0x1d1/0x200 [17036.263322] ? set_kthread_struct+0x70/0x70 [17036.266392] ret_from_fork+0x1f/0x30 [17040.057030] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [17040.062141] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17040.096783] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17040.106272] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17040.111143] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17040.114017] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17040.131853] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17040.136247] RAX: 0000000000005b0d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17040.144298] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [17040.150803] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [17040.154217] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [17040.159307] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [17040.173293] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [17040.183703] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17040.187568] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [17040.201733] Call Trace: [17040.209473] [17040.210507] ? show_regs.cold.9+0x22/0x2f [17040.212271] ? watchdog_timer_fn.cold.6+0x62/0x10a [17040.214339] ? __hrtimer_run_queues+0x1c2/0x4d0 [17040.223433] ? watchdog+0x40/0x40 [17040.225380] ? hrtimer_interrupt+0x128/0x2c0 [17040.229254] ? smp_apic_timer_interrupt+0xa6/0x270 [17040.233961] ? apic_timer_interrupt+0xf/0x20 [17040.238466] [17040.240744] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17040.244015] do_raw_spin_lock+0x177/0x190 [17040.247594] _raw_spin_lock+0x21/0x30 [17040.254336] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17040.263894] ? do_raw_spin_unlock+0x75/0x190 [17040.269464] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17040.275802] ? do_wait_intr+0xf0/0xf0 [17040.278426] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17040.282223] mdc_close+0x53b/0xd70 [mdc] [17040.286572] lmv_close+0x18d/0x490 [lmv] [17040.290901] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [17040.296349] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17040.303286] ll_md_real_close+0xa5/0x2a0 [lustre] [17040.307825] ll_md_close+0x274/0x800 [lustre] [17040.310234] ll_file_release+0x128/0xc50 [lustre] [17040.317176] __fput+0x10d/0x440 [17040.323674] ____fput+0x12/0x20 [17040.325399] task_work_run+0xc6/0x110 [17040.332469] exit_to_usermode_loop+0x1dd/0x1f0 [17040.338776] do_syscall_64+0x3d6/0x440 [17040.339915] entry_SYSCALL_64_after_hwframe+0x49/0xae [17040.345306] RIP: 0033:0x7f151bee3cd8 [17040.347333] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17040.358300] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17040.364486] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [17040.368028] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [17040.373289] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17040.376111] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17040.378596] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17044.052059] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [17044.055019] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17044.132798] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17044.154041] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17044.160428] RIP: 0010:native_safe_halt+0x2e/0x50 [17044.172459] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17044.201992] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17044.219656] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17044.229598] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [17044.247248] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [17044.258775] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [17044.272158] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [17044.278225] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [17044.296400] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17044.307641] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [17044.315417] Call Trace: [17044.321084] [17044.325291] ? show_regs.cold.9+0x22/0x2f [17044.329344] ? watchdog_timer_fn.cold.6+0x62/0x10a [17044.335229] ? __hrtimer_run_queues+0x1c2/0x4d0 [17044.337436] ? watchdog+0x40/0x40 [17044.343670] ? hrtimer_interrupt+0x128/0x2c0 [17044.351141] ? smp_apic_timer_interrupt+0xa6/0x270 [17044.356320] ? apic_timer_interrupt+0xf/0x20 [17044.360332] [17044.365214] ? native_safe_halt+0x2e/0x50 [17044.371280] kvm_wait+0xcd/0xe0 [17044.375162] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17044.379334] do_raw_spin_lock+0x177/0x190 [17044.384524] _raw_spin_lock+0x21/0x30 [17044.388930] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17044.395288] ? do_raw_spin_unlock+0x75/0x190 [17044.400267] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17044.406102] ? do_wait_intr+0xf0/0xf0 [17044.408204] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17044.410724] mdc_reint+0x2d/0x110 [mdc] [17044.412201] mdc_unlink+0x658/0x870 [mdc] [17044.413649] lmv_unlink+0x34d/0xbd0 [lmv] [17044.415959] ll_unlink+0x24d/0x7f0 [lustre] [17044.417506] vfs_unlink+0x202/0x330 [17044.419582] do_unlinkat+0x2fa/0x460 [17044.421622] __x64_sys_unlink+0x37/0x50 [17044.424185] do_syscall_64+0xc1/0x440 [17044.425890] entry_SYSCALL_64_after_hwframe+0x49/0xae [17044.427584] RIP: 0033:0x7f151bdee10b [17044.429542] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [17044.439347] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [17044.443574] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [17044.447332] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [17044.451255] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17044.455162] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17044.458522] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17048.545119] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15117s! [17048.549576] Showing busy workqueues and worker pools: [17048.553623] workqueue events: flags=0x0 [17048.556303] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17048.562398] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [17048.568308] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17048.571646] pending: kfree_rcu_monitor [17048.574338] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17048.578098] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [17048.585425] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17048.589122] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [17048.592865] workqueue events_long: flags=0x0 [17048.596261] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17048.600398] pending: lu_objects_flush [obdclass] [17048.603988] workqueue events_unbound: flags=0x2 [17048.607350] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [17048.611195] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17048.617162] workqueue events_power_efficient: flags=0x80 [17048.620592] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17048.626154] pending: do_cache_clean, neigh_periodic_work, check_lifetime [17048.632107] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17048.637495] pending: neigh_periodic_work [17048.640891] workqueue mm_percpu_wq: flags=0x8 [17048.645614] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17048.650361] pending: vmstat_update, lru_add_drain_per_cpu [17048.657130] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17048.665426] pending: vmstat_update [17048.671766] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17048.676537] pending: lru_add_drain_per_cpu [17048.679356] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [17048.683689] pending: lru_add_drain_per_cpu BAR(48) [17048.687628] workqueue writeback: flags=0x4a [17048.691841] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [17048.695819] pending: wb_workfn [17048.697410] workqueue xprtiod: flags=0xa [17048.700212] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17048.704325] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [17048.708809] workqueue ipv6_addrconf: flags=0x40008 [17048.711812] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17048.715631] pending: addrconf_verify_work [17048.718371] workqueue knbd0-recv: flags=0x1a [17048.722115] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17048.727575] in-flight: 658:recv_work [17048.729075] workqueue ptlrpc_pinger: flags=0x2 [17048.732509] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17048.742941] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [17048.747111] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15118s workers=4 idle: 32984 37 8 [17048.752118] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17033s workers=2 idle: 661 [17052.056385] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [17052.064034] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17052.158872] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17052.175558] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17052.188258] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17052.203053] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17052.216367] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17052.236897] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17052.243078] RAX: 00000000000051ab RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17052.255775] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [17052.270805] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [17052.278533] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [17052.296531] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [17052.308433] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [17052.311391] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17052.327545] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17052.334029] Call Trace: [17052.338676] [17052.339827] ? show_regs.cold.9+0x22/0x2f [17052.344486] ? watchdog_timer_fn.cold.6+0x62/0x10a [17052.358395] ? __hrtimer_run_queues+0x1c2/0x4d0 [17052.364372] ? watchdog+0x40/0x40 [17052.372026] ? hrtimer_interrupt+0x128/0x2c0 [17052.376391] ? smp_apic_timer_interrupt+0xa6/0x270 [17052.389891] ? apic_timer_interrupt+0xf/0x20 [17052.391472] [17052.392286] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17052.418875] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17052.428913] do_raw_spin_lock+0x177/0x190 [17052.433293] _raw_spin_lock+0x21/0x30 [17052.439238] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17052.441255] process_one_work+0x2c8/0x700 [17052.450592] worker_thread+0x296/0x6e0 [17052.451995] ? rescuer_thread+0x570/0x570 [17052.455140] kthread+0x1d1/0x200 [17052.456332] ? set_kthread_struct+0x70/0x70 [17052.459296] ret_from_fork+0x1f/0x30 [17064.055032] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [17064.057527] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17064.080799] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17064.087065] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17064.090123] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17064.092376] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17064.098592] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17064.106046] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [17064.115282] RDX: 00000000000077df RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [17064.123077] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [17064.125952] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [17064.129866] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17064.131328] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [17064.134154] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17064.135620] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17064.137681] Call Trace: [17064.138322] [17064.139139] ? show_regs.cold.9+0x22/0x2f [17064.140374] ? watchdog_timer_fn.cold.6+0x62/0x10a [17064.141578] ? __hrtimer_run_queues+0x1c2/0x4d0 [17064.143053] ? watchdog+0x40/0x40 [17064.144090] ? hrtimer_interrupt+0x128/0x2c0 [17064.145301] ? smp_apic_timer_interrupt+0xa6/0x270 [17064.146838] ? apic_timer_interrupt+0xf/0x20 [17064.147968] [17064.148421] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17064.149613] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17064.151136] do_raw_spin_lock+0x177/0x190 [17064.152118] _raw_spin_lock+0x21/0x30 [17064.152973] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17064.155327] obd_mod_free+0x46/0x140 [ptlrpc] [17064.159121] mdc_commit_open+0x91/0xb0 [mdc] [17064.160346] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17064.163504] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17064.164969] after_reply+0xe2c/0x1780 [ptlrpc] [17064.168000] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17064.171505] ? __next_timer_interrupt+0x160/0x160 [17064.173365] ptlrpcd+0xa82/0xe40 [ptlrpc] [17064.175959] ? do_wait_intr+0xf0/0xf0 [17064.177660] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17064.181674] kthread+0x1d1/0x200 [17064.182816] ? set_kthread_struct+0x70/0x70 [17064.184227] ret_from_fork+0x1f/0x30 [17068.060365] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [17068.067337] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17068.116824] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17068.129220] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17068.137884] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17068.148244] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17068.175573] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17068.186607] RAX: 0000000000006ed1 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17068.199482] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [17068.212007] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [17068.225334] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [17068.237030] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [17068.248550] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [17068.263248] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17068.273330] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [17068.281623] Call Trace: [17068.282766] [17068.284035] ? show_regs.cold.9+0x22/0x2f [17068.291187] ? watchdog_timer_fn.cold.6+0x62/0x10a [17068.298180] ? __hrtimer_run_queues+0x1c2/0x4d0 [17068.300257] ? watchdog+0x40/0x40 [17068.308138] ? hrtimer_interrupt+0x128/0x2c0 [17068.310607] ? smp_apic_timer_interrupt+0xa6/0x270 [17068.316758] ? apic_timer_interrupt+0xf/0x20 [17068.327070] [17068.327861] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17068.339645] do_raw_spin_lock+0x177/0x190 [17068.345173] _raw_spin_lock+0x21/0x30 [17068.351793] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17068.358141] ? do_raw_spin_unlock+0x75/0x190 [17068.360276] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17068.363090] ? do_wait_intr+0xf0/0xf0 [17068.368944] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17068.374765] mdc_close+0x53b/0xd70 [mdc] [17068.379194] lmv_close+0x18d/0x490 [lmv] [17068.380637] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [17068.391760] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17068.400644] ll_md_real_close+0xa5/0x2a0 [lustre] [17068.406084] ll_md_close+0x274/0x800 [lustre] [17068.417284] ll_file_release+0x128/0xc50 [lustre] [17068.425242] __fput+0x10d/0x440 [17068.429025] ____fput+0x12/0x20 [17068.434173] task_work_run+0xc6/0x110 [17068.437591] exit_to_usermode_loop+0x1dd/0x1f0 [17068.440712] do_syscall_64+0x3d6/0x440 [17068.445401] entry_SYSCALL_64_after_hwframe+0x49/0xae [17068.448790] RIP: 0033:0x7f151bee3cd8 [17068.450388] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17068.462072] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17068.465320] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [17068.469129] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [17068.471919] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17068.481561] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17068.489446] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17076.052017] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [17076.055391] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17076.078390] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17076.083633] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17076.091115] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17076.094813] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17076.108152] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17076.112056] RAX: 0000000000001464 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17076.115027] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [17076.118679] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [17076.122275] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [17076.126935] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [17076.131891] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [17076.136971] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17076.140646] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [17076.144673] Call Trace: [17076.145895] [17076.146998] ? show_regs.cold.9+0x22/0x2f [17076.149163] ? watchdog_timer_fn.cold.6+0x62/0x10a [17076.151684] ? __hrtimer_run_queues+0x1c2/0x4d0 [17076.153824] ? watchdog+0x40/0x40 [17076.156304] ? hrtimer_interrupt+0x128/0x2c0 [17076.158520] ? smp_apic_timer_interrupt+0xa6/0x270 [17076.161866] ? apic_timer_interrupt+0xf/0x20 [17076.165754] [17076.168175] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17076.173728] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17076.176821] do_raw_spin_lock+0x177/0x190 [17076.178792] _raw_spin_lock+0x21/0x30 [17076.181216] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17076.187141] ? do_raw_spin_unlock+0x75/0x190 [17076.190139] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17076.195131] ? do_wait_intr+0xf0/0xf0 [17076.199353] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17076.201958] mdc_reint+0x2d/0x110 [mdc] [17076.203756] mdc_unlink+0x658/0x870 [mdc] [17076.207527] lmv_unlink+0x34d/0xbd0 [lmv] [17076.209728] ll_unlink+0x24d/0x7f0 [lustre] [17076.211765] vfs_unlink+0x202/0x330 [17076.213105] do_unlinkat+0x2fa/0x460 [17076.215013] __x64_sys_unlink+0x37/0x50 [17076.216872] do_syscall_64+0xc1/0x440 [17076.222187] entry_SYSCALL_64_after_hwframe+0x49/0xae [17076.226109] RIP: 0033:0x7f151bdee10b [17076.232166] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [17076.247682] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [17076.251759] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [17076.259669] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [17076.272818] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17076.288107] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17076.292138] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17078.754467] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15148s! [17078.765320] Showing busy workqueues and worker pools: [17078.771016] workqueue events: flags=0x0 [17078.778611] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17078.786169] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [17078.790948] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17078.794108] pending: kfree_rcu_monitor [17078.796036] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17078.799323] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [17078.804034] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17078.807346] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [17078.811124] workqueue events_long: flags=0x0 [17078.813855] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17078.820864] pending: lu_objects_flush [obdclass] [17078.823758] workqueue events_unbound: flags=0x2 [17078.828043] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [17078.839165] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17078.844244] workqueue events_power_efficient: flags=0x80 [17078.848047] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17078.854559] pending: do_cache_clean, neigh_periodic_work, check_lifetime [17078.861171] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17078.865560] pending: neigh_periodic_work [17078.870076] workqueue mm_percpu_wq: flags=0x8 [17078.877129] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17078.882665] pending: vmstat_update, lru_add_drain_per_cpu [17078.888291] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17078.895291] pending: vmstat_update [17078.897683] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17078.906847] pending: lru_add_drain_per_cpu [17078.913701] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [17078.924027] pending: lru_add_drain_per_cpu BAR(48) [17078.930654] workqueue writeback: flags=0x4a [17078.937586] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [17078.947186] pending: wb_workfn [17078.950546] workqueue xprtiod: flags=0xa [17078.952807] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17078.966734] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [17078.972885] workqueue ipv6_addrconf: flags=0x40008 [17078.975796] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17078.979593] pending: addrconf_verify_work [17078.986094] workqueue knbd0-recv: flags=0x1a [17078.990107] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17078.995950] in-flight: 658:recv_work [17078.998930] workqueue ptlrpc_pinger: flags=0x2 [17079.001231] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17079.004606] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [17079.007638] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15148s workers=4 idle: 32984 37 8 [17079.011187] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17063s workers=2 idle: 661 [17080.059291] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [17080.067976] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17080.113908] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17080.123512] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17080.131037] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17080.137016] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17080.141221] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17080.159254] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17080.165741] RAX: 000000000000312b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17080.172558] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [17080.177716] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [17080.184047] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [17080.188246] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [17080.191349] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [17080.196240] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17080.202249] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17080.206836] Call Trace: [17080.207962] [17080.209055] ? show_regs.cold.9+0x22/0x2f [17080.211168] ? watchdog_timer_fn.cold.6+0x62/0x10a [17080.214079] ? __hrtimer_run_queues+0x1c2/0x4d0 [17080.216440] ? watchdog+0x40/0x40 [17080.218742] ? hrtimer_interrupt+0x128/0x2c0 [17080.221176] ? smp_apic_timer_interrupt+0xa6/0x270 [17080.233160] ? apic_timer_interrupt+0xf/0x20 [17080.246470] [17080.247342] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17080.258405] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17080.264016] do_raw_spin_lock+0x177/0x190 [17080.266878] _raw_spin_lock+0x21/0x30 [17080.270354] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17080.274221] process_one_work+0x2c8/0x700 [17080.282413] worker_thread+0x296/0x6e0 [17080.287315] ? rescuer_thread+0x570/0x570 [17080.292241] kthread+0x1d1/0x200 [17080.296850] ? set_kthread_struct+0x70/0x70 [17080.299075] ret_from_fork+0x1f/0x30 [17092.055027] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [17092.058003] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17092.079834] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17092.086225] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17092.093159] RIP: 0010:native_safe_halt+0x2e/0x50 [17092.107872] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17092.129587] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17092.136532] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [17092.139545] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [17092.143261] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [17092.145694] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [17092.150979] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17092.154170] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [17092.158050] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17092.160802] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17092.163826] Call Trace: [17092.164925] [17092.165953] ? show_regs.cold.9+0x22/0x2f [17092.167752] ? watchdog_timer_fn.cold.6+0x62/0x10a [17092.170487] ? __hrtimer_run_queues+0x1c2/0x4d0 [17092.172664] ? watchdog+0x40/0x40 [17092.174047] ? hrtimer_interrupt+0x128/0x2c0 [17092.181879] ? smp_apic_timer_interrupt+0xa6/0x270 [17092.183519] ? apic_timer_interrupt+0xf/0x20 [17092.186082] [17092.187068] ? native_safe_halt+0x2e/0x50 [17092.189854] kvm_wait+0xcd/0xe0 [17092.191166] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17092.196819] do_raw_spin_lock+0x177/0x190 [17092.199897] _raw_spin_lock+0x21/0x30 [17092.202119] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17092.208955] obd_mod_free+0x46/0x140 [ptlrpc] [17092.213742] mdc_commit_open+0x91/0xb0 [mdc] [17092.218608] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17092.224668] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17092.228681] after_reply+0xe2c/0x1780 [ptlrpc] [17092.232820] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17092.238042] ? __next_timer_interrupt+0x160/0x160 [17092.241127] ptlrpcd+0xa82/0xe40 [ptlrpc] [17092.246728] ? do_wait_intr+0xf0/0xf0 [17092.250975] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17092.254873] kthread+0x1d1/0x200 [17092.256871] ? set_kthread_struct+0x70/0x70 [17092.263155] ret_from_fork+0x1f/0x30 [17096.059697] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [17096.069189] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17096.088612] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17096.093139] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17096.096770] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17096.099044] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17096.107154] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17096.110132] RAX: 0000000000004d10 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17096.113160] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [17096.116026] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [17096.119711] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [17096.122629] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [17096.125657] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [17096.128871] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17096.131102] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [17096.134358] Call Trace: [17096.135133] [17096.136161] ? show_regs.cold.9+0x22/0x2f [17096.137685] ? watchdog_timer_fn.cold.6+0x62/0x10a [17096.140049] ? __hrtimer_run_queues+0x1c2/0x4d0 [17096.141860] ? watchdog+0x40/0x40 [17096.143510] ? hrtimer_interrupt+0x128/0x2c0 [17096.145104] ? smp_apic_timer_interrupt+0xa6/0x270 [17096.147228] ? apic_timer_interrupt+0xf/0x20 [17096.149160] [17096.149925] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17096.151978] do_raw_spin_lock+0x177/0x190 [17096.153900] _raw_spin_lock+0x21/0x30 [17096.155416] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17096.158157] ? do_raw_spin_unlock+0x75/0x190 [17096.160392] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17096.162880] ? do_wait_intr+0xf0/0xf0 [17096.164588] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17096.166710] mdc_close+0x53b/0xd70 [mdc] [17096.168758] lmv_close+0x18d/0x490 [lmv] [17096.170184] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [17096.172785] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17096.174604] ll_md_real_close+0xa5/0x2a0 [lustre] [17096.176336] ll_md_close+0x274/0x800 [lustre] [17096.177766] ll_file_release+0x128/0xc50 [lustre] [17096.179850] __fput+0x10d/0x440 [17096.180949] ____fput+0x12/0x20 [17096.182168] task_work_run+0xc6/0x110 [17096.183636] exit_to_usermode_loop+0x1dd/0x1f0 [17096.185490] do_syscall_64+0x3d6/0x440 [17096.186922] entry_SYSCALL_64_after_hwframe+0x49/0xae [17096.188923] RIP: 0033:0x7f151bee3cd8 [17096.190496] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17096.197721] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17096.200304] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [17096.202484] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [17096.204934] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17096.207561] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17096.210689] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17104.052028] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [17104.054439] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17104.067796] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17104.070870] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17104.073597] RIP: 0010:native_safe_halt+0x2e/0x50 [17104.074881] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17104.080937] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17104.083345] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17104.085576] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [17104.087611] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [17104.089725] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [17104.091905] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [17104.094222] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [17104.096544] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17104.098094] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [17104.100319] Call Trace: [17104.101029] [17104.101721] ? show_regs.cold.9+0x22/0x2f [17104.102914] ? watchdog_timer_fn.cold.6+0x62/0x10a [17104.104379] ? __hrtimer_run_queues+0x1c2/0x4d0 [17104.105641] ? watchdog+0x40/0x40 [17104.106569] ? hrtimer_interrupt+0x128/0x2c0 [17104.107740] ? smp_apic_timer_interrupt+0xa6/0x270 [17104.109062] ? apic_timer_interrupt+0xf/0x20 [17104.110326] [17104.110839] ? native_safe_halt+0x2e/0x50 [17104.112031] kvm_wait+0xcd/0xe0 [17104.112928] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17104.114500] do_raw_spin_lock+0x177/0x190 [17104.115729] _raw_spin_lock+0x21/0x30 [17104.116894] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17104.118684] ? do_raw_spin_unlock+0x75/0x190 [17104.119925] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17104.121737] ? do_wait_intr+0xf0/0xf0 [17104.122780] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17104.124500] mdc_reint+0x2d/0x110 [mdc] [17104.125705] mdc_unlink+0x658/0x870 [mdc] [17104.127018] lmv_unlink+0x34d/0xbd0 [lmv] [17104.128335] ll_unlink+0x24d/0x7f0 [lustre] [17104.129681] vfs_unlink+0x202/0x330 [17104.130766] do_unlinkat+0x2fa/0x460 [17104.131939] __x64_sys_unlink+0x37/0x50 [17104.133194] do_syscall_64+0xc1/0x440 [17104.134189] entry_SYSCALL_64_after_hwframe+0x49/0xae [17104.135824] RIP: 0033:0x7f151bdee10b [17104.136742] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [17104.141917] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [17104.143888] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [17104.145832] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [17104.147921] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17104.149783] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17104.152175] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17104.961034] rcu: INFO: rcu_sched self-detected stall on CPU [17104.962854] rcu: 1-...!: (9119088 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [17104.966050] (t=15180314 jiffies g=249657 q=7104) [17104.967694] rcu: rcu_sched kthread starved for 15175266 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [17104.970529] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [17104.973117] rcu: RCU grace-period kthread stack dump: [17104.975048] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [17104.978735] Call Trace: [17104.979769] __schedule+0x351/0xcb0 [17104.981657] schedule+0xc0/0x180 [17104.982636] schedule_timeout+0xb4/0x190 [17104.983941] ? do_raw_spin_unlock+0x75/0x190 [17104.986324] ? __next_timer_interrupt+0x160/0x160 [17104.988467] ? prepare_to_swait_event+0xf3/0x140 [17104.990600] rcu_gp_kthread+0x8dd/0x15b0 [17104.992640] ? force_qs_rnp+0x2c0/0x2c0 [17104.994687] kthread+0x1d1/0x200 [17104.996389] ? set_kthread_struct+0x70/0x70 [17104.998534] ret_from_fork+0x1f/0x30 [17104.999888] rcu: Stack dump where RCU GP kthread last ran: [17105.002815] NMI backtrace for cpu 1 [17105.005112] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17105.011310] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17105.017033] Call Trace: [17105.020814] [17105.022111] ? dump_stack+0xbb/0x10e [17105.024522] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [17105.026662] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [17105.028667] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [17105.033309] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [17105.035675] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [17105.037068] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [17105.038499] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [17105.043387] ? update_process_times+0x6d/0xc0 [17105.044484] ? tick_sched_handle.isra.18+0x27/0x90 [17105.045665] ? tick_sched_timer+0x53/0xb0 [17105.047226] ? __hrtimer_run_queues+0x1c2/0x4d0 [17105.048558] ? tick_sched_handle.isra.18+0x90/0x90 [17105.049730] ? hrtimer_interrupt+0x128/0x2c0 [17105.051016] ? smp_apic_timer_interrupt+0xa6/0x270 [17105.052361] ? apic_timer_interrupt+0xf/0x20 [17105.053443] [17105.054007] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17105.055592] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17105.057022] ? do_raw_spin_lock+0x177/0x190 [17105.058208] ? _raw_spin_lock+0x21/0x30 [17105.059297] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17105.060741] ? obd_mod_free+0x46/0x140 [ptlrpc] [17105.062186] ? mdc_commit_open+0x91/0xb0 [mdc] [17105.063642] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17105.065301] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17105.066705] ? after_reply+0xe2c/0x1780 [ptlrpc] [17105.068030] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17105.069443] ? __next_timer_interrupt+0x160/0x160 [17105.070530] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [17105.071699] ? do_wait_intr+0xf0/0xf0 [17105.072637] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17105.074066] ? kthread+0x1d1/0x200 [17105.075057] ? set_kthread_struct+0x70/0x70 [17105.076339] ? ret_from_fork+0x1f/0x30 [17105.077231] NMI backtrace for cpu 1 [17105.078017] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17105.081318] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17105.084711] Call Trace: [17105.085400] [17105.085875] ? dump_stack+0xbb/0x10e [17105.086843] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [17105.088289] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [17105.089513] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [17105.090839] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [17105.092296] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [17105.093631] ? rcu_dump_cpu_stacks+0x11b/0x171 [17105.096694] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [17105.099634] ? update_process_times+0x6d/0xc0 [17105.100720] ? tick_sched_handle.isra.18+0x27/0x90 [17105.102113] ? tick_sched_timer+0x53/0xb0 [17105.103287] ? __hrtimer_run_queues+0x1c2/0x4d0 [17105.104505] ? tick_sched_handle.isra.18+0x90/0x90 [17105.105655] ? hrtimer_interrupt+0x128/0x2c0 [17105.106795] ? smp_apic_timer_interrupt+0xa6/0x270 [17105.107912] ? apic_timer_interrupt+0xf/0x20 [17105.109069] [17105.109791] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17105.112082] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17105.113870] ? do_raw_spin_lock+0x177/0x190 [17105.115501] ? _raw_spin_lock+0x21/0x30 [17105.116817] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17105.119451] ? obd_mod_free+0x46/0x140 [ptlrpc] [17105.121236] ? mdc_commit_open+0x91/0xb0 [mdc] [17105.122714] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17105.124838] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17105.127842] ? after_reply+0xe2c/0x1780 [ptlrpc] [17105.129815] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17105.132437] ? __next_timer_interrupt+0x160/0x160 [17105.133790] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [17105.135597] ? do_wait_intr+0xf0/0xf0 [17105.137129] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17105.139019] ? kthread+0x1d1/0x200 [17105.139901] ? set_kthread_struct+0x70/0x70 [17105.142790] ? ret_from_fork+0x1f/0x30 [17108.058290] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [17108.060844] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17108.092013] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17108.095742] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17108.098571] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17108.102149] RIP: 0010:native_safe_halt+0x2e/0x50 [17108.105244] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17108.111742] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17108.117581] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17108.126706] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [17108.129787] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [17108.132943] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [17108.141975] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [17108.146158] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [17108.156255] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17108.159365] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17108.164651] Call Trace: [17108.169690] [17108.170554] ? show_regs.cold.9+0x22/0x2f [17108.172626] ? watchdog_timer_fn.cold.6+0x62/0x10a [17108.177349] ? __hrtimer_run_queues+0x1c2/0x4d0 [17108.180631] ? watchdog+0x40/0x40 [17108.184519] ? hrtimer_interrupt+0x128/0x2c0 [17108.187339] ? smp_apic_timer_interrupt+0xa6/0x270 [17108.189889] ? apic_timer_interrupt+0xf/0x20 [17108.191693] [17108.194575] ? native_safe_halt+0x2e/0x50 [17108.196390] kvm_wait+0xcd/0xe0 [17108.197696] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17108.201505] do_raw_spin_lock+0x177/0x190 [17108.207427] _raw_spin_lock+0x21/0x30 [17108.209914] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17108.213820] process_one_work+0x2c8/0x700 [17108.217816] worker_thread+0x296/0x6e0 [17108.221466] ? rescuer_thread+0x570/0x570 [17108.225273] kthread+0x1d1/0x200 [17108.228318] ? set_kthread_struct+0x70/0x70 [17108.230066] ret_from_fork+0x1f/0x30 [17109.473038] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15178s! [17109.477384] Showing busy workqueues and worker pools: [17109.480777] workqueue events: flags=0x0 [17109.482442] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17109.485579] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [17109.489422] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17109.491470] pending: kfree_rcu_monitor [17109.494332] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17109.497982] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [17109.502014] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17109.506625] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [17109.510647] workqueue events_long: flags=0x0 [17109.513045] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17109.515597] pending: lu_objects_flush [obdclass] [17109.518146] workqueue events_unbound: flags=0x2 [17109.519609] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [17109.523399] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17109.530748] workqueue events_power_efficient: flags=0x80 [17109.532702] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17109.538577] pending: do_cache_clean, neigh_periodic_work, check_lifetime [17109.543257] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17109.547688] pending: neigh_periodic_work [17109.550439] workqueue mm_percpu_wq: flags=0x8 [17109.552423] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17109.555406] pending: vmstat_update, lru_add_drain_per_cpu [17109.557351] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17109.560622] pending: vmstat_update [17109.562249] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17109.565270] pending: lru_add_drain_per_cpu [17109.568218] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [17109.571869] pending: lru_add_drain_per_cpu BAR(48) [17109.574120] workqueue writeback: flags=0x4a [17109.576820] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [17109.579516] pending: wb_workfn [17109.581056] workqueue xprtiod: flags=0xa [17109.584057] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17109.586608] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [17109.590510] workqueue ipv6_addrconf: flags=0x40008 [17109.593237] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17109.596800] pending: addrconf_verify_work [17109.598117] workqueue knbd0-recv: flags=0x1a [17109.601063] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17109.605894] in-flight: 658:recv_work [17109.607420] workqueue ptlrpc_pinger: flags=0x2 [17109.609885] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17109.612592] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [17109.615050] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15179s workers=4 idle: 32984 37 8 [17109.618238] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17094s workers=2 idle: 661 [17124.060859] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [17124.063848] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17124.096105] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17124.101291] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17124.105807] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17124.108730] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17124.115889] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17124.118759] RAX: 0000000000007085 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17124.121258] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [17124.123663] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [17124.126246] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [17124.128743] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [17124.131173] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [17124.134546] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17124.137159] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [17124.140522] Call Trace: [17124.141566] [17124.142519] ? show_regs.cold.9+0x22/0x2f [17124.144702] ? watchdog_timer_fn.cold.6+0x62/0x10a [17124.167645] ? __hrtimer_run_queues+0x1c2/0x4d0 [17124.178669] ? watchdog+0x40/0x40 [17124.184694] ? hrtimer_interrupt+0x128/0x2c0 [17124.195055] ? smp_apic_timer_interrupt+0xa6/0x270 [17124.196628] ? apic_timer_interrupt+0xf/0x20 [17124.197968] [17124.213561] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17124.225657] do_raw_spin_lock+0x177/0x190 [17124.235175] _raw_spin_lock+0x21/0x30 [17124.240799] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17124.246254] ? do_raw_spin_unlock+0x75/0x190 [17124.252311] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17124.260904] ? do_wait_intr+0xf0/0xf0 [17124.262108] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17124.264766] mdc_close+0x53b/0xd70 [mdc] [17124.268099] lmv_close+0x18d/0x490 [lmv] [17124.271539] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [17124.276903] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17124.294543] ll_md_real_close+0xa5/0x2a0 [lustre] [17124.296259] ll_md_close+0x274/0x800 [lustre] [17124.304652] ll_file_release+0x128/0xc50 [lustre] [17124.307645] __fput+0x10d/0x440 [17124.308591] ____fput+0x12/0x20 [17124.318254] task_work_run+0xc6/0x110 [17124.319304] exit_to_usermode_loop+0x1dd/0x1f0 [17124.320563] do_syscall_64+0x3d6/0x440 [17124.331718] entry_SYSCALL_64_after_hwframe+0x49/0xae [17124.333250] RIP: 0033:0x7f151bee3cd8 [17124.342174] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17124.358137] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17124.361226] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [17124.364453] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [17124.369730] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17124.378888] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17124.383926] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17132.055009] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [17132.063109] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17132.088848] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17132.098546] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17132.101712] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17132.105760] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17132.113980] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17132.116539] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [17132.119789] RDX: 0000000000000ce4 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [17132.124672] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [17132.126783] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [17132.128549] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17132.134553] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [17132.138452] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17132.142616] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17132.146542] Call Trace: [17132.147524] [17132.149411] ? show_regs.cold.9+0x22/0x2f [17132.151467] ? watchdog_timer_fn.cold.6+0x62/0x10a [17132.156316] ? __hrtimer_run_queues+0x1c2/0x4d0 [17132.160435] ? watchdog+0x40/0x40 [17132.166220] ? hrtimer_interrupt+0x128/0x2c0 [17132.168518] ? smp_apic_timer_interrupt+0xa6/0x270 [17132.172421] ? apic_timer_interrupt+0xf/0x20 [17132.176448] [17132.177478] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17132.180429] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17132.181714] do_raw_spin_lock+0x177/0x190 [17132.182603] _raw_spin_lock+0x21/0x30 [17132.185709] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17132.187884] obd_mod_free+0x46/0x140 [ptlrpc] [17132.193408] mdc_commit_open+0x91/0xb0 [mdc] [17132.195512] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17132.198453] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17132.203848] after_reply+0xe2c/0x1780 [ptlrpc] [17132.205751] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17132.213167] ? __next_timer_interrupt+0x160/0x160 [17132.216927] ptlrpcd+0xa82/0xe40 [ptlrpc] [17132.218758] ? do_wait_intr+0xf0/0xf0 [17132.231419] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17132.235243] kthread+0x1d1/0x200 [17132.236617] ? set_kthread_struct+0x70/0x70 [17132.238689] ret_from_fork+0x1f/0x30 [17136.052370] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [17136.056111] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [17136.058449] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17136.058530] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17136.058538] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17136.058542] RIP: 0010:native_safe_halt+0x2e/0x50 [17136.058555] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17136.058560] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17136.058564] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17136.058567] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [17136.058569] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [17136.058571] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [17136.058574] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [17136.058577] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [17136.058580] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17136.058583] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [17136.058592] Call Trace: [17136.058597] [17136.058603] ? show_regs.cold.9+0x22/0x2f [17136.058613] ? watchdog_timer_fn.cold.6+0x62/0x10a [17136.058619] ? __hrtimer_run_queues+0x1c2/0x4d0 [17136.058623] ? watchdog+0x40/0x40 [17136.058627] ? hrtimer_interrupt+0x128/0x2c0 [17136.058631] ? smp_apic_timer_interrupt+0xa6/0x270 [17136.058637] ? apic_timer_interrupt+0xf/0x20 [17136.058642] [17136.058645] ? native_safe_halt+0x2e/0x50 [17136.058648] kvm_wait+0xcd/0xe0 [17136.058655] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17136.058663] do_raw_spin_lock+0x177/0x190 [17136.058667] _raw_spin_lock+0x21/0x30 [17136.058671] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17136.071492] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17136.071569] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17136.071576] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17136.071583] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17136.111844] ? do_raw_spin_unlock+0x75/0x190 [17136.111865] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17136.112344] ? do_wait_intr+0xf0/0xf0 [17136.112353] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17136.112718] mdc_reint+0x2d/0x110 [mdc] [17136.112766] mdc_unlink+0x658/0x870 [mdc] [17136.112796] lmv_unlink+0x34d/0xbd0 [lmv] [17136.112884] ll_unlink+0x24d/0x7f0 [lustre] [17136.113003] vfs_unlink+0x202/0x330 [17136.113010] do_unlinkat+0x2fa/0x460 [17136.113015] __x64_sys_unlink+0x37/0x50 [17136.113018] do_syscall_64+0xc1/0x440 [17136.113025] entry_SYSCALL_64_after_hwframe+0x49/0xae [17136.113032] RIP: 0033:0x7f151bdee10b [17136.113039] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [17136.113043] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [17136.113048] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [17136.113051] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [17136.113053] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17136.113055] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17136.113057] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17136.508034] [17136.508910] RIP: 0010:native_safe_halt+0x2e/0x50 [17136.516160] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17136.532162] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17136.543191] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17136.547209] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [17136.554318] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [17136.558125] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [17136.567549] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [17136.584633] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [17136.590819] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17136.598255] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17136.602546] Call Trace: [17136.603726] [17136.605170] ? show_regs.cold.9+0x22/0x2f [17136.607304] ? watchdog_timer_fn.cold.6+0x62/0x10a [17136.609691] ? __hrtimer_run_queues+0x1c2/0x4d0 [17136.614524] ? watchdog+0x40/0x40 [17136.616346] ? hrtimer_interrupt+0x128/0x2c0 [17136.618984] ? smp_apic_timer_interrupt+0xa6/0x270 [17136.628300] ? apic_timer_interrupt+0xf/0x20 [17136.631706] [17136.637247] ? native_safe_halt+0x2e/0x50 [17136.639674] kvm_wait+0xcd/0xe0 [17136.643039] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17136.645584] do_raw_spin_lock+0x177/0x190 [17136.647114] _raw_spin_lock+0x21/0x30 [17136.648245] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17136.650682] process_one_work+0x2c8/0x700 [17136.654139] worker_thread+0x296/0x6e0 [17136.655508] ? rescuer_thread+0x570/0x570 [17136.657015] kthread+0x1d1/0x200 [17136.658028] ? set_kthread_struct+0x70/0x70 [17136.659462] ret_from_fork+0x1f/0x30 [17139.682442] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15209s! [17139.693559] Showing busy workqueues and worker pools: [17139.705104] workqueue events: flags=0x0 [17139.709124] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17139.714056] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [17139.718549] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17139.722436] pending: kfree_rcu_monitor [17139.727186] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17139.733380] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [17139.743827] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17139.753770] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [17139.762660] workqueue events_long: flags=0x0 [17139.765237] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17139.768597] pending: lu_objects_flush [obdclass] [17139.771582] workqueue events_unbound: flags=0x2 [17139.773530] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [17139.777076] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17139.786437] workqueue events_power_efficient: flags=0x80 [17139.790575] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17139.795626] pending: do_cache_clean, neigh_periodic_work, check_lifetime [17139.797808] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17139.801126] pending: neigh_periodic_work [17139.802515] workqueue mm_percpu_wq: flags=0x8 [17139.804428] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17139.806734] pending: vmstat_update, lru_add_drain_per_cpu [17139.810744] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17139.815326] pending: vmstat_update [17139.816573] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17139.818777] pending: lru_add_drain_per_cpu [17139.820694] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [17139.823418] pending: lru_add_drain_per_cpu BAR(48) [17139.827156] workqueue writeback: flags=0x4a [17139.829701] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [17139.835209] pending: wb_workfn [17139.836307] workqueue xprtiod: flags=0xa [17139.840663] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17139.842527] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [17139.852587] workqueue ipv6_addrconf: flags=0x40008 [17139.853934] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17139.861130] pending: addrconf_verify_work [17139.863608] workqueue knbd0-recv: flags=0x1a [17139.866957] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17139.869622] in-flight: 658:recv_work [17139.871565] workqueue ptlrpc_pinger: flags=0x2 [17139.873441] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17139.877096] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [17139.880364] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15209s workers=4 idle: 32984 37 8 [17139.887014] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17124s workers=2 idle: 661 [17152.057021] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [17152.059742] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17152.074918] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17152.078647] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17152.081504] RIP: 0010:native_safe_halt+0x2e/0x50 [17152.083460] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17152.095290] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17152.100947] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17152.104313] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [17152.107147] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [17152.110404] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [17152.120031] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [17152.123682] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [17152.131296] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17152.135585] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [17152.141252] Call Trace: [17152.143465] [17152.144378] ? show_regs.cold.9+0x22/0x2f [17152.148011] ? watchdog_timer_fn.cold.6+0x62/0x10a [17152.151981] ? __hrtimer_run_queues+0x1c2/0x4d0 [17152.155373] ? watchdog+0x40/0x40 [17152.158301] ? hrtimer_interrupt+0x128/0x2c0 [17152.162904] ? smp_apic_timer_interrupt+0xa6/0x270 [17152.165753] ? apic_timer_interrupt+0xf/0x20 [17152.167761] [17152.168610] ? native_safe_halt+0x2e/0x50 [17152.169927] kvm_wait+0xcd/0xe0 [17152.171227] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17152.172781] do_raw_spin_lock+0x177/0x190 [17152.176740] _raw_spin_lock+0x21/0x30 [17152.177898] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17152.181350] ? do_raw_spin_unlock+0x75/0x190 [17152.183458] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17152.186351] ? do_wait_intr+0xf0/0xf0 [17152.187977] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17152.190452] mdc_close+0x53b/0xd70 [mdc] [17152.192648] lmv_close+0x18d/0x490 [lmv] [17152.194210] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [17152.196373] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17152.201382] ll_md_real_close+0xa5/0x2a0 [lustre] [17152.203652] ll_md_close+0x274/0x800 [lustre] [17152.205167] ll_file_release+0x128/0xc50 [lustre] [17152.209898] __fput+0x10d/0x440 [17152.212438] ____fput+0x12/0x20 [17152.213599] task_work_run+0xc6/0x110 [17152.217243] exit_to_usermode_loop+0x1dd/0x1f0 [17152.218924] do_syscall_64+0x3d6/0x440 [17152.222804] entry_SYSCALL_64_after_hwframe+0x49/0xae [17152.226695] RIP: 0033:0x7f151bee3cd8 [17152.230192] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17152.236272] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17152.239506] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [17152.242405] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [17152.245018] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17152.247513] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17152.249939] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17160.055020] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [17160.058097] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17160.073101] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17160.076496] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17160.079360] RIP: 0010:native_safe_halt+0x2e/0x50 [17160.080672] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17160.086579] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17160.089488] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [17160.091665] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [17160.093876] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [17160.095988] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [17160.097984] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17160.099929] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [17160.102171] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17160.103809] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17160.106168] Call Trace: [17160.106893] [17160.108243] ? show_regs.cold.9+0x22/0x2f [17160.109403] ? watchdog_timer_fn.cold.6+0x62/0x10a [17160.110687] ? __hrtimer_run_queues+0x1c2/0x4d0 [17160.112121] ? watchdog+0x40/0x40 [17160.113186] ? hrtimer_interrupt+0x128/0x2c0 [17160.114359] ? smp_apic_timer_interrupt+0xa6/0x270 [17160.115886] ? apic_timer_interrupt+0xf/0x20 [17160.117093] [17160.117705] ? native_safe_halt+0x2e/0x50 [17160.118852] kvm_wait+0xcd/0xe0 [17160.119746] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17160.121169] do_raw_spin_lock+0x177/0x190 [17160.122176] _raw_spin_lock+0x21/0x30 [17160.123342] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17160.125026] obd_mod_free+0x46/0x140 [ptlrpc] [17160.126669] mdc_commit_open+0x91/0xb0 [mdc] [17160.128111] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17160.130217] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17160.131992] after_reply+0xe2c/0x1780 [ptlrpc] [17160.133235] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17160.135096] ? __next_timer_interrupt+0x160/0x160 [17160.136288] ptlrpcd+0xa82/0xe40 [ptlrpc] [17160.137804] ? do_wait_intr+0xf0/0xf0 [17160.138942] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17160.140405] kthread+0x1d1/0x200 [17160.141487] ? set_kthread_struct+0x70/0x70 [17160.142655] ret_from_fork+0x1f/0x30 [17164.052076] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [17164.055031] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul [17164.056036] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [17164.055137] crc32c_intel [17164.076997] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic [17164.087607] ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17164.087635] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17164.087643] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17164.087646] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17164.087661] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17164.087665] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17164.087670] RAX: 0000000000005a85 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17164.087672] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [17164.087674] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [17164.087676] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [17164.087678] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [17164.087690] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [17164.087694] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17164.087696] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [17164.087703] Call Trace: [17164.087708] [17164.087713] ? show_regs.cold.9+0x22/0x2f [17164.087728] ? watchdog_timer_fn.cold.6+0x62/0x10a [17164.087734] ? __hrtimer_run_queues+0x1c2/0x4d0 [17164.087738] ? watchdog+0x40/0x40 [17164.087742] ? hrtimer_interrupt+0x128/0x2c0 [17164.087746] ? smp_apic_timer_interrupt+0xa6/0x270 [17164.087754] ? apic_timer_interrupt+0xf/0x20 [17164.087758] [17164.087760] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17164.087765] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17164.087769] do_raw_spin_lock+0x177/0x190 [17164.087774] _raw_spin_lock+0x21/0x30 [17164.087781] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17164.091990] crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17164.092020] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17164.092029] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17164.092066] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17164.094002] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17164.094019] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17164.094023] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17164.094028] RAX: 0000000000006db1 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17164.094031] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [17164.094033] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [17164.094035] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [17164.094037] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [17164.094041] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [17164.094044] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17164.094047] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17164.094054] Call Trace: [17164.138613] ? do_raw_spin_unlock+0x75/0x190 [17164.138634] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17164.139921] ? do_wait_intr+0xf0/0xf0 [17164.139931] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17164.140145] mdc_reint+0x2d/0x110 [mdc] [17164.140182] mdc_unlink+0x658/0x870 [mdc] [17164.140209] lmv_unlink+0x34d/0xbd0 [lmv] [17164.140240] ll_unlink+0x24d/0x7f0 [lustre] [17164.141520] vfs_unlink+0x202/0x330 [17164.141535] do_unlinkat+0x2fa/0x460 [17164.141541] __x64_sys_unlink+0x37/0x50 [17164.141546] do_syscall_64+0xc1/0x440 [17164.141814] entry_SYSCALL_64_after_hwframe+0x49/0xae [17164.141828] RIP: 0033:0x7f151bdee10b [17164.141874] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [17164.141883] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [17164.141889] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [17164.141891] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [17164.141894] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17164.141896] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17164.141898] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17164.455645] [17164.456718] ? show_regs.cold.9+0x22/0x2f [17164.458804] ? watchdog_timer_fn.cold.6+0x62/0x10a [17164.461469] ? __hrtimer_run_queues+0x1c2/0x4d0 [17164.463907] ? watchdog+0x40/0x40 [17164.465939] ? hrtimer_interrupt+0x128/0x2c0 [17164.468352] ? smp_apic_timer_interrupt+0xa6/0x270 [17164.470792] ? apic_timer_interrupt+0xf/0x20 [17164.473278] [17164.474537] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17164.478128] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17164.481229] do_raw_spin_lock+0x177/0x190 [17164.482857] _raw_spin_lock+0x21/0x30 [17164.484645] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17164.487022] process_one_work+0x2c8/0x700 [17164.488864] worker_thread+0x296/0x6e0 [17164.490482] ? rescuer_thread+0x570/0x570 [17164.492335] kthread+0x1d1/0x200 [17164.493861] ? set_kthread_struct+0x70/0x70 [17164.496042] ret_from_fork+0x1f/0x30 [17169.890097] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15239s! [17169.901940] Showing busy workqueues and worker pools: [17169.909112] workqueue events: flags=0x0 [17169.914179] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17169.921403] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [17169.928236] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17169.944427] pending: kfree_rcu_monitor [17169.949179] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17169.963066] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [17169.978699] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17169.997545] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [17170.009542] workqueue events_long: flags=0x0 [17170.016882] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17170.023121] pending: lu_objects_flush [obdclass] [17170.025161] workqueue events_unbound: flags=0x2 [17170.035749] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [17170.040040] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17170.054409] workqueue events_power_efficient: flags=0x80 [17170.061239] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17170.069705] pending: do_cache_clean, neigh_periodic_work, check_lifetime [17170.079411] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17170.087673] pending: neigh_periodic_work [17170.091139] workqueue mm_percpu_wq: flags=0x8 [17170.092790] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17170.102253] pending: vmstat_update, lru_add_drain_per_cpu [17170.110340] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17170.116311] pending: vmstat_update [17170.120260] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17170.125693] pending: lru_add_drain_per_cpu [17170.132164] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [17170.142498] pending: lru_add_drain_per_cpu BAR(48) [17170.144466] workqueue writeback: flags=0x4a [17170.156361] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [17170.164122] pending: wb_workfn [17170.166492] workqueue xprtiod: flags=0xa [17170.175107] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17170.177467] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [17170.188718] workqueue ipv6_addrconf: flags=0x40008 [17170.193750] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17170.198162] pending: addrconf_verify_work [17170.201193] workqueue knbd0-recv: flags=0x1a [17170.206471] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17170.210280] in-flight: 658:recv_work [17170.212822] workqueue ptlrpc_pinger: flags=0x2 [17170.215460] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17170.219606] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [17170.226136] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15239s workers=4 idle: 32984 37 8 [17170.229314] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17154s workers=2 idle: 661 [17180.060353] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [17180.069831] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17180.105646] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17180.117543] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17180.129054] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17180.136820] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17180.152352] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17180.156806] RAX: 0000000000006986 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17180.160083] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [17180.170058] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [17180.179894] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [17180.186940] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [17180.195566] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [17180.198377] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17180.201983] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [17180.207682] Call Trace: [17180.209465] [17180.210219] ? show_regs.cold.9+0x22/0x2f [17180.211371] ? watchdog_timer_fn.cold.6+0x62/0x10a [17180.215442] ? __hrtimer_run_queues+0x1c2/0x4d0 [17180.218153] ? watchdog+0x40/0x40 [17180.219822] ? hrtimer_interrupt+0x128/0x2c0 [17180.222583] ? smp_apic_timer_interrupt+0xa6/0x270 [17180.225071] ? apic_timer_interrupt+0xf/0x20 [17180.229735] [17180.232740] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17180.237881] do_raw_spin_lock+0x177/0x190 [17180.243290] _raw_spin_lock+0x21/0x30 [17180.245664] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17180.251442] ? do_raw_spin_unlock+0x75/0x190 [17180.253513] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17180.257623] ? do_wait_intr+0xf0/0xf0 [17180.259721] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17180.264612] mdc_close+0x53b/0xd70 [mdc] [17180.268326] lmv_close+0x18d/0x490 [lmv] [17180.271979] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [17180.278165] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17180.281310] ll_md_real_close+0xa5/0x2a0 [lustre] [17180.286403] ll_md_close+0x274/0x800 [lustre] [17180.291839] ll_file_release+0x128/0xc50 [lustre] [17180.298540] __fput+0x10d/0x440 [17180.301462] ____fput+0x12/0x20 [17180.311600] task_work_run+0xc6/0x110 [17180.317380] exit_to_usermode_loop+0x1dd/0x1f0 [17180.325553] do_syscall_64+0x3d6/0x440 [17180.328301] entry_SYSCALL_64_after_hwframe+0x49/0xae [17180.334514] RIP: 0033:0x7f151bee3cd8 [17180.337869] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17180.358314] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17180.361008] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [17180.365353] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [17180.372393] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17180.374508] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17180.380641] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17188.055016] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [17188.059051] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17188.123087] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17188.134273] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17188.142792] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17188.148293] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17188.164431] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17188.173133] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [17188.180466] RDX: 0000000000006906 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [17188.186927] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [17188.194960] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [17188.201209] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17188.205309] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [17188.216114] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17188.220442] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17188.225914] Call Trace: [17188.227947] [17188.229005] ? show_regs.cold.9+0x22/0x2f [17188.233561] ? watchdog_timer_fn.cold.6+0x62/0x10a [17188.237687] ? __hrtimer_run_queues+0x1c2/0x4d0 [17188.241332] ? watchdog+0x40/0x40 [17188.243077] ? hrtimer_interrupt+0x128/0x2c0 [17188.245052] ? smp_apic_timer_interrupt+0xa6/0x270 [17188.247303] ? apic_timer_interrupt+0xf/0x20 [17188.249389] [17188.250615] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17188.252760] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17188.254256] do_raw_spin_lock+0x177/0x190 [17188.255497] _raw_spin_lock+0x21/0x30 [17188.257348] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17188.261476] obd_mod_free+0x46/0x140 [ptlrpc] [17188.264270] mdc_commit_open+0x91/0xb0 [mdc] [17188.267377] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17188.270770] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17188.274006] after_reply+0xe2c/0x1780 [ptlrpc] [17188.276350] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17188.279793] ? __next_timer_interrupt+0x160/0x160 [17188.282797] ptlrpcd+0xa82/0xe40 [ptlrpc] [17188.285195] ? do_wait_intr+0xf0/0xf0 [17188.287036] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17188.289991] kthread+0x1d1/0x200 [17188.291249] ? set_kthread_struct+0x70/0x70 [17188.293214] ret_from_fork+0x1f/0x30 [17192.056028] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [17192.059056] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17192.106716] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17192.118204] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17192.124277] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17192.129103] RIP: 0010:native_safe_halt+0x2e/0x50 [17192.134818] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17192.148849] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17192.154293] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17192.165843] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [17192.174139] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [17192.179862] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [17192.186541] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [17192.194821] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [17192.198970] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17192.202120] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17192.208905] Call Trace: [17192.209740] [17192.214030] ? show_regs.cold.9+0x22/0x2f [17192.218316] ? watchdog_timer_fn.cold.6+0x62/0x10a [17192.220754] ? __hrtimer_run_queues+0x1c2/0x4d0 [17192.223180] ? watchdog+0x40/0x40 [17192.224320] ? hrtimer_interrupt+0x128/0x2c0 [17192.229938] ? smp_apic_timer_interrupt+0xa6/0x270 [17192.233929] ? apic_timer_interrupt+0xf/0x20 [17192.235998] [17192.237108] ? native_safe_halt+0x2e/0x50 [17192.239032] kvm_wait+0xcd/0xe0 [17192.240660] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17192.246935] do_raw_spin_lock+0x177/0x190 [17192.249175] _raw_spin_lock+0x21/0x30 [17192.251196] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17192.254523] process_one_work+0x2c8/0x700 [17192.256466] worker_thread+0x296/0x6e0 [17192.258892] ? rescuer_thread+0x570/0x570 [17192.260534] kthread+0x1d1/0x200 [17192.262706] ? set_kthread_struct+0x70/0x70 [17192.265232] ret_from_fork+0x1f/0x30 [17196.052057] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [17196.055226] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17196.085626] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17196.090054] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17196.092930] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17196.095028] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17196.114447] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17196.120378] RAX: 0000000000004432 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17196.124701] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [17196.126924] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [17196.129896] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [17196.131725] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [17196.139016] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [17196.148226] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17196.153222] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [17196.157155] Call Trace: [17196.159957] [17196.160761] ? show_regs.cold.9+0x22/0x2f [17196.162392] ? watchdog_timer_fn.cold.6+0x62/0x10a [17196.165025] ? __hrtimer_run_queues+0x1c2/0x4d0 [17196.170124] ? watchdog+0x40/0x40 [17196.172995] ? hrtimer_interrupt+0x128/0x2c0 [17196.175894] ? smp_apic_timer_interrupt+0xa6/0x270 [17196.178459] ? apic_timer_interrupt+0xf/0x20 [17196.180964] [17196.181882] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17196.183926] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17196.185960] do_raw_spin_lock+0x177/0x190 [17196.187901] _raw_spin_lock+0x21/0x30 [17196.189893] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17196.192689] ? do_raw_spin_unlock+0x75/0x190 [17196.193910] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17196.195553] ? do_wait_intr+0xf0/0xf0 [17196.196812] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17196.198457] mdc_reint+0x2d/0x110 [mdc] [17196.199747] mdc_unlink+0x658/0x870 [mdc] [17196.201146] lmv_unlink+0x34d/0xbd0 [lmv] [17196.202555] ll_unlink+0x24d/0x7f0 [lustre] [17196.204216] vfs_unlink+0x202/0x330 [17196.205416] do_unlinkat+0x2fa/0x460 [17196.206448] __x64_sys_unlink+0x37/0x50 [17196.207506] do_syscall_64+0xc1/0x440 [17196.208702] entry_SYSCALL_64_after_hwframe+0x49/0xae [17196.210177] RIP: 0033:0x7f151bdee10b [17196.211170] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [17196.218727] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [17196.221696] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [17196.224352] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [17196.226939] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17196.229518] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17196.231981] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17200.610404] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15270s! [17200.616150] Showing busy workqueues and worker pools: [17200.622984] workqueue events: flags=0x0 [17200.628941] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17200.638045] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [17200.653660] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17200.662448] pending: kfree_rcu_monitor [17200.667883] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17200.677427] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [17200.687277] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17200.697047] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [17200.702866] workqueue events_long: flags=0x0 [17200.709051] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17200.716055] pending: lu_objects_flush [obdclass] [17200.719991] workqueue events_unbound: flags=0x2 [17200.724010] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [17200.729283] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17200.735786] workqueue events_power_efficient: flags=0x80 [17200.738957] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17200.743385] pending: do_cache_clean, neigh_periodic_work, check_lifetime [17200.747043] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17200.750254] pending: neigh_periodic_work [17200.752046] workqueue mm_percpu_wq: flags=0x8 [17200.753668] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17200.762045] pending: vmstat_update, lru_add_drain_per_cpu [17200.765977] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17200.770140] pending: vmstat_update [17200.773523] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17200.777124] pending: lru_add_drain_per_cpu [17200.779861] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [17200.786802] pending: lru_add_drain_per_cpu BAR(48) [17200.789421] workqueue writeback: flags=0x4a [17200.791683] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [17200.794943] pending: wb_workfn [17200.796228] workqueue xprtiod: flags=0xa [17200.798708] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17200.800893] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [17200.804715] workqueue ipv6_addrconf: flags=0x40008 [17200.806705] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17200.813421] pending: addrconf_verify_work [17200.816289] workqueue knbd0-recv: flags=0x1a [17200.818664] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17200.821777] in-flight: 658:recv_work [17200.824146] workqueue ptlrpc_pinger: flags=0x2 [17200.826584] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17200.831719] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [17200.834299] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15270s workers=4 idle: 32984 37 8 [17200.839355] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17185s workers=2 idle: 661 [17208.060700] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [17208.066739] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17208.092095] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17208.096928] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17208.101100] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17208.104852] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17208.113235] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17208.116578] RAX: 0000000000006087 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17208.120058] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [17208.123559] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [17208.127036] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [17208.130593] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [17208.134073] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [17208.138435] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17208.142344] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [17208.146449] Call Trace: [17208.147786] [17208.148662] ? show_regs.cold.9+0x22/0x2f [17208.150692] ? watchdog_timer_fn.cold.6+0x62/0x10a [17208.153692] ? __hrtimer_run_queues+0x1c2/0x4d0 [17208.155650] ? watchdog+0x40/0x40 [17208.157880] ? hrtimer_interrupt+0x128/0x2c0 [17208.160327] ? smp_apic_timer_interrupt+0xa6/0x270 [17208.162318] ? apic_timer_interrupt+0xf/0x20 [17208.163902] [17208.166849] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17208.170567] do_raw_spin_lock+0x177/0x190 [17208.173350] _raw_spin_lock+0x21/0x30 [17208.175245] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17208.178318] ? do_raw_spin_unlock+0x75/0x190 [17208.181154] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17208.183759] ? do_wait_intr+0xf0/0xf0 [17208.185706] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17208.188431] mdc_close+0x53b/0xd70 [mdc] [17208.191015] lmv_close+0x18d/0x490 [lmv] [17208.194340] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [17208.196926] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17208.199672] ll_md_real_close+0xa5/0x2a0 [lustre] [17208.201799] ll_md_close+0x274/0x800 [lustre] [17208.203853] ll_file_release+0x128/0xc50 [lustre] [17208.206781] __fput+0x10d/0x440 [17208.208453] ____fput+0x12/0x20 [17208.210328] task_work_run+0xc6/0x110 [17208.212111] exit_to_usermode_loop+0x1dd/0x1f0 [17208.214881] do_syscall_64+0x3d6/0x440 [17208.216921] entry_SYSCALL_64_after_hwframe+0x49/0xae [17208.219490] RIP: 0033:0x7f151bee3cd8 [17208.221533] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17208.231716] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17208.235382] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [17208.238724] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [17208.242900] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17208.247924] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17208.251653] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17216.055658] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [17216.062030] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17216.119865] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17216.136250] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17216.157166] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17216.163666] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17216.184429] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17216.194868] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [17216.204208] RDX: 00000000000054ad RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [17216.208978] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [17216.216238] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [17216.224819] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17216.230269] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [17216.236967] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17216.245763] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17216.251964] Call Trace: [17216.254592] [17216.255916] ? show_regs.cold.9+0x22/0x2f [17216.261528] ? watchdog_timer_fn.cold.6+0x62/0x10a [17216.269262] ? __hrtimer_run_queues+0x1c2/0x4d0 [17216.272756] ? watchdog+0x40/0x40 [17216.274572] ? hrtimer_interrupt+0x128/0x2c0 [17216.283048] ? smp_apic_timer_interrupt+0xa6/0x270 [17216.286820] ? apic_timer_interrupt+0xf/0x20 [17216.289251] [17216.290706] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17216.294171] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17216.300542] do_raw_spin_lock+0x177/0x190 [17216.304911] _raw_spin_lock+0x21/0x30 [17216.307629] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17216.319019] obd_mod_free+0x46/0x140 [ptlrpc] [17216.324910] mdc_commit_open+0x91/0xb0 [mdc] [17216.328166] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17216.338869] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17216.342025] after_reply+0xe2c/0x1780 [ptlrpc] [17216.348736] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17216.360048] ? __next_timer_interrupt+0x160/0x160 [17216.361721] ptlrpcd+0xa82/0xe40 [ptlrpc] [17216.384309] ? do_wait_intr+0xf0/0xf0 [17216.385497] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17216.401383] kthread+0x1d1/0x200 [17216.402554] ? set_kthread_struct+0x70/0x70 [17216.408980] ret_from_fork+0x1f/0x30 [17220.056155] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [17220.065000] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17220.121200] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17220.132902] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17220.142884] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17220.150647] RIP: 0010:native_safe_halt+0x2e/0x50 [17220.156894] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17220.177980] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [17220.185755] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17220.194374] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [17220.200751] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [17220.208412] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [17220.216075] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [17220.224799] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [17220.234380] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17220.241706] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17220.250737] Call Trace: [17220.253902] [17220.257230] ? show_regs.cold.9+0x22/0x2f [17220.262527] ? watchdog_timer_fn.cold.6+0x62/0x10a [17220.267542] ? __hrtimer_run_queues+0x1c2/0x4d0 [17220.272003] ? watchdog+0x40/0x40 [17220.275146] ? hrtimer_interrupt+0x128/0x2c0 [17220.280428] ? smp_apic_timer_interrupt+0xa6/0x270 [17220.287368] ? apic_timer_interrupt+0xf/0x20 [17220.292654] [17220.295595] ? native_safe_halt+0x2e/0x50 [17220.301234] kvm_wait+0xcd/0xe0 [17220.304656] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17220.309755] do_raw_spin_lock+0x177/0x190 [17220.313974] _raw_spin_lock+0x21/0x30 [17220.318021] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17220.324351] process_one_work+0x2c8/0x700 [17220.327692] worker_thread+0x296/0x6e0 [17220.331032] ? rescuer_thread+0x570/0x570 [17220.334726] kthread+0x1d1/0x200 [17220.337513] ? set_kthread_struct+0x70/0x70 [17220.341727] ret_from_fork+0x1f/0x30 [17228.054213] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [17228.066526] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17228.126188] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17228.146367] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17228.163309] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17228.172929] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17228.197752] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17228.212433] RAX: 0000000000002990 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17228.224019] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [17228.232007] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [17228.236465] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [17228.241043] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [17228.244704] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [17228.249523] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17228.252788] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [17228.256594] Call Trace: [17228.258018] [17228.260004] ? show_regs.cold.9+0x22/0x2f [17228.262034] ? watchdog_timer_fn.cold.6+0x62/0x10a [17228.264444] ? __hrtimer_run_queues+0x1c2/0x4d0 [17228.266881] ? watchdog+0x40/0x40 [17228.268963] ? hrtimer_interrupt+0x128/0x2c0 [17228.271258] ? smp_apic_timer_interrupt+0xa6/0x270 [17228.273031] ? apic_timer_interrupt+0xf/0x20 [17228.274701] [17228.275622] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17228.278043] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17228.280453] do_raw_spin_lock+0x177/0x190 [17228.281823] _raw_spin_lock+0x21/0x30 [17228.283502] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17228.288135] ? do_raw_spin_unlock+0x75/0x190 [17228.291089] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17228.295694] ? do_wait_intr+0xf0/0xf0 [17228.298683] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17228.301974] mdc_reint+0x2d/0x110 [mdc] [17228.303835] mdc_unlink+0x658/0x870 [mdc] [17228.306164] lmv_unlink+0x34d/0xbd0 [lmv] [17228.309030] ll_unlink+0x24d/0x7f0 [lustre] [17228.312460] vfs_unlink+0x202/0x330 [17228.314090] do_unlinkat+0x2fa/0x460 [17228.315848] __x64_sys_unlink+0x37/0x50 [17228.317761] do_syscall_64+0xc1/0x440 [17228.319532] entry_SYSCALL_64_after_hwframe+0x49/0xae [17228.323272] RIP: 0033:0x7f151bdee10b [17228.324901] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [17228.338325] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [17228.342529] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [17228.345934] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [17228.349849] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17228.353602] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17228.357136] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17231.332536] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15300s! [17231.349142] Showing busy workqueues and worker pools: [17231.352511] workqueue events: flags=0x0 [17231.359354] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17231.374025] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [17231.393165] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17231.403512] pending: kfree_rcu_monitor [17231.406577] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17231.423221] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [17231.443153] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17231.454564] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [17231.476630] workqueue events_long: flags=0x0 [17231.482489] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17231.495939] pending: lu_objects_flush [obdclass] [17231.504635] workqueue events_unbound: flags=0x2 [17231.509577] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [17231.518707] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17231.545285] workqueue events_power_efficient: flags=0x80 [17231.556019] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17231.567463] pending: do_cache_clean, neigh_periodic_work, check_lifetime [17231.574828] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17231.588525] pending: neigh_periodic_work [17231.594289] workqueue mm_percpu_wq: flags=0x8 [17231.598364] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17231.606375] pending: vmstat_update, lru_add_drain_per_cpu [17231.612811] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17231.621153] pending: vmstat_update [17231.627856] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17231.639941] pending: lru_add_drain_per_cpu [17231.654627] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [17231.666865] pending: lru_add_drain_per_cpu BAR(48) [17231.675426] workqueue writeback: flags=0x4a [17231.683141] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [17231.692797] pending: wb_workfn [17231.703753] workqueue xprtiod: flags=0xa [17231.706171] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17231.717741] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [17231.729081] workqueue ipv6_addrconf: flags=0x40008 [17231.735183] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17231.739394] pending: addrconf_verify_work [17231.741120] workqueue knbd0-recv: flags=0x1a [17231.743303] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17231.746537] in-flight: 658:recv_work [17231.748545] workqueue ptlrpc_pinger: flags=0x2 [17231.750986] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17231.764064] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [17231.768782] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15301s workers=4 idle: 32984 37 8 [17231.778783] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17216s workers=2 idle: 661 [17236.057975] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [17236.062663] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17236.098180] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17236.111219] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17236.119320] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17236.126043] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17236.137076] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17236.140542] RAX: 0000000000007087 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17236.147566] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [17236.155711] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [17236.167147] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [17236.170204] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [17236.173077] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [17236.180572] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17236.186544] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [17236.193559] Call Trace: [17236.194789] [17236.196219] ? show_regs.cold.9+0x22/0x2f [17236.198455] ? watchdog_timer_fn.cold.6+0x62/0x10a [17236.200667] ? __hrtimer_run_queues+0x1c2/0x4d0 [17236.202658] ? watchdog+0x40/0x40 [17236.204216] ? hrtimer_interrupt+0x128/0x2c0 [17236.210036] ? smp_apic_timer_interrupt+0xa6/0x270 [17236.212004] ? apic_timer_interrupt+0xf/0x20 [17236.214050] [17236.217034] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17236.219766] do_raw_spin_lock+0x177/0x190 [17236.221800] _raw_spin_lock+0x21/0x30 [17236.223097] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17236.230978] ? do_raw_spin_unlock+0x75/0x190 [17236.233140] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17236.239191] ? do_wait_intr+0xf0/0xf0 [17236.240491] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17236.248840] mdc_close+0x53b/0xd70 [mdc] [17236.255931] lmv_close+0x18d/0x490 [lmv] [17236.259754] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [17236.267334] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17236.270315] ll_md_real_close+0xa5/0x2a0 [lustre] [17236.273002] ll_md_close+0x274/0x800 [lustre] [17236.275311] ll_file_release+0x128/0xc50 [lustre] [17236.280347] __fput+0x10d/0x440 [17236.281878] ____fput+0x12/0x20 [17236.285747] task_work_run+0xc6/0x110 [17236.291175] exit_to_usermode_loop+0x1dd/0x1f0 [17236.297661] do_syscall_64+0x3d6/0x440 [17236.299095] entry_SYSCALL_64_after_hwframe+0x49/0xae [17236.302148] RIP: 0033:0x7f151bee3cd8 [17236.303745] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17236.314263] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17236.317771] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [17236.322232] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [17236.330105] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17236.336464] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17236.340544] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17244.055310] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [17244.066611] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17244.112286] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17244.126117] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17244.140318] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17244.143509] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17244.162751] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17244.169748] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [17244.178483] RDX: 0000000000001ef3 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [17244.185517] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [17244.193628] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [17244.199705] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17244.208288] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [17244.215434] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17244.219634] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17244.225544] Call Trace: [17244.228475] [17244.230342] ? show_regs.cold.9+0x22/0x2f [17244.234626] ? watchdog_timer_fn.cold.6+0x62/0x10a [17244.240361] ? __hrtimer_run_queues+0x1c2/0x4d0 [17244.243689] ? watchdog+0x40/0x40 [17244.246427] ? hrtimer_interrupt+0x128/0x2c0 [17244.249078] ? smp_apic_timer_interrupt+0xa6/0x270 [17244.251169] ? apic_timer_interrupt+0xf/0x20 [17244.253272] [17244.254333] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17244.259246] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17244.265941] do_raw_spin_lock+0x177/0x190 [17244.269126] _raw_spin_lock+0x21/0x30 [17244.271869] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17244.275018] obd_mod_free+0x46/0x140 [ptlrpc] [17244.280888] mdc_commit_open+0x91/0xb0 [mdc] [17244.285813] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17244.288649] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17244.291162] after_reply+0xe2c/0x1780 [ptlrpc] [17244.299234] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17244.302195] ? __next_timer_interrupt+0x160/0x160 [17244.306507] ptlrpcd+0xa82/0xe40 [ptlrpc] [17244.311240] ? do_wait_intr+0xf0/0xf0 [17244.315740] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17244.318008] kthread+0x1d1/0x200 [17244.322388] ? set_kthread_struct+0x70/0x70 [17244.324286] ret_from_fork+0x1f/0x30 [17248.056653] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [17248.061446] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17248.097874] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17248.104662] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17248.108860] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17248.111076] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17248.114528] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17248.129309] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17248.133988] RAX: 0000000000002964 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17248.136562] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [17248.139498] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [17248.142749] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [17248.146482] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [17248.149495] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [17248.151991] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17248.157179] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17248.163112] Call Trace: [17248.165534] [17248.167155] ? show_regs.cold.9+0x22/0x2f [17248.170318] ? watchdog_timer_fn.cold.6+0x62/0x10a [17248.175267] ? __hrtimer_run_queues+0x1c2/0x4d0 [17248.179534] ? watchdog+0x40/0x40 [17248.181592] ? hrtimer_interrupt+0x128/0x2c0 [17248.183685] ? smp_apic_timer_interrupt+0xa6/0x270 [17248.188664] ? apic_timer_interrupt+0xf/0x20 [17248.191243] [17248.192085] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17248.194160] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17248.200255] do_raw_spin_lock+0x177/0x190 [17248.204621] _raw_spin_lock+0x21/0x30 [17248.208281] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17248.214380] process_one_work+0x2c8/0x700 [17248.216804] worker_thread+0x296/0x6e0 [17248.221184] ? rescuer_thread+0x570/0x570 [17248.223527] kthread+0x1d1/0x200 [17248.226187] ? set_kthread_struct+0x70/0x70 [17248.230448] ret_from_fork+0x1f/0x30 [17256.052083] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [17256.055825] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17256.078511] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17256.084230] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17256.088256] RIP: 0010:native_safe_halt+0x2e/0x50 [17256.090381] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17256.099722] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17256.104263] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17256.108143] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [17256.111395] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [17256.115174] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [17256.118866] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [17256.122895] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [17256.126950] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17256.129516] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [17256.132783] Call Trace: [17256.134177] [17256.135104] ? show_regs.cold.9+0x22/0x2f [17256.137302] ? watchdog_timer_fn.cold.6+0x62/0x10a [17256.139630] ? __hrtimer_run_queues+0x1c2/0x4d0 [17256.142139] ? watchdog+0x40/0x40 [17256.143616] ? hrtimer_interrupt+0x128/0x2c0 [17256.146178] ? smp_apic_timer_interrupt+0xa6/0x270 [17256.148358] ? apic_timer_interrupt+0xf/0x20 [17256.150239] [17256.151237] ? native_safe_halt+0x2e/0x50 [17256.153271] kvm_wait+0xcd/0xe0 [17256.154606] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17256.157404] do_raw_spin_lock+0x177/0x190 [17256.159303] _raw_spin_lock+0x21/0x30 [17256.161166] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17256.164381] ? do_raw_spin_unlock+0x75/0x190 [17256.166340] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17256.168988] ? do_wait_intr+0xf0/0xf0 [17256.170798] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17256.173983] mdc_reint+0x2d/0x110 [mdc] [17256.175442] mdc_unlink+0x658/0x870 [mdc] [17256.177906] lmv_unlink+0x34d/0xbd0 [lmv] [17256.179822] ll_unlink+0x24d/0x7f0 [lustre] [17256.182236] vfs_unlink+0x202/0x330 [17256.183877] do_unlinkat+0x2fa/0x460 [17256.185733] __x64_sys_unlink+0x37/0x50 [17256.187465] do_syscall_64+0xc1/0x440 [17256.189884] entry_SYSCALL_64_after_hwframe+0x49/0xae [17256.192411] RIP: 0033:0x7f151bdee10b [17256.194190] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [17256.202378] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [17256.205562] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [17256.209740] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [17256.213520] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17256.217143] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17256.220163] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17262.055136] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15331s! [17262.075147] Showing busy workqueues and worker pools: [17262.077200] workqueue events: flags=0x0 [17262.088377] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17262.108093] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [17262.117803] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17262.125814] pending: kfree_rcu_monitor [17262.138890] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17262.149150] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [17262.161949] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17262.174103] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [17262.187172] workqueue events_long: flags=0x0 [17262.200588] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17262.210999] pending: lu_objects_flush [obdclass] [17262.218621] workqueue events_unbound: flags=0x2 [17262.232461] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [17262.245177] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17262.261130] workqueue events_power_efficient: flags=0x80 [17262.265719] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17262.278480] pending: do_cache_clean, neigh_periodic_work, check_lifetime [17262.291590] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17262.299421] pending: neigh_periodic_work [17262.308860] workqueue mm_percpu_wq: flags=0x8 [17262.316793] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17262.331433] pending: vmstat_update, lru_add_drain_per_cpu [17262.349272] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17262.376147] pending: vmstat_update [17262.390481] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17262.410146] pending: lru_add_drain_per_cpu [17262.421585] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [17262.426165] pending: lru_add_drain_per_cpu BAR(48) [17262.433566] workqueue writeback: flags=0x4a [17262.438317] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [17262.448364] pending: wb_workfn [17262.459762] workqueue xprtiod: flags=0xa [17262.461668] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17262.467356] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [17262.479028] workqueue ipv6_addrconf: flags=0x40008 [17262.487367] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17262.492280] pending: addrconf_verify_work [17262.494576] workqueue knbd0-recv: flags=0x1a [17262.498444] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17262.501464] in-flight: 658:recv_work [17262.513807] workqueue ptlrpc_pinger: flags=0x2 [17262.515333] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17262.531369] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [17262.539304] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15331s workers=4 idle: 32984 37 8 [17262.545832] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17247s workers=2 idle: 661 [17264.057096] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [17264.066357] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17264.115902] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17264.121972] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17264.127804] RIP: 0010:native_safe_halt+0x2e/0x50 [17264.138162] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17264.156135] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17264.158633] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17264.168283] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [17264.173652] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [17264.177531] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [17264.181007] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [17264.186098] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [17264.190354] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17264.193365] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [17264.200281] Call Trace: [17264.202686] [17264.203963] ? show_regs.cold.9+0x22/0x2f [17264.210264] ? watchdog_timer_fn.cold.6+0x62/0x10a [17264.212392] ? __hrtimer_run_queues+0x1c2/0x4d0 [17264.217085] ? watchdog+0x40/0x40 [17264.219023] ? hrtimer_interrupt+0x128/0x2c0 [17264.226657] ? smp_apic_timer_interrupt+0xa6/0x270 [17264.228455] ? apic_timer_interrupt+0xf/0x20 [17264.237899] [17264.240874] ? native_safe_halt+0x2e/0x50 [17264.245878] kvm_wait+0xcd/0xe0 [17264.248287] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17264.250392] do_raw_spin_lock+0x177/0x190 [17264.252755] _raw_spin_lock+0x21/0x30 [17264.255321] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17264.265323] ? do_raw_spin_unlock+0x75/0x190 [17264.276184] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17264.279373] ? do_wait_intr+0xf0/0xf0 [17264.288025] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17264.296856] mdc_close+0x53b/0xd70 [mdc] [17264.302938] lmv_close+0x18d/0x490 [lmv] [17264.306012] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [17264.310376] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17264.322879] ll_md_real_close+0xa5/0x2a0 [lustre] [17264.332326] ll_md_close+0x274/0x800 [lustre] [17264.338979] ll_file_release+0x128/0xc50 [lustre] [17264.344131] __fput+0x10d/0x440 [17264.345886] ____fput+0x12/0x20 [17264.347208] task_work_run+0xc6/0x110 [17264.349345] exit_to_usermode_loop+0x1dd/0x1f0 [17264.352050] do_syscall_64+0x3d6/0x440 [17264.357420] entry_SYSCALL_64_after_hwframe+0x49/0xae [17264.362323] RIP: 0033:0x7f151bee3cd8 [17264.368685] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17264.394500] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17264.409658] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [17264.413139] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [17264.434096] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17264.441431] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17264.444252] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17272.059811] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [17272.079917] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17272.128464] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17272.148279] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17272.168353] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17272.175443] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17272.216152] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17272.223655] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [17272.234524] RDX: 00000000000032a8 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [17272.253985] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [17272.258304] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [17272.261680] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17272.265627] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [17272.270242] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17272.278405] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17272.281670] Call Trace: [17272.283078] [17272.284141] ? show_regs.cold.9+0x22/0x2f [17272.286043] ? watchdog_timer_fn.cold.6+0x62/0x10a [17272.288153] ? __hrtimer_run_queues+0x1c2/0x4d0 [17272.289983] ? watchdog+0x40/0x40 [17272.291458] ? hrtimer_interrupt+0x128/0x2c0 [17272.293438] ? smp_apic_timer_interrupt+0xa6/0x270 [17272.297092] ? apic_timer_interrupt+0xf/0x20 [17272.302673] [17272.304243] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17272.310376] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17272.316958] do_raw_spin_lock+0x177/0x190 [17272.320590] _raw_spin_lock+0x21/0x30 [17272.321544] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17272.331134] obd_mod_free+0x46/0x140 [ptlrpc] [17272.340737] mdc_commit_open+0x91/0xb0 [mdc] [17272.346159] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17272.360757] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17272.365148] after_reply+0xe2c/0x1780 [ptlrpc] [17272.373003] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17272.380252] ? __next_timer_interrupt+0x160/0x160 [17272.384347] ptlrpcd+0xa82/0xe40 [ptlrpc] [17272.386152] ? do_wait_intr+0xf0/0xf0 [17272.388866] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17272.395827] kthread+0x1d1/0x200 [17272.399387] ? set_kthread_struct+0x70/0x70 [17272.402436] ret_from_fork+0x1f/0x30 [17276.056030] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [17276.060378] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17276.089435] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17276.096005] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17276.101427] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17276.105838] RIP: 0010:native_safe_halt+0x2e/0x50 [17276.108010] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17276.121579] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17276.124990] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17276.129056] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [17276.133415] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [17276.139777] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [17276.144159] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [17276.147916] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [17276.153294] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17276.157717] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17276.161309] Call Trace: [17276.164071] [17276.165322] ? show_regs.cold.9+0x22/0x2f [17276.168914] ? watchdog_timer_fn.cold.6+0x62/0x10a [17276.171023] ? __hrtimer_run_queues+0x1c2/0x4d0 [17276.173437] ? watchdog+0x40/0x40 [17276.175376] ? hrtimer_interrupt+0x128/0x2c0 [17276.177467] ? smp_apic_timer_interrupt+0xa6/0x270 [17276.179204] ? apic_timer_interrupt+0xf/0x20 [17276.181037] [17276.182040] ? native_safe_halt+0x2e/0x50 [17276.185194] kvm_wait+0xcd/0xe0 [17276.187219] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17276.190268] do_raw_spin_lock+0x177/0x190 [17276.192438] _raw_spin_lock+0x21/0x30 [17276.195540] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17276.198742] process_one_work+0x2c8/0x700 [17276.201052] worker_thread+0x296/0x6e0 [17276.203761] ? rescuer_thread+0x570/0x570 [17276.205917] kthread+0x1d1/0x200 [17276.208251] ? set_kthread_struct+0x70/0x70 [17276.210351] ret_from_fork+0x1f/0x30 [17284.964880] rcu: INFO: rcu_sched self-detected stall on CPU [17284.969176] rcu: 1-...!: (9230709 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [17284.973185] (t=15360317 jiffies g=249657 q=7156) [17284.975262] rcu: rcu_sched kthread starved for 15355269 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [17284.979597] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [17284.983209] rcu: RCU grace-period kthread stack dump: [17284.985566] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [17284.989841] Call Trace: [17284.990920] __schedule+0x351/0xcb0 [17284.992297] schedule+0xc0/0x180 [17284.993658] schedule_timeout+0xb4/0x190 [17284.995029] ? do_raw_spin_unlock+0x75/0x190 [17284.997071] ? __next_timer_interrupt+0x160/0x160 [17284.999047] ? prepare_to_swait_event+0xf3/0x140 [17285.000989] rcu_gp_kthread+0x8dd/0x15b0 [17285.002568] ? force_qs_rnp+0x2c0/0x2c0 [17285.004159] kthread+0x1d1/0x200 [17285.005460] ? set_kthread_struct+0x70/0x70 [17285.007802] ret_from_fork+0x1f/0x30 [17285.009088] rcu: Stack dump where RCU GP kthread last ran: [17285.011443] Sending NMI from CPU 1 to CPUs 2: [17285.015196] NMI backtrace for cpu 2 [17285.015202] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17285.015204] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17285.015206] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17285.015209] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17285.015210] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17285.015213] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 [17285.015215] RAX: 0000000000004f85 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17285.015217] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [17285.015218] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [17285.015220] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [17285.015221] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [17285.015223] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [17285.015225] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17285.015226] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17285.015227] Call Trace: [17285.015228] [17285.015229] ? show_regs.cold.9+0x22/0x2f [17285.015230] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [17285.015232] ? nmi_cpu_backtrace_handler+0x15/0x40 [17285.015233] ? nmi_handle+0xbd/0x210 [17285.015234] ? default_do_nmi+0x4b/0x1b0 [17285.015235] ? do_nmi+0x280/0x380 [17285.015236] ? end_repeat_nmi+0x16/0x1f [17285.015237] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17285.015239] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17285.015240] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17285.015241] [17285.015243] do_raw_spin_lock+0x177/0x190 [17285.015244] _raw_spin_lock+0x21/0x30 [17285.015245] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17285.015246] process_one_work+0x2c8/0x700 [17285.015247] worker_thread+0x296/0x6e0 [17285.015248] ? rescuer_thread+0x570/0x570 [17285.015249] kthread+0x1d1/0x200 [17285.015250] ? set_kthread_struct+0x70/0x70 [17285.015251] ret_from_fork+0x1f/0x30 [17285.015257] NMI backtrace for cpu 1 [17285.166937] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17285.173020] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17285.179886] Call Trace: [17285.180924] [17285.181809] ? dump_stack+0xbb/0x10e [17285.187534] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [17285.189591] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [17285.195551] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [17285.201186] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [17285.203770] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [17285.209233] ? rcu_dump_cpu_stacks+0x11b/0x171 [17285.214882] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [17285.216803] ? update_process_times+0x6d/0xc0 [17285.220575] ? tick_sched_handle.isra.18+0x27/0x90 [17285.225188] ? tick_sched_timer+0x53/0xb0 [17285.229007] ? __hrtimer_run_queues+0x1c2/0x4d0 [17285.231359] ? tick_sched_handle.isra.18+0x90/0x90 [17285.235656] ? hrtimer_interrupt+0x128/0x2c0 [17285.239476] ? smp_apic_timer_interrupt+0xa6/0x270 [17285.244072] ? apic_timer_interrupt+0xf/0x20 [17285.247227] [17285.250006] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17285.252472] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17285.258142] ? do_raw_spin_lock+0x177/0x190 [17285.262551] ? _raw_spin_lock+0x21/0x30 [17285.265885] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17285.270222] ? obd_mod_free+0x46/0x140 [ptlrpc] [17285.274176] ? mdc_commit_open+0x91/0xb0 [mdc] [17285.279427] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17285.285130] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17285.289199] ? after_reply+0xe2c/0x1780 [ptlrpc] [17285.293831] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17285.300336] ? __next_timer_interrupt+0x160/0x160 [17285.305017] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [17285.310007] ? do_wait_intr+0xf0/0xf0 [17285.313342] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17285.318921] ? kthread+0x1d1/0x200 [17285.321588] ? set_kthread_struct+0x70/0x70 [17285.324255] ? ret_from_fork+0x1f/0x30 [17288.052106] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [17288.056768] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17288.096824] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17288.111637] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17288.123423] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17288.131067] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17288.148438] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17288.164078] RAX: 0000000000003885 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17288.172313] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [17288.181109] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [17288.188899] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [17288.194354] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [17288.199234] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [17288.204158] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17288.209049] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [17288.215171] Call Trace: [17288.219393] [17288.221146] ? show_regs.cold.9+0x22/0x2f [17288.223341] ? watchdog_timer_fn.cold.6+0x62/0x10a [17288.226025] ? __hrtimer_run_queues+0x1c2/0x4d0 [17288.228259] ? watchdog+0x40/0x40 [17288.230109] ? hrtimer_interrupt+0x128/0x2c0 [17288.232508] ? smp_apic_timer_interrupt+0xa6/0x270 [17288.234694] ? apic_timer_interrupt+0xf/0x20 [17288.236836] [17288.238058] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17288.243139] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17288.247214] do_raw_spin_lock+0x177/0x190 [17288.249377] _raw_spin_lock+0x21/0x30 [17288.255004] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17288.261280] ? do_raw_spin_unlock+0x75/0x190 [17288.264279] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17288.268076] ? do_wait_intr+0xf0/0xf0 [17288.270543] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17288.274896] mdc_reint+0x2d/0x110 [mdc] [17288.279473] mdc_unlink+0x658/0x870 [mdc] [17288.283943] lmv_unlink+0x34d/0xbd0 [lmv] [17288.291042] ll_unlink+0x24d/0x7f0 [lustre] [17288.294599] vfs_unlink+0x202/0x330 [17288.299466] do_unlinkat+0x2fa/0x460 [17288.304050] __x64_sys_unlink+0x37/0x50 [17288.306922] do_syscall_64+0xc1/0x440 [17288.308158] entry_SYSCALL_64_after_hwframe+0x49/0xae [17288.313224] RIP: 0033:0x7f151bdee10b [17288.315019] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [17288.324328] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [17288.327959] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [17288.331416] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [17288.334768] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17288.337991] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17288.341104] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17292.057020] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [17292.060038] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17292.082998] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17292.087970] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17292.091695] RIP: 0010:native_safe_halt+0x2e/0x50 [17292.093163] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17292.100028] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17292.103857] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17292.107085] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [17292.128998] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [17292.139746] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [17292.148268] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [17292.153622] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [17292.165958] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17292.176028] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [17292.190794] Call Trace: [17292.193877] [17292.200917] ? show_regs.cold.9+0x22/0x2f [17292.202960] ? watchdog_timer_fn.cold.6+0x62/0x10a [17292.208039] ? __hrtimer_run_queues+0x1c2/0x4d0 [17292.209942] ? watchdog+0x40/0x40 [17292.211621] ? hrtimer_interrupt+0x128/0x2c0 [17292.213779] ? smp_apic_timer_interrupt+0xa6/0x270 [17292.215919] ? apic_timer_interrupt+0xf/0x20 [17292.217954] [17292.218792] ? native_safe_halt+0x2e/0x50 [17292.220247] kvm_wait+0xcd/0xe0 [17292.222711] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17292.227110] do_raw_spin_lock+0x177/0x190 [17292.233706] _raw_spin_lock+0x21/0x30 [17292.237995] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17292.242625] ? do_raw_spin_unlock+0x75/0x190 [17292.257075] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17292.260766] ? do_wait_intr+0xf0/0xf0 [17292.262491] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17292.264996] mdc_close+0x53b/0xd70 [mdc] [17292.267139] lmv_close+0x18d/0x490 [lmv] [17292.270583] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [17292.287590] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17292.295537] ll_md_real_close+0xa5/0x2a0 [lustre] [17292.302588] ll_md_close+0x274/0x800 [lustre] [17292.304062] ll_file_release+0x128/0xc50 [lustre] [17292.308097] __fput+0x10d/0x440 [17292.309556] ____fput+0x12/0x20 [17292.310962] task_work_run+0xc6/0x110 [17292.312996] exit_to_usermode_loop+0x1dd/0x1f0 [17292.319611] do_syscall_64+0x3d6/0x440 [17292.324910] entry_SYSCALL_64_after_hwframe+0x49/0xae [17292.328750] RIP: 0033:0x7f151bee3cd8 [17292.341089] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17292.357259] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17292.360156] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [17292.363092] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [17292.365838] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17292.368816] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17292.371664] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17292.771021] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15362s! [17292.788014] Showing busy workqueues and worker pools: [17292.793145] workqueue events: flags=0x0 [17292.794400] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17292.804889] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [17292.825212] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17292.829729] pending: kfree_rcu_monitor [17292.832626] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17292.841368] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [17292.847849] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17292.852823] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [17292.861888] workqueue events_long: flags=0x0 [17292.868487] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17292.871844] pending: lu_objects_flush [obdclass] [17292.874401] workqueue events_unbound: flags=0x2 [17292.876000] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [17292.878001] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17292.893642] workqueue events_power_efficient: flags=0x80 [17292.899377] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17292.902608] pending: do_cache_clean, neigh_periodic_work, check_lifetime [17292.904715] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17292.909590] pending: neigh_periodic_work [17292.915168] workqueue mm_percpu_wq: flags=0x8 [17292.917710] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17292.926653] pending: vmstat_update, lru_add_drain_per_cpu [17292.932864] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17292.938729] pending: vmstat_update [17292.940518] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17292.943303] pending: lru_add_drain_per_cpu [17292.948120] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [17292.954780] pending: lru_add_drain_per_cpu BAR(48) [17292.961875] workqueue writeback: flags=0x4a [17292.964135] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [17292.968477] pending: wb_workfn [17292.971717] workqueue xprtiod: flags=0xa [17292.975423] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17292.982361] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [17292.989888] workqueue ipv6_addrconf: flags=0x40008 [17292.994154] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17292.996058] pending: addrconf_verify_work [17292.999665] workqueue knbd0-recv: flags=0x1a [17293.006172] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17293.015598] in-flight: 658:recv_work [17293.016800] workqueue ptlrpc_pinger: flags=0x2 [17293.018384] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17293.026428] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [17293.033791] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15362s workers=4 idle: 32984 37 8 [17293.045769] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17277s workers=2 idle: 661 [17312.056610] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [17312.062340] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17312.063459] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [17312.094749] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17312.097666] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17312.097726] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17312.097732] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17312.097737] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17312.098149] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17312.098216] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17312.098221] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17312.098225] RAX: 0000000000005184 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17312.098228] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [17312.098340] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [17312.098342] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [17312.098345] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [17312.098405] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [17312.098409] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17312.098411] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17312.098419] Call Trace: [17312.098425] [17312.098430] ? show_regs.cold.9+0x22/0x2f [17312.098438] ? watchdog_timer_fn.cold.6+0x62/0x10a [17312.098443] ? __hrtimer_run_queues+0x1c2/0x4d0 [17312.098446] ? watchdog+0x40/0x40 [17312.098450] ? hrtimer_interrupt+0x128/0x2c0 [17312.098455] ? smp_apic_timer_interrupt+0xa6/0x270 [17312.098462] ? apic_timer_interrupt+0xf/0x20 [17312.098467] [17312.098470] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17312.098474] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17312.098480] do_raw_spin_lock+0x177/0x190 [17312.098485] _raw_spin_lock+0x21/0x30 [17312.098490] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17312.098644] process_one_work+0x2c8/0x700 [17312.098650] worker_thread+0x296/0x6e0 [17312.098653] ? rescuer_thread+0x570/0x570 [17312.098656] kthread+0x1d1/0x200 [17312.098659] ? set_kthread_struct+0x70/0x70 [17312.098661] ret_from_fork+0x1f/0x30 [17312.282506] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17312.288837] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17312.293959] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17312.306927] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17312.310819] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [17312.313366] RDX: 00000000000025fa RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [17312.316283] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [17312.318818] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [17312.324205] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17312.326718] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [17312.331816] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17312.334053] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17312.337767] Call Trace: [17312.338601] [17312.339461] ? show_regs.cold.9+0x22/0x2f [17312.341108] ? watchdog_timer_fn.cold.6+0x62/0x10a [17312.342714] ? __hrtimer_run_queues+0x1c2/0x4d0 [17312.345687] ? watchdog+0x40/0x40 [17312.346813] ? hrtimer_interrupt+0x128/0x2c0 [17312.348335] ? smp_apic_timer_interrupt+0xa6/0x270 [17312.349866] ? apic_timer_interrupt+0xf/0x20 [17312.351725] [17312.352519] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17312.355065] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17312.357378] do_raw_spin_lock+0x177/0x190 [17312.359333] _raw_spin_lock+0x21/0x30 [17312.361860] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17312.364278] obd_mod_free+0x46/0x140 [ptlrpc] [17312.367761] mdc_commit_open+0x91/0xb0 [mdc] [17312.369461] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17312.371437] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17312.374426] after_reply+0xe2c/0x1780 [ptlrpc] [17312.377908] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17312.379993] ? __next_timer_interrupt+0x160/0x160 [17312.382010] ptlrpcd+0xa82/0xe40 [ptlrpc] [17312.384769] ? do_wait_intr+0xf0/0xf0 [17312.386933] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17312.389467] kthread+0x1d1/0x200 [17312.390690] ? set_kthread_struct+0x70/0x70 [17312.392178] ret_from_fork+0x1f/0x30 [17320.056684] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [17320.059236] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17320.060006] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [17320.086778] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17320.086794] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17320.086797] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17320.086812] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17320.086816] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17320.086822] RAX: 000000000000410a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17320.086825] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [17320.086827] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [17320.086829] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [17320.086831] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [17320.086835] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [17320.086838] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17320.086840] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [17320.086849] Call Trace: [17320.086855] [17320.086860] ? show_regs.cold.9+0x22/0x2f [17320.086869] ? watchdog_timer_fn.cold.6+0x62/0x10a [17320.086876] ? __hrtimer_run_queues+0x1c2/0x4d0 [17320.086880] ? watchdog+0x40/0x40 [17320.086885] ? hrtimer_interrupt+0x128/0x2c0 [17320.086888] ? smp_apic_timer_interrupt+0xa6/0x270 [17320.086898] ? apic_timer_interrupt+0xf/0x20 [17320.086902] [17320.086905] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17320.086909] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17320.086914] do_raw_spin_lock+0x177/0x190 [17320.086918] _raw_spin_lock+0x21/0x30 [17320.086926] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17320.087473] ? do_raw_spin_unlock+0x75/0x190 [17320.087480] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17320.087682] ? do_wait_intr+0xf0/0xf0 [17320.087690] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17320.087887] mdc_reint+0x2d/0x110 [mdc] [17320.087922] mdc_unlink+0x658/0x870 [mdc] [17320.087952] lmv_unlink+0x34d/0xbd0 [lmv] [17320.087985] ll_unlink+0x24d/0x7f0 [lustre] [17320.088189] vfs_unlink+0x202/0x330 [17320.088196] do_unlinkat+0x2fa/0x460 [17320.088202] __x64_sys_unlink+0x37/0x50 [17320.088207] do_syscall_64+0xc1/0x440 [17320.093740] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17320.093821] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17320.093829] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17320.093832] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17320.103695] entry_SYSCALL_64_after_hwframe+0x49/0xae [17320.103724] RIP: 0033:0x7f151bdee10b [17320.103734] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [17320.103739] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [17320.103745] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [17320.103747] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [17320.103749] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17320.103751] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17320.103754] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17320.274018] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17320.282978] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17320.285907] RAX: 0000000000003c84 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17320.289665] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [17320.293664] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [17320.297387] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [17320.300291] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [17320.310426] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [17320.314655] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17320.317436] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [17320.322201] Call Trace: [17320.323247] [17320.324724] ? show_regs.cold.9+0x22/0x2f [17320.326421] ? watchdog_timer_fn.cold.6+0x62/0x10a [17320.330578] ? __hrtimer_run_queues+0x1c2/0x4d0 [17320.339494] ? watchdog+0x40/0x40 [17320.340689] ? hrtimer_interrupt+0x128/0x2c0 [17320.344252] ? smp_apic_timer_interrupt+0xa6/0x270 [17320.346238] ? apic_timer_interrupt+0xf/0x20 [17320.350422] [17320.353200] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17320.356643] do_raw_spin_lock+0x177/0x190 [17320.358749] _raw_spin_lock+0x21/0x30 [17320.361745] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17320.367172] ? do_raw_spin_unlock+0x75/0x190 [17320.375788] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17320.377787] ? do_wait_intr+0xf0/0xf0 [17320.384656] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17320.386685] mdc_close+0x53b/0xd70 [mdc] [17320.391928] lmv_close+0x18d/0x490 [lmv] [17320.396831] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [17320.402805] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17320.411747] ll_md_real_close+0xa5/0x2a0 [lustre] [17320.413635] ll_md_close+0x274/0x800 [lustre] [17320.417968] ll_file_release+0x128/0xc50 [lustre] [17320.419298] __fput+0x10d/0x440 [17320.420060] ____fput+0x12/0x20 [17320.424536] task_work_run+0xc6/0x110 [17320.425740] exit_to_usermode_loop+0x1dd/0x1f0 [17320.430819] do_syscall_64+0x3d6/0x440 [17320.434366] entry_SYSCALL_64_after_hwframe+0x49/0xae [17320.437825] RIP: 0033:0x7f151bee3cd8 [17320.440527] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17320.451564] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17320.460762] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [17320.470728] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [17320.475788] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17320.480755] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17320.484214] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17323.489058] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15392s! [17323.492128] Showing busy workqueues and worker pools: [17323.496465] workqueue events: flags=0x0 [17323.498050] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17323.502700] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [17323.508315] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17323.512317] pending: kfree_rcu_monitor [17323.516926] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17323.522478] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [17323.527469] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17323.531749] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [17323.536424] workqueue events_long: flags=0x0 [17323.538419] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17323.543312] pending: lu_objects_flush [obdclass] [17323.547105] workqueue events_unbound: flags=0x2 [17323.549893] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [17323.552946] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17323.557812] workqueue events_power_efficient: flags=0x80 [17323.562927] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17323.570482] pending: do_cache_clean, neigh_periodic_work, check_lifetime [17323.575847] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17323.578880] pending: neigh_periodic_work [17323.582919] workqueue mm_percpu_wq: flags=0x8 [17323.584495] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17323.587621] pending: vmstat_update, lru_add_drain_per_cpu [17323.590010] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17323.594510] pending: vmstat_update [17323.597622] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17323.601365] pending: lru_add_drain_per_cpu [17323.604292] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [17323.610926] pending: lru_add_drain_per_cpu BAR(48) [17323.613086] workqueue writeback: flags=0x4a [17323.614918] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [17323.617864] pending: wb_workfn [17323.620242] workqueue xprtiod: flags=0xa [17323.621923] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17323.627713] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [17323.631390] workqueue ipv6_addrconf: flags=0x40008 [17323.633539] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17323.638800] pending: addrconf_verify_work [17323.642064] workqueue knbd0-recv: flags=0x1a [17323.648340] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17323.654986] in-flight: 658:recv_work [17323.657425] workqueue ptlrpc_pinger: flags=0x2 [17323.663492] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17323.670795] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [17323.678925] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15393s workers=4 idle: 32984 37 8 [17323.685390] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17308s workers=2 idle: 661 [17340.055961] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [17340.058671] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [17340.067759] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17340.067835] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17340.067855] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17340.067858] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17340.067871] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17340.067884] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17340.067889] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [17340.067892] RDX: 0000000000002c6a RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [17340.067894] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [17340.067896] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [17340.067898] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17340.067901] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [17340.067904] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17340.067906] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17340.067913] Call Trace: [17340.067918] [17340.067924] ? show_regs.cold.9+0x22/0x2f [17340.067935] ? watchdog_timer_fn.cold.6+0x62/0x10a [17340.067942] ? __hrtimer_run_queues+0x1c2/0x4d0 [17340.067946] ? watchdog+0x40/0x40 [17340.067951] ? hrtimer_interrupt+0x128/0x2c0 [17340.067955] ? smp_apic_timer_interrupt+0xa6/0x270 [17340.067962] ? apic_timer_interrupt+0xf/0x20 [17340.067968] [17340.067971] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17340.067988] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17340.067994] do_raw_spin_lock+0x177/0x190 [17340.067999] _raw_spin_lock+0x21/0x30 [17340.068005] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17340.068825] obd_mod_free+0x46/0x140 [ptlrpc] [17340.069100] mdc_commit_open+0x91/0xb0 [mdc] [17340.069279] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17340.069646] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17340.069835] after_reply+0xe2c/0x1780 [ptlrpc] [17340.070473] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17340.070678] ? __next_timer_interrupt+0x160/0x160 [17340.070689] ptlrpcd+0xa82/0xe40 [ptlrpc] [17340.071104] ? do_wait_intr+0xf0/0xf0 [17340.071114] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17340.071472] kthread+0x1d1/0x200 [17340.071479] ? set_kthread_struct+0x70/0x70 [17340.071483] ret_from_fork+0x1f/0x30 [17340.645246] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17340.722123] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17340.739735] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17340.756627] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17340.764071] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17340.773674] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17340.802606] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17340.810445] RAX: 0000000000004a09 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17340.816717] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [17340.827684] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [17340.838485] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [17340.847990] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [17340.858405] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [17340.867614] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17340.871148] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17340.882719] Call Trace: [17340.888818] [17340.893722] ? show_regs.cold.9+0x22/0x2f [17340.896067] ? watchdog_timer_fn.cold.6+0x62/0x10a [17340.898452] ? __hrtimer_run_queues+0x1c2/0x4d0 [17340.900524] ? watchdog+0x40/0x40 [17340.901753] ? hrtimer_interrupt+0x128/0x2c0 [17340.905048] ? smp_apic_timer_interrupt+0xa6/0x270 [17340.908059] ? apic_timer_interrupt+0xf/0x20 [17340.910641] [17340.911695] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17340.915594] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17340.919224] do_raw_spin_lock+0x177/0x190 [17340.925663] _raw_spin_lock+0x21/0x30 [17340.926957] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17340.929600] process_one_work+0x2c8/0x700 [17340.931347] worker_thread+0x296/0x6e0 [17340.932895] ? rescuer_thread+0x570/0x570 [17340.934665] kthread+0x1d1/0x200 [17340.936046] ? set_kthread_struct+0x70/0x70 [17340.938600] ret_from_fork+0x1f/0x30 [17348.054421] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [17348.057732] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [17348.067460] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17348.067542] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17348.067550] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17348.067553] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17348.067569] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17348.067574] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17348.067578] RAX: 0000000000005d8e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17348.067581] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [17348.067583] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [17348.067586] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [17348.067588] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [17348.067592] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [17348.067595] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17348.067598] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [17348.067606] Call Trace: [17348.067612] [17348.067619] ? show_regs.cold.9+0x22/0x2f [17348.067629] ? watchdog_timer_fn.cold.6+0x62/0x10a [17348.067707] ? __hrtimer_run_queues+0x1c2/0x4d0 [17348.067714] ? watchdog+0x40/0x40 [17348.067720] ? hrtimer_interrupt+0x128/0x2c0 [17348.067724] ? smp_apic_timer_interrupt+0xa6/0x270 [17348.067734] ? apic_timer_interrupt+0xf/0x20 [17348.067739] [17348.067742] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17348.067747] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17348.067752] do_raw_spin_lock+0x177/0x190 [17348.067757] _raw_spin_lock+0x21/0x30 [17348.067763] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17348.068052] ? do_raw_spin_unlock+0x75/0x190 [17348.068059] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17348.070778] ? do_wait_intr+0xf0/0xf0 [17348.070788] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17348.071015] mdc_reint+0x2d/0x110 [mdc] [17348.071050] mdc_unlink+0x658/0x870 [mdc] [17348.071079] lmv_unlink+0x34d/0xbd0 [lmv] [17348.071614] ll_unlink+0x24d/0x7f0 [lustre] [17348.071817] vfs_unlink+0x202/0x330 [17348.071828] do_unlinkat+0x2fa/0x460 [17348.071834] __x64_sys_unlink+0x37/0x50 [17348.071838] do_syscall_64+0xc1/0x440 [17348.071845] entry_SYSCALL_64_after_hwframe+0x49/0xae [17348.071855] RIP: 0033:0x7f151bdee10b [17348.071863] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [17348.071868] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [17348.071873] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [17348.071876] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [17348.071879] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17348.071882] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17348.071884] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17348.538268] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17348.617484] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17348.644226] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17348.659440] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17348.673576] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17348.730521] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [17348.737674] RAX: 0000000000004e18 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17348.746321] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [17348.750665] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [17348.760489] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [17348.769413] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [17348.783853] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [17348.795251] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17348.805016] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [17348.810121] Call Trace: [17348.811477] [17348.813980] ? show_regs.cold.9+0x22/0x2f [17348.818364] ? watchdog_timer_fn.cold.6+0x62/0x10a [17348.825664] ? __hrtimer_run_queues+0x1c2/0x4d0 [17348.835272] ? watchdog+0x40/0x40 [17348.840108] ? hrtimer_interrupt+0x128/0x2c0 [17348.842984] ? smp_apic_timer_interrupt+0xa6/0x270 [17348.846246] ? apic_timer_interrupt+0xf/0x20 [17348.853323] [17348.856493] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17348.864564] do_raw_spin_lock+0x177/0x190 [17348.865835] _raw_spin_lock+0x21/0x30 [17348.872153] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17348.875356] ? do_raw_spin_unlock+0x75/0x190 [17348.879976] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17348.881509] ? do_wait_intr+0xf0/0xf0 [17348.890483] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17348.894716] mdc_close+0x53b/0xd70 [mdc] [17348.898399] lmv_close+0x18d/0x490 [lmv] [17348.908209] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [17348.915401] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17348.926900] ll_md_real_close+0xa5/0x2a0 [lustre] [17348.928807] ll_md_close+0x274/0x800 [lustre] [17348.940284] ll_file_release+0x128/0xc50 [lustre] [17348.950202] __fput+0x10d/0x440 [17348.951324] ____fput+0x12/0x20 [17348.952448] task_work_run+0xc6/0x110 [17348.965181] exit_to_usermode_loop+0x1dd/0x1f0 [17348.968293] do_syscall_64+0x3d6/0x440 [17348.975514] entry_SYSCALL_64_after_hwframe+0x49/0xae [17348.992336] RIP: 0033:0x7f151bee3cd8 [17348.996943] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17349.007268] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17349.010058] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [17349.013000] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [17349.015941] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17349.018563] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17349.021227] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17353.697245] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15423s! [17353.700305] Showing busy workqueues and worker pools: [17353.702704] workqueue events: flags=0x0 [17353.708120] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17353.723510] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [17353.735248] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17353.740639] pending: kfree_rcu_monitor [17353.742803] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17353.747376] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [17353.758993] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17353.764500] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [17353.777967] workqueue events_long: flags=0x0 [17353.783276] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17353.791944] pending: lu_objects_flush [obdclass] [17353.797121] workqueue events_unbound: flags=0x2 [17353.802423] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [17353.807482] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17353.815643] workqueue events_power_efficient: flags=0x80 [17353.820818] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17353.827659] pending: do_cache_clean, neigh_periodic_work, check_lifetime [17353.833200] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17353.842414] pending: neigh_periodic_work [17353.850533] workqueue mm_percpu_wq: flags=0x8 [17353.856805] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17353.860175] pending: vmstat_update, lru_add_drain_per_cpu [17353.862334] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17353.870333] pending: vmstat_update [17353.873835] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17353.879444] pending: lru_add_drain_per_cpu [17353.882248] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [17353.894291] pending: lru_add_drain_per_cpu BAR(48) [17353.900467] workqueue writeback: flags=0x4a [17353.905428] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [17353.913445] pending: wb_workfn [17353.915616] workqueue xprtiod: flags=0xa [17353.917114] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17353.926561] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [17353.934298] workqueue ipv6_addrconf: flags=0x40008 [17353.938704] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17353.948341] pending: addrconf_verify_work [17353.953159] workqueue knbd0-recv: flags=0x1a [17353.962391] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17353.975128] in-flight: 658:recv_work [17353.980505] workqueue ptlrpc_pinger: flags=0x2 [17353.987177] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17353.990683] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [17353.996482] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15423s workers=4 idle: 32984 37 8 [17354.005429] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17338s workers=2 idle: 661 [17368.055295] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [17368.059682] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17368.061801] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [17368.076728] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17368.076751] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17368.076755] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17368.076769] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17368.076774] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17368.076788] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [17368.076791] RDX: 0000000000007cc1 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [17368.076793] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [17368.076795] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [17368.076798] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17368.076801] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [17368.076805] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17368.076807] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17368.076814] Call Trace: [17368.076820] [17368.076829] ? show_regs.cold.9+0x22/0x2f [17368.076839] ? watchdog_timer_fn.cold.6+0x62/0x10a [17368.076845] ? __hrtimer_run_queues+0x1c2/0x4d0 [17368.076850] ? watchdog+0x40/0x40 [17368.076857] ? hrtimer_interrupt+0x128/0x2c0 [17368.076867] ? smp_apic_timer_interrupt+0xa6/0x270 [17368.076876] ? apic_timer_interrupt+0xf/0x20 [17368.076881] [17368.076884] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17368.076889] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17368.076894] do_raw_spin_lock+0x177/0x190 [17368.076898] _raw_spin_lock+0x21/0x30 [17368.076916] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17368.077354] obd_mod_free+0x46/0x140 [ptlrpc] [17368.077577] mdc_commit_open+0x91/0xb0 [mdc] [17368.077618] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17368.077859] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17368.078155] after_reply+0xe2c/0x1780 [ptlrpc] [17368.078657] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17368.079010] ? __next_timer_interrupt+0x160/0x160 [17368.079023] ptlrpcd+0xa82/0xe40 [ptlrpc] [17368.079272] ? do_wait_intr+0xf0/0xf0 [17368.079289] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17368.079542] kthread+0x1d1/0x200 [17368.079548] ? set_kthread_struct+0x70/0x70 [17368.079552] ret_from_fork+0x1f/0x30 [17368.250170] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17368.309366] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17368.313274] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17368.316959] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17368.320140] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17368.324119] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17368.332855] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17368.336444] RAX: 0000000000006404 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17368.339867] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [17368.343008] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [17368.346880] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [17368.350182] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [17368.353527] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [17368.357373] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17368.360122] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17368.363476] Call Trace: [17368.364745] [17368.365725] ? show_regs.cold.9+0x22/0x2f [17368.368187] ? watchdog_timer_fn.cold.6+0x62/0x10a [17368.370820] ? __hrtimer_run_queues+0x1c2/0x4d0 [17368.373519] ? watchdog+0x40/0x40 [17368.375001] ? hrtimer_interrupt+0x128/0x2c0 [17368.377013] ? smp_apic_timer_interrupt+0xa6/0x270 [17368.378548] ? apic_timer_interrupt+0xf/0x20 [17368.380312] [17368.381152] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17368.382892] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17368.384709] do_raw_spin_lock+0x177/0x190 [17368.385819] _raw_spin_lock+0x21/0x30 [17368.386878] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17368.389282] process_one_work+0x2c8/0x700 [17368.390493] worker_thread+0x296/0x6e0 [17368.391682] ? rescuer_thread+0x570/0x570 [17368.393245] kthread+0x1d1/0x200 [17368.394861] ? set_kthread_struct+0x70/0x70 [17368.397071] ret_from_fork+0x1f/0x30 [17376.057022] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [17376.059406] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17376.096635] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17376.104108] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17376.108106] RIP: 0010:native_safe_halt+0x2e/0x50 [17376.112128] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17376.124482] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17376.127650] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17376.130954] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [17376.133887] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [17376.137422] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [17376.141340] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [17376.145346] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [17376.149965] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17376.153098] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [17376.156292] Call Trace: [17376.158711] [17376.159480] ? show_regs.cold.9+0x22/0x2f [17376.163758] ? watchdog_timer_fn.cold.6+0x62/0x10a [17376.165408] ? __hrtimer_run_queues+0x1c2/0x4d0 [17376.169143] ? watchdog+0x40/0x40 [17376.175831] ? hrtimer_interrupt+0x128/0x2c0 [17376.183164] ? smp_apic_timer_interrupt+0xa6/0x270 [17376.185657] ? apic_timer_interrupt+0xf/0x20 [17376.191416] [17376.192201] ? native_safe_halt+0x2e/0x50 [17376.193375] kvm_wait+0xcd/0xe0 [17376.194368] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17376.204439] do_raw_spin_lock+0x177/0x190 [17376.213237] _raw_spin_lock+0x21/0x30 [17376.220677] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17376.225092] ? do_raw_spin_unlock+0x75/0x190 [17376.226886] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17376.228337] ? do_wait_intr+0xf0/0xf0 [17376.229251] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17376.231278] mdc_close+0x53b/0xd70 [mdc] [17376.232408] lmv_close+0x18d/0x490 [lmv] [17376.233409] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [17376.235420] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17376.237361] ll_md_real_close+0xa5/0x2a0 [lustre] [17376.239919] ll_md_close+0x274/0x800 [lustre] [17376.242146] ll_file_release+0x128/0xc50 [lustre] [17376.244701] __fput+0x10d/0x440 [17376.246874] ____fput+0x12/0x20 [17376.248844] task_work_run+0xc6/0x110 [17376.251032] exit_to_usermode_loop+0x1dd/0x1f0 [17376.253189] do_syscall_64+0x3d6/0x440 [17376.255057] entry_SYSCALL_64_after_hwframe+0x49/0xae [17376.257424] RIP: 0033:0x7f151bee3cd8 [17376.259166] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17376.274191] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17376.279411] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [17376.283212] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [17376.286582] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17376.290330] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17376.295057] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17380.052029] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [17380.054320] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17380.087265] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17380.097429] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17380.103185] RIP: 0010:native_safe_halt+0x2e/0x50 [17380.108383] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17380.122316] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17380.127278] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17380.133441] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [17380.137419] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [17380.144789] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [17380.151350] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [17380.158194] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [17380.171765] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17380.173200] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [17380.176175] Call Trace: [17380.177146] [17380.178237] ? show_regs.cold.9+0x22/0x2f [17380.180209] ? watchdog_timer_fn.cold.6+0x62/0x10a [17380.182189] ? __hrtimer_run_queues+0x1c2/0x4d0 [17380.186125] ? watchdog+0x40/0x40 [17380.189231] ? hrtimer_interrupt+0x128/0x2c0 [17380.193349] ? smp_apic_timer_interrupt+0xa6/0x270 [17380.196432] ? apic_timer_interrupt+0xf/0x20 [17380.197999] [17380.200062] ? native_safe_halt+0x2e/0x50 [17380.202349] kvm_wait+0xcd/0xe0 [17380.203516] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17380.206093] do_raw_spin_lock+0x177/0x190 [17380.208149] _raw_spin_lock+0x21/0x30 [17380.209856] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17380.212126] ? do_raw_spin_unlock+0x75/0x190 [17380.216173] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17380.220154] ? do_wait_intr+0xf0/0xf0 [17380.222418] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17380.226364] mdc_reint+0x2d/0x110 [mdc] [17380.228321] mdc_unlink+0x658/0x870 [mdc] [17380.230520] lmv_unlink+0x34d/0xbd0 [lmv] [17380.238506] ll_unlink+0x24d/0x7f0 [lustre] [17380.243175] vfs_unlink+0x202/0x330 [17380.244947] do_unlinkat+0x2fa/0x460 [17380.249118] __x64_sys_unlink+0x37/0x50 [17380.255378] do_syscall_64+0xc1/0x440 [17380.256652] entry_SYSCALL_64_after_hwframe+0x49/0xae [17380.262978] RIP: 0033:0x7f151bdee10b [17380.264003] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [17380.275178] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [17380.281146] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [17380.287670] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [17380.292102] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17380.294560] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17380.296401] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17384.424472] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15453s! [17384.437115] Showing busy workqueues and worker pools: [17384.439115] workqueue events: flags=0x0 [17384.440398] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17384.445779] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [17384.460294] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17384.465861] pending: kfree_rcu_monitor [17384.468709] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17384.474278] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [17384.486564] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17384.497315] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [17384.503932] workqueue events_long: flags=0x0 [17384.508951] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17384.513042] pending: lu_objects_flush [obdclass] [17384.522942] workqueue events_unbound: flags=0x2 [17384.532064] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [17384.543259] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17384.555294] workqueue events_power_efficient: flags=0x80 [17384.559358] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17384.567787] pending: do_cache_clean, neigh_periodic_work, check_lifetime [17384.573630] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17384.579192] pending: neigh_periodic_work [17384.583398] workqueue mm_percpu_wq: flags=0x8 [17384.585213] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17384.599888] pending: vmstat_update, lru_add_drain_per_cpu [17384.604707] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17384.607937] pending: vmstat_update [17384.609247] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17384.616308] pending: lru_add_drain_per_cpu [17384.622312] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [17384.624720] pending: lru_add_drain_per_cpu BAR(48) [17384.626745] workqueue writeback: flags=0x4a [17384.628780] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [17384.634218] pending: wb_workfn [17384.636726] workqueue xprtiod: flags=0xa [17384.638154] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17384.640267] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [17384.645281] workqueue ipv6_addrconf: flags=0x40008 [17384.651383] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17384.656376] pending: addrconf_verify_work [17384.659958] workqueue knbd0-recv: flags=0x1a [17384.662975] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17384.666685] in-flight: 658:recv_work [17384.670233] workqueue ptlrpc_pinger: flags=0x2 [17384.673874] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17384.678753] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [17384.682508] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15454s workers=4 idle: 32984 37 8 [17384.692227] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17369s workers=2 idle: 661 [17396.056453] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [17396.059159] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [17396.064666] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17396.064745] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17396.064753] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17396.064758] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17396.065034] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17396.065044] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17396.065048] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17396.065053] RAX: 0000000000001332 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17396.065055] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [17396.065058] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [17396.065060] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [17396.065062] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [17396.065065] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [17396.065068] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17396.065071] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17396.065080] Call Trace: [17396.065085] [17396.065089] ? show_regs.cold.9+0x22/0x2f [17396.065099] ? watchdog_timer_fn.cold.6+0x62/0x10a [17396.065105] ? __hrtimer_run_queues+0x1c2/0x4d0 [17396.065110] ? watchdog+0x40/0x40 [17396.065115] ? hrtimer_interrupt+0x128/0x2c0 [17396.065119] ? smp_apic_timer_interrupt+0xa6/0x270 [17396.065127] ? apic_timer_interrupt+0xf/0x20 [17396.065132] [17396.065135] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17396.065140] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17396.065145] do_raw_spin_lock+0x177/0x190 [17396.065150] _raw_spin_lock+0x21/0x30 [17396.065154] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17396.065352] process_one_work+0x2c8/0x700 [17396.065359] worker_thread+0x296/0x6e0 [17396.065364] ? rescuer_thread+0x570/0x570 [17396.065368] kthread+0x1d1/0x200 [17396.065373] ? set_kthread_struct+0x70/0x70 [17396.065376] ret_from_fork+0x1f/0x30 [17396.255102] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17396.290060] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17396.294500] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17396.298066] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17396.300190] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17396.307138] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17396.309746] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [17396.312058] RDX: 000000000000316b RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [17396.314743] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [17396.317156] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [17396.319609] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17396.321901] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [17396.324701] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17396.326688] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17396.328557] Call Trace: [17396.329560] [17396.330191] ? show_regs.cold.9+0x22/0x2f [17396.331594] ? watchdog_timer_fn.cold.6+0x62/0x10a [17396.333722] ? __hrtimer_run_queues+0x1c2/0x4d0 [17396.335357] ? watchdog+0x40/0x40 [17396.336523] ? hrtimer_interrupt+0x128/0x2c0 [17396.337980] ? smp_apic_timer_interrupt+0xa6/0x270 [17396.339695] ? apic_timer_interrupt+0xf/0x20 [17396.341118] [17396.341893] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17396.343585] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17396.344927] do_raw_spin_lock+0x177/0x190 [17396.345933] _raw_spin_lock+0x21/0x30 [17396.347030] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17396.348907] obd_mod_free+0x46/0x140 [ptlrpc] [17396.350288] mdc_commit_open+0x91/0xb0 [mdc] [17396.351487] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17396.353144] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17396.355037] after_reply+0xe2c/0x1780 [ptlrpc] [17396.357277] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17396.359265] ? __next_timer_interrupt+0x160/0x160 [17396.362739] ptlrpcd+0xa82/0xe40 [ptlrpc] [17396.364323] ? do_wait_intr+0xf0/0xf0 [17396.367364] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17396.369801] kthread+0x1d1/0x200 [17396.371358] ? set_kthread_struct+0x70/0x70 [17396.372904] ret_from_fork+0x1f/0x30 [17404.057026] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [17404.064495] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17404.088082] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17404.097146] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17404.101812] RIP: 0010:native_safe_halt+0x2e/0x50 [17404.103004] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17404.110226] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17404.113075] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17404.116248] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [17404.119872] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [17404.122867] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [17404.126243] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [17404.128731] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [17404.131875] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17404.133992] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [17404.136671] Call Trace: [17404.137652] [17404.138179] ? show_regs.cold.9+0x22/0x2f [17404.139714] ? watchdog_timer_fn.cold.6+0x62/0x10a [17404.141173] ? __hrtimer_run_queues+0x1c2/0x4d0 [17404.143530] ? watchdog+0x40/0x40 [17404.144425] ? hrtimer_interrupt+0x128/0x2c0 [17404.145991] ? smp_apic_timer_interrupt+0xa6/0x270 [17404.147558] ? apic_timer_interrupt+0xf/0x20 [17404.149054] [17404.149746] ? native_safe_halt+0x2e/0x50 [17404.151042] kvm_wait+0xcd/0xe0 [17404.152095] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17404.153865] do_raw_spin_lock+0x177/0x190 [17404.155320] _raw_spin_lock+0x21/0x30 [17404.156927] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17404.158910] ? do_raw_spin_unlock+0x75/0x190 [17404.160425] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17404.161962] ? do_wait_intr+0xf0/0xf0 [17404.163281] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17404.165039] mdc_close+0x53b/0xd70 [mdc] [17404.166436] lmv_close+0x18d/0x490 [lmv] [17404.167861] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [17404.170050] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17404.172146] ll_md_real_close+0xa5/0x2a0 [lustre] [17404.174002] ll_md_close+0x274/0x800 [lustre] [17404.176113] ll_file_release+0x128/0xc50 [lustre] [17404.177883] __fput+0x10d/0x440 [17404.179034] ____fput+0x12/0x20 [17404.180167] task_work_run+0xc6/0x110 [17404.181566] exit_to_usermode_loop+0x1dd/0x1f0 [17404.183096] do_syscall_64+0x3d6/0x440 [17404.184455] entry_SYSCALL_64_after_hwframe+0x49/0xae [17404.186215] RIP: 0033:0x7f151bee3cd8 [17404.187621] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17404.194320] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17404.197435] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [17404.199711] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [17404.202088] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17404.204439] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17404.207071] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17412.052264] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [17412.055860] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17412.078973] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17412.084363] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17412.091851] RIP: 0010:native_safe_halt+0x2e/0x50 [17412.094396] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17412.103553] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17412.112920] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17412.117626] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [17412.124751] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [17412.134882] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [17412.144895] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [17412.156778] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [17412.159403] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17412.166547] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [17412.173819] Call Trace: [17412.175023] [17412.176036] ? show_regs.cold.9+0x22/0x2f [17412.179722] ? watchdog_timer_fn.cold.6+0x62/0x10a [17412.181455] ? __hrtimer_run_queues+0x1c2/0x4d0 [17412.184450] ? watchdog+0x40/0x40 [17412.186304] ? hrtimer_interrupt+0x128/0x2c0 [17412.188464] ? smp_apic_timer_interrupt+0xa6/0x270 [17412.194821] ? apic_timer_interrupt+0xf/0x20 [17412.198751] [17412.201288] ? native_safe_halt+0x2e/0x50 [17412.203482] kvm_wait+0xcd/0xe0 [17412.209305] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17412.217443] do_raw_spin_lock+0x177/0x190 [17412.219169] _raw_spin_lock+0x21/0x30 [17412.221645] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17412.225905] ? do_raw_spin_unlock+0x75/0x190 [17412.228000] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17412.234524] ? do_wait_intr+0xf0/0xf0 [17412.236009] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17412.239654] mdc_reint+0x2d/0x110 [mdc] [17412.241107] mdc_unlink+0x658/0x870 [mdc] [17412.242699] lmv_unlink+0x34d/0xbd0 [lmv] [17412.245375] ll_unlink+0x24d/0x7f0 [lustre] [17412.246676] vfs_unlink+0x202/0x330 [17412.247724] do_unlinkat+0x2fa/0x460 [17412.248998] __x64_sys_unlink+0x37/0x50 [17412.250084] do_syscall_64+0xc1/0x440 [17412.251670] entry_SYSCALL_64_after_hwframe+0x49/0xae [17412.253320] RIP: 0033:0x7f151bdee10b [17412.254480] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [17412.260315] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [17412.263019] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [17412.265326] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [17412.267474] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17412.269768] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17412.272694] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17415.139332] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15484s! [17415.143644] Showing busy workqueues and worker pools: [17415.147854] workqueue events: flags=0x0 [17415.153327] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17415.158744] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [17415.167976] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17415.176592] pending: kfree_rcu_monitor [17415.179592] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17415.187731] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [17415.198283] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17415.205889] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [17415.210086] workqueue events_long: flags=0x0 [17415.219990] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17415.223537] pending: lu_objects_flush [obdclass] [17415.225517] workqueue events_unbound: flags=0x2 [17415.233528] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [17415.239729] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17415.250536] workqueue events_power_efficient: flags=0x80 [17415.257550] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17415.264309] pending: do_cache_clean, neigh_periodic_work, check_lifetime [17415.267517] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17415.271709] pending: neigh_periodic_work [17415.278191] workqueue mm_percpu_wq: flags=0x8 [17415.279782] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17415.284277] pending: vmstat_update, lru_add_drain_per_cpu [17415.291322] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17415.293658] pending: vmstat_update [17415.298531] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17415.300868] pending: lru_add_drain_per_cpu [17415.313469] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [17415.317616] pending: lru_add_drain_per_cpu BAR(48) [17415.320530] workqueue writeback: flags=0x4a [17415.322845] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [17415.327348] pending: wb_workfn [17415.331030] workqueue xprtiod: flags=0xa [17415.333397] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17415.336768] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [17415.350540] workqueue ipv6_addrconf: flags=0x40008 [17415.356815] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17415.361558] pending: addrconf_verify_work [17415.365882] workqueue knbd0-recv: flags=0x1a [17415.370226] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17415.373761] in-flight: 658:recv_work [17415.375210] workqueue ptlrpc_pinger: flags=0x2 [17415.382085] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17415.386016] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [17415.391956] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15484s workers=4 idle: 32984 37 8 [17415.397318] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17400s workers=2 idle: 661 [17424.055669] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [17424.056032] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [17424.061905] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver [17424.066525] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17424.066614] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17424.066622] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17424.066628] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17424.067111] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17424.067138] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17424.067145] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17424.067150] RAX: 0000000000003450 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17424.067153] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [17424.067155] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [17424.067157] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [17424.067172] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [17424.067176] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [17424.067179] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17424.067182] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17424.067191] Call Trace: [17424.067196] [17424.067202] ? show_regs.cold.9+0x22/0x2f [17424.067212] ? watchdog_timer_fn.cold.6+0x62/0x10a [17424.067225] ? __hrtimer_run_queues+0x1c2/0x4d0 [17424.067231] ? watchdog+0x40/0x40 [17424.067236] ? hrtimer_interrupt+0x128/0x2c0 [17424.067240] ? smp_apic_timer_interrupt+0xa6/0x270 [17424.067248] ? apic_timer_interrupt+0xf/0x20 [17424.067253] [17424.067257] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17424.067262] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17424.067272] do_raw_spin_lock+0x177/0x190 [17424.067278] _raw_spin_lock+0x21/0x30 [17424.067283] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17424.067529] process_one_work+0x2c8/0x700 [17424.067536] worker_thread+0x296/0x6e0 [17424.067541] ? rescuer_thread+0x570/0x570 [17424.067549] kthread+0x1d1/0x200 [17424.067553] ? set_kthread_struct+0x70/0x70 [17424.067557] ret_from_fork+0x1f/0x30 [17424.295613] intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17424.316430] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17424.330026] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17424.335548] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17424.339017] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17424.356279] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17424.362305] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [17424.371038] RDX: 0000000000002e7b RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [17424.377517] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [17424.380927] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [17424.384384] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17424.390612] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [17424.395982] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17424.400751] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17424.406377] Call Trace: [17424.408572] [17424.409589] ? show_regs.cold.9+0x22/0x2f [17424.411452] ? watchdog_timer_fn.cold.6+0x62/0x10a [17424.415248] ? __hrtimer_run_queues+0x1c2/0x4d0 [17424.417941] ? watchdog+0x40/0x40 [17424.420868] ? hrtimer_interrupt+0x128/0x2c0 [17424.425669] ? smp_apic_timer_interrupt+0xa6/0x270 [17424.428654] ? apic_timer_interrupt+0xf/0x20 [17424.432680] [17424.434915] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17424.438965] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17424.441795] do_raw_spin_lock+0x177/0x190 [17424.443873] _raw_spin_lock+0x21/0x30 [17424.445850] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17424.448777] obd_mod_free+0x46/0x140 [ptlrpc] [17424.453925] mdc_commit_open+0x91/0xb0 [mdc] [17424.455360] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17424.457274] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17424.459385] after_reply+0xe2c/0x1780 [ptlrpc] [17424.461368] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17424.463660] ? __next_timer_interrupt+0x160/0x160 [17424.466571] ptlrpcd+0xa82/0xe40 [ptlrpc] [17424.468593] ? do_wait_intr+0xf0/0xf0 [17424.469698] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17424.472442] kthread+0x1d1/0x200 [17424.474694] ? set_kthread_struct+0x70/0x70 [17424.477250] ret_from_fork+0x1f/0x30 [17432.057534] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [17432.066816] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17432.114942] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17432.127255] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17432.135932] RIP: 0010:native_safe_halt+0x2e/0x50 [17432.145493] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17432.162336] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17432.166851] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17432.171789] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [17432.175800] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [17432.180332] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [17432.183629] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [17432.187686] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [17432.193254] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17432.196848] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [17432.199878] Call Trace: [17432.203970] [17432.205679] ? show_regs.cold.9+0x22/0x2f [17432.208887] ? watchdog_timer_fn.cold.6+0x62/0x10a [17432.212836] ? __hrtimer_run_queues+0x1c2/0x4d0 [17432.215883] ? watchdog+0x40/0x40 [17432.217826] ? hrtimer_interrupt+0x128/0x2c0 [17432.220594] ? smp_apic_timer_interrupt+0xa6/0x270 [17432.222950] ? apic_timer_interrupt+0xf/0x20 [17432.225388] [17432.226750] ? native_safe_halt+0x2e/0x50 [17432.229371] kvm_wait+0xcd/0xe0 [17432.230854] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17432.233722] do_raw_spin_lock+0x177/0x190 [17432.235815] _raw_spin_lock+0x21/0x30 [17432.237830] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17432.241452] ? do_raw_spin_unlock+0x75/0x190 [17432.243922] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17432.246513] ? do_wait_intr+0xf0/0xf0 [17432.248454] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17432.250844] mdc_close+0x53b/0xd70 [mdc] [17432.252809] lmv_close+0x18d/0x490 [lmv] [17432.255228] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [17432.257223] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17432.259530] ll_md_real_close+0xa5/0x2a0 [lustre] [17432.262468] ll_md_close+0x274/0x800 [lustre] [17432.264781] ll_file_release+0x128/0xc50 [lustre] [17432.267570] __fput+0x10d/0x440 [17432.269529] ____fput+0x12/0x20 [17432.271276] task_work_run+0xc6/0x110 [17432.273359] exit_to_usermode_loop+0x1dd/0x1f0 [17432.275610] do_syscall_64+0x3d6/0x440 [17432.277909] entry_SYSCALL_64_after_hwframe+0x49/0xae [17432.279668] RIP: 0033:0x7f151bee3cd8 [17432.280820] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17432.287953] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17432.290614] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [17432.292939] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [17432.295423] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17432.297804] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17432.300728] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17440.052077] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [17440.055672] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17440.087027] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17440.094526] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17440.097581] RIP: 0010:native_safe_halt+0x2e/0x50 [17440.099323] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17440.119331] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17440.135119] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17440.140823] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [17440.148498] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [17440.159464] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [17440.166820] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [17440.173576] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [17440.182315] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17440.184310] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [17440.189416] Call Trace: [17440.190398] [17440.191272] ? show_regs.cold.9+0x22/0x2f [17440.194323] ? watchdog_timer_fn.cold.6+0x62/0x10a [17440.198548] ? __hrtimer_run_queues+0x1c2/0x4d0 [17440.202543] ? watchdog+0x40/0x40 [17440.205641] ? hrtimer_interrupt+0x128/0x2c0 [17440.209388] ? smp_apic_timer_interrupt+0xa6/0x270 [17440.211576] ? apic_timer_interrupt+0xf/0x20 [17440.214463] [17440.219274] ? native_safe_halt+0x2e/0x50 [17440.228355] kvm_wait+0xcd/0xe0 [17440.231799] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17440.235391] do_raw_spin_lock+0x177/0x190 [17440.239703] _raw_spin_lock+0x21/0x30 [17440.247308] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17440.250747] ? do_raw_spin_unlock+0x75/0x190 [17440.253520] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17440.259024] ? do_wait_intr+0xf0/0xf0 [17440.260153] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17440.267844] mdc_reint+0x2d/0x110 [mdc] [17440.270604] mdc_unlink+0x658/0x870 [mdc] [17440.272059] lmv_unlink+0x34d/0xbd0 [lmv] [17440.274327] ll_unlink+0x24d/0x7f0 [lustre] [17440.276548] vfs_unlink+0x202/0x330 [17440.280520] do_unlinkat+0x2fa/0x460 [17440.282853] __x64_sys_unlink+0x37/0x50 [17440.285705] do_syscall_64+0xc1/0x440 [17440.288360] entry_SYSCALL_64_after_hwframe+0x49/0xae [17440.293321] RIP: 0033:0x7f151bdee10b [17440.295306] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [17440.314480] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [17440.318306] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [17440.323137] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [17440.329138] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17440.338826] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17440.347675] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17445.860276] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15515s! [17445.866591] Showing busy workqueues and worker pools: [17445.869782] workqueue events: flags=0x0 [17445.871882] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17445.876105] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [17445.886912] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17445.891313] pending: kfree_rcu_monitor [17445.892769] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17445.896933] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [17445.901014] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17445.906037] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [17445.911029] workqueue events_long: flags=0x0 [17445.917581] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17445.922368] pending: lu_objects_flush [obdclass] [17445.925863] workqueue events_unbound: flags=0x2 [17445.929223] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [17445.933627] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17445.940381] workqueue events_power_efficient: flags=0x80 [17445.943077] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17445.946501] pending: do_cache_clean, neigh_periodic_work, check_lifetime [17445.950014] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17445.953053] pending: neigh_periodic_work [17445.955464] workqueue mm_percpu_wq: flags=0x8 [17445.957449] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17445.961621] pending: vmstat_update, lru_add_drain_per_cpu [17445.965259] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17445.970674] pending: vmstat_update [17445.974536] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17445.978704] pending: lru_add_drain_per_cpu [17445.983031] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [17445.987349] pending: lru_add_drain_per_cpu BAR(48) [17446.008533] workqueue writeback: flags=0x4a [17446.017723] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [17446.021543] pending: wb_workfn [17446.027376] workqueue xprtiod: flags=0xa [17446.036296] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17446.043239] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [17446.049932] workqueue ipv6_addrconf: flags=0x40008 [17446.052751] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17446.057304] pending: addrconf_verify_work [17446.063016] workqueue knbd0-recv: flags=0x1a [17446.068026] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17446.076424] in-flight: 658:recv_work [17446.081276] workqueue ptlrpc_pinger: flags=0x2 [17446.088799] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17446.090984] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [17446.104517] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15515s workers=4 idle: 32984 37 8 [17446.116057] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17430s workers=2 idle: 661 [17452.061709] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [17452.062136] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [17452.068031] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17452.068114] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17452.068122] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17452.068125] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17452.068140] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17452.068144] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17452.068149] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [17452.068151] RDX: 0000000000003635 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [17452.068153] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [17452.068155] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [17452.068157] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17452.068160] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [17452.068163] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17452.068165] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17452.068174] Call Trace: [17452.068180] [17452.068186] ? show_regs.cold.9+0x22/0x2f [17452.068196] ? watchdog_timer_fn.cold.6+0x62/0x10a [17452.068204] ? __hrtimer_run_queues+0x1c2/0x4d0 [17452.068208] ? watchdog+0x40/0x40 [17452.068213] ? hrtimer_interrupt+0x128/0x2c0 [17452.068217] ? smp_apic_timer_interrupt+0xa6/0x270 [17452.068225] ? apic_timer_interrupt+0xf/0x20 [17452.068230] [17452.068233] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17452.068238] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17452.068243] do_raw_spin_lock+0x177/0x190 [17452.068248] _raw_spin_lock+0x21/0x30 [17452.068253] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17452.068527] obd_mod_free+0x46/0x140 [ptlrpc] [17452.068755] mdc_commit_open+0x91/0xb0 [mdc] [17452.068793] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17452.069006] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17452.069216] after_reply+0xe2c/0x1780 [ptlrpc] [17452.069417] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17452.069611] ? __next_timer_interrupt+0x160/0x160 [17452.069620] ptlrpcd+0xa82/0xe40 [ptlrpc] [17452.069822] ? do_wait_intr+0xf0/0xf0 [17452.069831] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17452.070042] kthread+0x1d1/0x200 [17452.070048] ? set_kthread_struct+0x70/0x70 [17452.070052] ret_from_fork+0x1f/0x30 [17452.307626] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17452.327373] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17452.332118] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17452.336077] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17452.339151] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17452.342270] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17452.351508] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17452.355396] RAX: 0000000000005c90 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17452.358862] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [17452.362454] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [17452.365713] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [17452.370122] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [17452.374055] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [17452.378014] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17452.384557] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17452.391282] Call Trace: [17452.392969] [17452.393701] ? show_regs.cold.9+0x22/0x2f [17452.399922] ? watchdog_timer_fn.cold.6+0x62/0x10a [17452.404906] ? __hrtimer_run_queues+0x1c2/0x4d0 [17452.406524] ? watchdog+0x40/0x40 [17452.407680] ? hrtimer_interrupt+0x128/0x2c0 [17452.410746] ? smp_apic_timer_interrupt+0xa6/0x270 [17452.413476] ? apic_timer_interrupt+0xf/0x20 [17452.415820] [17452.416915] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17452.419560] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17452.422340] do_raw_spin_lock+0x177/0x190 [17452.425472] _raw_spin_lock+0x21/0x30 [17452.428636] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17452.433089] process_one_work+0x2c8/0x700 [17452.437563] worker_thread+0x296/0x6e0 [17452.441619] ? rescuer_thread+0x570/0x570 [17452.443829] kthread+0x1d1/0x200 [17452.445864] ? set_kthread_struct+0x70/0x70 [17452.450115] ret_from_fork+0x1f/0x30 [17460.062953] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [17460.065179] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17460.082600] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17460.087112] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17460.090570] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17460.093032] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17460.102097] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17460.105414] RAX: 0000000000003b85 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17460.108358] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [17460.112043] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [17460.115428] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [17460.118473] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [17460.121362] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [17460.125907] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17460.128324] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [17460.131483] Call Trace: [17460.132569] [17460.133555] ? show_regs.cold.9+0x22/0x2f [17460.135658] ? watchdog_timer_fn.cold.6+0x62/0x10a [17460.138002] ? __hrtimer_run_queues+0x1c2/0x4d0 [17460.139840] ? watchdog+0x40/0x40 [17460.141326] ? hrtimer_interrupt+0x128/0x2c0 [17460.142921] ? smp_apic_timer_interrupt+0xa6/0x270 [17460.144951] ? apic_timer_interrupt+0xf/0x20 [17460.147033] [17460.147664] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17460.152979] do_raw_spin_lock+0x177/0x190 [17460.154425] _raw_spin_lock+0x21/0x30 [17460.159142] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17460.161552] ? do_raw_spin_unlock+0x75/0x190 [17460.163429] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17460.165542] ? do_wait_intr+0xf0/0xf0 [17460.167368] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17460.169355] mdc_close+0x53b/0xd70 [mdc] [17460.170588] lmv_close+0x18d/0x490 [lmv] [17460.172668] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [17460.174961] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17460.177044] ll_md_real_close+0xa5/0x2a0 [lustre] [17460.179143] ll_md_close+0x274/0x800 [lustre] [17460.180897] ll_file_release+0x128/0xc50 [lustre] [17460.182615] __fput+0x10d/0x440 [17460.184287] ____fput+0x12/0x20 [17460.185283] task_work_run+0xc6/0x110 [17460.186280] exit_to_usermode_loop+0x1dd/0x1f0 [17460.187665] do_syscall_64+0x3d6/0x440 [17460.189097] entry_SYSCALL_64_after_hwframe+0x49/0xae [17460.190486] RIP: 0033:0x7f151bee3cd8 [17460.191654] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17460.199023] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17460.203305] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [17460.206926] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [17460.210609] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17460.214381] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17460.218335] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17464.970229] rcu: INFO: rcu_sched self-detected stall on CPU [17464.976121] rcu: 1-...!: (9344971 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [17464.989449] (t=15540323 jiffies g=249657 q=7206) [17464.996480] rcu: rcu_sched kthread starved for 15535275 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [17465.008229] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [17465.019552] rcu: RCU grace-period kthread stack dump: [17465.024483] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [17465.034308] Call Trace: [17465.035277] __schedule+0x351/0xcb0 [17465.036890] schedule+0xc0/0x180 [17465.043432] schedule_timeout+0xb4/0x190 [17465.046928] ? do_raw_spin_unlock+0x75/0x190 [17465.049495] ? __next_timer_interrupt+0x160/0x160 [17465.054504] ? prepare_to_swait_event+0xf3/0x140 [17465.063861] rcu_gp_kthread+0x8dd/0x15b0 [17465.065278] ? force_qs_rnp+0x2c0/0x2c0 [17465.066604] kthread+0x1d1/0x200 [17465.071017] ? set_kthread_struct+0x70/0x70 [17465.077504] ret_from_fork+0x1f/0x30 [17465.082016] rcu: Stack dump where RCU GP kthread last ran: [17465.085457] Sending NMI from CPU 1 to CPUs 2: [17465.087675] NMI backtrace for cpu 2 skipped: idling at native_safe_halt+0x2e/0x50 [17465.092398] NMI backtrace for cpu 1 [17465.102929] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17465.108586] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17465.113719] Call Trace: [17465.115058] [17465.116254] ? dump_stack+0xbb/0x10e [17465.118115] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [17465.121141] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [17465.123238] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [17465.127598] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [17465.133330] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [17465.136813] ? rcu_dump_cpu_stacks+0x11b/0x171 [17465.139603] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [17465.141997] ? update_process_times+0x6d/0xc0 [17465.144012] ? tick_sched_handle.isra.18+0x27/0x90 [17465.146799] ? tick_sched_timer+0x53/0xb0 [17465.148838] ? __hrtimer_run_queues+0x1c2/0x4d0 [17465.150433] ? tick_sched_handle.isra.18+0x90/0x90 [17465.152832] ? hrtimer_interrupt+0x128/0x2c0 [17465.161011] ? smp_apic_timer_interrupt+0xa6/0x270 [17465.166379] ? apic_timer_interrupt+0xf/0x20 [17465.172176] [17465.174283] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17465.180602] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17465.188179] ? do_raw_spin_lock+0x177/0x190 [17465.192929] ? _raw_spin_lock+0x21/0x30 [17465.196229] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17465.198889] ? obd_mod_free+0x46/0x140 [ptlrpc] [17465.201301] ? mdc_commit_open+0x91/0xb0 [mdc] [17465.209968] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17465.220098] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17465.231199] ? after_reply+0xe2c/0x1780 [ptlrpc] [17465.239546] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17465.246359] ? __next_timer_interrupt+0x160/0x160 [17465.251418] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [17465.254635] ? do_wait_intr+0xf0/0xf0 [17465.258045] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17465.267502] ? kthread+0x1d1/0x200 [17465.271385] ? set_kthread_struct+0x70/0x70 [17465.273549] ? ret_from_fork+0x1f/0x30 [17472.057153] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [17472.064469] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17472.130836] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17472.142214] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17472.159081] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17472.169469] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17472.197399] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17472.207228] RAX: 0000000000007e13 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17472.216706] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [17472.230333] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [17472.234873] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [17472.249370] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [17472.263422] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [17472.283444] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17472.295059] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [17472.306416] Call Trace: [17472.311044] [17472.316460] ? show_regs.cold.9+0x22/0x2f [17472.321957] ? watchdog_timer_fn.cold.6+0x62/0x10a [17472.332327] ? __hrtimer_run_queues+0x1c2/0x4d0 [17472.338984] ? watchdog+0x40/0x40 [17472.341211] ? hrtimer_interrupt+0x128/0x2c0 [17472.346444] ? smp_apic_timer_interrupt+0xa6/0x270 [17472.351976] ? apic_timer_interrupt+0xf/0x20 [17472.363580] [17472.364361] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17472.368320] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17472.370288] do_raw_spin_lock+0x177/0x190 [17472.386263] _raw_spin_lock+0x21/0x30 [17472.399098] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17472.411971] ? do_raw_spin_unlock+0x75/0x190 [17472.417540] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17472.431166] ? do_wait_intr+0xf0/0xf0 [17472.432438] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17472.438027] mdc_reint+0x2d/0x110 [mdc] [17472.440078] mdc_unlink+0x658/0x870 [mdc] [17472.441598] lmv_unlink+0x34d/0xbd0 [lmv] [17472.445347] ll_unlink+0x24d/0x7f0 [lustre] [17472.450432] vfs_unlink+0x202/0x330 [17472.454504] do_unlinkat+0x2fa/0x460 [17472.458072] __x64_sys_unlink+0x37/0x50 [17472.461821] do_syscall_64+0xc1/0x440 [17472.464570] entry_SYSCALL_64_after_hwframe+0x49/0xae [17472.466510] RIP: 0033:0x7f151bdee10b [17472.468812] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [17472.488462] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [17472.497985] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [17472.504269] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [17472.512230] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17472.520095] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17472.525509] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17476.577038] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15545s! [17476.580283] Showing busy workqueues and worker pools: [17476.581917] workqueue events: flags=0x0 [17476.584310] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17476.589190] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [17476.594020] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17476.596161] pending: kfree_rcu_monitor [17476.597485] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17476.599976] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [17476.603551] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17476.606497] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [17476.609451] workqueue events_long: flags=0x0 [17476.611040] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17476.613233] pending: lu_objects_flush [obdclass] [17476.615161] workqueue events_unbound: flags=0x2 [17476.616677] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [17476.618935] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17476.622470] workqueue events_power_efficient: flags=0x80 [17476.624490] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17476.627171] pending: do_cache_clean, neigh_periodic_work, check_lifetime [17476.629856] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17476.633473] pending: neigh_periodic_work [17476.636115] workqueue mm_percpu_wq: flags=0x8 [17476.638352] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17476.643811] pending: vmstat_update, lru_add_drain_per_cpu [17476.646267] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17476.648521] pending: vmstat_update [17476.650178] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17476.653394] pending: lru_add_drain_per_cpu [17476.655389] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [17476.658062] pending: lru_add_drain_per_cpu BAR(48) [17476.659823] workqueue writeback: flags=0x4a [17476.661493] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [17476.663683] pending: wb_workfn [17476.664937] workqueue xprtiod: flags=0xa [17476.666165] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17476.668660] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [17476.671299] workqueue ipv6_addrconf: flags=0x40008 [17476.673188] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17476.675358] pending: addrconf_verify_work [17476.677275] workqueue knbd0-recv: flags=0x1a [17476.678955] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17476.681112] in-flight: 658:recv_work [17476.682365] workqueue ptlrpc_pinger: flags=0x2 [17476.683903] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17476.685934] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [17476.688966] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15546s workers=4 idle: 32984 37 8 [17476.693050] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17461s workers=2 idle: 661 [17488.060886] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [17488.064338] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17488.095715] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17488.106662] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17488.112984] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17488.118231] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17488.135688] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17488.141256] RAX: 0000000000005619 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17488.152354] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [17488.172899] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [17488.179081] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [17488.183654] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [17488.201912] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [17488.214308] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17488.220285] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [17488.225823] Call Trace: [17488.227717] [17488.228771] ? show_regs.cold.9+0x22/0x2f [17488.233931] ? watchdog_timer_fn.cold.6+0x62/0x10a [17488.238229] ? __hrtimer_run_queues+0x1c2/0x4d0 [17488.243185] ? watchdog+0x40/0x40 [17488.247171] ? hrtimer_interrupt+0x128/0x2c0 [17488.253378] ? smp_apic_timer_interrupt+0xa6/0x270 [17488.258883] ? apic_timer_interrupt+0xf/0x20 [17488.265053] [17488.268371] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17488.274287] do_raw_spin_lock+0x177/0x190 [17488.279177] _raw_spin_lock+0x21/0x30 [17488.280807] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17488.287871] ? do_raw_spin_unlock+0x75/0x190 [17488.295785] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17488.303759] ? do_wait_intr+0xf0/0xf0 [17488.308043] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17488.315672] mdc_close+0x53b/0xd70 [mdc] [17488.319344] lmv_close+0x18d/0x490 [lmv] [17488.323271] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [17488.326908] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17488.334026] ll_md_real_close+0xa5/0x2a0 [lustre] [17488.342689] ll_md_close+0x274/0x800 [lustre] [17488.349002] ll_file_release+0x128/0xc50 [lustre] [17488.351917] __fput+0x10d/0x440 [17488.354358] ____fput+0x12/0x20 [17488.356126] task_work_run+0xc6/0x110 [17488.366215] exit_to_usermode_loop+0x1dd/0x1f0 [17488.375376] do_syscall_64+0x3d6/0x440 [17488.380264] entry_SYSCALL_64_after_hwframe+0x49/0xae [17488.387930] RIP: 0033:0x7f151bee3cd8 [17488.395027] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17488.426930] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17488.431908] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [17488.435051] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [17488.445001] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17488.453094] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17488.461522] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17492.055130] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [17492.060724] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [17492.061286] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17492.061355] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17492.061362] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17492.061364] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17492.061391] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17492.061472] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17492.061477] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [17492.061479] RDX: 00000000000050a1 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [17492.061481] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [17492.061483] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [17492.061485] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17492.061488] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [17492.061507] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17492.061510] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17492.061518] Call Trace: [17492.061523] [17492.061528] ? show_regs.cold.9+0x22/0x2f [17492.061538] ? watchdog_timer_fn.cold.6+0x62/0x10a [17492.061543] ? __hrtimer_run_queues+0x1c2/0x4d0 [17492.061552] ? watchdog+0x40/0x40 [17492.061631] ? hrtimer_interrupt+0x128/0x2c0 [17492.061636] ? smp_apic_timer_interrupt+0xa6/0x270 [17492.061644] ? apic_timer_interrupt+0xf/0x20 [17492.061649] [17492.061652] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17492.061656] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17492.061661] do_raw_spin_lock+0x177/0x190 [17492.061667] _raw_spin_lock+0x21/0x30 [17492.061672] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17492.062167] obd_mod_free+0x46/0x140 [ptlrpc] [17492.062549] mdc_commit_open+0x91/0xb0 [mdc] [17492.062737] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17492.063109] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17492.063719] after_reply+0xe2c/0x1780 [ptlrpc] [17492.064160] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17492.064668] ? __next_timer_interrupt+0x160/0x160 [17492.064677] ptlrpcd+0xa82/0xe40 [ptlrpc] [17492.064981] ? do_wait_intr+0xf0/0xf0 [17492.064991] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17492.070752] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17492.070834] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17492.070841] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17492.070848] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17492.071378] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17492.071389] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17492.071393] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17492.071655] RAX: 00000000000028ea RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17492.071657] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [17492.071660] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [17492.071662] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [17492.071664] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [17492.071667] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [17492.071670] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17492.071672] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17492.071681] Call Trace: [17492.071692] [17492.071698] ? show_regs.cold.9+0x22/0x2f [17492.071707] ? watchdog_timer_fn.cold.6+0x62/0x10a [17492.071713] ? __hrtimer_run_queues+0x1c2/0x4d0 [17492.071718] ? watchdog+0x40/0x40 [17492.071722] ? hrtimer_interrupt+0x128/0x2c0 [17492.071725] ? smp_apic_timer_interrupt+0xa6/0x270 [17492.071734] ? apic_timer_interrupt+0xf/0x20 [17492.071738] [17492.071741] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17492.071745] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17492.071751] do_raw_spin_lock+0x177/0x190 [17492.071755] _raw_spin_lock+0x21/0x30 [17492.071760] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17492.072142] process_one_work+0x2c8/0x700 [17492.072150] worker_thread+0x296/0x6e0 [17492.072155] ? rescuer_thread+0x570/0x570 [17492.072159] kthread+0x1d1/0x200 [17492.072163] ? set_kthread_struct+0x70/0x70 [17492.072167] ret_from_fork+0x1f/0x30 [17492.468630] kthread+0x1d1/0x200 [17492.470267] ? set_kthread_struct+0x70/0x70 [17492.479611] ret_from_fork+0x1f/0x30 [17504.052058] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [17504.057949] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17504.080542] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17504.090766] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17504.097559] RIP: 0010:native_safe_halt+0x2e/0x50 [17504.104621] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17504.121798] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17504.126618] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17504.132139] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [17504.138036] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [17504.143223] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [17504.148098] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [17504.154067] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [17504.168960] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17504.175983] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [17504.179049] Call Trace: [17504.182945] [17504.185309] ? show_regs.cold.9+0x22/0x2f [17504.187708] ? watchdog_timer_fn.cold.6+0x62/0x10a [17504.189882] ? __hrtimer_run_queues+0x1c2/0x4d0 [17504.196764] ? watchdog+0x40/0x40 [17504.198961] ? hrtimer_interrupt+0x128/0x2c0 [17504.203120] ? smp_apic_timer_interrupt+0xa6/0x270 [17504.206985] ? apic_timer_interrupt+0xf/0x20 [17504.211654] [17504.213634] ? native_safe_halt+0x2e/0x50 [17504.215891] kvm_wait+0xcd/0xe0 [17504.218448] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17504.220464] do_raw_spin_lock+0x177/0x190 [17504.222037] _raw_spin_lock+0x21/0x30 [17504.223587] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17504.226017] ? do_raw_spin_unlock+0x75/0x190 [17504.227675] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17504.229919] ? do_wait_intr+0xf0/0xf0 [17504.231344] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17504.233460] mdc_reint+0x2d/0x110 [mdc] [17504.235008] mdc_unlink+0x658/0x870 [mdc] [17504.236527] lmv_unlink+0x34d/0xbd0 [lmv] [17504.238017] ll_unlink+0x24d/0x7f0 [lustre] [17504.239982] vfs_unlink+0x202/0x330 [17504.241594] do_unlinkat+0x2fa/0x460 [17504.242968] __x64_sys_unlink+0x37/0x50 [17504.244895] do_syscall_64+0xc1/0x440 [17504.246245] entry_SYSCALL_64_after_hwframe+0x49/0xae [17504.248444] RIP: 0033:0x7f151bdee10b [17504.249721] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [17504.259047] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [17504.262837] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [17504.265643] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [17504.268573] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17504.272025] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17504.276081] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17506.785411] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15576s! [17506.792118] Showing busy workqueues and worker pools: [17506.793611] workqueue events: flags=0x0 [17506.794683] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17506.796395] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [17506.802202] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17506.808901] pending: kfree_rcu_monitor [17506.811179] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17506.818873] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [17506.825489] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17506.830004] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [17506.837092] workqueue events_long: flags=0x0 [17506.839940] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17506.850981] pending: lu_objects_flush [obdclass] [17506.856900] workqueue events_unbound: flags=0x2 [17506.861285] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [17506.862846] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17506.874146] workqueue events_power_efficient: flags=0x80 [17506.882729] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17506.887736] pending: do_cache_clean, neigh_periodic_work, check_lifetime [17506.894184] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17506.898101] pending: neigh_periodic_work [17506.900979] workqueue mm_percpu_wq: flags=0x8 [17506.909895] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17506.922079] pending: vmstat_update, lru_add_drain_per_cpu [17506.930816] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17506.945249] pending: vmstat_update [17506.947293] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17506.960307] pending: lru_add_drain_per_cpu [17506.961903] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [17506.975136] pending: lru_add_drain_per_cpu BAR(48) [17506.984869] workqueue writeback: flags=0x4a [17506.986948] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [17506.993087] pending: wb_workfn [17506.997052] workqueue xprtiod: flags=0xa [17507.005931] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17507.013781] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [17507.024307] workqueue ipv6_addrconf: flags=0x40008 [17507.028684] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17507.032010] pending: addrconf_verify_work [17507.039861] workqueue knbd0-recv: flags=0x1a [17507.043472] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17507.048752] in-flight: 658:recv_work [17507.051853] workqueue ptlrpc_pinger: flags=0x2 [17507.054382] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17507.057176] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [17507.060981] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15576s workers=4 idle: 32984 37 8 [17507.065513] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17491s workers=2 idle: 661 [17516.058462] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [17516.071556] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17516.090519] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17516.095582] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17516.099011] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17516.101580] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17516.109178] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17516.111817] RAX: 0000000000004178 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17516.114708] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [17516.117880] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [17516.120609] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [17516.123031] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [17516.125713] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [17516.129057] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17516.131384] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [17516.134031] Call Trace: [17516.135108] [17516.135883] ? show_regs.cold.9+0x22/0x2f [17516.137112] ? watchdog_timer_fn.cold.6+0x62/0x10a [17516.138904] ? __hrtimer_run_queues+0x1c2/0x4d0 [17516.140895] ? watchdog+0x40/0x40 [17516.142239] ? hrtimer_interrupt+0x128/0x2c0 [17516.143951] ? smp_apic_timer_interrupt+0xa6/0x270 [17516.145721] ? apic_timer_interrupt+0xf/0x20 [17516.147397] [17516.148025] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17516.149283] do_raw_spin_lock+0x177/0x190 [17516.150388] _raw_spin_lock+0x21/0x30 [17516.151683] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17516.154079] ? do_raw_spin_unlock+0x75/0x190 [17516.155606] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17516.157630] ? do_wait_intr+0xf0/0xf0 [17516.158917] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17516.160557] mdc_close+0x53b/0xd70 [mdc] [17516.161751] lmv_close+0x18d/0x490 [lmv] [17516.162950] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [17516.165103] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17516.167689] ll_md_real_close+0xa5/0x2a0 [lustre] [17516.169417] ll_md_close+0x274/0x800 [lustre] [17516.170993] ll_file_release+0x128/0xc50 [lustre] [17516.173093] __fput+0x10d/0x440 [17516.174286] ____fput+0x12/0x20 [17516.175904] task_work_run+0xc6/0x110 [17516.177453] exit_to_usermode_loop+0x1dd/0x1f0 [17516.179123] do_syscall_64+0x3d6/0x440 [17516.180533] entry_SYSCALL_64_after_hwframe+0x49/0xae [17516.182312] RIP: 0033:0x7f151bee3cd8 [17516.183640] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17516.190194] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17516.193508] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [17516.196687] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [17516.199724] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17516.203053] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17516.206759] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17520.056024] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [17520.057824] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17520.058446] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [17520.085887] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17520.085903] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17520.085909] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17520.086198] RIP: 0010:native_safe_halt+0x2e/0x50 [17520.086208] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17520.086212] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17520.086217] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17520.086220] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [17520.098022] Modules linked in: [17520.101922] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [17520.101932] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [17520.101935] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [17520.101938] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [17520.101941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17520.101943] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17520.101951] Call Trace: [17520.101957] [17520.101963] ? show_regs.cold.9+0x22/0x2f [17520.101977] ? watchdog_timer_fn.cold.6+0x62/0x10a [17520.101991] ? __hrtimer_run_queues+0x1c2/0x4d0 [17520.101996] ? watchdog+0x40/0x40 [17520.102000] ? hrtimer_interrupt+0x128/0x2c0 [17520.102005] ? smp_apic_timer_interrupt+0xa6/0x270 [17520.102013] ? apic_timer_interrupt+0xf/0x20 [17520.102017] [17520.102021] ? native_safe_halt+0x2e/0x50 [17520.102025] kvm_wait+0xcd/0xe0 [17520.102032] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17520.102043] do_raw_spin_lock+0x177/0x190 [17520.102047] _raw_spin_lock+0x21/0x30 [17520.102052] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17520.102665] process_one_work+0x2c8/0x700 [17520.102675] worker_thread+0x296/0x6e0 [17520.102680] ? rescuer_thread+0x570/0x570 [17520.102684] kthread+0x1d1/0x200 [17520.102689] ? set_kthread_struct+0x70/0x70 [17520.102692] ret_from_fork+0x1f/0x30 [17520.222408] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17520.244935] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17520.251976] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17520.256026] RIP: 0010:native_safe_halt+0x2e/0x50 [17520.260152] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17520.275416] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17520.278099] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [17520.280828] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [17520.283259] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [17520.285870] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [17520.288725] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17520.291308] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [17520.294486] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17520.296747] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17520.299416] Call Trace: [17520.300383] [17520.301674] ? show_regs.cold.9+0x22/0x2f [17520.303496] ? watchdog_timer_fn.cold.6+0x62/0x10a [17520.305077] ? __hrtimer_run_queues+0x1c2/0x4d0 [17520.306499] ? watchdog+0x40/0x40 [17520.307666] ? hrtimer_interrupt+0x128/0x2c0 [17520.309159] ? smp_apic_timer_interrupt+0xa6/0x270 [17520.310965] ? apic_timer_interrupt+0xf/0x20 [17520.312678] [17520.313538] ? native_safe_halt+0x2e/0x50 [17520.315045] kvm_wait+0xcd/0xe0 [17520.316359] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17520.318383] do_raw_spin_lock+0x177/0x190 [17520.320091] _raw_spin_lock+0x21/0x30 [17520.321119] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17520.323502] obd_mod_free+0x46/0x140 [ptlrpc] [17520.325212] mdc_commit_open+0x91/0xb0 [mdc] [17520.329950] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17520.335635] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17520.345712] after_reply+0xe2c/0x1780 [ptlrpc] [17520.347540] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17520.351790] ? __next_timer_interrupt+0x160/0x160 [17520.353987] ptlrpcd+0xa82/0xe40 [ptlrpc] [17520.355998] ? do_wait_intr+0xf0/0xf0 [17520.358365] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17520.360461] kthread+0x1d1/0x200 [17520.366031] ? set_kthread_struct+0x70/0x70 [17520.368547] ret_from_fork+0x1f/0x30 [17532.052506] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [17532.062130] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17532.091878] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17532.098261] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17532.103372] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17532.106646] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17532.119429] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [17532.125497] RAX: 000000000000592b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17532.129020] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [17532.133070] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [17532.139236] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [17532.144208] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [17532.147597] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [17532.151736] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17532.154859] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [17532.160153] Call Trace: [17532.161317] [17532.162260] ? show_regs.cold.9+0x22/0x2f [17532.164130] ? watchdog_timer_fn.cold.6+0x62/0x10a [17532.166093] ? __hrtimer_run_queues+0x1c2/0x4d0 [17532.167957] ? watchdog+0x40/0x40 [17532.169448] ? hrtimer_interrupt+0x128/0x2c0 [17532.172193] ? smp_apic_timer_interrupt+0xa6/0x270 [17532.173685] ? apic_timer_interrupt+0xf/0x20 [17532.175995] [17532.176588] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17532.181911] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17532.187192] do_raw_spin_lock+0x177/0x190 [17532.189192] _raw_spin_lock+0x21/0x30 [17532.190751] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17532.195600] ? do_raw_spin_unlock+0x75/0x190 [17532.198271] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17532.201478] ? do_wait_intr+0xf0/0xf0 [17532.206240] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17532.213261] mdc_reint+0x2d/0x110 [mdc] [17532.216227] mdc_unlink+0x658/0x870 [mdc] [17532.218471] lmv_unlink+0x34d/0xbd0 [lmv] [17532.220600] ll_unlink+0x24d/0x7f0 [lustre] [17532.222979] vfs_unlink+0x202/0x330 [17532.224099] do_unlinkat+0x2fa/0x460 [17532.226202] __x64_sys_unlink+0x37/0x50 [17532.230958] do_syscall_64+0xc1/0x440 [17532.232881] entry_SYSCALL_64_after_hwframe+0x49/0xae [17532.238277] RIP: 0033:0x7f151bdee10b [17532.240604] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [17532.257742] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [17532.262442] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [17532.270822] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [17532.279704] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17532.287212] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17532.292239] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17537.505057] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15606s! [17537.509224] Showing busy workqueues and worker pools: [17537.511812] workqueue events: flags=0x0 [17537.513456] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17537.516459] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [17537.520701] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17537.523659] pending: kfree_rcu_monitor [17537.525555] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17537.528840] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [17537.533868] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17537.536785] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [17537.540726] workqueue events_long: flags=0x0 [17537.543023] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17537.546284] pending: lu_objects_flush [obdclass] [17537.549205] workqueue events_unbound: flags=0x2 [17537.551120] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [17537.553712] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17537.559286] workqueue events_power_efficient: flags=0x80 [17537.561819] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17537.565822] pending: do_cache_clean, neigh_periodic_work, check_lifetime [17537.569677] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17537.572441] pending: neigh_periodic_work [17537.574101] workqueue mm_percpu_wq: flags=0x8 [17537.575754] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17537.579283] pending: vmstat_update, lru_add_drain_per_cpu [17537.581761] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17537.584286] pending: vmstat_update [17537.585687] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17537.588199] pending: lru_add_drain_per_cpu [17537.589805] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [17537.592654] pending: lru_add_drain_per_cpu BAR(48) [17537.594775] workqueue writeback: flags=0x4a [17537.596703] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [17537.599872] pending: wb_workfn [17537.601529] workqueue xprtiod: flags=0xa [17537.603231] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17537.605663] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [17537.608619] workqueue ipv6_addrconf: flags=0x40008 [17537.610233] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17537.612436] pending: addrconf_verify_work [17537.614035] workqueue knbd0-recv: flags=0x1a [17537.615632] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17537.618574] in-flight: 658:recv_work [17537.620180] workqueue ptlrpc_pinger: flags=0x2 [17537.622280] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17537.625118] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [17537.628236] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15607s workers=4 idle: 32984 37 8 [17537.631808] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17522s workers=2 idle: 661 [17544.057076] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [17544.069523] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17544.104184] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17544.123224] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17544.129111] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17544.136540] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17544.160957] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17544.166492] RAX: 0000000000005b95 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17544.172453] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [17544.176266] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [17544.184053] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [17544.192759] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [17544.196006] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [17544.206513] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17544.216734] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [17544.225011] Call Trace: [17544.229149] [17544.230584] ? show_regs.cold.9+0x22/0x2f [17544.239145] ? watchdog_timer_fn.cold.6+0x62/0x10a [17544.240903] ? __hrtimer_run_queues+0x1c2/0x4d0 [17544.250627] ? watchdog+0x40/0x40 [17544.253582] ? hrtimer_interrupt+0x128/0x2c0 [17544.257977] ? smp_apic_timer_interrupt+0xa6/0x270 [17544.264626] ? apic_timer_interrupt+0xf/0x20 [17544.267525] [17544.270615] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17544.274471] do_raw_spin_lock+0x177/0x190 [17544.278166] _raw_spin_lock+0x21/0x30 [17544.283803] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17544.287139] ? do_raw_spin_unlock+0x75/0x190 [17544.295038] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17544.311543] ? do_wait_intr+0xf0/0xf0 [17544.319933] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17544.328627] mdc_close+0x53b/0xd70 [mdc] [17544.333716] lmv_close+0x18d/0x490 [lmv] [17544.337721] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [17544.342986] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17544.346619] ll_md_real_close+0xa5/0x2a0 [lustre] [17544.351407] ll_md_close+0x274/0x800 [lustre] [17544.363331] ll_file_release+0x128/0xc50 [lustre] [17544.370516] __fput+0x10d/0x440 [17544.374132] ____fput+0x12/0x20 [17544.377548] task_work_run+0xc6/0x110 [17544.382528] exit_to_usermode_loop+0x1dd/0x1f0 [17544.384797] do_syscall_64+0x3d6/0x440 [17544.391508] entry_SYSCALL_64_after_hwframe+0x49/0xae [17544.398345] RIP: 0033:0x7f151bee3cd8 [17544.401840] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17544.426276] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17544.438611] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [17544.443699] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [17544.454542] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17544.458391] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17544.462487] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17548.055027] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [17548.056028] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [17548.057062] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17548.057142] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17548.057152] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17548.057155] RIP: 0010:native_safe_halt+0x2e/0x50 [17548.057168] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17548.057173] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17548.057178] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [17548.057180] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [17548.057182] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [17548.057184] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [17548.057186] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17548.057190] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [17548.057193] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17548.057195] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17548.057202] Call Trace: [17548.057208] [17548.057214] ? show_regs.cold.9+0x22/0x2f [17548.057224] ? watchdog_timer_fn.cold.6+0x62/0x10a [17548.057231] ? __hrtimer_run_queues+0x1c2/0x4d0 [17548.057236] ? watchdog+0x40/0x40 [17548.057240] ? hrtimer_interrupt+0x128/0x2c0 [17548.057244] ? smp_apic_timer_interrupt+0xa6/0x270 [17548.057251] ? apic_timer_interrupt+0xf/0x20 [17548.057256] [17548.057259] ? native_safe_halt+0x2e/0x50 [17548.057268] kvm_wait+0xcd/0xe0 [17548.057277] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17548.057287] do_raw_spin_lock+0x177/0x190 [17548.057291] _raw_spin_lock+0x21/0x30 [17548.057297] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17548.057669] obd_mod_free+0x46/0x140 [ptlrpc] [17548.066804] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17548.066895] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17548.066903] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17548.066909] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17548.067302] RIP: 0010:native_safe_halt+0x2e/0x50 [17548.067314] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17548.067318] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17548.067323] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17548.067325] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [17548.067327] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [17548.067330] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [17548.067332] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [17548.067336] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [17548.067339] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17548.067341] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17548.067349] Call Trace: [17548.067354] [17548.067359] ? show_regs.cold.9+0x22/0x2f [17548.067368] ? watchdog_timer_fn.cold.6+0x62/0x10a [17548.067375] ? __hrtimer_run_queues+0x1c2/0x4d0 [17548.067379] ? watchdog+0x40/0x40 [17548.067384] ? hrtimer_interrupt+0x128/0x2c0 [17548.067388] ? smp_apic_timer_interrupt+0xa6/0x270 [17548.067394] ? apic_timer_interrupt+0xf/0x20 [17548.067399] [17548.067402] ? native_safe_halt+0x2e/0x50 [17548.067406] kvm_wait+0xcd/0xe0 [17548.067413] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17548.067422] do_raw_spin_lock+0x177/0x190 [17548.067426] _raw_spin_lock+0x21/0x30 [17548.067430] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17548.067631] process_one_work+0x2c8/0x700 [17548.067637] worker_thread+0x296/0x6e0 [17548.067641] ? rescuer_thread+0x570/0x570 [17548.067645] kthread+0x1d1/0x200 [17548.067649] ? set_kthread_struct+0x70/0x70 [17548.067653] ret_from_fork+0x1f/0x30 [17548.285193] mdc_commit_open+0x91/0xb0 [mdc] [17548.286505] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17548.290684] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17548.292752] after_reply+0xe2c/0x1780 [ptlrpc] [17548.295976] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17548.303643] ? __next_timer_interrupt+0x160/0x160 [17548.308105] ptlrpcd+0xa82/0xe40 [ptlrpc] [17548.311299] ? do_wait_intr+0xf0/0xf0 [17548.315359] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17548.318529] kthread+0x1d1/0x200 [17548.320327] ? set_kthread_struct+0x70/0x70 [17548.322578] ret_from_fork+0x1f/0x30 [17564.054420] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [17564.066621] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17564.127972] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17564.139769] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17564.144543] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17564.146058] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17564.159985] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17564.163900] RAX: 0000000000006b87 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17564.167193] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [17564.175244] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [17564.181329] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [17564.186621] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [17564.191389] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [17564.201076] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17564.207063] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [17564.210335] Call Trace: [17564.211585] [17564.214304] ? show_regs.cold.9+0x22/0x2f [17564.219423] ? watchdog_timer_fn.cold.6+0x62/0x10a [17564.221720] ? __hrtimer_run_queues+0x1c2/0x4d0 [17564.223822] ? watchdog+0x40/0x40 [17564.225724] ? hrtimer_interrupt+0x128/0x2c0 [17564.227676] ? smp_apic_timer_interrupt+0xa6/0x270 [17564.229714] ? apic_timer_interrupt+0xf/0x20 [17564.232155] [17564.239336] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17564.241230] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17564.246498] do_raw_spin_lock+0x177/0x190 [17564.258457] _raw_spin_lock+0x21/0x30 [17564.260279] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17564.265258] ? do_raw_spin_unlock+0x75/0x190 [17564.269613] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17564.275695] ? do_wait_intr+0xf0/0xf0 [17564.278182] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17564.282648] mdc_reint+0x2d/0x110 [mdc] [17564.289647] mdc_unlink+0x658/0x870 [mdc] [17564.299151] lmv_unlink+0x34d/0xbd0 [lmv] [17564.300916] ll_unlink+0x24d/0x7f0 [lustre] [17564.302454] vfs_unlink+0x202/0x330 [17564.310903] do_unlinkat+0x2fa/0x460 [17564.313017] __x64_sys_unlink+0x37/0x50 [17564.320582] do_syscall_64+0xc1/0x440 [17564.322169] entry_SYSCALL_64_after_hwframe+0x49/0xae [17564.326423] RIP: 0033:0x7f151bdee10b [17564.330333] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [17564.351110] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [17564.357579] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [17564.360298] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [17564.370303] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17564.380200] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17564.387513] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17567.717952] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15637s! [17567.726135] Showing busy workqueues and worker pools: [17567.731600] workqueue events: flags=0x0 [17567.738380] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17567.747480] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [17567.764117] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17567.770987] pending: kfree_rcu_monitor [17567.774169] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17567.783851] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [17567.795813] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17567.807730] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [17567.814173] workqueue events_long: flags=0x0 [17567.820171] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17567.830502] pending: lu_objects_flush [obdclass] [17567.839332] workqueue events_unbound: flags=0x2 [17567.845137] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [17567.849858] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17567.860455] workqueue events_power_efficient: flags=0x80 [17567.865114] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17567.869616] pending: do_cache_clean, neigh_periodic_work, check_lifetime [17567.875394] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17567.885789] pending: neigh_periodic_work [17567.888120] workqueue mm_percpu_wq: flags=0x8 [17567.893652] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17567.901951] pending: vmstat_update, lru_add_drain_per_cpu [17567.906202] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17567.912840] pending: vmstat_update [17567.914128] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17567.922955] pending: lru_add_drain_per_cpu [17567.929361] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [17567.936703] pending: lru_add_drain_per_cpu BAR(48) [17567.945383] workqueue writeback: flags=0x4a [17567.948493] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [17567.957394] pending: wb_workfn [17567.961981] workqueue xprtiod: flags=0xa [17567.963955] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17567.974147] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [17567.986506] workqueue ipv6_addrconf: flags=0x40008 [17567.995952] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17568.005507] pending: addrconf_verify_work [17568.009091] workqueue knbd0-recv: flags=0x1a [17568.014624] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17568.021529] in-flight: 658:recv_work [17568.024156] workqueue ptlrpc_pinger: flags=0x2 [17568.030461] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17568.039917] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [17568.044623] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15637s workers=4 idle: 32984 37 8 [17568.051452] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17552s workers=2 idle: 661 [17572.062042] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [17572.071063] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17572.132092] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17572.145144] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17572.153991] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17572.159470] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17572.176564] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17572.188396] RAX: 0000000000007083 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17572.192372] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [17572.197996] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [17572.201779] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [17572.207661] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [17572.214274] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [17572.221320] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17572.225332] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [17572.235998] Call Trace: [17572.241692] [17572.242836] ? show_regs.cold.9+0x22/0x2f [17572.248885] ? watchdog_timer_fn.cold.6+0x62/0x10a [17572.252879] ? __hrtimer_run_queues+0x1c2/0x4d0 [17572.257529] ? watchdog+0x40/0x40 [17572.259392] ? hrtimer_interrupt+0x128/0x2c0 [17572.263696] ? smp_apic_timer_interrupt+0xa6/0x270 [17572.265011] ? apic_timer_interrupt+0xf/0x20 [17572.268121] [17572.271413] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17572.277454] do_raw_spin_lock+0x177/0x190 [17572.278975] _raw_spin_lock+0x21/0x30 [17572.281140] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17572.284067] ? do_raw_spin_unlock+0x75/0x190 [17572.290925] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17572.296132] ? do_wait_intr+0xf0/0xf0 [17572.297974] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17572.302335] mdc_close+0x53b/0xd70 [mdc] [17572.307049] lmv_close+0x18d/0x490 [lmv] [17572.310922] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [17572.315605] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17572.322970] ll_md_real_close+0xa5/0x2a0 [lustre] [17572.328933] ll_md_close+0x274/0x800 [lustre] [17572.336392] ll_file_release+0x128/0xc50 [lustre] [17572.341425] __fput+0x10d/0x440 [17572.342591] ____fput+0x12/0x20 [17572.344098] task_work_run+0xc6/0x110 [17572.350283] exit_to_usermode_loop+0x1dd/0x1f0 [17572.351590] do_syscall_64+0x3d6/0x440 [17572.353350] entry_SYSCALL_64_after_hwframe+0x49/0xae [17572.356409] RIP: 0033:0x7f151bee3cd8 [17572.363061] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17572.378647] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17572.393885] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [17572.398051] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [17572.406326] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17572.417843] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17572.423813] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17576.055680] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [17576.056460] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [17576.068613] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17576.071771] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17576.071780] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17576.071783] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17576.071797] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17576.071801] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17576.071805] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [17576.071808] RDX: 0000000000003fd5 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [17576.071810] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [17576.071812] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [17576.071814] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17576.071817] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [17576.071820] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17576.071822] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17576.071831] Call Trace: [17576.071836] [17576.071842] ? show_regs.cold.9+0x22/0x2f [17576.071851] ? watchdog_timer_fn.cold.6+0x62/0x10a [17576.071858] ? __hrtimer_run_queues+0x1c2/0x4d0 [17576.071863] ? watchdog+0x40/0x40 [17576.071868] ? hrtimer_interrupt+0x128/0x2c0 [17576.071883] ? smp_apic_timer_interrupt+0xa6/0x270 [17576.071893] ? apic_timer_interrupt+0xf/0x20 [17576.071898] [17576.071901] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17576.071905] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17576.071910] do_raw_spin_lock+0x177/0x190 [17576.071915] _raw_spin_lock+0x21/0x30 [17576.071921] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17576.072209] obd_mod_free+0x46/0x140 [ptlrpc] [17576.072401] mdc_commit_open+0x91/0xb0 [mdc] [17576.072438] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17576.072647] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17576.072853] after_reply+0xe2c/0x1780 [ptlrpc] [17576.073050] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17576.073266] ? __next_timer_interrupt+0x160/0x160 [17576.073275] ptlrpcd+0xa82/0xe40 [ptlrpc] [17576.087063] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17576.087140] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17576.087147] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17576.087154] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17576.090112] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17576.090129] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17576.090133] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17576.090138] RAX: 000000000000470b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17576.090141] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [17576.090143] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [17576.090145] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [17576.090148] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [17576.090151] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [17576.090154] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17576.090157] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17576.090166] Call Trace: [17576.090171] [17576.090177] ? show_regs.cold.9+0x22/0x2f [17576.090187] ? watchdog_timer_fn.cold.6+0x62/0x10a [17576.090194] ? __hrtimer_run_queues+0x1c2/0x4d0 [17576.090199] ? watchdog+0x40/0x40 [17576.090204] ? hrtimer_interrupt+0x128/0x2c0 [17576.090208] ? smp_apic_timer_interrupt+0xa6/0x270 [17576.090218] ? apic_timer_interrupt+0xf/0x20 [17576.090222] [17576.090225] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17576.090230] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17576.090238] do_raw_spin_lock+0x177/0x190 [17576.090243] _raw_spin_lock+0x21/0x30 [17576.092947] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17576.093240] process_one_work+0x2c8/0x700 [17576.094038] worker_thread+0x296/0x6e0 [17576.094049] ? rescuer_thread+0x570/0x570 [17576.094053] kthread+0x1d1/0x200 [17576.094058] ? set_kthread_struct+0x70/0x70 [17576.094062] ret_from_fork+0x1f/0x30 [17576.743571] ? do_wait_intr+0xf0/0xf0 [17576.747453] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17576.755129] kthread+0x1d1/0x200 [17576.757376] ? set_kthread_struct+0x70/0x70 [17576.759377] ret_from_fork+0x1f/0x30 [17596.053012] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [17596.055874] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17596.082884] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17596.087361] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17596.092766] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17596.095731] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17596.104847] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17596.107738] RAX: 0000000000000b23 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17596.110569] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [17596.115113] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [17596.118813] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [17596.124653] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [17596.130010] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [17596.135177] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17596.136607] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [17596.139106] Call Trace: [17596.139816] [17596.140342] ? show_regs.cold.9+0x22/0x2f [17596.141623] ? watchdog_timer_fn.cold.6+0x62/0x10a [17596.143287] ? __hrtimer_run_queues+0x1c2/0x4d0 [17596.144584] ? watchdog+0x40/0x40 [17596.145670] ? hrtimer_interrupt+0x128/0x2c0 [17596.146874] ? smp_apic_timer_interrupt+0xa6/0x270 [17596.148214] ? apic_timer_interrupt+0xf/0x20 [17596.149600] [17596.157237] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17596.165145] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17596.166627] do_raw_spin_lock+0x177/0x190 [17596.173248] _raw_spin_lock+0x21/0x30 [17596.174317] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17596.176002] ? do_raw_spin_unlock+0x75/0x190 [17596.185620] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17596.187803] ? do_wait_intr+0xf0/0xf0 [17596.190532] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17596.192641] mdc_reint+0x2d/0x110 [mdc] [17596.193980] mdc_unlink+0x658/0x870 [mdc] [17596.195406] lmv_unlink+0x34d/0xbd0 [lmv] [17596.197072] ll_unlink+0x24d/0x7f0 [lustre] [17596.198608] vfs_unlink+0x202/0x330 [17596.199788] do_unlinkat+0x2fa/0x460 [17596.201209] __x64_sys_unlink+0x37/0x50 [17596.202885] do_syscall_64+0xc1/0x440 [17596.204304] entry_SYSCALL_64_after_hwframe+0x49/0xae [17596.206131] RIP: 0033:0x7f151bdee10b [17596.207347] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [17596.219821] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [17596.224619] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [17596.231127] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [17596.234804] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17596.237240] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17596.240134] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17598.438766] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15667s! [17598.450558] Showing busy workqueues and worker pools: [17598.454059] workqueue events: flags=0x0 [17598.460253] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17598.471456] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [17598.493355] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17598.505838] pending: kfree_rcu_monitor [17598.514478] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17598.529811] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [17598.545656] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17598.566021] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [17598.574864] workqueue events_long: flags=0x0 [17598.583148] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17598.595204] pending: lu_objects_flush [obdclass] [17598.606246] workqueue events_unbound: flags=0x2 [17598.612295] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [17598.619179] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17598.630129] workqueue events_power_efficient: flags=0x80 [17598.635437] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17598.641085] pending: do_cache_clean, neigh_periodic_work, check_lifetime [17598.645898] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17598.650863] pending: neigh_periodic_work [17598.654062] workqueue mm_percpu_wq: flags=0x8 [17598.657888] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17598.673138] pending: vmstat_update, lru_add_drain_per_cpu [17598.681276] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17598.687181] pending: vmstat_update [17598.696768] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17598.710884] pending: lru_add_drain_per_cpu [17598.719335] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [17598.733390] pending: lru_add_drain_per_cpu BAR(48) [17598.750314] workqueue writeback: flags=0x4a [17598.764029] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [17598.768890] pending: wb_workfn [17598.785641] workqueue xprtiod: flags=0xa [17598.786961] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17598.795307] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [17598.811839] workqueue ipv6_addrconf: flags=0x40008 [17598.814838] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17598.826929] pending: addrconf_verify_work [17598.834722] workqueue knbd0-recv: flags=0x1a [17598.842573] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17598.852759] in-flight: 658:recv_work [17598.858054] workqueue ptlrpc_pinger: flags=0x2 [17598.867121] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17598.877130] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [17598.887656] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15668s workers=4 idle: 32984 37 8 [17598.898984] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17583s workers=2 idle: 661 [17600.057298] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [17600.061519] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17600.097253] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17600.105374] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17600.113469] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17600.119815] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17600.135857] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17600.150179] RAX: 000000000000092b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17600.158114] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [17600.162437] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [17600.171697] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [17600.182597] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [17600.192257] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [17600.200728] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17600.207234] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [17600.217429] Call Trace: [17600.218588] [17600.221808] ? show_regs.cold.9+0x22/0x2f [17600.226274] ? watchdog_timer_fn.cold.6+0x62/0x10a [17600.230283] ? __hrtimer_run_queues+0x1c2/0x4d0 [17600.232760] ? watchdog+0x40/0x40 [17600.234683] ? hrtimer_interrupt+0x128/0x2c0 [17600.236830] ? smp_apic_timer_interrupt+0xa6/0x270 [17600.240204] ? apic_timer_interrupt+0xf/0x20 [17600.242665] [17600.243791] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17600.246747] do_raw_spin_lock+0x177/0x190 [17600.248690] _raw_spin_lock+0x21/0x30 [17600.250765] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17600.253136] ? do_raw_spin_unlock+0x75/0x190 [17600.255228] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17600.261724] ? do_wait_intr+0xf0/0xf0 [17600.263912] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17600.271895] mdc_close+0x53b/0xd70 [mdc] [17600.275200] lmv_close+0x18d/0x490 [lmv] [17600.278395] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [17600.280871] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17600.285649] ll_md_real_close+0xa5/0x2a0 [lustre] [17600.293684] ll_md_close+0x274/0x800 [lustre] [17600.296718] ll_file_release+0x128/0xc50 [lustre] [17600.302099] __fput+0x10d/0x440 [17600.303471] ____fput+0x12/0x20 [17600.309492] task_work_run+0xc6/0x110 [17600.310906] exit_to_usermode_loop+0x1dd/0x1f0 [17600.318388] do_syscall_64+0x3d6/0x440 [17600.325604] entry_SYSCALL_64_after_hwframe+0x49/0xae [17600.330705] RIP: 0033:0x7f151bee3cd8 [17600.338327] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17600.349761] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17600.353403] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [17600.358578] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [17600.367584] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17600.371033] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17600.382023] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17604.055072] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [17604.058169] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17604.058700] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [17604.077665] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17604.077680] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17604.077683] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17604.077700] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17604.077705] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17604.077713] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [17604.077716] RDX: 0000000000001e9d RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [17604.077719] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [17604.077721] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [17604.077723] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17604.077726] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [17604.077729] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17604.077731] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17604.077737] Call Trace: [17604.077742] [17604.077747] ? show_regs.cold.9+0x22/0x2f [17604.077756] ? watchdog_timer_fn.cold.6+0x62/0x10a [17604.077763] ? __hrtimer_run_queues+0x1c2/0x4d0 [17604.077767] ? watchdog+0x40/0x40 [17604.077771] ? hrtimer_interrupt+0x128/0x2c0 [17604.077774] ? smp_apic_timer_interrupt+0xa6/0x270 [17604.077782] ? apic_timer_interrupt+0xf/0x20 [17604.077787] [17604.077790] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17604.077794] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17604.077799] do_raw_spin_lock+0x177/0x190 [17604.077803] _raw_spin_lock+0x21/0x30 [17604.077811] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17604.087056] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common [17604.091986] obd_mod_free+0x46/0x140 [ptlrpc] [17604.092354] mdc_commit_open+0x91/0xb0 [mdc] [17604.092400] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17604.092646] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17604.092994] after_reply+0xe2c/0x1780 [ptlrpc] [17604.093521] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17604.093869] ? __next_timer_interrupt+0x160/0x160 [17604.093936] ptlrpcd+0xa82/0xe40 [ptlrpc] [17604.094186] ? do_wait_intr+0xf0/0xf0 [17604.094197] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17604.094497] kthread+0x1d1/0x200 [17604.094503] ? set_kthread_struct+0x70/0x70 [17604.094508] ret_from_fork+0x1f/0x30 [17604.321476] sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17604.336237] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17604.345843] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17604.350878] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17604.355362] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17604.359364] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17604.379901] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17604.383946] RAX: 0000000000003e47 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17604.386086] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [17604.390761] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [17604.394866] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [17604.399364] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [17604.412383] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [17604.417812] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17604.421425] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17604.427492] Call Trace: [17604.428919] [17604.430261] ? show_regs.cold.9+0x22/0x2f [17604.438570] ? watchdog_timer_fn.cold.6+0x62/0x10a [17604.445242] ? __hrtimer_run_queues+0x1c2/0x4d0 [17604.446842] ? watchdog+0x40/0x40 [17604.450643] ? hrtimer_interrupt+0x128/0x2c0 [17604.453811] ? smp_apic_timer_interrupt+0xa6/0x270 [17604.457786] ? apic_timer_interrupt+0xf/0x20 [17604.461009] [17604.465348] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17604.472489] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17604.483221] do_raw_spin_lock+0x177/0x190 [17604.486592] _raw_spin_lock+0x21/0x30 [17604.490084] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17604.500457] process_one_work+0x2c8/0x700 [17604.503284] worker_thread+0x296/0x6e0 [17604.505661] ? rescuer_thread+0x570/0x570 [17604.507917] kthread+0x1d1/0x200 [17604.510573] ? set_kthread_struct+0x70/0x70 [17604.516228] ret_from_fork+0x1f/0x30 [17624.055361] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [17624.066344] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17624.112336] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17624.128255] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17624.137206] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17624.140994] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17624.148866] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17624.152067] RAX: 0000000000001e8b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17624.154470] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [17624.157187] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [17624.159723] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [17624.162910] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [17624.165013] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [17624.167722] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17624.178834] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [17624.187677] Call Trace: [17624.188663] [17624.191530] ? show_regs.cold.9+0x22/0x2f [17624.195144] ? watchdog_timer_fn.cold.6+0x62/0x10a [17624.199661] ? __hrtimer_run_queues+0x1c2/0x4d0 [17624.203492] ? watchdog+0x40/0x40 [17624.206160] ? hrtimer_interrupt+0x128/0x2c0 [17624.217442] ? smp_apic_timer_interrupt+0xa6/0x270 [17624.224420] ? apic_timer_interrupt+0xf/0x20 [17624.229752] [17624.231871] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17624.238311] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17624.247627] do_raw_spin_lock+0x177/0x190 [17624.250173] _raw_spin_lock+0x21/0x30 [17624.257748] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17624.262576] ? do_raw_spin_unlock+0x75/0x190 [17624.267442] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17624.273779] ? do_wait_intr+0xf0/0xf0 [17624.275781] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17624.281170] mdc_reint+0x2d/0x110 [mdc] [17624.284418] mdc_unlink+0x658/0x870 [mdc] [17624.287567] lmv_unlink+0x34d/0xbd0 [lmv] [17624.291045] ll_unlink+0x24d/0x7f0 [lustre] [17624.294146] vfs_unlink+0x202/0x330 [17624.303776] do_unlinkat+0x2fa/0x460 [17624.305059] __x64_sys_unlink+0x37/0x50 [17624.308454] do_syscall_64+0xc1/0x440 [17624.312081] entry_SYSCALL_64_after_hwframe+0x49/0xae [17624.314681] RIP: 0033:0x7f151bdee10b [17624.316540] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [17624.331967] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [17624.336759] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [17624.341060] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [17624.352719] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17624.359525] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17624.363041] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17628.070141] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [17628.088380] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17628.138175] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17628.148146] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17628.170658] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17628.177485] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17628.204393] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17628.213805] RAX: 0000000000006687 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17628.222739] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [17628.234930] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [17628.251729] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [17628.257671] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [17628.269938] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [17628.281101] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17628.284613] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [17628.295836] Call Trace: [17628.299824] [17628.303297] ? show_regs.cold.9+0x22/0x2f [17628.305863] ? watchdog_timer_fn.cold.6+0x62/0x10a [17628.309722] ? __hrtimer_run_queues+0x1c2/0x4d0 [17628.313519] ? watchdog+0x40/0x40 [17628.316108] ? hrtimer_interrupt+0x128/0x2c0 [17628.324384] ? smp_apic_timer_interrupt+0xa6/0x270 [17628.333458] ? apic_timer_interrupt+0xf/0x20 [17628.346017] [17628.350689] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17628.360386] do_raw_spin_lock+0x177/0x190 [17628.369451] _raw_spin_lock+0x21/0x30 [17628.371573] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17628.377008] ? do_raw_spin_unlock+0x75/0x190 [17628.382858] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17628.388167] ? do_wait_intr+0xf0/0xf0 [17628.389548] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17628.395418] mdc_close+0x53b/0xd70 [mdc] [17628.396884] lmv_close+0x18d/0x490 [lmv] [17628.399103] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [17628.412863] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17628.415073] ll_md_real_close+0xa5/0x2a0 [lustre] [17628.421506] ll_md_close+0x274/0x800 [lustre] [17628.424930] ll_file_release+0x128/0xc50 [lustre] [17628.428755] __fput+0x10d/0x440 [17628.431347] ____fput+0x12/0x20 [17628.432852] task_work_run+0xc6/0x110 [17628.436627] exit_to_usermode_loop+0x1dd/0x1f0 [17628.439392] do_syscall_64+0x3d6/0x440 [17628.446150] entry_SYSCALL_64_after_hwframe+0x49/0xae [17628.447961] RIP: 0033:0x7f151bee3cd8 [17628.453323] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17628.474655] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17628.485560] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [17628.489707] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [17628.500172] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17628.505081] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17628.513324] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17629.161523] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15698s! [17629.172854] Showing busy workqueues and worker pools: [17629.180310] workqueue events: flags=0x0 [17629.183406] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17629.191660] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [17629.201283] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17629.205484] pending: kfree_rcu_monitor [17629.207738] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17629.218304] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [17629.230498] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17629.237409] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [17629.247285] workqueue events_long: flags=0x0 [17629.249617] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17629.262841] pending: lu_objects_flush [obdclass] [17629.273185] workqueue events_unbound: flags=0x2 [17629.275707] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [17629.281542] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17629.298340] workqueue events_power_efficient: flags=0x80 [17629.306137] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17629.315579] pending: do_cache_clean, neigh_periodic_work, check_lifetime [17629.331016] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17629.333720] pending: neigh_periodic_work [17629.341637] workqueue mm_percpu_wq: flags=0x8 [17629.343514] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17629.353449] pending: vmstat_update, lru_add_drain_per_cpu [17629.358562] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17629.370601] pending: vmstat_update [17629.374235] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17629.382691] pending: lru_add_drain_per_cpu [17629.395141] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [17629.402283] pending: lru_add_drain_per_cpu BAR(48) [17629.404217] workqueue writeback: flags=0x4a [17629.411355] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [17629.418190] pending: wb_workfn [17629.421604] workqueue xprtiod: flags=0xa [17629.423694] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17629.433728] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [17629.442456] workqueue ipv6_addrconf: flags=0x40008 [17629.447969] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17629.460326] pending: addrconf_verify_work [17629.461970] workqueue knbd0-recv: flags=0x1a [17629.468719] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17629.474400] in-flight: 658:recv_work [17629.476300] workqueue ptlrpc_pinger: flags=0x2 [17629.481140] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17629.495919] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [17629.504986] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15698s workers=4 idle: 32984 37 8 [17629.507863] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17614s workers=2 idle: 661 [17632.055374] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [17632.056870] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [17632.060266] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17632.060344] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17632.060351] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17632.060354] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17632.060368] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17632.060372] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17632.060377] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [17632.060379] RDX: 0000000000002753 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [17632.060382] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [17632.060384] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [17632.060386] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17632.060389] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [17632.060391] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17632.060393] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17632.060402] Call Trace: [17632.060406] [17632.060412] ? show_regs.cold.9+0x22/0x2f [17632.060420] ? watchdog_timer_fn.cold.6+0x62/0x10a [17632.060427] ? __hrtimer_run_queues+0x1c2/0x4d0 [17632.060431] ? watchdog+0x40/0x40 [17632.060435] ? hrtimer_interrupt+0x128/0x2c0 [17632.060438] ? smp_apic_timer_interrupt+0xa6/0x270 [17632.060446] ? apic_timer_interrupt+0xf/0x20 [17632.060450] [17632.060453] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17632.060457] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17632.068615] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17632.068697] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17632.068705] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17632.068712] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17632.098316] do_raw_spin_lock+0x177/0x190 [17632.098338] _raw_spin_lock+0x21/0x30 [17632.098348] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17632.098662] obd_mod_free+0x46/0x140 [ptlrpc] [17632.098899] mdc_commit_open+0x91/0xb0 [mdc] [17632.098943] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17632.099271] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17632.099494] after_reply+0xe2c/0x1780 [ptlrpc] [17632.099713] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17632.099924] ? __next_timer_interrupt+0x160/0x160 [17632.099934] ptlrpcd+0xa82/0xe40 [ptlrpc] [17632.100370] ? do_wait_intr+0xf0/0xf0 [17632.100381] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17632.100818] kthread+0x1d1/0x200 [17632.100824] ? set_kthread_struct+0x70/0x70 [17632.100892] ret_from_fork+0x1f/0x30 [17632.452528] [17632.453034] RIP: 0010:native_safe_halt+0x2e/0x50 [17632.454275] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17632.470648] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17632.473927] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17632.478188] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [17632.486020] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [17632.497238] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [17632.508866] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [17632.518157] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [17632.538020] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17632.543223] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17632.552509] Call Trace: [17632.555628] [17632.556414] ? show_regs.cold.9+0x22/0x2f [17632.557889] ? watchdog_timer_fn.cold.6+0x62/0x10a [17632.564108] ? __hrtimer_run_queues+0x1c2/0x4d0 [17632.571685] ? watchdog+0x40/0x40 [17632.578908] ? hrtimer_interrupt+0x128/0x2c0 [17632.581290] ? smp_apic_timer_interrupt+0xa6/0x270 [17632.589105] ? apic_timer_interrupt+0xf/0x20 [17632.594445] [17632.597529] ? native_safe_halt+0x2e/0x50 [17632.607193] kvm_wait+0xcd/0xe0 [17632.613528] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17632.618218] do_raw_spin_lock+0x177/0x190 [17632.620905] _raw_spin_lock+0x21/0x30 [17632.629814] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17632.632796] process_one_work+0x2c8/0x700 [17632.636485] worker_thread+0x296/0x6e0 [17632.638637] ? rescuer_thread+0x570/0x570 [17632.640996] kthread+0x1d1/0x200 [17632.643263] ? set_kthread_struct+0x70/0x70 [17632.648742] ret_from_fork+0x1f/0x30 [17644.974060] rcu: INFO: rcu_sched self-detected stall on CPU [17644.978182] rcu: 1-...!: (9448089 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [17644.996009] (t=15720327 jiffies g=249657 q=7300) [17645.010108] rcu: rcu_sched kthread starved for 15715279 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [17645.023577] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [17645.034459] rcu: RCU grace-period kthread stack dump: [17645.037490] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [17645.051082] Call Trace: [17645.052213] __schedule+0x351/0xcb0 [17645.060115] schedule+0xc0/0x180 [17645.062662] schedule_timeout+0xb4/0x190 [17645.069555] ? do_raw_spin_unlock+0x75/0x190 [17645.079651] ? __next_timer_interrupt+0x160/0x160 [17645.085430] ? prepare_to_swait_event+0xf3/0x140 [17645.095932] rcu_gp_kthread+0x8dd/0x15b0 [17645.100738] ? force_qs_rnp+0x2c0/0x2c0 [17645.108323] kthread+0x1d1/0x200 [17645.111076] ? set_kthread_struct+0x70/0x70 [17645.115962] ret_from_fork+0x1f/0x30 [17645.122136] rcu: Stack dump where RCU GP kthread last ran: [17645.125345] NMI backtrace for cpu 1 [17645.136640] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17645.147527] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17645.162552] Call Trace: [17645.164107] [17645.165226] ? dump_stack+0xbb/0x10e [17645.170251] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [17645.183819] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [17645.194119] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [17645.207293] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [17645.211909] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [17645.216002] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [17645.219401] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [17645.223576] ? update_process_times+0x6d/0xc0 [17645.228053] ? tick_sched_handle.isra.18+0x27/0x90 [17645.236056] ? tick_sched_timer+0x53/0xb0 [17645.239798] ? __hrtimer_run_queues+0x1c2/0x4d0 [17645.249084] ? tick_sched_handle.isra.18+0x90/0x90 [17645.250908] ? hrtimer_interrupt+0x128/0x2c0 [17645.255214] ? smp_apic_timer_interrupt+0xa6/0x270 [17645.258044] ? apic_timer_interrupt+0xf/0x20 [17645.264318] [17645.265092] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17645.270118] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17645.284707] ? do_raw_spin_lock+0x177/0x190 [17645.288166] ? _raw_spin_lock+0x21/0x30 [17645.294099] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17645.304631] ? obd_mod_free+0x46/0x140 [ptlrpc] [17645.312486] ? mdc_commit_open+0x91/0xb0 [mdc] [17645.318173] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17645.321483] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17645.325025] ? after_reply+0xe2c/0x1780 [ptlrpc] [17645.328708] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17645.333113] ? __next_timer_interrupt+0x160/0x160 [17645.338579] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [17645.344192] ? do_wait_intr+0xf0/0xf0 [17645.345668] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17645.349619] ? kthread+0x1d1/0x200 [17645.350818] ? set_kthread_struct+0x70/0x70 [17645.357135] ? ret_from_fork+0x1f/0x30 [17645.359674] NMI backtrace for cpu 1 [17645.371514] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17645.387355] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17645.406487] Call Trace: [17645.407616] [17645.409177] ? dump_stack+0xbb/0x10e [17645.411170] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [17645.416102] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [17645.422205] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [17645.426075] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [17645.427808] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [17645.431228] ? rcu_dump_cpu_stacks+0x11b/0x171 [17645.435210] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [17645.438670] ? update_process_times+0x6d/0xc0 [17645.441658] ? tick_sched_handle.isra.18+0x27/0x90 [17645.446785] ? tick_sched_timer+0x53/0xb0 [17645.450744] ? __hrtimer_run_queues+0x1c2/0x4d0 [17645.453761] ? tick_sched_handle.isra.18+0x90/0x90 [17645.457834] ? hrtimer_interrupt+0x128/0x2c0 [17645.459408] ? smp_apic_timer_interrupt+0xa6/0x270 [17645.469022] ? apic_timer_interrupt+0xf/0x20 [17645.483201] [17645.483939] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17645.494544] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17645.503355] ? do_raw_spin_lock+0x177/0x190 [17645.505804] ? _raw_spin_lock+0x21/0x30 [17645.510159] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17645.515633] ? obd_mod_free+0x46/0x140 [ptlrpc] [17645.520519] ? mdc_commit_open+0x91/0xb0 [mdc] [17645.526795] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17645.529912] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17645.533802] ? after_reply+0xe2c/0x1780 [ptlrpc] [17645.543025] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17645.550761] ? __next_timer_interrupt+0x160/0x160 [17645.556671] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [17645.561113] ? do_wait_intr+0xf0/0xf0 [17645.566400] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17645.571192] ? kthread+0x1d1/0x200 [17645.573270] ? set_kthread_struct+0x70/0x70 [17645.576251] ? ret_from_fork+0x1f/0x30 [17656.052394] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [17656.055609] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17656.057030] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [17656.071503] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17656.071519] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17656.071522] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17656.076024] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) [17656.079377] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17656.079387] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [17656.079393] RAX: 000000000000724b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17656.079395] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [17656.079398] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [17656.079399] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [17656.079402] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [17656.079405] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [17656.079407] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17656.079410] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [17656.079417] Call Trace: [17656.079422] [17656.079427] ? show_regs.cold.9+0x22/0x2f [17656.079440] ? watchdog_timer_fn.cold.6+0x62/0x10a [17656.079446] ? __hrtimer_run_queues+0x1c2/0x4d0 [17656.079450] ? watchdog+0x40/0x40 [17656.079454] ? hrtimer_interrupt+0x128/0x2c0 [17656.079458] ? smp_apic_timer_interrupt+0xa6/0x270 [17656.079466] ? apic_timer_interrupt+0xf/0x20 [17656.079469] [17656.079472] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17656.079478] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17656.079482] do_raw_spin_lock+0x177/0x190 [17656.079486] _raw_spin_lock+0x21/0x30 [17656.079503] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17656.083596] obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17656.083646] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17656.083653] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17656.083656] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17656.083669] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17656.083673] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17656.083678] RAX: 0000000000006aa9 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17656.083681] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [17656.083683] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [17656.083685] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [17656.083688] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [17656.083691] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [17656.083694] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17656.083696] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [17656.083704] Call Trace: [17656.083728] [17656.083736] ? show_regs.cold.9+0x22/0x2f [17656.083745] ? watchdog_timer_fn.cold.6+0x62/0x10a [17656.083751] ? __hrtimer_run_queues+0x1c2/0x4d0 [17656.083754] ? watchdog+0x40/0x40 [17656.083758] ? hrtimer_interrupt+0x128/0x2c0 [17656.083762] ? smp_apic_timer_interrupt+0xa6/0x270 [17656.083769] ? apic_timer_interrupt+0xf/0x20 [17656.083773] [17656.083853] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17656.083863] do_raw_spin_lock+0x177/0x190 [17656.083868] _raw_spin_lock+0x21/0x30 [17656.083873] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17656.086552] ? do_raw_spin_unlock+0x75/0x190 [17656.092956] ? do_raw_spin_unlock+0x75/0x190 [17656.102987] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17656.103268] ? do_wait_intr+0xf0/0xf0 [17656.103278] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17656.103472] mdc_reint+0x2d/0x110 [mdc] [17656.103514] mdc_unlink+0x658/0x870 [mdc] [17656.103547] lmv_unlink+0x34d/0xbd0 [lmv] [17656.103586] ll_unlink+0x24d/0x7f0 [lustre] [17656.103705] vfs_unlink+0x202/0x330 [17656.103713] do_unlinkat+0x2fa/0x460 [17656.103718] __x64_sys_unlink+0x37/0x50 [17656.103721] do_syscall_64+0xc1/0x440 [17656.103728] entry_SYSCALL_64_after_hwframe+0x49/0xae [17656.103736] RIP: 0033:0x7f151bdee10b [17656.103743] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [17656.103747] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [17656.103752] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [17656.103754] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [17656.103756] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17656.103758] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17656.103760] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17656.304189] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17656.306330] ? do_wait_intr+0xf0/0xf0 [17656.308015] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17656.309954] mdc_close+0x53b/0xd70 [mdc] [17656.311250] lmv_close+0x18d/0x490 [lmv] [17656.312601] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [17656.314729] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17656.317253] ll_md_real_close+0xa5/0x2a0 [lustre] [17656.318779] ll_md_close+0x274/0x800 [lustre] [17656.320472] ll_file_release+0x128/0xc50 [lustre] [17656.322389] __fput+0x10d/0x440 [17656.323667] ____fput+0x12/0x20 [17656.325264] task_work_run+0xc6/0x110 [17656.326281] exit_to_usermode_loop+0x1dd/0x1f0 [17656.328739] do_syscall_64+0x3d6/0x440 [17656.331122] entry_SYSCALL_64_after_hwframe+0x49/0xae [17656.333321] RIP: 0033:0x7f151bee3cd8 [17656.335213] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17656.342775] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17656.346091] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [17656.348211] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [17656.351546] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17656.354530] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17656.357591] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17659.873061] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15729s! [17659.877245] Showing busy workqueues and worker pools: [17659.879321] workqueue events: flags=0x0 [17659.880514] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17659.882806] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [17659.886328] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17659.888674] pending: kfree_rcu_monitor [17659.890142] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17659.892578] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [17659.896174] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17659.898475] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [17659.901227] workqueue events_long: flags=0x0 [17659.902700] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17659.905189] pending: lu_objects_flush [obdclass] [17659.907235] workqueue events_unbound: flags=0x2 [17659.909030] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [17659.911298] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17659.915281] workqueue events_power_efficient: flags=0x80 [17659.917330] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17659.920078] pending: do_cache_clean, neigh_periodic_work, check_lifetime [17659.922441] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17659.924697] pending: neigh_periodic_work [17659.926345] workqueue mm_percpu_wq: flags=0x8 [17659.927752] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17659.930239] pending: vmstat_update, lru_add_drain_per_cpu [17659.932372] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17659.934496] pending: vmstat_update [17659.936176] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17659.938620] pending: lru_add_drain_per_cpu [17659.940197] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [17659.942675] pending: lru_add_drain_per_cpu BAR(48) [17659.944418] workqueue writeback: flags=0x4a [17659.945799] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [17659.948330] pending: wb_workfn [17659.949434] workqueue xprtiod: flags=0xa [17659.951038] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17659.953279] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [17659.955762] workqueue ipv6_addrconf: flags=0x40008 [17659.957636] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17659.960087] pending: addrconf_verify_work [17659.961438] workqueue knbd0-recv: flags=0x1a [17659.963048] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17659.965212] in-flight: 658:recv_work [17659.966473] workqueue ptlrpc_pinger: flags=0x2 [17659.968140] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17659.970416] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [17659.972939] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15729s workers=4 idle: 32984 37 8 [17659.975685] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17644s workers=2 idle: 661 [17660.056051] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [17660.059516] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17660.078314] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17660.082658] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17660.086177] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17660.088893] RIP: 0010:native_safe_halt+0x2e/0x50 [17660.090217] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17660.096856] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17660.100348] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17660.102998] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [17660.106052] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [17660.108424] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [17660.111732] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [17660.114831] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [17660.117597] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17660.120116] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17660.123117] Call Trace: [17660.124021] [17660.124565] ? show_regs.cold.9+0x22/0x2f [17660.126353] ? watchdog_timer_fn.cold.6+0x62/0x10a [17660.128221] ? __hrtimer_run_queues+0x1c2/0x4d0 [17660.130243] ? watchdog+0x40/0x40 [17660.131435] ? hrtimer_interrupt+0x128/0x2c0 [17660.133160] ? smp_apic_timer_interrupt+0xa6/0x270 [17660.134689] ? apic_timer_interrupt+0xf/0x20 [17660.136226] [17660.137171] ? native_safe_halt+0x2e/0x50 [17660.138645] kvm_wait+0xcd/0xe0 [17660.140172] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17660.142357] do_raw_spin_lock+0x177/0x190 [17660.144086] _raw_spin_lock+0x21/0x30 [17660.145334] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17660.147441] process_one_work+0x2c8/0x700 [17660.149073] worker_thread+0x296/0x6e0 [17660.150470] ? rescuer_thread+0x570/0x570 [17660.152143] kthread+0x1d1/0x200 [17660.153339] ? set_kthread_struct+0x70/0x70 [17660.155028] ret_from_fork+0x1f/0x30 [17672.056146] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [17672.060041] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17672.092077] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17672.096399] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17672.100002] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17672.103049] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17672.111129] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17672.114405] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [17672.117826] RDX: 00000000000074ed RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [17672.120844] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [17672.124124] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [17672.127693] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17672.130650] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [17672.134203] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17672.136530] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17672.139367] Call Trace: [17672.140472] [17672.141254] ? show_regs.cold.9+0x22/0x2f [17672.142923] ? watchdog_timer_fn.cold.6+0x62/0x10a [17672.144723] ? __hrtimer_run_queues+0x1c2/0x4d0 [17672.146648] ? watchdog+0x40/0x40 [17672.147984] ? hrtimer_interrupt+0x128/0x2c0 [17672.149604] ? smp_apic_timer_interrupt+0xa6/0x270 [17672.151421] ? apic_timer_interrupt+0xf/0x20 [17672.153193] [17672.153878] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17672.155352] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17672.157151] do_raw_spin_lock+0x177/0x190 [17672.158917] _raw_spin_lock+0x21/0x30 [17672.160430] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17672.162588] obd_mod_free+0x46/0x140 [ptlrpc] [17672.164368] mdc_commit_open+0x91/0xb0 [mdc] [17672.165427] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17672.167465] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17672.169988] after_reply+0xe2c/0x1780 [ptlrpc] [17672.172014] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17672.174409] ? __next_timer_interrupt+0x160/0x160 [17672.176306] ptlrpcd+0xa82/0xe40 [ptlrpc] [17672.178259] ? do_wait_intr+0xf0/0xf0 [17672.179643] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17672.181805] kthread+0x1d1/0x200 [17672.183378] ? set_kthread_struct+0x70/0x70 [17672.185166] ret_from_fork+0x1f/0x30 [17684.054690] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [17684.057040] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [17684.058600] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17684.058685] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17684.058692] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17684.060139] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17684.064378] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17684.064481] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17684.064489] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17684.064492] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17684.064515] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17684.064520] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17684.064524] RAX: 0000000000003909 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17684.064527] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [17684.064529] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [17684.064532] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [17684.064534] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [17684.064538] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [17684.064541] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17684.064544] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [17684.064552] Call Trace: [17684.064559] [17684.064564] ? show_regs.cold.9+0x22/0x2f [17684.064572] ? watchdog_timer_fn.cold.6+0x62/0x10a [17684.064578] ? __hrtimer_run_queues+0x1c2/0x4d0 [17684.064582] ? watchdog+0x40/0x40 [17684.064585] ? hrtimer_interrupt+0x128/0x2c0 [17684.064589] ? smp_apic_timer_interrupt+0xa6/0x270 [17684.064596] ? apic_timer_interrupt+0xf/0x20 [17684.064600] [17684.064603] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17684.064608] do_raw_spin_lock+0x177/0x190 [17684.064612] _raw_spin_lock+0x21/0x30 [17684.064617] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17684.065069] ? do_raw_spin_unlock+0x75/0x190 [17684.065076] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17684.065276] ? do_wait_intr+0xf0/0xf0 [17684.065285] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17684.065688] mdc_close+0x53b/0xd70 [mdc] [17684.065726] lmv_close+0x18d/0x490 [lmv] [17684.065757] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [17684.091089] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17684.091102] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17684.091109] RAX: 0000000000000b45 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17684.091111] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [17684.091113] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [17684.091115] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [17684.091118] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [17684.091121] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [17684.091124] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17684.091127] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [17684.091135] Call Trace: [17684.091140] [17684.091145] ? show_regs.cold.9+0x22/0x2f [17684.091157] ? watchdog_timer_fn.cold.6+0x62/0x10a [17684.091162] ? __hrtimer_run_queues+0x1c2/0x4d0 [17684.091166] ? watchdog+0x40/0x40 [17684.091169] ? hrtimer_interrupt+0x128/0x2c0 [17684.091173] ? smp_apic_timer_interrupt+0xa6/0x270 [17684.091180] ? apic_timer_interrupt+0xf/0x20 [17684.091185] [17684.091188] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17684.091194] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17684.091198] do_raw_spin_lock+0x177/0x190 [17684.091203] _raw_spin_lock+0x21/0x30 [17684.091210] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17684.096011] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17684.096133] ll_md_real_close+0xa5/0x2a0 [lustre] [17684.096217] ll_md_close+0x274/0x800 [lustre] [17684.096302] ll_file_release+0x128/0xc50 [lustre] [17684.096388] __fput+0x10d/0x440 [17684.096397] ____fput+0x12/0x20 [17684.096402] task_work_run+0xc6/0x110 [17684.096410] exit_to_usermode_loop+0x1dd/0x1f0 [17684.096428] do_syscall_64+0x3d6/0x440 [17684.096434] entry_SYSCALL_64_after_hwframe+0x49/0xae [17684.096443] RIP: 0033:0x7f151bee3cd8 [17684.096451] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17684.096455] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17684.096460] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [17684.096462] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [17684.096465] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17684.096661] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17684.096664] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17684.280765] ? do_raw_spin_unlock+0x75/0x190 [17684.282285] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17684.284151] ? do_wait_intr+0xf0/0xf0 [17684.285336] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17684.287237] mdc_reint+0x2d/0x110 [mdc] [17684.288447] mdc_unlink+0x658/0x870 [mdc] [17684.289747] lmv_unlink+0x34d/0xbd0 [lmv] [17684.291126] ll_unlink+0x24d/0x7f0 [lustre] [17684.292440] vfs_unlink+0x202/0x330 [17684.293614] do_unlinkat+0x2fa/0x460 [17684.295108] __x64_sys_unlink+0x37/0x50 [17684.296373] do_syscall_64+0xc1/0x440 [17684.297742] entry_SYSCALL_64_after_hwframe+0x49/0xae [17684.299297] RIP: 0033:0x7f151bdee10b [17684.300311] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [17684.308205] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [17684.311437] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [17684.314399] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [17684.317401] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17684.321415] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17684.332399] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17688.059298] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [17688.064575] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17688.110742] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17688.123783] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17688.139111] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17688.149800] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17688.156128] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17688.173346] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17688.183255] RAX: 0000000000007009 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17688.192774] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [17688.200877] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [17688.209445] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [17688.216190] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [17688.218662] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [17688.229221] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17688.232531] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17688.237793] Call Trace: [17688.240760] [17688.242729] ? show_regs.cold.9+0x22/0x2f [17688.246162] ? watchdog_timer_fn.cold.6+0x62/0x10a [17688.252118] ? __hrtimer_run_queues+0x1c2/0x4d0 [17688.257209] ? watchdog+0x40/0x40 [17688.262145] ? hrtimer_interrupt+0x128/0x2c0 [17688.266996] ? smp_apic_timer_interrupt+0xa6/0x270 [17688.272431] ? apic_timer_interrupt+0xf/0x20 [17688.275150] [17688.275920] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17688.283782] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17688.290151] do_raw_spin_lock+0x177/0x190 [17688.295381] _raw_spin_lock+0x21/0x30 [17688.299862] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17688.306561] process_one_work+0x2c8/0x700 [17688.312187] worker_thread+0x296/0x6e0 [17688.315797] ? rescuer_thread+0x570/0x570 [17688.318852] kthread+0x1d1/0x200 [17688.321863] ? set_kthread_struct+0x70/0x70 [17688.323483] ret_from_fork+0x1f/0x30 [17690.080528] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15759s! [17690.085512] Showing busy workqueues and worker pools: [17690.088772] workqueue events: flags=0x0 [17690.092053] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17690.097270] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [17690.104157] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17690.107912] pending: kfree_rcu_monitor [17690.110557] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17690.114231] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [17690.120045] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17690.123236] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [17690.127757] workqueue events_long: flags=0x0 [17690.129913] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17690.134616] pending: lu_objects_flush [obdclass] [17690.138379] workqueue events_unbound: flags=0x2 [17690.142541] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [17690.148205] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17690.154374] workqueue events_power_efficient: flags=0x80 [17690.157519] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17690.161180] pending: do_cache_clean, neigh_periodic_work, check_lifetime [17690.165440] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17690.169046] pending: neigh_periodic_work [17690.171467] workqueue mm_percpu_wq: flags=0x8 [17690.174095] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17690.177255] pending: vmstat_update, lru_add_drain_per_cpu [17690.181373] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17690.185037] pending: vmstat_update [17690.187133] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17690.190320] pending: lru_add_drain_per_cpu [17690.192718] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [17690.196587] pending: lru_add_drain_per_cpu BAR(48) [17690.199476] workqueue writeback: flags=0x4a [17690.201882] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [17690.204777] pending: wb_workfn [17690.207329] workqueue xprtiod: flags=0xa [17690.212787] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17690.216173] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [17690.220464] workqueue ipv6_addrconf: flags=0x40008 [17690.222610] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17690.226264] pending: addrconf_verify_work [17690.228438] workqueue knbd0-recv: flags=0x1a [17690.230359] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17690.233688] in-flight: 658:recv_work [17690.235742] workqueue ptlrpc_pinger: flags=0x2 [17690.238288] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17690.243325] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [17690.247985] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15759s workers=4 idle: 32984 37 8 [17690.252233] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17675s workers=2 idle: 661 [17700.055090] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [17700.058698] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17700.083838] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17700.093683] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17700.103926] RIP: 0010:native_safe_halt+0x2e/0x50 [17700.108936] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17700.134073] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17700.140042] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [17700.149454] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [17700.153987] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [17700.164750] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [17700.178345] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17700.192288] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [17700.203940] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17700.215315] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17700.225721] Call Trace: [17700.226849] [17700.232446] ? show_regs.cold.9+0x22/0x2f [17700.239020] ? watchdog_timer_fn.cold.6+0x62/0x10a [17700.240648] ? __hrtimer_run_queues+0x1c2/0x4d0 [17700.249644] ? watchdog+0x40/0x40 [17700.252187] ? hrtimer_interrupt+0x128/0x2c0 [17700.257952] ? smp_apic_timer_interrupt+0xa6/0x270 [17700.261187] ? apic_timer_interrupt+0xf/0x20 [17700.264431] [17700.265846] ? native_safe_halt+0x2e/0x50 [17700.267124] kvm_wait+0xcd/0xe0 [17700.270718] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17700.277888] do_raw_spin_lock+0x177/0x190 [17700.282700] _raw_spin_lock+0x21/0x30 [17700.287066] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17700.295712] obd_mod_free+0x46/0x140 [ptlrpc] [17700.305879] mdc_commit_open+0x91/0xb0 [mdc] [17700.310600] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17700.317714] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17700.324573] after_reply+0xe2c/0x1780 [ptlrpc] [17700.327306] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17700.329288] ? __next_timer_interrupt+0x160/0x160 [17700.335724] ptlrpcd+0xa82/0xe40 [ptlrpc] [17700.340721] ? do_wait_intr+0xf0/0xf0 [17700.343955] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17700.349812] kthread+0x1d1/0x200 [17700.351960] ? set_kthread_struct+0x70/0x70 [17700.354139] ret_from_fork+0x1f/0x30 [17712.057017] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [17712.074135] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17712.102389] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17712.108794] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17712.115922] RIP: 0010:native_safe_halt+0x2e/0x50 [17712.118161] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17712.130468] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17712.137131] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17712.143150] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [17712.149713] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [17712.152437] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [17712.156297] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [17712.160757] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [17712.165053] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17712.168632] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [17712.171186] Call Trace: [17712.172083] [17712.174012] ? show_regs.cold.9+0x22/0x2f [17712.177252] ? watchdog_timer_fn.cold.6+0x62/0x10a [17712.179161] ? __hrtimer_run_queues+0x1c2/0x4d0 [17712.183505] ? watchdog+0x40/0x40 [17712.185164] ? hrtimer_interrupt+0x128/0x2c0 [17712.190944] ? smp_apic_timer_interrupt+0xa6/0x270 [17712.194211] ? apic_timer_interrupt+0xf/0x20 [17712.198526] [17712.199955] ? native_safe_halt+0x2e/0x50 [17712.202379] kvm_wait+0xcd/0xe0 [17712.207713] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17712.212105] do_raw_spin_lock+0x177/0x190 [17712.214623] _raw_spin_lock+0x21/0x30 [17712.219048] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17712.224182] ? do_raw_spin_unlock+0x75/0x190 [17712.226465] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17712.236466] ? do_wait_intr+0xf0/0xf0 [17712.239720] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17712.243948] mdc_close+0x53b/0xd70 [mdc] [17712.248187] lmv_close+0x18d/0x490 [lmv] [17712.252459] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [17712.256193] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17712.264558] ll_md_real_close+0xa5/0x2a0 [lustre] [17712.266443] ll_md_close+0x274/0x800 [lustre] [17712.270340] ll_file_release+0x128/0xc50 [lustre] [17712.275590] __fput+0x10d/0x440 [17712.277136] ____fput+0x12/0x20 [17712.282500] task_work_run+0xc6/0x110 [17712.284972] exit_to_usermode_loop+0x1dd/0x1f0 [17712.288679] do_syscall_64+0x3d6/0x440 [17712.290048] entry_SYSCALL_64_after_hwframe+0x49/0xae [17712.291879] RIP: 0033:0x7f151bee3cd8 [17712.300052] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17712.315517] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17712.319760] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [17712.323554] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [17712.330202] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17712.336515] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17712.342769] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17716.052086] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [17716.061074] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17716.067345] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [17716.133202] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17716.133429] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17716.133433] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17716.133448] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17716.133452] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17716.133458] RAX: 0000000000005696 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17716.133461] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [17716.133463] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [17716.133466] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [17716.133468] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [17716.133471] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [17716.133474] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17716.133476] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [17716.133486] Call Trace: [17716.133491] [17716.133497] ? show_regs.cold.9+0x22/0x2f [17716.133508] ? watchdog_timer_fn.cold.6+0x62/0x10a [17716.133513] ? __hrtimer_run_queues+0x1c2/0x4d0 [17716.133518] ? watchdog+0x40/0x40 [17716.133522] ? hrtimer_interrupt+0x128/0x2c0 [17716.133527] ? smp_apic_timer_interrupt+0xa6/0x270 [17716.133536] ? apic_timer_interrupt+0xf/0x20 [17716.133541] [17716.133544] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17716.133549] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17716.133554] do_raw_spin_lock+0x177/0x190 [17716.133559] _raw_spin_lock+0x21/0x30 [17716.133568] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17716.141409] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17716.141492] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17716.141499] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17716.141506] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17716.142345] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17716.142357] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17716.142365] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17716.142369] RAX: 0000000000007f12 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17716.142372] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [17716.142375] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [17716.142378] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [17716.142380] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [17716.142384] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [17716.142387] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17716.142389] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17716.142397] Call Trace: [17716.142402] [17716.142411] ? show_regs.cold.9+0x22/0x2f [17716.142421] ? watchdog_timer_fn.cold.6+0x62/0x10a [17716.142427] ? __hrtimer_run_queues+0x1c2/0x4d0 [17716.142435] ? watchdog+0x40/0x40 [17716.142439] ? hrtimer_interrupt+0x128/0x2c0 [17716.142442] ? smp_apic_timer_interrupt+0xa6/0x270 [17716.142451] ? apic_timer_interrupt+0xf/0x20 [17716.142456] [17716.142461] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17716.142466] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17716.142471] do_raw_spin_lock+0x177/0x190 [17716.142476] _raw_spin_lock+0x21/0x30 [17716.142481] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17716.142988] process_one_work+0x2c8/0x700 [17716.142996] worker_thread+0x296/0x6e0 [17716.143000] ? rescuer_thread+0x570/0x570 [17716.143004] kthread+0x1d1/0x200 [17716.143009] ? set_kthread_struct+0x70/0x70 [17716.143013] ret_from_fork+0x1f/0x30 [17716.601499] ? do_raw_spin_unlock+0x75/0x190 [17716.604054] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17716.606696] ? do_wait_intr+0xf0/0xf0 [17716.608836] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17716.611751] mdc_reint+0x2d/0x110 [mdc] [17716.613701] mdc_unlink+0x658/0x870 [mdc] [17716.615917] lmv_unlink+0x34d/0xbd0 [lmv] [17716.617937] ll_unlink+0x24d/0x7f0 [lustre] [17716.620496] vfs_unlink+0x202/0x330 [17716.622144] do_unlinkat+0x2fa/0x460 [17716.624000] __x64_sys_unlink+0x37/0x50 [17716.626151] do_syscall_64+0xc1/0x440 [17716.628205] entry_SYSCALL_64_after_hwframe+0x49/0xae [17716.630785] RIP: 0033:0x7f151bdee10b [17716.632708] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [17716.641965] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [17716.646213] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [17716.649934] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [17716.654636] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17716.658141] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17716.662081] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17720.290725] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15789s! [17720.294787] Showing busy workqueues and worker pools: [17720.300016] workqueue events: flags=0x0 [17720.302112] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17720.313099] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [17720.323780] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17720.329887] pending: kfree_rcu_monitor [17720.337722] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17720.345741] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [17720.357716] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17720.368630] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [17720.379412] workqueue events_long: flags=0x0 [17720.381371] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17720.390872] pending: lu_objects_flush [obdclass] [17720.401320] workqueue events_unbound: flags=0x2 [17720.403051] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [17720.408590] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17720.423826] workqueue events_power_efficient: flags=0x80 [17720.428767] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17720.434065] pending: do_cache_clean, neigh_periodic_work, check_lifetime [17720.441974] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17720.446743] pending: neigh_periodic_work [17720.451308] workqueue mm_percpu_wq: flags=0x8 [17720.455070] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17720.466487] pending: vmstat_update, lru_add_drain_per_cpu [17720.473774] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17720.481605] pending: vmstat_update [17720.485052] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17720.495430] pending: lru_add_drain_per_cpu [17720.497746] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [17720.506868] pending: lru_add_drain_per_cpu BAR(48) [17720.509090] workqueue writeback: flags=0x4a [17720.511651] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [17720.517203] pending: wb_workfn [17720.519216] workqueue xprtiod: flags=0xa [17720.523363] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17720.528694] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [17720.542773] workqueue ipv6_addrconf: flags=0x40008 [17720.551147] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17720.560187] pending: addrconf_verify_work [17720.561846] workqueue knbd0-recv: flags=0x1a [17720.567854] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17720.578141] in-flight: 658:recv_work [17720.586164] workqueue ptlrpc_pinger: flags=0x2 [17720.594297] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17720.608727] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [17720.620031] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15790s workers=4 idle: 32984 37 8 [17720.634532] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17705s workers=2 idle: 661 [17728.055967] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [17728.059878] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17728.098465] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17728.112275] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17728.118278] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17728.121064] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17728.135459] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17728.139639] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [17728.149887] RDX: 0000000000007f83 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [17728.154458] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [17728.158727] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [17728.165825] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17728.172866] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [17728.193951] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17728.202501] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17728.204707] Call Trace: [17728.205452] [17728.205946] ? show_regs.cold.9+0x22/0x2f [17728.209953] ? watchdog_timer_fn.cold.6+0x62/0x10a [17728.212836] ? __hrtimer_run_queues+0x1c2/0x4d0 [17728.217284] ? watchdog+0x40/0x40 [17728.218717] ? hrtimer_interrupt+0x128/0x2c0 [17728.220899] ? smp_apic_timer_interrupt+0xa6/0x270 [17728.226430] ? apic_timer_interrupt+0xf/0x20 [17728.230587] [17728.232635] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17728.236800] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17728.241544] do_raw_spin_lock+0x177/0x190 [17728.245124] _raw_spin_lock+0x21/0x30 [17728.247038] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17728.251679] obd_mod_free+0x46/0x140 [ptlrpc] [17728.260673] mdc_commit_open+0x91/0xb0 [mdc] [17728.263069] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17728.266360] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17728.284569] after_reply+0xe2c/0x1780 [ptlrpc] [17728.287654] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17728.290824] ? __next_timer_interrupt+0x160/0x160 [17728.295470] ptlrpcd+0xa82/0xe40 [ptlrpc] [17728.298271] ? do_wait_intr+0xf0/0xf0 [17728.300628] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17728.309898] kthread+0x1d1/0x200 [17728.314597] ? set_kthread_struct+0x70/0x70 [17728.316977] ret_from_fork+0x1f/0x30 [17740.059886] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [17740.071361] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17740.116413] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17740.131343] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17740.133948] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17740.138727] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17740.155305] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17740.165782] RAX: 00000000000071f9 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17740.168560] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [17740.174556] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [17740.179866] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [17740.183517] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [17740.187291] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [17740.191413] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17740.195831] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [17740.199379] Call Trace: [17740.200665] [17740.201495] ? show_regs.cold.9+0x22/0x2f [17740.204868] ? watchdog_timer_fn.cold.6+0x62/0x10a [17740.208538] ? __hrtimer_run_queues+0x1c2/0x4d0 [17740.211315] ? watchdog+0x40/0x40 [17740.216409] ? hrtimer_interrupt+0x128/0x2c0 [17740.218873] ? smp_apic_timer_interrupt+0xa6/0x270 [17740.221303] ? apic_timer_interrupt+0xf/0x20 [17740.224772] [17740.228897] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17740.230861] do_raw_spin_lock+0x177/0x190 [17740.235822] _raw_spin_lock+0x21/0x30 [17740.237934] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17740.241386] ? do_raw_spin_unlock+0x75/0x190 [17740.244728] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17740.247744] ? do_wait_intr+0xf0/0xf0 [17740.249607] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17740.253747] mdc_close+0x53b/0xd70 [mdc] [17740.262865] lmv_close+0x18d/0x490 [lmv] [17740.269745] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [17740.273860] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17740.278084] ll_md_real_close+0xa5/0x2a0 [lustre] [17740.281362] ll_md_close+0x274/0x800 [lustre] [17740.282901] ll_file_release+0x128/0xc50 [lustre] [17740.286263] __fput+0x10d/0x440 [17740.288161] ____fput+0x12/0x20 [17740.289775] task_work_run+0xc6/0x110 [17740.292262] exit_to_usermode_loop+0x1dd/0x1f0 [17740.298595] do_syscall_64+0x3d6/0x440 [17740.302704] entry_SYSCALL_64_after_hwframe+0x49/0xae [17740.304770] RIP: 0033:0x7f151bee3cd8 [17740.306278] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17740.314405] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17740.317689] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [17740.320561] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [17740.323173] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17740.327766] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17740.331760] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17744.056556] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [17744.061957] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17744.098481] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17744.103092] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17744.106129] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17744.109711] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17744.112749] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17744.120712] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17744.124562] RAX: 0000000000004a0e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17744.129156] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [17744.132696] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [17744.136628] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [17744.139778] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [17744.143773] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [17744.148828] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17744.152053] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17744.156042] Call Trace: [17744.157453] [17744.158577] ? show_regs.cold.9+0x22/0x2f [17744.160736] ? watchdog_timer_fn.cold.6+0x62/0x10a [17744.163865] ? __hrtimer_run_queues+0x1c2/0x4d0 [17744.166604] ? watchdog+0x40/0x40 [17744.168584] ? hrtimer_interrupt+0x128/0x2c0 [17744.171023] ? smp_apic_timer_interrupt+0xa6/0x270 [17744.173867] ? apic_timer_interrupt+0xf/0x20 [17744.175964] [17744.177359] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17744.180868] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17744.183946] do_raw_spin_lock+0x177/0x190 [17744.186036] _raw_spin_lock+0x21/0x30 [17744.188826] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17744.192109] process_one_work+0x2c8/0x700 [17744.194091] worker_thread+0x296/0x6e0 [17744.196538] ? rescuer_thread+0x570/0x570 [17744.198717] kthread+0x1d1/0x200 [17744.200988] ? set_kthread_struct+0x70/0x70 [17744.204411] ret_from_fork+0x1f/0x30 [17748.052937] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [17748.059874] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17748.095394] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17748.106946] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17748.118660] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17748.124239] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17748.137340] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17748.141197] RAX: 0000000000000ae0 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17748.144676] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [17748.148244] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [17748.151503] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [17748.154759] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [17748.158806] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [17748.162552] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17748.165357] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [17748.169751] Call Trace: [17748.175033] [17748.176426] ? show_regs.cold.9+0x22/0x2f [17748.178733] ? watchdog_timer_fn.cold.6+0x62/0x10a [17748.184997] ? __hrtimer_run_queues+0x1c2/0x4d0 [17748.191298] ? watchdog+0x40/0x40 [17748.196777] ? hrtimer_interrupt+0x128/0x2c0 [17748.202541] ? smp_apic_timer_interrupt+0xa6/0x270 [17748.210104] ? apic_timer_interrupt+0xf/0x20 [17748.217578] [17748.218658] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17748.222298] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17748.235591] do_raw_spin_lock+0x177/0x190 [17748.242472] _raw_spin_lock+0x21/0x30 [17748.247111] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17748.256621] ? do_raw_spin_unlock+0x75/0x190 [17748.262275] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17748.269066] ? do_wait_intr+0xf0/0xf0 [17748.272811] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17748.283647] mdc_reint+0x2d/0x110 [mdc] [17748.291594] mdc_unlink+0x658/0x870 [mdc] [17748.297557] lmv_unlink+0x34d/0xbd0 [lmv] [17748.303384] ll_unlink+0x24d/0x7f0 [lustre] [17748.306647] vfs_unlink+0x202/0x330 [17748.309122] do_unlinkat+0x2fa/0x460 [17748.310787] __x64_sys_unlink+0x37/0x50 [17748.317980] do_syscall_64+0xc1/0x440 [17748.319660] entry_SYSCALL_64_after_hwframe+0x49/0xae [17748.325099] RIP: 0033:0x7f151bdee10b [17748.332150] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [17748.344249] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [17748.355267] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [17748.360069] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [17748.365828] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17748.372269] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17748.381957] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17751.013116] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15820s! [17751.018484] Showing busy workqueues and worker pools: [17751.020827] workqueue events: flags=0x0 [17751.022921] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17751.026208] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [17751.031655] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17751.040670] pending: kfree_rcu_monitor [17751.044631] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17751.048598] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [17751.056310] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17751.059543] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [17751.068316] workqueue events_long: flags=0x0 [17751.075109] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17751.085576] pending: lu_objects_flush [obdclass] [17751.088415] workqueue events_unbound: flags=0x2 [17751.091887] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [17751.094926] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17751.100526] workqueue events_power_efficient: flags=0x80 [17751.103130] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17751.106457] pending: do_cache_clean, neigh_periodic_work, check_lifetime [17751.109548] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17751.113074] pending: neigh_periodic_work [17751.114404] workqueue mm_percpu_wq: flags=0x8 [17751.115977] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17751.118516] pending: vmstat_update, lru_add_drain_per_cpu [17751.120677] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17751.123443] pending: vmstat_update [17751.125363] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17751.128279] pending: lru_add_drain_per_cpu [17751.130097] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [17751.133271] pending: lru_add_drain_per_cpu BAR(48) [17751.135231] workqueue writeback: flags=0x4a [17751.136817] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [17751.139442] pending: wb_workfn [17751.141692] workqueue xprtiod: flags=0xa [17751.143522] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17751.145923] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [17751.148620] workqueue ipv6_addrconf: flags=0x40008 [17751.151190] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17751.155212] pending: addrconf_verify_work [17751.156983] workqueue knbd0-recv: flags=0x1a [17751.158830] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17751.161431] in-flight: 658:recv_work [17751.163458] workqueue ptlrpc_pinger: flags=0x2 [17751.165248] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17751.168050] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [17751.170393] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15820s workers=4 idle: 32984 37 8 [17751.174360] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17735s workers=2 idle: 661 [17756.055025] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [17756.065657] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17756.130592] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17756.148693] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17756.160367] RIP: 0010:native_safe_halt+0x2e/0x50 [17756.164227] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17756.176121] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17756.179856] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [17756.182223] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [17756.185303] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [17756.188254] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [17756.195438] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17756.199736] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [17756.203258] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17756.206139] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17756.209305] Call Trace: [17756.210539] [17756.211372] ? show_regs.cold.9+0x22/0x2f [17756.212755] ? watchdog_timer_fn.cold.6+0x62/0x10a [17756.214511] ? __hrtimer_run_queues+0x1c2/0x4d0 [17756.216398] ? watchdog+0x40/0x40 [17756.217974] ? hrtimer_interrupt+0x128/0x2c0 [17756.220269] ? smp_apic_timer_interrupt+0xa6/0x270 [17756.222655] ? apic_timer_interrupt+0xf/0x20 [17756.224773] [17756.225696] ? native_safe_halt+0x2e/0x50 [17756.231145] kvm_wait+0xcd/0xe0 [17756.233161] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17756.236969] do_raw_spin_lock+0x177/0x190 [17756.238355] _raw_spin_lock+0x21/0x30 [17756.242889] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17756.245362] obd_mod_free+0x46/0x140 [ptlrpc] [17756.249776] mdc_commit_open+0x91/0xb0 [mdc] [17756.251468] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17756.254506] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17756.260781] after_reply+0xe2c/0x1780 [ptlrpc] [17756.265735] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17756.271209] ? __next_timer_interrupt+0x160/0x160 [17756.273132] ptlrpcd+0xa82/0xe40 [ptlrpc] [17756.275653] ? do_wait_intr+0xf0/0xf0 [17756.277559] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17756.280654] kthread+0x1d1/0x200 [17756.281798] ? set_kthread_struct+0x70/0x70 [17756.292515] ret_from_fork+0x1f/0x30 [17768.065363] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [17768.074253] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17768.116330] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17768.124986] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17768.130200] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17768.139121] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17768.162523] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17768.170379] RAX: 000000000000531f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17768.174833] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [17768.179046] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [17768.182964] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [17768.186227] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [17768.191798] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [17768.198536] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17768.206264] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [17768.215838] Call Trace: [17768.216757] [17768.217531] ? show_regs.cold.9+0x22/0x2f [17768.222244] ? watchdog_timer_fn.cold.6+0x62/0x10a [17768.224512] ? __hrtimer_run_queues+0x1c2/0x4d0 [17768.227373] ? watchdog+0x40/0x40 [17768.229104] ? hrtimer_interrupt+0x128/0x2c0 [17768.233260] ? smp_apic_timer_interrupt+0xa6/0x270 [17768.234839] ? apic_timer_interrupt+0xf/0x20 [17768.241914] [17768.242502] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17768.244228] do_raw_spin_lock+0x177/0x190 [17768.248319] _raw_spin_lock+0x21/0x30 [17768.250589] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17768.257418] ? do_raw_spin_unlock+0x75/0x190 [17768.263059] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17768.265069] ? do_wait_intr+0xf0/0xf0 [17768.271760] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17768.277819] mdc_close+0x53b/0xd70 [mdc] [17768.280093] lmv_close+0x18d/0x490 [lmv] [17768.287021] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [17768.293507] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17768.299899] ll_md_real_close+0xa5/0x2a0 [lustre] [17768.301588] ll_md_close+0x274/0x800 [lustre] [17768.309257] ll_file_release+0x128/0xc50 [lustre] [17768.310966] __fput+0x10d/0x440 [17768.314228] ____fput+0x12/0x20 [17768.328935] task_work_run+0xc6/0x110 [17768.330110] exit_to_usermode_loop+0x1dd/0x1f0 [17768.334522] do_syscall_64+0x3d6/0x440 [17768.335985] entry_SYSCALL_64_after_hwframe+0x49/0xae [17768.340325] RIP: 0033:0x7f151bee3cd8 [17768.342370] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17768.364121] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17768.375440] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [17768.391477] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [17768.405455] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17768.417176] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17768.431131] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17772.056474] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [17772.062243] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17772.115466] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17772.131454] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17772.141209] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17772.147710] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17772.153428] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17772.173510] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17772.177805] RAX: 000000000000538e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17772.181077] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [17772.184314] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [17772.187197] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [17772.190157] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [17772.192873] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [17772.196044] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17772.198924] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17772.202102] Call Trace: [17772.203108] [17772.203655] ? show_regs.cold.9+0x22/0x2f [17772.205442] ? watchdog_timer_fn.cold.6+0x62/0x10a [17772.207622] ? __hrtimer_run_queues+0x1c2/0x4d0 [17772.209802] ? watchdog+0x40/0x40 [17772.211579] ? hrtimer_interrupt+0x128/0x2c0 [17772.213776] ? smp_apic_timer_interrupt+0xa6/0x270 [17772.216383] ? apic_timer_interrupt+0xf/0x20 [17772.218399] [17772.219493] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17772.223860] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17772.227324] do_raw_spin_lock+0x177/0x190 [17772.231331] _raw_spin_lock+0x21/0x30 [17772.233154] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17772.236360] process_one_work+0x2c8/0x700 [17772.238356] worker_thread+0x296/0x6e0 [17772.240820] ? rescuer_thread+0x570/0x570 [17772.243118] kthread+0x1d1/0x200 [17772.245252] ? set_kthread_struct+0x70/0x70 [17772.247827] ret_from_fork+0x1f/0x30 [17776.052026] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [17776.055179] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17776.148344] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17776.178379] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17776.195563] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17776.204428] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17776.231863] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17776.257684] RAX: 000000000000762f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17776.269098] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [17776.278056] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [17776.296220] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [17776.300595] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [17776.310417] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [17776.314139] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17776.317292] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [17776.321467] Call Trace: [17776.322928] [17776.323837] ? show_regs.cold.9+0x22/0x2f [17776.325195] ? watchdog_timer_fn.cold.6+0x62/0x10a [17776.326454] ? __hrtimer_run_queues+0x1c2/0x4d0 [17776.328424] ? watchdog+0x40/0x40 [17776.329446] ? hrtimer_interrupt+0x128/0x2c0 [17776.330950] ? smp_apic_timer_interrupt+0xa6/0x270 [17776.332489] ? apic_timer_interrupt+0xf/0x20 [17776.334458] [17776.335498] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17776.338404] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17776.340858] do_raw_spin_lock+0x177/0x190 [17776.346074] _raw_spin_lock+0x21/0x30 [17776.347424] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17776.352420] ? do_raw_spin_unlock+0x75/0x190 [17776.356063] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17776.359192] ? do_wait_intr+0xf0/0xf0 [17776.363600] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17776.365650] mdc_reint+0x2d/0x110 [mdc] [17776.368376] mdc_unlink+0x658/0x870 [mdc] [17776.370500] lmv_unlink+0x34d/0xbd0 [lmv] [17776.375514] ll_unlink+0x24d/0x7f0 [lustre] [17776.378005] vfs_unlink+0x202/0x330 [17776.379441] do_unlinkat+0x2fa/0x460 [17776.385589] __x64_sys_unlink+0x37/0x50 [17776.388036] do_syscall_64+0xc1/0x440 [17776.391160] entry_SYSCALL_64_after_hwframe+0x49/0xae [17776.407415] RIP: 0033:0x7f151bdee10b [17776.415506] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [17776.474723] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [17776.482610] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [17776.494971] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [17776.507271] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17776.515055] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17776.528589] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17781.217797] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15850s! [17781.227195] Showing busy workqueues and worker pools: [17781.230201] workqueue events: flags=0x0 [17781.235195] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17781.244964] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [17781.264270] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17781.274350] pending: kfree_rcu_monitor [17781.280309] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17781.298589] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [17781.323179] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17781.331452] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [17781.341013] workqueue events_long: flags=0x0 [17781.342616] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17781.347441] pending: lu_objects_flush [obdclass] [17781.352168] workqueue events_unbound: flags=0x2 [17781.354160] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [17781.358029] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17781.364187] workqueue events_power_efficient: flags=0x80 [17781.367374] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17781.378945] pending: do_cache_clean, neigh_periodic_work, check_lifetime [17781.383219] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17781.388355] pending: neigh_periodic_work [17781.393251] workqueue mm_percpu_wq: flags=0x8 [17781.397748] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17781.404335] pending: vmstat_update, lru_add_drain_per_cpu [17781.410305] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17781.415463] pending: vmstat_update [17781.423010] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17781.430134] pending: lru_add_drain_per_cpu [17781.431665] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [17781.435905] pending: lru_add_drain_per_cpu BAR(48) [17781.439338] workqueue writeback: flags=0x4a [17781.442215] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [17781.446359] pending: wb_workfn [17781.448758] workqueue xprtiod: flags=0xa [17781.450377] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17781.454190] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [17781.465055] workqueue ipv6_addrconf: flags=0x40008 [17781.471105] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17781.475320] pending: addrconf_verify_work [17781.478049] workqueue knbd0-recv: flags=0x1a [17781.480172] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17781.486852] in-flight: 658:recv_work [17781.488366] workqueue ptlrpc_pinger: flags=0x2 [17781.493434] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17781.500450] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [17781.509193] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15850s workers=4 idle: 32984 37 8 [17781.515048] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17766s workers=2 idle: 661 [17784.055024] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [17784.058182] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17784.081858] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17784.086837] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17784.090702] RIP: 0010:native_safe_halt+0x2e/0x50 [17784.092721] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17784.100862] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17784.104072] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [17784.107266] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [17784.110266] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [17784.113436] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [17784.116899] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17784.121687] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [17784.125120] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17784.127684] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17784.130794] Call Trace: [17784.132508] [17784.133314] ? show_regs.cold.9+0x22/0x2f [17784.136302] ? watchdog_timer_fn.cold.6+0x62/0x10a [17784.139029] ? __hrtimer_run_queues+0x1c2/0x4d0 [17784.141956] ? watchdog+0x40/0x40 [17784.143467] ? hrtimer_interrupt+0x128/0x2c0 [17784.145300] ? smp_apic_timer_interrupt+0xa6/0x270 [17784.147416] ? apic_timer_interrupt+0xf/0x20 [17784.149319] [17784.150302] ? native_safe_halt+0x2e/0x50 [17784.151985] kvm_wait+0xcd/0xe0 [17784.153278] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17784.155769] do_raw_spin_lock+0x177/0x190 [17784.157479] _raw_spin_lock+0x21/0x30 [17784.159043] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17784.161274] obd_mod_free+0x46/0x140 [ptlrpc] [17784.163520] mdc_commit_open+0x91/0xb0 [mdc] [17784.165521] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17784.167980] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17784.170655] after_reply+0xe2c/0x1780 [ptlrpc] [17784.172356] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17784.174899] ? __next_timer_interrupt+0x160/0x160 [17784.176934] ptlrpcd+0xa82/0xe40 [ptlrpc] [17784.179046] ? do_wait_intr+0xf0/0xf0 [17784.180708] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17784.183139] kthread+0x1d1/0x200 [17784.184680] ? set_kthread_struct+0x70/0x70 [17784.186607] ret_from_fork+0x1f/0x30 [17796.057277] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [17796.061291] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17796.089761] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17796.094006] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17796.097438] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17796.099889] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17796.107134] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17796.110077] RAX: 0000000000006421 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17796.113041] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [17796.115783] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [17796.119079] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [17796.123148] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [17796.128082] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [17796.130756] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17796.133270] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [17796.136430] Call Trace: [17796.137481] [17796.138269] ? show_regs.cold.9+0x22/0x2f [17796.139940] ? watchdog_timer_fn.cold.6+0x62/0x10a [17796.141804] ? __hrtimer_run_queues+0x1c2/0x4d0 [17796.143713] ? watchdog+0x40/0x40 [17796.145092] ? hrtimer_interrupt+0x128/0x2c0 [17796.146853] ? smp_apic_timer_interrupt+0xa6/0x270 [17796.148724] ? apic_timer_interrupt+0xf/0x20 [17796.150238] [17796.150976] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17796.153192] do_raw_spin_lock+0x177/0x190 [17796.154676] _raw_spin_lock+0x21/0x30 [17796.156200] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17796.158628] ? do_raw_spin_unlock+0x75/0x190 [17796.160356] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17796.162414] ? do_wait_intr+0xf0/0xf0 [17796.164126] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17796.166212] mdc_close+0x53b/0xd70 [mdc] [17796.167940] lmv_close+0x18d/0x490 [lmv] [17796.169355] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [17796.171668] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17796.174257] ll_md_real_close+0xa5/0x2a0 [lustre] [17796.176371] ll_md_close+0x274/0x800 [lustre] [17796.178323] ll_file_release+0x128/0xc50 [lustre] [17796.180613] __fput+0x10d/0x440 [17796.181904] ____fput+0x12/0x20 [17796.183481] task_work_run+0xc6/0x110 [17796.184861] exit_to_usermode_loop+0x1dd/0x1f0 [17796.186325] do_syscall_64+0x3d6/0x440 [17796.188121] entry_SYSCALL_64_after_hwframe+0x49/0xae [17796.189911] RIP: 0033:0x7f151bee3cd8 [17796.191269] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17796.198971] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17796.202202] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [17796.204116] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [17796.207129] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17796.209765] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17796.212322] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17800.056294] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [17800.066032] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17800.106948] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17800.113093] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17800.123726] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17800.127232] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17800.130073] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17800.138093] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [17800.142869] RAX: 0000000000007990 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17800.148773] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [17800.151507] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [17800.154757] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [17800.160334] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [17800.162620] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [17800.169250] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17800.172098] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17800.175233] Call Trace: [17800.176185] [17800.177218] ? show_regs.cold.9+0x22/0x2f [17800.178850] ? watchdog_timer_fn.cold.6+0x62/0x10a [17800.181186] ? __hrtimer_run_queues+0x1c2/0x4d0 [17800.183117] ? watchdog+0x40/0x40 [17800.184259] ? hrtimer_interrupt+0x128/0x2c0 [17800.188870] ? smp_apic_timer_interrupt+0xa6/0x270 [17800.191034] ? apic_timer_interrupt+0xf/0x20 [17800.192819] [17800.193947] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17800.197353] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17800.207713] do_raw_spin_lock+0x177/0x190 [17800.211666] _raw_spin_lock+0x21/0x30 [17800.216138] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17800.220149] process_one_work+0x2c8/0x700 [17800.223261] worker_thread+0x296/0x6e0 [17800.226364] ? rescuer_thread+0x570/0x570 [17800.228640] kthread+0x1d1/0x200 [17800.230747] ? set_kthread_struct+0x70/0x70 [17800.232738] ret_from_fork+0x1f/0x30 [17808.052871] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [17808.056420] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17808.106147] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17808.118843] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17808.127897] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17808.134678] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17808.160609] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17808.164651] RAX: 000000000000250b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17808.168392] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [17808.172843] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [17808.176967] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [17808.180177] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [17808.182632] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [17808.185635] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17808.188593] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [17808.191535] Call Trace: [17808.192590] [17808.193399] ? show_regs.cold.9+0x22/0x2f [17808.195157] ? watchdog_timer_fn.cold.6+0x62/0x10a [17808.196926] ? __hrtimer_run_queues+0x1c2/0x4d0 [17808.198713] ? watchdog+0x40/0x40 [17808.199930] ? hrtimer_interrupt+0x128/0x2c0 [17808.201655] ? smp_apic_timer_interrupt+0xa6/0x270 [17808.203515] ? apic_timer_interrupt+0xf/0x20 [17808.204730] [17808.205549] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17808.207493] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17808.209014] do_raw_spin_lock+0x177/0x190 [17808.210679] _raw_spin_lock+0x21/0x30 [17808.211839] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17808.214085] ? do_raw_spin_unlock+0x75/0x190 [17808.215906] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17808.217848] ? do_wait_intr+0xf0/0xf0 [17808.219267] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17808.221236] mdc_reint+0x2d/0x110 [mdc] [17808.222870] mdc_unlink+0x658/0x870 [mdc] [17808.224510] lmv_unlink+0x34d/0xbd0 [lmv] [17808.226102] ll_unlink+0x24d/0x7f0 [lustre] [17808.227755] vfs_unlink+0x202/0x330 [17808.229191] do_unlinkat+0x2fa/0x460 [17808.230779] __x64_sys_unlink+0x37/0x50 [17808.232370] do_syscall_64+0xc1/0x440 [17808.233774] entry_SYSCALL_64_after_hwframe+0x49/0xae [17808.235806] RIP: 0033:0x7f151bdee10b [17808.237409] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [17808.253529] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [17808.260661] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [17808.267818] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [17808.274428] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17808.280607] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17808.287212] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17811.940649] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15881s! [17811.948042] Showing busy workqueues and worker pools: [17811.952808] workqueue events: flags=0x0 [17811.958075] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17811.964763] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [17811.975654] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17811.990032] pending: kfree_rcu_monitor [17811.991690] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17811.995931] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [17812.008900] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17812.012958] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [17812.025443] workqueue events_long: flags=0x0 [17812.029951] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17812.034100] pending: lu_objects_flush [obdclass] [17812.036745] workqueue events_unbound: flags=0x2 [17812.040042] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [17812.044370] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17812.052552] workqueue events_power_efficient: flags=0x80 [17812.060770] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [17812.060978] Modules linked in: [17812.065103] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17812.065126] pending: do_cache_clean, neigh_periodic_work, check_lifetime [17812.065165] pwq 4: cpus=2 [17812.068043] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17812.068120] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17812.068127] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17812.068130] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17812.068144] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17812.068149] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17812.068153] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [17812.068156] RDX: 0000000000001dee RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [17812.068159] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [17812.068161] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [17812.068163] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17812.068166] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [17812.068169] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17812.068171] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17812.068180] Call Trace: [17812.068185] [17812.068191] ? show_regs.cold.9+0x22/0x2f [17812.068200] ? watchdog_timer_fn.cold.6+0x62/0x10a [17812.068206] ? __hrtimer_run_queues+0x1c2/0x4d0 [17812.068211] ? watchdog+0x40/0x40 [17812.068215] ? hrtimer_interrupt+0x128/0x2c0 [17812.068219] ? smp_apic_timer_interrupt+0xa6/0x270 [17812.068585] ? apic_timer_interrupt+0xf/0x20 [17812.068590] [17812.068594] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17812.068599] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17812.068603] do_raw_spin_lock+0x177/0x190 [17812.068608] _raw_spin_lock+0x21/0x30 [17812.068614] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17812.071815] node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17812.071834] pending: neigh_periodic_work [17812.071874] workqueue mm_percpu_wq: flags=0x8 [17812.071878] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17812.071890] pending: vmstat_update, lru_add_drain_per_cpu [17812.071905] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17812.071912] pending: vmstat_update [17812.071917] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17812.071925] pending: lru_add_drain_per_cpu [17812.071931] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [17812.071940] pending: lru_add_drain_per_cpu BAR(48) [17812.071952] workqueue writeback: flags=0x4a [17812.071954] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [17812.071961] pending: wb_workfn [17812.071974] workqueue xprtiod: flags=0xa [17812.071976] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17812.071982] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [17812.072044] workqueue ipv6_addrconf: flags=0x40008 [17812.072049] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17812.072057] pending: addrconf_verify_work [17812.072068] workqueue knbd0-recv: flags=0x1a [17812.072071] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17812.072078] in-flight: 658:recv_work [17812.072088] workqueue ptlrpc_pinger: flags=0x2 [17812.072091] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17812.072096] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [17812.072390] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15881s workers=4 idle: 32984 37 8 [17812.072890] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17796s workers=2 idle: 661 [17812.486059] obd_mod_free+0x46/0x140 [ptlrpc] [17812.495125] mdc_commit_open+0x91/0xb0 [mdc] [17812.500148] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17812.511111] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17812.521024] after_reply+0xe2c/0x1780 [ptlrpc] [17812.522831] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17812.532913] ? __next_timer_interrupt+0x160/0x160 [17812.534738] ptlrpcd+0xa82/0xe40 [ptlrpc] [17812.541117] ? do_wait_intr+0xf0/0xf0 [17812.542413] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17812.551501] kthread+0x1d1/0x200 [17812.553888] ? set_kthread_struct+0x70/0x70 [17812.561649] ret_from_fork+0x1f/0x30 [17824.057033] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [17824.062794] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17824.088375] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17824.093752] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17824.098453] RIP: 0010:native_safe_halt+0x2e/0x50 [17824.101067] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17824.121052] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17824.133172] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17824.142344] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [17824.146059] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [17824.152073] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [17824.154609] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [17824.158238] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [17824.168686] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17824.171889] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [17824.175450] Call Trace: [17824.176572] [17824.177795] ? show_regs.cold.9+0x22/0x2f [17824.179859] ? watchdog_timer_fn.cold.6+0x62/0x10a [17824.181568] ? __hrtimer_run_queues+0x1c2/0x4d0 [17824.184837] ? watchdog+0x40/0x40 [17824.186650] ? hrtimer_interrupt+0x128/0x2c0 [17824.190061] ? smp_apic_timer_interrupt+0xa6/0x270 [17824.192851] ? apic_timer_interrupt+0xf/0x20 [17824.195227] [17824.196212] ? native_safe_halt+0x2e/0x50 [17824.198123] kvm_wait+0xcd/0xe0 [17824.199492] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17824.201757] do_raw_spin_lock+0x177/0x190 [17824.203377] _raw_spin_lock+0x21/0x30 [17824.205111] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17824.212117] ? do_raw_spin_unlock+0x75/0x190 [17824.220971] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17824.225820] ? do_wait_intr+0xf0/0xf0 [17824.230486] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17824.238903] mdc_close+0x53b/0xd70 [mdc] [17824.242644] lmv_close+0x18d/0x490 [lmv] [17824.245884] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [17824.248967] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17824.251260] ll_md_real_close+0xa5/0x2a0 [lustre] [17824.254154] ll_md_close+0x274/0x800 [lustre] [17824.260700] ll_file_release+0x128/0xc50 [lustre] [17824.262673] __fput+0x10d/0x440 [17824.264092] ____fput+0x12/0x20 [17824.265129] task_work_run+0xc6/0x110 [17824.267031] exit_to_usermode_loop+0x1dd/0x1f0 [17824.269004] do_syscall_64+0x3d6/0x440 [17824.270607] entry_SYSCALL_64_after_hwframe+0x49/0xae [17824.272713] RIP: 0033:0x7f151bee3cd8 [17824.274130] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17824.285066] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17824.294305] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [17824.301866] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [17824.306205] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17824.315224] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17824.324655] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17824.977226] rcu: INFO: rcu_sched self-detected stall on CPU [17824.984594] rcu: 1-...!: (9551668 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [17824.990577] (t=15900330 jiffies g=249657 q=7368) [17824.993721] rcu: rcu_sched kthread starved for 15895282 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [17824.999198] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [17825.004307] rcu: RCU grace-period kthread stack dump: [17825.006705] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [17825.011581] Call Trace: [17825.012604] __schedule+0x351/0xcb0 [17825.014650] schedule+0xc0/0x180 [17825.016203] schedule_timeout+0xb4/0x190 [17825.017926] ? do_raw_spin_unlock+0x75/0x190 [17825.020103] ? __next_timer_interrupt+0x160/0x160 [17825.022577] ? prepare_to_swait_event+0xf3/0x140 [17825.024865] rcu_gp_kthread+0x8dd/0x15b0 [17825.026877] ? force_qs_rnp+0x2c0/0x2c0 [17825.028452] kthread+0x1d1/0x200 [17825.030580] ? set_kthread_struct+0x70/0x70 [17825.033084] ret_from_fork+0x1f/0x30 [17825.036774] rcu: Stack dump where RCU GP kthread last ran: [17825.040387] Sending NMI from CPU 1 to CPUs 2: [17825.043576] NMI backtrace for cpu 2 [17825.043582] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17825.043584] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17825.043585] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17825.043588] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17825.043590] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17825.043592] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 [17825.043594] RAX: 0000000000007711 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17825.043595] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [17825.043596] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [17825.043598] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [17825.043599] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [17825.043600] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [17825.043601] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17825.043603] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17825.043604] Call Trace: [17825.043605] [17825.043605] ? show_regs.cold.9+0x22/0x2f [17825.043607] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [17825.043608] ? nmi_cpu_backtrace_handler+0x15/0x40 [17825.043609] ? nmi_handle+0xbd/0x210 [17825.043610] ? default_do_nmi+0x4b/0x1b0 [17825.043611] ? do_nmi+0x280/0x380 [17825.043612] ? end_repeat_nmi+0x16/0x1f [17825.043613] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17825.043614] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17825.043615] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17825.043616] [17825.043616] do_raw_spin_lock+0x177/0x190 [17825.043617] _raw_spin_lock+0x21/0x30 [17825.043618] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17825.043619] process_one_work+0x2c8/0x700 [17825.043620] worker_thread+0x296/0x6e0 [17825.043621] ? rescuer_thread+0x570/0x570 [17825.043622] kthread+0x1d1/0x200 [17825.043622] ? set_kthread_struct+0x70/0x70 [17825.043623] ret_from_fork+0x1f/0x30 [17825.043990] NMI backtrace for cpu 1 [17825.213470] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17825.220771] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17825.224651] Call Trace: [17825.230048] [17825.231428] ? dump_stack+0xbb/0x10e [17825.235303] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [17825.238789] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [17825.241754] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [17825.243079] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [17825.245456] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [17825.251125] ? rcu_dump_cpu_stacks+0x11b/0x171 [17825.255215] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [17825.258474] ? update_process_times+0x6d/0xc0 [17825.265954] ? tick_sched_handle.isra.18+0x27/0x90 [17825.268117] ? tick_sched_timer+0x53/0xb0 [17825.273218] ? __hrtimer_run_queues+0x1c2/0x4d0 [17825.275608] ? tick_sched_handle.isra.18+0x90/0x90 [17825.278012] ? hrtimer_interrupt+0x128/0x2c0 [17825.283146] ? smp_apic_timer_interrupt+0xa6/0x270 [17825.286294] ? apic_timer_interrupt+0xf/0x20 [17825.288560] [17825.289639] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17825.291665] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17825.296737] ? do_raw_spin_lock+0x177/0x190 [17825.298218] ? _raw_spin_lock+0x21/0x30 [17825.300079] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17825.302170] ? obd_mod_free+0x46/0x140 [ptlrpc] [17825.304479] ? mdc_commit_open+0x91/0xb0 [mdc] [17825.306153] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17825.311996] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17825.314527] ? after_reply+0xe2c/0x1780 [ptlrpc] [17825.317616] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17825.320926] ? __next_timer_interrupt+0x160/0x160 [17825.323480] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [17825.325794] ? do_wait_intr+0xf0/0xf0 [17825.327511] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17825.331076] ? kthread+0x1d1/0x200 [17825.332566] ? set_kthread_struct+0x70/0x70 [17825.338976] ? ret_from_fork+0x1f/0x30 [17840.052388] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [17840.063237] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17840.105469] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17840.119512] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17840.128876] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17840.138537] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17840.157758] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17840.167329] RAX: 0000000000004c8a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17840.177221] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [17840.181469] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [17840.185202] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [17840.188500] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [17840.191411] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [17840.194562] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17840.198406] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [17840.200116] Call Trace: [17840.200752] [17840.201611] ? show_regs.cold.9+0x22/0x2f [17840.202972] ? watchdog_timer_fn.cold.6+0x62/0x10a [17840.204689] ? __hrtimer_run_queues+0x1c2/0x4d0 [17840.206422] ? watchdog+0x40/0x40 [17840.207584] ? hrtimer_interrupt+0x128/0x2c0 [17840.208930] ? smp_apic_timer_interrupt+0xa6/0x270 [17840.210547] ? apic_timer_interrupt+0xf/0x20 [17840.211956] [17840.212829] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17840.214690] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17840.216442] do_raw_spin_lock+0x177/0x190 [17840.217793] _raw_spin_lock+0x21/0x30 [17840.219200] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17840.221199] ? do_raw_spin_unlock+0x75/0x190 [17840.223169] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17840.227619] ? do_wait_intr+0xf0/0xf0 [17840.228889] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17840.231608] mdc_reint+0x2d/0x110 [mdc] [17840.232665] mdc_unlink+0x658/0x870 [mdc] [17840.233894] lmv_unlink+0x34d/0xbd0 [lmv] [17840.244646] ll_unlink+0x24d/0x7f0 [lustre] [17840.248302] vfs_unlink+0x202/0x330 [17840.250767] do_unlinkat+0x2fa/0x460 [17840.253966] __x64_sys_unlink+0x37/0x50 [17840.259048] do_syscall_64+0xc1/0x440 [17840.260789] entry_SYSCALL_64_after_hwframe+0x49/0xae [17840.263825] RIP: 0033:0x7f151bdee10b [17840.267802] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [17840.281620] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [17840.287642] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [17840.291148] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [17840.296384] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17840.302792] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17840.314947] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17842.145189] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15911s! [17842.150090] Showing busy workqueues and worker pools: [17842.152779] workqueue events: flags=0x0 [17842.155085] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17842.160832] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [17842.167606] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17842.173720] pending: kfree_rcu_monitor [17842.176865] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17842.180626] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [17842.187137] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17842.190781] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [17842.194421] workqueue events_long: flags=0x0 [17842.196477] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17842.200223] pending: lu_objects_flush [obdclass] [17842.202784] workqueue events_unbound: flags=0x2 [17842.204857] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [17842.207866] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17842.212228] workqueue events_power_efficient: flags=0x80 [17842.214814] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17842.218077] pending: do_cache_clean, neigh_periodic_work, check_lifetime [17842.221078] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17842.224165] pending: neigh_periodic_work [17842.226167] workqueue mm_percpu_wq: flags=0x8 [17842.228219] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17842.231360] pending: vmstat_update, lru_add_drain_per_cpu [17842.234335] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17842.236954] pending: vmstat_update [17842.238818] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17842.241525] pending: lru_add_drain_per_cpu [17842.243470] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [17842.246250] pending: lru_add_drain_per_cpu BAR(48) [17842.248909] workqueue writeback: flags=0x4a [17842.250727] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [17842.253420] pending: wb_workfn [17842.254792] workqueue xprtiod: flags=0xa [17842.256425] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17842.259163] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [17842.262940] workqueue ipv6_addrconf: flags=0x40008 [17842.265128] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17842.268124] pending: addrconf_verify_work [17842.270054] workqueue knbd0-recv: flags=0x1a [17842.271929] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17842.274819] in-flight: 658:recv_work [17842.276687] workqueue ptlrpc_pinger: flags=0x2 [17842.278507] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17842.281215] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [17842.284068] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15911s workers=4 idle: 32984 37 8 [17842.287730] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17827s workers=2 idle: 661 [17852.056065] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [17852.056225] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [17852.057034] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [17852.057042] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17852.057181] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17852.057187] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17852.057190] RIP: 0010:native_safe_halt+0x2e/0x50 [17852.057203] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17852.057207] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17852.057212] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17852.057214] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [17852.057217] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [17852.057219] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [17852.057222] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [17852.057226] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [17852.057229] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17852.057231] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [17852.057292] Call Trace: [17852.057297] [17852.057302] ? show_regs.cold.9+0x22/0x2f [17852.057310] ? watchdog_timer_fn.cold.6+0x62/0x10a [17852.057317] ? __hrtimer_run_queues+0x1c2/0x4d0 [17852.057321] ? watchdog+0x40/0x40 [17852.057325] ? hrtimer_interrupt+0x128/0x2c0 [17852.057330] ? smp_apic_timer_interrupt+0xa6/0x270 [17852.057336] ? apic_timer_interrupt+0xf/0x20 [17852.057341] [17852.057344] ? native_safe_halt+0x2e/0x50 [17852.057348] kvm_wait+0xcd/0xe0 [17852.057354] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17852.057362] do_raw_spin_lock+0x177/0x190 [17852.057366] _raw_spin_lock+0x21/0x30 [17852.057370] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17852.057640] ? do_raw_spin_unlock+0x75/0x190 [17852.057646] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17852.057986] ? do_wait_intr+0xf0/0xf0 [17852.057993] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17852.058444] mdc_close+0x53b/0xd70 [mdc] [17852.058478] lmv_close+0x18d/0x490 [lmv] [17852.058508] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [17852.058936] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17852.059058] ll_md_real_close+0xa5/0x2a0 [lustre] [17852.059598] Modules linked in: lustre(O) [17852.059215] ll_md_close+0x274/0x800 [lustre] [17852.059805] ll_file_release+0x128/0xc50 [lustre] [17852.059889] __fput+0x10d/0x440 [17852.059899] ____fput+0x12/0x20 [17852.059903] task_work_run+0xc6/0x110 [17852.059911] exit_to_usermode_loop+0x1dd/0x1f0 [17852.059920] do_syscall_64+0x3d6/0x440 [17852.059924] entry_SYSCALL_64_after_hwframe+0x49/0xae [17852.059932] RIP: 0033:0x7f151bee3cd8 [17852.059938] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17852.059942] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17852.059947] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [17852.059949] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [17852.059951] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17852.059953] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17852.059955] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17852.065788] Modules linked in: [17852.070289] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17852.070363] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17852.070370] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17852.070377] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17852.070656] RIP: 0010:native_safe_halt+0x2e/0x50 [17852.070667] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17852.070671] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17852.070676] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17852.070678] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [17852.070681] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [17852.070683] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [17852.070685] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [17852.070688] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [17852.070691] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17852.070694] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17852.070825] Call Trace: [17852.070832] [17852.070837] ? show_regs.cold.9+0x22/0x2f [17852.070849] ? watchdog_timer_fn.cold.6+0x62/0x10a [17852.070855] ? __hrtimer_run_queues+0x1c2/0x4d0 [17852.070859] ? watchdog+0x40/0x40 [17852.070864] ? hrtimer_interrupt+0x128/0x2c0 [17852.071024] ? smp_apic_timer_interrupt+0xa6/0x270 [17852.071033] ? apic_timer_interrupt+0xf/0x20 [17852.071038] [17852.071041] ? native_safe_halt+0x2e/0x50 [17852.071045] kvm_wait+0xcd/0xe0 [17852.071053] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17852.071062] do_raw_spin_lock+0x177/0x190 [17852.071066] _raw_spin_lock+0x21/0x30 [17852.071070] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17852.071278] process_one_work+0x2c8/0x700 [17852.071286] worker_thread+0x296/0x6e0 [17852.071291] ? rescuer_thread+0x570/0x570 [17852.071294] kthread+0x1d1/0x200 [17852.071299] ? set_kthread_struct+0x70/0x70 [17852.071302] ret_from_fork+0x1f/0x30 [17852.541847] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17852.560081] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17852.564727] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17852.568454] RIP: 0010:native_safe_halt+0x2e/0x50 [17852.570472] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17852.578579] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17852.581765] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [17852.584015] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [17852.587290] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [17852.590177] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [17852.593351] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17852.596324] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [17852.599500] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17852.602182] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17852.605304] Call Trace: [17852.606239] [17852.606829] ? show_regs.cold.9+0x22/0x2f [17852.608274] ? watchdog_timer_fn.cold.6+0x62/0x10a [17852.610310] ? __hrtimer_run_queues+0x1c2/0x4d0 [17852.612466] ? watchdog+0x40/0x40 [17852.614062] ? hrtimer_interrupt+0x128/0x2c0 [17852.615806] ? smp_apic_timer_interrupt+0xa6/0x270 [17852.618208] ? apic_timer_interrupt+0xf/0x20 [17852.620062] [17852.620711] ? native_safe_halt+0x2e/0x50 [17852.622580] kvm_wait+0xcd/0xe0 [17852.623862] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17852.626285] do_raw_spin_lock+0x177/0x190 [17852.627770] _raw_spin_lock+0x21/0x30 [17852.629394] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17852.631857] obd_mod_free+0x46/0x140 [ptlrpc] [17852.633720] mdc_commit_open+0x91/0xb0 [mdc] [17852.635562] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17852.637838] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17852.640467] after_reply+0xe2c/0x1780 [ptlrpc] [17852.642638] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17852.645133] ? __next_timer_interrupt+0x160/0x160 [17852.646609] ptlrpcd+0xa82/0xe40 [ptlrpc] [17852.648709] ? do_wait_intr+0xf0/0xf0 [17852.651630] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17852.656272] kthread+0x1d1/0x200 [17852.657837] ? set_kthread_struct+0x70/0x70 [17852.660359] ret_from_fork+0x1f/0x30 [17868.052017] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [17868.058955] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17868.091502] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17868.099965] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17868.108549] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17868.112454] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17868.131810] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17868.136524] RAX: 000000000000368c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17868.142383] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [17868.146367] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [17868.150137] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [17868.154196] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [17868.160739] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [17868.165012] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17868.168649] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [17868.175302] Call Trace: [17868.176419] [17868.183047] ? show_regs.cold.9+0x22/0x2f [17868.187188] ? watchdog_timer_fn.cold.6+0x62/0x10a [17868.190576] ? __hrtimer_run_queues+0x1c2/0x4d0 [17868.192260] ? watchdog+0x40/0x40 [17868.193982] ? hrtimer_interrupt+0x128/0x2c0 [17868.196114] ? smp_apic_timer_interrupt+0xa6/0x270 [17868.198629] ? apic_timer_interrupt+0xf/0x20 [17868.200497] [17868.201813] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17868.203964] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17868.206300] do_raw_spin_lock+0x177/0x190 [17868.208183] _raw_spin_lock+0x21/0x30 [17868.209727] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17868.212095] ? do_raw_spin_unlock+0x75/0x190 [17868.214122] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17868.216396] ? do_wait_intr+0xf0/0xf0 [17868.217956] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17868.220452] mdc_reint+0x2d/0x110 [mdc] [17868.222148] mdc_unlink+0x658/0x870 [mdc] [17868.223573] lmv_unlink+0x34d/0xbd0 [lmv] [17868.225501] ll_unlink+0x24d/0x7f0 [lustre] [17868.227524] vfs_unlink+0x202/0x330 [17868.229105] do_unlinkat+0x2fa/0x460 [17868.231254] __x64_sys_unlink+0x37/0x50 [17868.233310] do_syscall_64+0xc1/0x440 [17868.235164] entry_SYSCALL_64_after_hwframe+0x49/0xae [17868.237584] RIP: 0033:0x7f151bdee10b [17868.239252] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [17868.248470] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [17868.252133] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [17868.258359] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [17868.263417] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17868.266889] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17868.270780] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17872.353044] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15941s! [17872.354839] Showing busy workqueues and worker pools: [17872.355674] workqueue events: flags=0x0 [17872.356372] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17872.359533] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [17872.361559] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17872.365882] pending: kfree_rcu_monitor [17872.367397] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17872.368539] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [17872.370700] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17872.372356] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [17872.379481] workqueue events_long: flags=0x0 [17872.380454] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17872.382881] pending: lu_objects_flush [obdclass] [17872.384483] workqueue events_unbound: flags=0x2 [17872.385488] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [17872.389483] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17872.392618] workqueue events_power_efficient: flags=0x80 [17872.395165] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17872.396895] pending: do_cache_clean, neigh_periodic_work, check_lifetime [17872.398263] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17872.399882] pending: neigh_periodic_work [17872.400593] workqueue mm_percpu_wq: flags=0x8 [17872.401947] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17872.404229] pending: vmstat_update, lru_add_drain_per_cpu [17872.409281] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17872.411631] pending: vmstat_update [17872.412301] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17872.416381] pending: lru_add_drain_per_cpu [17872.420623] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [17872.424352] pending: lru_add_drain_per_cpu BAR(48) [17872.426473] workqueue writeback: flags=0x4a [17872.427493] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [17872.429886] pending: wb_workfn [17872.431171] workqueue xprtiod: flags=0xa [17872.434623] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17872.436978] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [17872.440154] workqueue ipv6_addrconf: flags=0x40008 [17872.450858] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17872.460611] pending: addrconf_verify_work [17872.466017] workqueue knbd0-recv: flags=0x1a [17872.467547] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17872.469284] in-flight: 658:recv_work [17872.473336] workqueue ptlrpc_pinger: flags=0x2 [17872.475353] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17872.478412] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [17872.490428] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15941s workers=4 idle: 32984 37 8 [17872.495062] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17857s workers=2 idle: 661 [17880.055026] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [17880.056657] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [17880.057243] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17880.057311] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17880.057317] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17880.057320] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17880.057334] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17880.057338] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17880.057342] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [17880.057345] RDX: 0000000000002f1a RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [17880.057347] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [17880.057349] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [17880.057351] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17880.057354] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [17880.057357] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17880.057360] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17880.057368] Call Trace: [17880.057376] [17880.057377] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [17880.057385] Modules linked in: lustre(O) osp(O) ofd(O) [17880.057383] ? show_regs.cold.9+0x22/0x2f [17880.057395] lod(O) mdt(O) mdd(O) mgs(O) [17880.057401] ? watchdog_timer_fn.cold.6+0x62/0x10a [17880.057408] lquota(O) lfsck(O) obdecho(O) [17880.057414] ? __hrtimer_run_queues+0x1c2/0x4d0 [17880.057421] mgc(O) mdc(O) [17880.057426] ? watchdog+0x40/0x40 [17880.057429] lov(O) osc(O) lmv(O) [17880.057436] ? hrtimer_interrupt+0x128/0x2c0 [17880.057440] ec(O) fid(O) fld(O) ptlrpc_gss(O) [17880.057445] ? smp_apic_timer_interrupt+0xa6/0x270 [17880.057450] ptlrpc(O) obdclass(O) ksocklnd(O) [17880.057457] ? apic_timer_interrupt+0xf/0x20 [17880.057461] lnet(O) libcfs(O) [17880.057467] [17880.057469] rpcsec_gss_krb5 auth_rpcgss [17880.057473] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17880.057476] nfsv4 dns_resolver intel_rapl_msr [17880.057483] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17880.057486] intel_rapl_common [17880.057494] do_raw_spin_lock+0x177/0x190 [17880.057499] _raw_spin_lock+0x21/0x30 [17880.057505] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17880.057935] obd_mod_free+0x46/0x140 [ptlrpc] [17880.058125] mdc_commit_open+0x91/0xb0 [mdc] [17880.058166] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17880.058361] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17880.058551] after_reply+0xe2c/0x1780 [ptlrpc] [17880.057490] sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17880.059247] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17880.059253] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17880.059256] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17880.059267] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17880.059271] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17880.059277] RAX: 000000000000678f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17880.059279] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [17880.059281] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [17880.059284] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [17880.059286] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [17880.059290] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [17880.059293] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17880.059296] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [17880.059304] Call Trace: [17880.059307] [17880.059310] ? show_regs.cold.9+0x22/0x2f [17880.059319] ? watchdog_timer_fn.cold.6+0x62/0x10a [17880.059325] ? __hrtimer_run_queues+0x1c2/0x4d0 [17880.059330] ? watchdog+0x40/0x40 [17880.059335] ? hrtimer_interrupt+0x128/0x2c0 [17880.059339] ? smp_apic_timer_interrupt+0xa6/0x270 [17880.059348] ? apic_timer_interrupt+0xf/0x20 [17880.059353] [17880.059356] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17880.059363] do_raw_spin_lock+0x177/0x190 [17880.059367] _raw_spin_lock+0x21/0x30 [17880.059372] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17880.058759] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17880.060467] ? __next_timer_interrupt+0x160/0x160 [17880.060476] ptlrpcd+0xa82/0xe40 [ptlrpc] [17880.060692] ? do_wait_intr+0xf0/0xf0 [17880.060700] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17880.064699] kthread+0x1d1/0x200 [17880.064708] ? set_kthread_struct+0x70/0x70 [17880.064712] ret_from_fork+0x1f/0x30 [17880.067345] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17880.067426] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17880.067434] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17880.067440] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17880.067870] RIP: 0010:native_safe_halt+0x2e/0x50 [17880.067880] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17880.067883] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17880.067888] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17880.067890] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [17880.067892] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [17880.067894] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [17880.067896] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [17880.067899] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [17880.067901] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17880.067903] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17880.067914] Call Trace: [17880.067919] [17880.067924] ? show_regs.cold.9+0x22/0x2f [17880.067934] ? watchdog_timer_fn.cold.6+0x62/0x10a [17880.067951] ? __hrtimer_run_queues+0x1c2/0x4d0 [17880.067955] ? watchdog+0x40/0x40 [17880.067959] ? hrtimer_interrupt+0x128/0x2c0 [17880.067963] ? smp_apic_timer_interrupt+0xa6/0x270 [17880.067970] ? apic_timer_interrupt+0xf/0x20 [17880.067974] [17880.067977] ? native_safe_halt+0x2e/0x50 [17880.067981] kvm_wait+0xcd/0xe0 [17880.067988] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17880.067996] do_raw_spin_lock+0x177/0x190 [17880.068000] _raw_spin_lock+0x21/0x30 [17880.068004] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17880.068428] process_one_work+0x2c8/0x700 [17880.068436] worker_thread+0x296/0x6e0 [17880.068441] ? rescuer_thread+0x570/0x570 [17880.068445] kthread+0x1d1/0x200 [17880.068450] ? set_kthread_struct+0x70/0x70 [17880.068454] ret_from_fork+0x1f/0x30 [17880.769470] ? do_raw_spin_unlock+0x75/0x190 [17880.771434] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17880.774296] ? do_wait_intr+0xf0/0xf0 [17880.775979] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17880.785602] mdc_close+0x53b/0xd70 [mdc] [17880.787054] lmv_close+0x18d/0x490 [lmv] [17880.789172] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [17880.802269] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17880.806170] ll_md_real_close+0xa5/0x2a0 [lustre] [17880.808171] ll_md_close+0x274/0x800 [lustre] [17880.815250] ll_file_release+0x128/0xc50 [lustre] [17880.819004] __fput+0x10d/0x440 [17880.823548] ____fput+0x12/0x20 [17880.824607] task_work_run+0xc6/0x110 [17880.828570] exit_to_usermode_loop+0x1dd/0x1f0 [17880.837540] do_syscall_64+0x3d6/0x440 [17880.840141] entry_SYSCALL_64_after_hwframe+0x49/0xae [17880.846966] RIP: 0033:0x7f151bee3cd8 [17880.848287] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17880.860987] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17880.865561] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [17880.869903] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [17880.877890] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17880.887237] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17880.893882] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17900.052159] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [17900.055321] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17900.073049] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17900.077530] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17900.080927] RIP: 0010:native_safe_halt+0x2e/0x50 [17900.082873] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17900.090277] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17900.093295] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17900.096395] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [17900.099253] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [17900.102078] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [17900.104925] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [17900.107635] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [17900.110819] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17900.113078] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [17900.116056] Call Trace: [17900.117012] [17900.117827] ? show_regs.cold.9+0x22/0x2f [17900.119336] ? watchdog_timer_fn.cold.6+0x62/0x10a [17900.121274] ? __hrtimer_run_queues+0x1c2/0x4d0 [17900.123031] ? watchdog+0x40/0x40 [17900.124488] ? hrtimer_interrupt+0x128/0x2c0 [17900.126200] ? smp_apic_timer_interrupt+0xa6/0x270 [17900.128194] ? apic_timer_interrupt+0xf/0x20 [17900.129890] [17900.131229] ? native_safe_halt+0x2e/0x50 [17900.132853] kvm_wait+0xcd/0xe0 [17900.134124] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17900.136280] do_raw_spin_lock+0x177/0x190 [17900.137901] _raw_spin_lock+0x21/0x30 [17900.139513] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17900.141876] ? do_raw_spin_unlock+0x75/0x190 [17900.143604] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17900.145945] ? do_wait_intr+0xf0/0xf0 [17900.147579] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17900.149843] mdc_reint+0x2d/0x110 [mdc] [17900.151395] mdc_unlink+0x658/0x870 [mdc] [17900.153224] lmv_unlink+0x34d/0xbd0 [lmv] [17900.154922] ll_unlink+0x24d/0x7f0 [lustre] [17900.156848] vfs_unlink+0x202/0x330 [17900.158338] do_unlinkat+0x2fa/0x460 [17900.159879] __x64_sys_unlink+0x37/0x50 [17900.161393] do_syscall_64+0xc1/0x440 [17900.162921] entry_SYSCALL_64_after_hwframe+0x49/0xae [17900.164936] RIP: 0033:0x7f151bdee10b [17900.166357] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [17900.174150] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [17900.177355] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [17900.180237] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [17900.183015] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17900.185875] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17900.188707] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17902.560092] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15971s! [17902.567159] Showing busy workqueues and worker pools: [17902.568808] workqueue events: flags=0x0 [17902.569891] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17902.574849] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [17902.580772] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17902.583836] pending: kfree_rcu_monitor [17902.585975] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17902.589453] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [17902.594335] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17902.598615] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [17902.603924] workqueue events_long: flags=0x0 [17902.607225] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17902.611353] pending: lu_objects_flush [obdclass] [17902.613412] workqueue events_unbound: flags=0x2 [17902.615300] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [17902.618189] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17902.622445] workqueue events_power_efficient: flags=0x80 [17902.624347] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17902.630908] pending: do_cache_clean, neigh_periodic_work, check_lifetime [17902.637858] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17902.643418] pending: neigh_periodic_work [17902.649580] workqueue mm_percpu_wq: flags=0x8 [17902.651976] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17902.663955] pending: vmstat_update, lru_add_drain_per_cpu [17902.668993] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17902.676627] pending: vmstat_update [17902.678862] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17902.682143] pending: lru_add_drain_per_cpu [17902.687527] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [17902.692889] pending: lru_add_drain_per_cpu BAR(48) [17902.696545] workqueue writeback: flags=0x4a [17902.703595] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [17902.706874] pending: wb_workfn [17902.711647] workqueue xprtiod: flags=0xa [17902.720954] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17902.723876] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [17902.732337] workqueue ipv6_addrconf: flags=0x40008 [17902.736142] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17902.743450] pending: addrconf_verify_work [17902.748801] workqueue knbd0-recv: flags=0x1a [17902.752374] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17902.758123] in-flight: 658:recv_work [17902.763801] workqueue ptlrpc_pinger: flags=0x2 [17902.765686] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17902.768802] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [17902.775684] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15972s workers=4 idle: 32984 37 8 [17902.782301] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17887s workers=2 idle: 661 [17908.055242] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [17908.056110] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [17908.058555] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [17908.058613] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17908.058692] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17908.058698] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17908.058700] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17908.058715] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17908.058719] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17908.058723] RAX: 000000000000121a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17908.058725] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [17908.058728] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [17908.058730] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [17908.058733] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [17908.058736] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [17908.058738] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17908.058740] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [17908.058749] Call Trace: [17908.058755] [17908.058769] ? show_regs.cold.9+0x22/0x2f [17908.058782] ? watchdog_timer_fn.cold.6+0x62/0x10a [17908.058789] ? __hrtimer_run_queues+0x1c2/0x4d0 [17908.058793] ? watchdog+0x40/0x40 [17908.058801] ? hrtimer_interrupt+0x128/0x2c0 [17908.058805] ? smp_apic_timer_interrupt+0xa6/0x270 [17908.058813] ? apic_timer_interrupt+0xf/0x20 [17908.058819] [17908.058822] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17908.058829] do_raw_spin_lock+0x177/0x190 [17908.058833] _raw_spin_lock+0x21/0x30 [17908.058839] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17908.059266] ? do_raw_spin_unlock+0x75/0x190 [17908.059278] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17908.059914] ? do_wait_intr+0xf0/0xf0 [17908.059927] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17908.060169] mdc_close+0x53b/0xd70 [mdc] [17908.060224] lmv_close+0x18d/0x490 [lmv] [17908.060259] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [17908.060555] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17908.060776] ll_md_real_close+0xa5/0x2a0 [lustre] [17908.060962] ll_md_close+0x274/0x800 [lustre] [17908.061119] ll_file_release+0x128/0xc50 [lustre] [17908.061218] __fput+0x10d/0x440 [17908.061230] ____fput+0x12/0x20 [17908.061233] task_work_run+0xc6/0x110 [17908.061243] exit_to_usermode_loop+0x1dd/0x1f0 [17908.061253] do_syscall_64+0x3d6/0x440 [17908.061260] entry_SYSCALL_64_after_hwframe+0x49/0xae [17908.061271] RIP: 0033:0x7f151bee3cd8 [17908.061278] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17908.061287] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17908.061369] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [17908.061372] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [17908.061374] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17908.061376] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17908.061379] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17908.063802] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17908.063881] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17908.063888] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17908.063891] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17908.063904] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17908.063908] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17908.063912] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [17908.063915] RDX: 000000000000322f RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [17908.063917] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [17908.063919] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [17908.063922] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17908.063926] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [17908.063929] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17908.063931] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17908.063941] Call Trace: [17908.063946] [17908.063951] ? show_regs.cold.9+0x22/0x2f [17908.063960] ? watchdog_timer_fn.cold.6+0x62/0x10a [17908.067015] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17908.072709] ? __hrtimer_run_queues+0x1c2/0x4d0 [17908.072729] ? watchdog+0x40/0x40 [17908.072735] ? hrtimer_interrupt+0x128/0x2c0 [17908.072740] ? smp_apic_timer_interrupt+0xa6/0x270 [17908.072749] ? apic_timer_interrupt+0xf/0x20 [17908.072754] [17908.072759] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17908.072766] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17908.072770] do_raw_spin_lock+0x177/0x190 [17908.072776] _raw_spin_lock+0x21/0x30 [17908.072783] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17908.073057] obd_mod_free+0x46/0x140 [ptlrpc] [17908.092065] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17908.092082] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17908.092090] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17908.102001] mdc_commit_open+0x91/0xb0 [mdc] [17908.105550] [17908.105559] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17908.105576] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17908.105580] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17908.105586] RAX: 0000000000006796 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17908.105588] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [17908.105590] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [17908.105592] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [17908.105595] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [17908.105598] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [17908.105602] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17908.105604] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17908.105613] Call Trace: [17908.105619] [17908.105625] ? show_regs.cold.9+0x22/0x2f [17908.105634] ? watchdog_timer_fn.cold.6+0x62/0x10a [17908.105641] ? __hrtimer_run_queues+0x1c2/0x4d0 [17908.105645] ? watchdog+0x40/0x40 [17908.105649] ? hrtimer_interrupt+0x128/0x2c0 [17908.105653] ? smp_apic_timer_interrupt+0xa6/0x270 [17908.105661] ? apic_timer_interrupt+0xf/0x20 [17908.105666] [17908.105669] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17908.105674] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17908.105682] do_raw_spin_lock+0x177/0x190 [17908.105687] _raw_spin_lock+0x21/0x30 [17908.105692] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17908.108961] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17908.115106] process_one_work+0x2c8/0x700 [17908.119002] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17908.121746] worker_thread+0x296/0x6e0 [17908.125189] after_reply+0xe2c/0x1780 [ptlrpc] [17908.128508] ? rescuer_thread+0x570/0x570 [17908.128526] kthread+0x1d1/0x200 [17908.128531] ? set_kthread_struct+0x70/0x70 [17908.128534] ret_from_fork+0x1f/0x30 [17908.523400] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17908.527076] ? __next_timer_interrupt+0x160/0x160 [17908.530043] ptlrpcd+0xa82/0xe40 [ptlrpc] [17908.533020] ? do_wait_intr+0xf0/0xf0 [17908.535109] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17908.538201] kthread+0x1d1/0x200 [17908.540535] ? set_kthread_struct+0x70/0x70 [17908.543783] ret_from_fork+0x1f/0x30 [17928.052097] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [17928.058000] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17928.085188] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17928.090611] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17928.094648] RIP: 0010:native_safe_halt+0x2e/0x50 [17928.097203] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17928.109342] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17928.121150] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17928.124430] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [17928.128858] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [17928.131986] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [17928.138875] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [17928.146899] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [17928.152846] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17928.157886] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [17928.164658] Call Trace: [17928.166184] [17928.166825] ? show_regs.cold.9+0x22/0x2f [17928.168192] ? watchdog_timer_fn.cold.6+0x62/0x10a [17928.169967] ? __hrtimer_run_queues+0x1c2/0x4d0 [17928.175607] ? watchdog+0x40/0x40 [17928.178650] ? hrtimer_interrupt+0x128/0x2c0 [17928.182283] ? smp_apic_timer_interrupt+0xa6/0x270 [17928.188652] ? apic_timer_interrupt+0xf/0x20 [17928.192627] [17928.193938] ? native_safe_halt+0x2e/0x50 [17928.197780] kvm_wait+0xcd/0xe0 [17928.199371] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17928.202001] do_raw_spin_lock+0x177/0x190 [17928.203020] _raw_spin_lock+0x21/0x30 [17928.204562] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17928.206523] ? do_raw_spin_unlock+0x75/0x190 [17928.209661] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17928.211875] ? do_wait_intr+0xf0/0xf0 [17928.213643] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17928.216830] mdc_reint+0x2d/0x110 [mdc] [17928.227223] mdc_unlink+0x658/0x870 [mdc] [17928.229708] lmv_unlink+0x34d/0xbd0 [lmv] [17928.231267] ll_unlink+0x24d/0x7f0 [lustre] [17928.233040] vfs_unlink+0x202/0x330 [17928.234094] do_unlinkat+0x2fa/0x460 [17928.235446] __x64_sys_unlink+0x37/0x50 [17928.237748] do_syscall_64+0xc1/0x440 [17928.239981] entry_SYSCALL_64_after_hwframe+0x49/0xae [17928.242411] RIP: 0033:0x7f151bdee10b [17928.245672] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [17928.256617] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [17928.259990] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [17928.265451] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [17928.267407] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17928.278262] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17928.281415] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17933.281554] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16002s! [17933.286034] Showing busy workqueues and worker pools: [17933.288013] workqueue events: flags=0x0 [17933.289054] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17933.291768] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [17933.294383] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17933.296812] pending: kfree_rcu_monitor [17933.297767] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17933.299953] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [17933.302722] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17933.305454] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [17933.309283] workqueue events_long: flags=0x0 [17933.311129] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17933.315678] pending: lu_objects_flush [obdclass] [17933.318359] workqueue events_unbound: flags=0x2 [17933.320083] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [17933.322163] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17933.325866] workqueue events_power_efficient: flags=0x80 [17933.327421] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17933.329503] pending: do_cache_clean, neigh_periodic_work, check_lifetime [17933.331575] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17933.334933] pending: neigh_periodic_work [17933.337315] workqueue mm_percpu_wq: flags=0x8 [17933.339527] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17933.342916] pending: vmstat_update, lru_add_drain_per_cpu [17933.345556] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17933.348584] pending: vmstat_update [17933.350323] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17933.353546] pending: lru_add_drain_per_cpu [17933.355660] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [17933.358626] pending: lru_add_drain_per_cpu BAR(48) [17933.360760] workqueue writeback: flags=0x4a [17933.362922] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [17933.366311] pending: wb_workfn [17933.367613] workqueue xprtiod: flags=0xa [17933.369606] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17933.372301] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [17933.375646] workqueue ipv6_addrconf: flags=0x40008 [17933.378428] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17933.381304] pending: addrconf_verify_work [17933.383205] workqueue knbd0-recv: flags=0x1a [17933.384948] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17933.388591] in-flight: 658:recv_work [17933.390783] workqueue ptlrpc_pinger: flags=0x2 [17933.393233] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17933.396555] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [17933.399254] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16002s workers=4 idle: 32984 37 8 [17933.402959] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17918s workers=2 idle: 661 [17936.055340] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [17936.060531] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [17936.063975] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [17936.063987] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17936.064061] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17936.064066] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17936.064072] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17936.064369] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17936.064380] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17936.064384] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17936.064389] RAX: 0000000000001185 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17936.064392] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [17936.064394] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [17936.064397] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [17936.064400] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [17936.064403] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [17936.064406] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17936.064409] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17936.064432] Call Trace: [17936.064438] [17936.064451] ? show_regs.cold.9+0x22/0x2f [17936.065781] ? watchdog_timer_fn.cold.6+0x62/0x10a [17936.065798] ? __hrtimer_run_queues+0x1c2/0x4d0 [17936.065803] ? watchdog+0x40/0x40 [17936.065808] ? hrtimer_interrupt+0x128/0x2c0 [17936.065812] ? smp_apic_timer_interrupt+0xa6/0x270 [17936.065827] ? apic_timer_interrupt+0xf/0x20 [17936.065832] [17936.065835] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17936.065841] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17936.065848] do_raw_spin_lock+0x177/0x190 [17936.065853] _raw_spin_lock+0x21/0x30 [17936.065859] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17936.066119] process_one_work+0x2c8/0x700 [17936.066127] worker_thread+0x296/0x6e0 [17936.066132] ? rescuer_thread+0x570/0x570 [17936.066136] kthread+0x1d1/0x200 [17936.066141] ? set_kthread_struct+0x70/0x70 [17936.066144] ret_from_fork+0x1f/0x30 [17936.066269] Modules linked in: lustre(O) [17936.076392] Modules linked in: [17936.081591] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17936.081801] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17936.081996] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17936.081999] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17936.082014] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17936.082018] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17936.082023] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [17936.082025] RDX: 0000000000007281 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [17936.082027] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [17936.082030] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [17936.082032] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17936.082035] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [17936.082039] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17936.082042] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17936.082051] Call Trace: [17936.082057] [17936.082127] ? show_regs.cold.9+0x22/0x2f [17936.082137] ? watchdog_timer_fn.cold.6+0x62/0x10a [17936.082145] ? __hrtimer_run_queues+0x1c2/0x4d0 [17936.082149] ? watchdog+0x40/0x40 [17936.082154] ? hrtimer_interrupt+0x128/0x2c0 [17936.082158] ? smp_apic_timer_interrupt+0xa6/0x270 [17936.082166] ? apic_timer_interrupt+0xf/0x20 [17936.082171] [17936.082174] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17936.082179] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17936.082184] do_raw_spin_lock+0x177/0x190 [17936.082188] _raw_spin_lock+0x21/0x30 [17936.082208] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17936.082500] obd_mod_free+0x46/0x140 [ptlrpc] [17936.083135] mdc_commit_open+0x91/0xb0 [mdc] [17936.083172] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17936.083388] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17936.083739] after_reply+0xe2c/0x1780 [ptlrpc] [17936.084214] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17936.129028] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17936.129107] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17936.129115] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17936.129118] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17936.129133] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17936.129137] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17936.129142] RAX: 0000000000004d8a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17936.129144] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [17936.129146] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [17936.129148] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [17936.129150] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [17936.129154] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [17936.129156] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17936.129158] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [17936.129168] Call Trace: [17936.129174] [17936.129179] ? show_regs.cold.9+0x22/0x2f [17936.129189] ? watchdog_timer_fn.cold.6+0x62/0x10a [17936.129207] ? __hrtimer_run_queues+0x1c2/0x4d0 [17936.129211] ? watchdog+0x40/0x40 [17936.129216] ? hrtimer_interrupt+0x128/0x2c0 [17936.129219] ? smp_apic_timer_interrupt+0xa6/0x270 [17936.129301] ? apic_timer_interrupt+0xf/0x20 [17936.129312] [17936.129315] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17936.129322] do_raw_spin_lock+0x177/0x190 [17936.129327] _raw_spin_lock+0x21/0x30 [17936.129331] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17936.130337] ? do_raw_spin_unlock+0x75/0x190 [17936.130344] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17936.130561] ? do_wait_intr+0xf0/0xf0 [17936.130569] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17936.130789] mdc_close+0x53b/0xd70 [mdc] [17936.130830] lmv_close+0x18d/0x490 [lmv] [17936.130860] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [17936.131025] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17936.131145] ll_md_real_close+0xa5/0x2a0 [lustre] [17936.131243] ll_md_close+0x274/0x800 [lustre] [17936.131335] ll_file_release+0x128/0xc50 [lustre] [17936.131427] __fput+0x10d/0x440 [17936.131436] ____fput+0x12/0x20 [17936.131441] task_work_run+0xc6/0x110 [17936.131448] exit_to_usermode_loop+0x1dd/0x1f0 [17936.131455] do_syscall_64+0x3d6/0x440 [17936.131460] entry_SYSCALL_64_after_hwframe+0x49/0xae [17936.131467] RIP: 0033:0x7f151bee3cd8 [17936.131476] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17936.131479] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17936.131484] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [17936.131487] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [17936.131489] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17936.131491] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17936.131494] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17937.025428] ? __next_timer_interrupt+0x160/0x160 [17937.027522] ptlrpcd+0xa82/0xe40 [ptlrpc] [17937.029791] ? do_wait_intr+0xf0/0xf0 [17937.032622] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17937.036561] kthread+0x1d1/0x200 [17937.038133] ? set_kthread_struct+0x70/0x70 [17937.041382] ret_from_fork+0x1f/0x30 [17960.052028] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [17960.058815] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17960.081896] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17960.087991] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17960.091981] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17960.094448] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17960.101258] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17960.104568] RAX: 0000000000006ce7 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17960.107432] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [17960.110004] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [17960.113527] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [17960.116295] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [17960.119677] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [17960.123675] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17960.126502] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [17960.128767] Call Trace: [17960.129409] [17960.129922] ? show_regs.cold.9+0x22/0x2f [17960.131976] ? watchdog_timer_fn.cold.6+0x62/0x10a [17960.135040] ? __hrtimer_run_queues+0x1c2/0x4d0 [17960.138012] ? watchdog+0x40/0x40 [17960.139904] ? hrtimer_interrupt+0x128/0x2c0 [17960.141950] ? smp_apic_timer_interrupt+0xa6/0x270 [17960.144365] ? apic_timer_interrupt+0xf/0x20 [17960.146416] [17960.147106] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17960.148813] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17960.153320] do_raw_spin_lock+0x177/0x190 [17960.155245] _raw_spin_lock+0x21/0x30 [17960.156603] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17960.159270] ? do_raw_spin_unlock+0x75/0x190 [17960.161666] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17960.163747] ? do_wait_intr+0xf0/0xf0 [17960.165404] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17960.168019] mdc_reint+0x2d/0x110 [mdc] [17960.169467] mdc_unlink+0x658/0x870 [mdc] [17960.171207] lmv_unlink+0x34d/0xbd0 [lmv] [17960.172773] ll_unlink+0x24d/0x7f0 [lustre] [17960.174555] vfs_unlink+0x202/0x330 [17960.175848] do_unlinkat+0x2fa/0x460 [17960.177175] __x64_sys_unlink+0x37/0x50 [17960.178789] do_syscall_64+0xc1/0x440 [17960.180153] entry_SYSCALL_64_after_hwframe+0x49/0xae [17960.182125] RIP: 0033:0x7f151bdee10b [17960.183612] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [17960.190793] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [17960.192896] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [17960.196388] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [17960.199251] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17960.202308] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17960.205347] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17963.490081] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16032s! [17963.492771] Showing busy workqueues and worker pools: [17963.494779] workqueue events: flags=0x0 [17963.496414] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17963.499091] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [17963.502523] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17963.504724] pending: kfree_rcu_monitor [17963.506418] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17963.508786] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [17963.512516] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17963.514767] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [17963.517749] workqueue events_long: flags=0x0 [17963.519396] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17963.521655] pending: lu_objects_flush [obdclass] [17963.523718] workqueue events_unbound: flags=0x2 [17963.525299] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [17963.527523] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17963.531637] workqueue events_power_efficient: flags=0x80 [17963.533957] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17963.536697] pending: do_cache_clean, neigh_periodic_work, check_lifetime [17963.540153] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17963.543166] pending: neigh_periodic_work [17963.545107] workqueue mm_percpu_wq: flags=0x8 [17963.546724] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17963.549669] pending: vmstat_update, lru_add_drain_per_cpu [17963.552394] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17963.555161] pending: vmstat_update [17963.556487] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17963.558947] pending: lru_add_drain_per_cpu [17963.560491] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [17963.563535] pending: lru_add_drain_per_cpu BAR(48) [17963.565895] workqueue writeback: flags=0x4a [17963.567792] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [17963.570567] pending: wb_workfn [17963.572128] workqueue xprtiod: flags=0xa [17963.573675] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17963.576707] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [17963.580112] workqueue ipv6_addrconf: flags=0x40008 [17963.581646] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17963.584250] pending: addrconf_verify_work [17963.585839] workqueue knbd0-recv: flags=0x1a [17963.587401] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17963.589980] in-flight: 658:recv_work [17963.591527] workqueue ptlrpc_pinger: flags=0x2 [17963.592993] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17963.595301] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [17963.597620] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16033s workers=4 idle: 32984 37 8 [17963.600344] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17948s workers=2 idle: 661 [17964.055024] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [17964.057934] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [17964.059850] Modules linked in: [17964.060546] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [17964.060554] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17964.060992] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17964.060998] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17964.061002] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17964.061016] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17964.061021] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17964.061026] RAX: 000000000000410f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17964.061028] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [17964.061031] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [17964.061033] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [17964.061036] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [17964.061040] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [17964.061043] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17964.061046] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [17964.061228] Call Trace: [17964.061234] [17964.061240] ? show_regs.cold.9+0x22/0x2f [17964.061249] ? watchdog_timer_fn.cold.6+0x62/0x10a [17964.061255] ? __hrtimer_run_queues+0x1c2/0x4d0 [17964.061259] ? watchdog+0x40/0x40 [17964.061263] ? hrtimer_interrupt+0x128/0x2c0 [17964.061268] ? smp_apic_timer_interrupt+0xa6/0x270 [17964.061276] ? apic_timer_interrupt+0xf/0x20 [17964.061281] [17964.061283] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17964.061290] do_raw_spin_lock+0x177/0x190 [17964.061294] _raw_spin_lock+0x21/0x30 [17964.061299] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17964.061762] ? do_raw_spin_unlock+0x75/0x190 [17964.061769] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17964.062113] ? do_wait_intr+0xf0/0xf0 [17964.062123] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17964.062400] mdc_close+0x53b/0xd70 [mdc] [17964.062439] lmv_close+0x18d/0x490 [lmv] [17964.062730] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [17964.062906] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17964.063092] ll_md_real_close+0xa5/0x2a0 [lustre] [17964.063232] ll_md_close+0x274/0x800 [lustre] [17964.063315] ll_file_release+0x128/0xc50 [lustre] [17964.063398] __fput+0x10d/0x440 [17964.063407] ____fput+0x12/0x20 [17964.063412] task_work_run+0xc6/0x110 [17964.063419] exit_to_usermode_loop+0x1dd/0x1f0 [17964.063426] do_syscall_64+0x3d6/0x440 [17964.063431] entry_SYSCALL_64_after_hwframe+0x49/0xae [17964.063438] RIP: 0033:0x7f151bee3cd8 [17964.063447] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17964.063451] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17964.063457] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [17964.063460] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [17964.063462] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17964.063464] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17964.063538] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17964.067976] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17964.068055] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17964.068063] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17964.068069] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17964.076127] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17964.076318] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17964.076326] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17964.076329] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17964.076343] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17964.076348] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17964.076352] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [17964.076355] RDX: 00000000000000cd RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [17964.076357] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [17964.076360] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [17964.076362] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17964.076366] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [17964.076369] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17964.076372] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17964.076380] Call Trace: [17964.076386] [17964.076391] ? show_regs.cold.9+0x22/0x2f [17964.076400] ? watchdog_timer_fn.cold.6+0x62/0x10a [17964.076408] ? __hrtimer_run_queues+0x1c2/0x4d0 [17964.076413] ? watchdog+0x40/0x40 [17964.076417] ? hrtimer_interrupt+0x128/0x2c0 [17964.076421] ? smp_apic_timer_interrupt+0xa6/0x270 [17964.076429] ? apic_timer_interrupt+0xf/0x20 [17964.076434] [17964.076437] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17964.076443] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17964.076448] do_raw_spin_lock+0x177/0x190 [17964.076452] _raw_spin_lock+0x21/0x30 [17964.076459] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17964.076907] obd_mod_free+0x46/0x140 [ptlrpc] [17964.104017] [17964.104026] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17964.104043] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17964.104047] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17964.104052] RAX: 000000000000461d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17964.104055] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [17964.104057] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [17964.104061] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [17964.114209] mdc_commit_open+0x91/0xb0 [mdc] [17964.118323] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [17964.118342] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [17964.118345] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17964.118348] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17964.118367] Call Trace: [17964.118377] [17964.125206] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17964.136532] ? show_regs.cold.9+0x22/0x2f [17964.141669] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17964.151861] ? watchdog_timer_fn.cold.6+0x62/0x10a [17964.156104] after_reply+0xe2c/0x1780 [ptlrpc] [17964.159256] ? __hrtimer_run_queues+0x1c2/0x4d0 [17964.167055] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17964.167345] ? __next_timer_interrupt+0x160/0x160 [17964.167358] ptlrpcd+0xa82/0xe40 [ptlrpc] [17964.167766] ? do_wait_intr+0xf0/0xf0 [17964.167878] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17964.168245] kthread+0x1d1/0x200 [17964.168251] ? set_kthread_struct+0x70/0x70 [17964.168255] ret_from_fork+0x1f/0x30 [17964.632111] ? watchdog+0x40/0x40 [17964.635098] ? hrtimer_interrupt+0x128/0x2c0 [17964.637965] ? smp_apic_timer_interrupt+0xa6/0x270 [17964.642074] ? apic_timer_interrupt+0xf/0x20 [17964.645263] [17964.646583] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17964.649740] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17964.652512] do_raw_spin_lock+0x177/0x190 [17964.654152] _raw_spin_lock+0x21/0x30 [17964.656206] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17964.662318] process_one_work+0x2c8/0x700 [17964.665109] worker_thread+0x296/0x6e0 [17964.668098] ? rescuer_thread+0x570/0x570 [17964.670303] kthread+0x1d1/0x200 [17964.672304] ? set_kthread_struct+0x70/0x70 [17964.674109] ret_from_fork+0x1f/0x30 [17988.052025] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [17988.058217] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17988.086459] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17988.092101] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17988.096534] RIP: 0010:native_safe_halt+0x2e/0x50 [17988.099899] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17988.109565] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17988.113917] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17988.118136] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [17988.121609] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [17988.125343] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [17988.128855] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [17988.131661] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [17988.136829] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17988.139785] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [17988.146441] Call Trace: [17988.152492] [17988.153217] ? show_regs.cold.9+0x22/0x2f [17988.166098] ? watchdog_timer_fn.cold.6+0x62/0x10a [17988.170051] ? __hrtimer_run_queues+0x1c2/0x4d0 [17988.175129] ? watchdog+0x40/0x40 [17988.190953] ? hrtimer_interrupt+0x128/0x2c0 [17988.197256] ? smp_apic_timer_interrupt+0xa6/0x270 [17988.203114] ? apic_timer_interrupt+0xf/0x20 [17988.209006] [17988.215488] ? native_safe_halt+0x2e/0x50 [17988.219785] kvm_wait+0xcd/0xe0 [17988.220585] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17988.232578] do_raw_spin_lock+0x177/0x190 [17988.233840] _raw_spin_lock+0x21/0x30 [17988.234802] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17988.248757] ? do_raw_spin_unlock+0x75/0x190 [17988.254173] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17988.262881] ? do_wait_intr+0xf0/0xf0 [17988.265769] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17988.269593] mdc_reint+0x2d/0x110 [mdc] [17988.273206] mdc_unlink+0x658/0x870 [mdc] [17988.276846] lmv_unlink+0x34d/0xbd0 [lmv] [17988.278588] ll_unlink+0x24d/0x7f0 [lustre] [17988.281885] vfs_unlink+0x202/0x330 [17988.283829] do_unlinkat+0x2fa/0x460 [17988.285152] __x64_sys_unlink+0x37/0x50 [17988.287137] do_syscall_64+0xc1/0x440 [17988.289847] entry_SYSCALL_64_after_hwframe+0x49/0xae [17988.292085] RIP: 0033:0x7f151bdee10b [17988.293493] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [17988.306254] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [17988.311447] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [17988.319903] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [17988.328328] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17988.334703] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17988.345975] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17992.055742] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [17992.056396] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [17992.057346] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [17992.057355] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17992.057428] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17992.057434] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17992.057437] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17992.057450] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17992.057454] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17992.057459] RAX: 0000000000004d0f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17992.057461] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [17992.057463] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [17992.057466] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [17992.057468] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [17992.057472] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [17992.057475] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17992.057477] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [17992.057555] Call Trace: [17992.057561] [17992.057566] ? show_regs.cold.9+0x22/0x2f [17992.057576] ? watchdog_timer_fn.cold.6+0x62/0x10a [17992.057582] ? __hrtimer_run_queues+0x1c2/0x4d0 [17992.057586] ? watchdog+0x40/0x40 [17992.057589] ? hrtimer_interrupt+0x128/0x2c0 [17992.057593] ? smp_apic_timer_interrupt+0xa6/0x270 [17992.057601] ? apic_timer_interrupt+0xf/0x20 [17992.057606] [17992.057609] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17992.057615] do_raw_spin_lock+0x177/0x190 [17992.057620] _raw_spin_lock+0x21/0x30 [17992.057624] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17992.057915] ? do_raw_spin_unlock+0x75/0x190 [17992.057922] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17992.058130] ? do_wait_intr+0xf0/0xf0 [17992.058139] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17992.058347] mdc_close+0x53b/0xd70 [mdc] [17992.058386] lmv_close+0x18d/0x490 [lmv] [17992.058410] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [17992.059218] ll_close_inode_openhandle+0x493/0xea0 [lustre] [17992.059334] ll_md_real_close+0xa5/0x2a0 [lustre] [17992.059416] ll_md_close+0x274/0x800 [lustre] [17992.059626] ll_file_release+0x128/0xc50 [lustre] [17992.060069] __fput+0x10d/0x440 [17992.060079] ____fput+0x12/0x20 [17992.060084] task_work_run+0xc6/0x110 [17992.060091] exit_to_usermode_loop+0x1dd/0x1f0 [17992.060099] do_syscall_64+0x3d6/0x440 [17992.060103] entry_SYSCALL_64_after_hwframe+0x49/0xae [17992.060111] RIP: 0033:0x7f151bee3cd8 [17992.060118] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [17992.060122] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [17992.060127] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [17992.060130] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [17992.060132] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [17992.060134] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [17992.060136] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [17992.062329] Modules linked in: [17992.067453] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17992.067543] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17992.067551] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17992.067557] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17992.067890] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17992.067901] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17992.067906] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17992.067917] RAX: 0000000000005c8c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [17992.067920] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [17992.067922] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [17992.067924] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [17992.067926] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [17992.067930] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [17992.067933] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17992.067935] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17992.067945] Call Trace: [17992.067952] [17992.067957] ? show_regs.cold.9+0x22/0x2f [17992.067967] ? watchdog_timer_fn.cold.6+0x62/0x10a [17992.067973] ? __hrtimer_run_queues+0x1c2/0x4d0 [17992.067978] ? watchdog+0x40/0x40 [17992.067982] ? hrtimer_interrupt+0x128/0x2c0 [17992.067986] ? smp_apic_timer_interrupt+0xa6/0x270 [17992.067995] ? apic_timer_interrupt+0xf/0x20 [17992.068000] [17992.068003] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17992.068008] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17992.068013] do_raw_spin_lock+0x177/0x190 [17992.068018] _raw_spin_lock+0x21/0x30 [17992.068023] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17992.068236] process_one_work+0x2c8/0x700 [17992.068242] worker_thread+0x296/0x6e0 [17992.068246] ? rescuer_thread+0x570/0x570 [17992.068250] kthread+0x1d1/0x200 [17992.068254] ? set_kthread_struct+0x70/0x70 [17992.068258] ret_from_fork+0x1f/0x30 [17992.693947] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17992.771179] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17992.780988] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17992.795164] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17992.798980] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17992.813583] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17992.820132] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [17992.824755] RDX: 0000000000000adc RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [17992.829027] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [17992.837827] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [17992.848275] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17992.855202] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [17992.859118] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17992.866100] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [17992.873381] Call Trace: [17992.878002] [17992.879243] ? show_regs.cold.9+0x22/0x2f [17992.884731] ? watchdog_timer_fn.cold.6+0x62/0x10a [17992.886787] ? __hrtimer_run_queues+0x1c2/0x4d0 [17992.890037] ? watchdog+0x40/0x40 [17992.890966] ? hrtimer_interrupt+0x128/0x2c0 [17992.892215] ? smp_apic_timer_interrupt+0xa6/0x270 [17992.894371] ? apic_timer_interrupt+0xf/0x20 [17992.896236] [17992.897420] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17992.899914] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17992.904748] do_raw_spin_lock+0x177/0x190 [17992.905969] _raw_spin_lock+0x21/0x30 [17992.907134] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17992.911503] obd_mod_free+0x46/0x140 [ptlrpc] [17992.916233] mdc_commit_open+0x91/0xb0 [mdc] [17992.918007] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17992.920666] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17992.924871] after_reply+0xe2c/0x1780 [ptlrpc] [17992.928167] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17992.930950] ? __next_timer_interrupt+0x160/0x160 [17992.937263] ptlrpcd+0xa82/0xe40 [ptlrpc] [17992.940827] ? do_wait_intr+0xf0/0xf0 [17992.943299] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17992.946712] kthread+0x1d1/0x200 [17992.947972] ? set_kthread_struct+0x70/0x70 [17992.949263] ret_from_fork+0x1f/0x30 [17993.699824] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16063s! [17993.708741] Showing busy workqueues and worker pools: [17993.712134] workqueue events: flags=0x0 [17993.720342] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17993.728455] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [17993.747516] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17993.749908] pending: kfree_rcu_monitor [17993.753112] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17993.757942] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [17993.768226] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17993.777959] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [17993.794716] workqueue events_long: flags=0x0 [17993.799626] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17993.807373] pending: lu_objects_flush [obdclass] [17993.811288] workqueue events_unbound: flags=0x2 [17993.817904] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [17993.824228] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [17993.832437] workqueue events_power_efficient: flags=0x80 [17993.837097] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17993.843483] pending: do_cache_clean, neigh_periodic_work, check_lifetime [17993.847007] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17993.856734] pending: neigh_periodic_work [17993.868294] workqueue mm_percpu_wq: flags=0x8 [17993.869922] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17993.881460] pending: vmstat_update, lru_add_drain_per_cpu [17993.890809] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17993.902853] pending: vmstat_update [17993.907025] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17993.916169] pending: lru_add_drain_per_cpu [17993.917591] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [17993.924619] pending: lru_add_drain_per_cpu BAR(48) [17993.935530] workqueue writeback: flags=0x4a [17993.939907] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [17993.944044] pending: wb_workfn [17993.945265] workqueue xprtiod: flags=0xa [17993.957435] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [17993.969895] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [17993.979873] workqueue ipv6_addrconf: flags=0x40008 [17993.987665] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17993.999413] pending: addrconf_verify_work [17994.006501] workqueue knbd0-recv: flags=0x1a [17994.021801] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17994.024172] in-flight: 658:recv_work [17994.035629] workqueue ptlrpc_pinger: flags=0x2 [17994.044475] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17994.048725] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [17994.056182] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16063s workers=4 idle: 32984 37 8 [17994.067135] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17978s workers=2 idle: 661 [18004.981045] rcu: INFO: rcu_sched self-detected stall on CPU [18004.984315] rcu: 1-...!: (9661112 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [18004.996073] (t=16080334 jiffies g=249657 q=7430) [18005.002378] rcu: rcu_sched kthread starved for 16075286 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [18005.008971] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [18005.015863] rcu: RCU grace-period kthread stack dump: [18005.021367] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [18005.026535] Call Trace: [18005.027770] __schedule+0x351/0xcb0 [18005.031746] schedule+0xc0/0x180 [18005.032962] schedule_timeout+0xb4/0x190 [18005.034431] ? do_raw_spin_unlock+0x75/0x190 [18005.042385] ? __next_timer_interrupt+0x160/0x160 [18005.045253] ? prepare_to_swait_event+0xf3/0x140 [18005.053580] rcu_gp_kthread+0x8dd/0x15b0 [18005.056115] ? force_qs_rnp+0x2c0/0x2c0 [18005.064275] kthread+0x1d1/0x200 [18005.068302] ? set_kthread_struct+0x70/0x70 [18005.071342] ret_from_fork+0x1f/0x30 [18005.075902] rcu: Stack dump where RCU GP kthread last ran: [18005.082891] Sending NMI from CPU 1 to CPUs 3: [18005.087924] NMI backtrace for cpu 3 [18005.087931] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18005.087933] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18005.087935] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18005.087937] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18005.087939] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 [18005.087942] RAX: 0000000000007ab2 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18005.087943] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [18005.087944] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [18005.087946] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [18005.087953] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [18005.087955] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [18005.087956] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18005.087957] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [18005.087959] Call Trace: [18005.087959] [18005.087960] ? show_regs.cold.9+0x22/0x2f [18005.087961] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [18005.087962] ? nmi_cpu_backtrace_handler+0x15/0x40 [18005.087963] ? nmi_handle+0xbd/0x210 [18005.087965] ? default_do_nmi+0x4b/0x1b0 [18005.087966] ? do_nmi+0x280/0x380 [18005.087967] ? end_repeat_nmi+0x16/0x1f [18005.087968] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18005.087970] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18005.087971] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18005.087972] [18005.087973] do_raw_spin_lock+0x177/0x190 [18005.087974] _raw_spin_lock+0x21/0x30 [18005.087975] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18005.087977] ? do_raw_spin_unlock+0x75/0x190 [18005.087978] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18005.087979] ? do_wait_intr+0xf0/0xf0 [18005.087981] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18005.087982] mdc_close+0x53b/0xd70 [mdc] [18005.087983] lmv_close+0x18d/0x490 [lmv] [18005.087984] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [18005.087985] ll_close_inode_openhandle+0x493/0xea0 [lustre] [18005.087986] ll_md_real_close+0xa5/0x2a0 [lustre] [18005.087987] ll_md_close+0x274/0x800 [lustre] [18005.087989] ll_file_release+0x128/0xc50 [lustre] [18005.087990] __fput+0x10d/0x440 [18005.087991] ____fput+0x12/0x20 [18005.087992] task_work_run+0xc6/0x110 [18005.087993] exit_to_usermode_loop+0x1dd/0x1f0 [18005.087994] do_syscall_64+0x3d6/0x440 [18005.087995] entry_SYSCALL_64_after_hwframe+0x49/0xae [18005.087996] RIP: 0033:0x7f151bee3cd8 [18005.087997] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [18005.087999] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [18005.088002] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [18005.088003] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [18005.088004] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18005.088006] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18005.088007] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18005.091063] NMI backtrace for cpu 1 [18005.375188] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18005.384019] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18005.389171] Call Trace: [18005.392878] [18005.394297] ? dump_stack+0xbb/0x10e [18005.399240] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [18005.406173] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [18005.410681] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [18005.419962] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [18005.427470] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [18005.431987] ? rcu_dump_cpu_stacks+0x11b/0x171 [18005.438268] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [18005.443900] ? update_process_times+0x6d/0xc0 [18005.448864] ? tick_sched_handle.isra.18+0x27/0x90 [18005.452976] ? tick_sched_timer+0x53/0xb0 [18005.456862] ? __hrtimer_run_queues+0x1c2/0x4d0 [18005.458663] ? tick_sched_handle.isra.18+0x90/0x90 [18005.462030] ? hrtimer_interrupt+0x128/0x2c0 [18005.465862] ? smp_apic_timer_interrupt+0xa6/0x270 [18005.471682] ? apic_timer_interrupt+0xf/0x20 [18005.475966] [18005.478461] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18005.480282] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18005.486829] ? do_raw_spin_lock+0x177/0x190 [18005.489076] ? _raw_spin_lock+0x21/0x30 [18005.491840] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18005.495268] ? obd_mod_free+0x46/0x140 [ptlrpc] [18005.498111] ? mdc_commit_open+0x91/0xb0 [mdc] [18005.500146] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18005.502962] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18005.506235] ? after_reply+0xe2c/0x1780 [ptlrpc] [18005.508951] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18005.511763] ? __next_timer_interrupt+0x160/0x160 [18005.513981] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [18005.516133] ? do_wait_intr+0xf0/0xf0 [18005.521783] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18005.526008] ? kthread+0x1d1/0x200 [18005.527838] ? set_kthread_struct+0x70/0x70 [18005.529856] ? ret_from_fork+0x1f/0x30 [18020.052212] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [18020.056046] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [18020.058034] Modules linked in: [18020.060980] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18020.061107] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18020.061113] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18020.061123] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18020.061672] RIP: 0010:native_safe_halt+0x2e/0x50 [18020.061683] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18020.061692] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18020.061708] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18020.061714] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [18020.061717] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [18020.061719] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [18020.061722] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [18020.061725] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [18020.061732] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18020.061734] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18020.061750] Call Trace: [18020.061755] [18020.061760] ? show_regs.cold.9+0x22/0x2f [18020.061770] ? watchdog_timer_fn.cold.6+0x62/0x10a [18020.061783] ? __hrtimer_run_queues+0x1c2/0x4d0 [18020.061788] ? watchdog+0x40/0x40 [18020.061792] ? hrtimer_interrupt+0x128/0x2c0 [18020.061799] ? smp_apic_timer_interrupt+0xa6/0x270 [18020.061805] ? apic_timer_interrupt+0xf/0x20 [18020.061809] [18020.061812] ? native_safe_halt+0x2e/0x50 [18020.061816] kvm_wait+0xcd/0xe0 [18020.061822] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18020.061830] do_raw_spin_lock+0x177/0x190 [18020.061834] _raw_spin_lock+0x21/0x30 [18020.061838] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18020.062068] process_one_work+0x2c8/0x700 [18020.062075] worker_thread+0x296/0x6e0 [18020.062079] ? rescuer_thread+0x570/0x570 [18020.062082] kthread+0x1d1/0x200 [18020.062090] ? set_kthread_struct+0x70/0x70 [18020.062097] ret_from_fork+0x1f/0x30 [18020.161999] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18020.179516] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18020.183825] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18020.187732] RIP: 0010:native_safe_halt+0x2e/0x50 [18020.189913] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18020.197819] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18020.200864] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18020.204135] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [18020.207018] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [18020.210127] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [18020.214104] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [18020.216921] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [18020.220194] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18020.222537] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [18020.225417] Call Trace: [18020.226241] [18020.227205] ? show_regs.cold.9+0x22/0x2f [18020.228896] ? watchdog_timer_fn.cold.6+0x62/0x10a [18020.230923] ? __hrtimer_run_queues+0x1c2/0x4d0 [18020.233173] ? watchdog+0x40/0x40 [18020.234434] ? hrtimer_interrupt+0x128/0x2c0 [18020.236483] ? smp_apic_timer_interrupt+0xa6/0x270 [18020.240740] ? apic_timer_interrupt+0xf/0x20 [18020.242904] [18020.243729] ? native_safe_halt+0x2e/0x50 [18020.245300] kvm_wait+0xcd/0xe0 [18020.246781] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18020.248965] do_raw_spin_lock+0x177/0x190 [18020.250835] _raw_spin_lock+0x21/0x30 [18020.252411] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18020.255021] ? do_raw_spin_unlock+0x75/0x190 [18020.256708] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18020.258790] ? do_wait_intr+0xf0/0xf0 [18020.260176] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18020.262445] mdc_reint+0x2d/0x110 [mdc] [18020.264133] mdc_unlink+0x658/0x870 [mdc] [18020.265649] lmv_unlink+0x34d/0xbd0 [lmv] [18020.267118] ll_unlink+0x24d/0x7f0 [lustre] [18020.269299] vfs_unlink+0x202/0x330 [18020.270817] do_unlinkat+0x2fa/0x460 [18020.272299] __x64_sys_unlink+0x37/0x50 [18020.273939] do_syscall_64+0xc1/0x440 [18020.275551] entry_SYSCALL_64_after_hwframe+0x49/0xae [18020.277361] RIP: 0033:0x7f151bdee10b [18020.279108] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [18020.287849] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [18020.291315] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [18020.294808] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [18020.297895] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18020.300739] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18020.303794] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18024.416240] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16093s! [18024.419987] Showing busy workqueues and worker pools: [18024.422362] workqueue events: flags=0x0 [18024.424113] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18024.426855] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [18024.430686] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18024.433739] pending: kfree_rcu_monitor [18024.435604] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18024.438528] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [18024.442986] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18024.446014] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [18024.449099] workqueue events_long: flags=0x0 [18024.450894] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18024.453786] pending: lu_objects_flush [obdclass] [18024.455946] workqueue events_unbound: flags=0x2 [18024.458075] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [18024.464179] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [18024.471137] workqueue events_power_efficient: flags=0x80 [18024.473740] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18024.476745] pending: do_cache_clean, neigh_periodic_work, check_lifetime [18024.479912] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18024.482712] pending: neigh_periodic_work [18024.484721] workqueue mm_percpu_wq: flags=0x8 [18024.486575] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18024.490549] pending: vmstat_update, lru_add_drain_per_cpu [18024.493086] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18024.495970] pending: vmstat_update [18024.497750] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18024.500410] pending: lru_add_drain_per_cpu [18024.502189] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [18024.504820] pending: lru_add_drain_per_cpu BAR(48) [18024.506545] workqueue writeback: flags=0x4a [18024.509920] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [18024.513942] pending: wb_workfn [18024.515257] workqueue xprtiod: flags=0xa [18024.516439] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [18024.518434] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [18024.521192] workqueue ipv6_addrconf: flags=0x40008 [18024.522929] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18024.525283] pending: addrconf_verify_work [18024.526765] workqueue knbd0-recv: flags=0x1a [18024.528175] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18024.530349] in-flight: 658:recv_work [18024.531629] workqueue ptlrpc_pinger: flags=0x2 [18024.532882] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18024.534819] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [18024.536815] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16093s workers=4 idle: 32984 37 8 [18024.539320] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18009s workers=2 idle: 661 [18032.057556] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [18032.063786] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18032.064587] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [18032.102455] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18032.102467] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18032.102470] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18032.102483] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18032.102488] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18032.102495] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [18032.102498] RDX: 0000000000001033 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [18032.102500] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [18032.102503] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [18032.102506] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18032.102509] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [18032.102512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18032.102514] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18032.102523] Call Trace: [18032.102528] [18032.102534] ? show_regs.cold.9+0x22/0x2f [18032.102544] ? watchdog_timer_fn.cold.6+0x62/0x10a [18032.102551] ? __hrtimer_run_queues+0x1c2/0x4d0 [18032.102555] ? watchdog+0x40/0x40 [18032.102559] ? hrtimer_interrupt+0x128/0x2c0 [18032.102564] ? smp_apic_timer_interrupt+0xa6/0x270 [18032.102573] ? apic_timer_interrupt+0xf/0x20 [18032.102577] [18032.102580] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18032.102585] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18032.102590] do_raw_spin_lock+0x177/0x190 [18032.102595] _raw_spin_lock+0x21/0x30 [18032.102602] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18032.102857] obd_mod_free+0x46/0x140 [ptlrpc] [18032.103003] mdc_commit_open+0x91/0xb0 [mdc] [18032.103031] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18032.103155] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18032.117120] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18032.118502] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18032.118510] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18032.118513] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18032.118527] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18032.118531] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18032.118536] RAX: 0000000000006488 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18032.118538] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [18032.118540] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [18032.118543] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [18032.118546] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [18032.118549] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [18032.118553] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18032.118555] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [18032.118563] Call Trace: [18032.118570] [18032.118576] ? show_regs.cold.9+0x22/0x2f [18032.118587] ? watchdog_timer_fn.cold.6+0x62/0x10a [18032.118594] ? __hrtimer_run_queues+0x1c2/0x4d0 [18032.118598] ? watchdog+0x40/0x40 [18032.118603] ? hrtimer_interrupt+0x128/0x2c0 [18032.118607] ? smp_apic_timer_interrupt+0xa6/0x270 [18032.118617] ? apic_timer_interrupt+0xf/0x20 [18032.118621] [18032.118625] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18032.118631] do_raw_spin_lock+0x177/0x190 [18032.118635] _raw_spin_lock+0x21/0x30 [18032.118640] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18032.125396] after_reply+0xe2c/0x1780 [ptlrpc] [18032.129538] ? do_raw_spin_unlock+0x75/0x190 [18032.129566] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18032.133438] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18032.134087] ? __next_timer_interrupt+0x160/0x160 [18032.134098] ptlrpcd+0xa82/0xe40 [ptlrpc] [18032.134374] ? do_wait_intr+0xf0/0xf0 [18032.134383] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18032.134920] kthread+0x1d1/0x200 [18032.134927] ? set_kthread_struct+0x70/0x70 [18032.134931] ret_from_fork+0x1f/0x30 [18032.477058] ? do_wait_intr+0xf0/0xf0 [18032.478833] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18032.483954] mdc_close+0x53b/0xd70 [mdc] [18032.486044] lmv_close+0x18d/0x490 [lmv] [18032.496891] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [18032.503998] ll_close_inode_openhandle+0x493/0xea0 [lustre] [18032.510597] ll_md_real_close+0xa5/0x2a0 [lustre] [18032.513779] ll_md_close+0x274/0x800 [lustre] [18032.518251] ll_file_release+0x128/0xc50 [lustre] [18032.520942] __fput+0x10d/0x440 [18032.527001] ____fput+0x12/0x20 [18032.528760] task_work_run+0xc6/0x110 [18032.531293] exit_to_usermode_loop+0x1dd/0x1f0 [18032.533288] do_syscall_64+0x3d6/0x440 [18032.534832] entry_SYSCALL_64_after_hwframe+0x49/0xae [18032.538251] RIP: 0033:0x7f151bee3cd8 [18032.539609] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [18032.554502] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [18032.567825] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [18032.572613] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [18032.587590] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18032.591047] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18032.598250] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18048.056030] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [18048.059188] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18048.080611] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18048.086401] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18048.090703] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18048.094260] RIP: 0010:native_safe_halt+0x2e/0x50 [18048.096245] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18048.105184] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18048.108770] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18048.112513] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [18048.116592] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [18048.120101] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [18048.123535] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [18048.127047] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [18048.131382] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18048.134242] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18048.137691] Call Trace: [18048.138713] [18048.139706] ? show_regs.cold.9+0x22/0x2f [18048.141615] ? watchdog_timer_fn.cold.6+0x62/0x10a [18048.143642] ? __hrtimer_run_queues+0x1c2/0x4d0 [18048.146123] ? watchdog+0x40/0x40 [18048.147538] ? hrtimer_interrupt+0x128/0x2c0 [18048.149440] ? smp_apic_timer_interrupt+0xa6/0x270 [18048.151531] ? apic_timer_interrupt+0xf/0x20 [18048.153531] [18048.154396] ? native_safe_halt+0x2e/0x50 [18048.156166] kvm_wait+0xcd/0xe0 [18048.157649] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18048.159972] do_raw_spin_lock+0x177/0x190 [18048.161539] _raw_spin_lock+0x21/0x30 [18048.163726] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18048.166640] process_one_work+0x2c8/0x700 [18048.168804] worker_thread+0x296/0x6e0 [18048.170777] ? rescuer_thread+0x570/0x570 [18048.172761] kthread+0x1d1/0x200 [18048.174559] ? set_kthread_struct+0x70/0x70 [18048.176576] ret_from_fork+0x1f/0x30 [18052.052044] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [18052.055294] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18052.080300] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18052.088352] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18052.091983] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18052.097525] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18052.108348] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18052.113851] RAX: 0000000000004a0d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18052.116740] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [18052.120111] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [18052.123164] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [18052.126872] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [18052.131875] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [18052.138286] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18052.143419] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [18052.145808] Call Trace: [18052.147567] [18052.149885] ? show_regs.cold.9+0x22/0x2f [18052.155030] ? watchdog_timer_fn.cold.6+0x62/0x10a [18052.157477] ? __hrtimer_run_queues+0x1c2/0x4d0 [18052.159033] ? watchdog+0x40/0x40 [18052.165051] ? hrtimer_interrupt+0x128/0x2c0 [18052.168828] ? smp_apic_timer_interrupt+0xa6/0x270 [18052.172885] ? apic_timer_interrupt+0xf/0x20 [18052.178687] [18052.180305] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18052.188314] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18052.191754] do_raw_spin_lock+0x177/0x190 [18052.194529] _raw_spin_lock+0x21/0x30 [18052.196339] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18052.199194] ? do_raw_spin_unlock+0x75/0x190 [18052.201642] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18052.204182] ? do_wait_intr+0xf0/0xf0 [18052.205769] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18052.208526] mdc_reint+0x2d/0x110 [mdc] [18052.211212] mdc_unlink+0x658/0x870 [mdc] [18052.213533] lmv_unlink+0x34d/0xbd0 [lmv] [18052.215816] ll_unlink+0x24d/0x7f0 [lustre] [18052.218618] vfs_unlink+0x202/0x330 [18052.220250] do_unlinkat+0x2fa/0x460 [18052.221676] __x64_sys_unlink+0x37/0x50 [18052.223459] do_syscall_64+0xc1/0x440 [18052.225583] entry_SYSCALL_64_after_hwframe+0x49/0xae [18052.231829] RIP: 0033:0x7f151bdee10b [18052.234716] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [18052.256907] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [18052.262717] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [18052.265637] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [18052.274598] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18052.284313] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18052.288551] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18054.625312] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16124s! [18054.630152] Showing busy workqueues and worker pools: [18054.634200] workqueue events: flags=0x0 [18054.636988] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18054.643830] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [18054.650731] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18054.654380] pending: kfree_rcu_monitor [18054.656514] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18054.660332] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [18054.664214] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18054.668251] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [18054.671738] workqueue events_long: flags=0x0 [18054.674174] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18054.677560] pending: lu_objects_flush [obdclass] [18054.681352] workqueue events_unbound: flags=0x2 [18054.684880] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [18054.691435] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [18054.702885] workqueue events_power_efficient: flags=0x80 [18054.706229] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18054.710432] pending: do_cache_clean, neigh_periodic_work, check_lifetime [18054.714395] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18054.724578] pending: neigh_periodic_work [18054.731858] workqueue mm_percpu_wq: flags=0x8 [18054.735144] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18054.739251] pending: vmstat_update, lru_add_drain_per_cpu [18054.742646] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18054.747305] pending: vmstat_update [18054.749977] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18054.753262] pending: lru_add_drain_per_cpu [18054.755149] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [18054.758188] pending: lru_add_drain_per_cpu BAR(48) [18054.760581] workqueue writeback: flags=0x4a [18054.762416] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [18054.765132] pending: wb_workfn [18054.766540] workqueue xprtiod: flags=0xa [18054.767632] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [18054.772889] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [18054.779790] workqueue ipv6_addrconf: flags=0x40008 [18054.781535] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18054.786225] pending: addrconf_verify_work [18054.787535] workqueue knbd0-recv: flags=0x1a [18054.789578] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18054.791811] in-flight: 658:recv_work [18054.793177] workqueue ptlrpc_pinger: flags=0x2 [18054.794697] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18054.797243] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [18054.801883] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16124s workers=4 idle: 32984 37 8 [18054.805421] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18039s workers=2 idle: 661 [18060.055551] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [18060.058286] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [18060.059618] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18060.059721] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18060.059732] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18060.059735] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18060.059753] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18060.059757] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18060.059762] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [18060.059767] RDX: 00000000000057fd RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [18060.059770] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [18060.059772] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [18060.059774] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18060.059778] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [18060.059784] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18060.059787] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18060.059797] Call Trace: [18060.059807] [18060.059813] ? show_regs.cold.9+0x22/0x2f [18060.059830] ? watchdog_timer_fn.cold.6+0x62/0x10a [18060.059836] ? __hrtimer_run_queues+0x1c2/0x4d0 [18060.059844] ? watchdog+0x40/0x40 [18060.059848] ? hrtimer_interrupt+0x128/0x2c0 [18060.059852] ? smp_apic_timer_interrupt+0xa6/0x270 [18060.059863] ? apic_timer_interrupt+0xf/0x20 [18060.059868] [18060.059943] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18060.059950] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18060.059959] do_raw_spin_lock+0x177/0x190 [18060.059963] _raw_spin_lock+0x21/0x30 [18060.059973] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18060.062109] obd_mod_free+0x46/0x140 [ptlrpc] [18060.062350] mdc_commit_open+0x91/0xb0 [mdc] [18060.062408] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18060.068241] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl [18060.096274] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18060.096612] after_reply+0xe2c/0x1780 [ptlrpc] [18060.103779] i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18060.103827] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18060.103836] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18060.103839] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18060.103853] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18060.103857] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18060.103862] RAX: 0000000000001111 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18060.103864] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [18060.103866] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [18060.103869] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [18060.103871] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [18060.103874] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [18060.103883] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18060.103886] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [18060.103894] Call Trace: [18060.103974] [18060.103981] ? show_regs.cold.9+0x22/0x2f [18060.104004] ? watchdog_timer_fn.cold.6+0x62/0x10a [18060.104012] ? __hrtimer_run_queues+0x1c2/0x4d0 [18060.104017] ? watchdog+0x40/0x40 [18060.104023] ? hrtimer_interrupt+0x128/0x2c0 [18060.104028] ? smp_apic_timer_interrupt+0xa6/0x270 [18060.104110] ? apic_timer_interrupt+0xf/0x20 [18060.104116] [18060.104119] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18060.104132] do_raw_spin_lock+0x177/0x190 [18060.104142] _raw_spin_lock+0x21/0x30 [18060.104147] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18060.104621] ? do_raw_spin_unlock+0x75/0x190 [18060.104628] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18060.106151] ? do_wait_intr+0xf0/0xf0 [18060.106176] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18060.106523] mdc_close+0x53b/0xd70 [mdc] [18060.106577] lmv_close+0x18d/0x490 [lmv] [18060.106615] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [18060.110540] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18060.114812] ll_close_inode_openhandle+0x493/0xea0 [lustre] [18060.122099] ? __next_timer_interrupt+0x160/0x160 [18060.127585] ll_md_real_close+0xa5/0x2a0 [lustre] [18060.127708] ll_md_close+0x274/0x800 [lustre] [18060.128057] ll_file_release+0x128/0xc50 [lustre] [18060.128137] __fput+0x10d/0x440 [18060.128146] ____fput+0x12/0x20 [18060.128151] task_work_run+0xc6/0x110 [18060.128157] exit_to_usermode_loop+0x1dd/0x1f0 [18060.128164] do_syscall_64+0x3d6/0x440 [18060.128168] entry_SYSCALL_64_after_hwframe+0x49/0xae [18060.128175] RIP: 0033:0x7f151bee3cd8 [18060.128183] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [18060.128186] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [18060.128192] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [18060.128194] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [18060.128196] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18060.128199] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18060.128201] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18060.301724] ptlrpcd+0xa82/0xe40 [ptlrpc] [18060.303809] ? do_wait_intr+0xf0/0xf0 [18060.304812] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18060.306963] kthread+0x1d1/0x200 [18060.307982] ? set_kthread_struct+0x70/0x70 [18060.309422] ret_from_fork+0x1f/0x30 [18076.057255] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [18076.058865] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18076.076922] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18076.080888] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18076.087534] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18076.090968] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18076.092880] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18076.100693] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18076.103158] RAX: 0000000000007d82 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18076.105363] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [18076.107511] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [18076.109401] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [18076.114536] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [18076.117470] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [18076.119710] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18076.125014] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18076.129637] Call Trace: [18076.131376] [18076.132627] ? show_regs.cold.9+0x22/0x2f [18076.136572] ? watchdog_timer_fn.cold.6+0x62/0x10a [18076.138582] ? __hrtimer_run_queues+0x1c2/0x4d0 [18076.141012] ? watchdog+0x40/0x40 [18076.143001] ? hrtimer_interrupt+0x128/0x2c0 [18076.147574] ? smp_apic_timer_interrupt+0xa6/0x270 [18076.153301] ? apic_timer_interrupt+0xf/0x20 [18076.157060] [18076.159065] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18076.163683] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18076.168078] do_raw_spin_lock+0x177/0x190 [18076.171379] _raw_spin_lock+0x21/0x30 [18076.176049] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18076.183012] process_one_work+0x2c8/0x700 [18076.187622] worker_thread+0x296/0x6e0 [18076.191566] ? rescuer_thread+0x570/0x570 [18076.193930] kthread+0x1d1/0x200 [18076.197227] ? set_kthread_struct+0x70/0x70 [18076.198666] ret_from_fork+0x1f/0x30 [18080.053020] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [18080.060681] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18080.082502] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18080.087606] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18080.090694] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18080.094764] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18080.104989] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18080.108250] RAX: 0000000000001aed RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18080.111234] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [18080.114302] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [18080.117632] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [18080.120809] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [18080.124063] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [18080.127472] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18080.129466] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [18080.133394] Call Trace: [18080.134604] [18080.135944] ? show_regs.cold.9+0x22/0x2f [18080.137447] ? watchdog_timer_fn.cold.6+0x62/0x10a [18080.139801] ? __hrtimer_run_queues+0x1c2/0x4d0 [18080.141452] ? watchdog+0x40/0x40 [18080.147248] ? hrtimer_interrupt+0x128/0x2c0 [18080.149471] ? smp_apic_timer_interrupt+0xa6/0x270 [18080.151764] ? apic_timer_interrupt+0xf/0x20 [18080.153338] [18080.154405] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18080.156884] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18080.161857] do_raw_spin_lock+0x177/0x190 [18080.164568] _raw_spin_lock+0x21/0x30 [18080.166846] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18080.169472] ? do_raw_spin_unlock+0x75/0x190 [18080.174477] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18080.176912] ? do_wait_intr+0xf0/0xf0 [18080.178544] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18080.182151] mdc_reint+0x2d/0x110 [mdc] [18080.183550] mdc_unlink+0x658/0x870 [mdc] [18080.184900] lmv_unlink+0x34d/0xbd0 [lmv] [18080.186722] ll_unlink+0x24d/0x7f0 [lustre] [18080.188812] vfs_unlink+0x202/0x330 [18080.190519] do_unlinkat+0x2fa/0x460 [18080.196077] __x64_sys_unlink+0x37/0x50 [18080.197344] do_syscall_64+0xc1/0x440 [18080.204774] entry_SYSCALL_64_after_hwframe+0x49/0xae [18080.212729] RIP: 0033:0x7f151bdee10b [18080.217545] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [18080.235195] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [18080.237556] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [18080.241381] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [18080.244805] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18080.247859] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18080.252326] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18084.833160] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16154s! [18084.835880] Showing busy workqueues and worker pools: [18084.838060] workqueue events: flags=0x0 [18084.839827] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18084.842352] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [18084.846915] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18084.849421] pending: kfree_rcu_monitor [18084.851480] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18084.854027] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [18084.857902] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18084.860149] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [18084.862980] workqueue events_long: flags=0x0 [18084.864453] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18084.867820] pending: lu_objects_flush [obdclass] [18084.870011] workqueue events_unbound: flags=0x2 [18084.872557] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [18084.874778] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [18084.878400] workqueue events_power_efficient: flags=0x80 [18084.880885] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18084.883120] pending: do_cache_clean, neigh_periodic_work, check_lifetime [18084.885276] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18084.888284] pending: neigh_periodic_work [18084.889594] workqueue mm_percpu_wq: flags=0x8 [18084.891127] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18084.893325] pending: vmstat_update, lru_add_drain_per_cpu [18084.895194] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18084.897928] pending: vmstat_update [18084.899040] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18084.901767] pending: lru_add_drain_per_cpu [18084.903473] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [18084.906144] pending: lru_add_drain_per_cpu BAR(48) [18084.908290] workqueue writeback: flags=0x4a [18084.910013] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [18084.911959] pending: wb_workfn [18084.913305] workqueue xprtiod: flags=0xa [18084.914608] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [18084.916731] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [18084.919518] workqueue ipv6_addrconf: flags=0x40008 [18084.921078] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18084.923272] pending: addrconf_verify_work [18084.924846] workqueue knbd0-recv: flags=0x1a [18084.926153] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18084.928269] in-flight: 658:recv_work [18084.929594] workqueue ptlrpc_pinger: flags=0x2 [18084.931089] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18084.933172] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [18084.935374] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16154s workers=4 idle: 32984 37 8 [18084.938060] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18069s workers=2 idle: 661 [18088.057034] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [18088.058885] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [18088.061088] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18088.061173] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18088.061184] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18088.061187] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18088.061208] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18088.061213] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18088.061217] RAX: 0000000000003b05 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18088.061220] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [18088.061222] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [18088.061227] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [18088.061232] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [18088.061236] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [18088.061352] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18088.061355] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [18088.061364] Call Trace: [18088.061369] [18088.061375] ? show_regs.cold.9+0x22/0x2f [18088.061388] ? watchdog_timer_fn.cold.6+0x62/0x10a [18088.061394] ? __hrtimer_run_queues+0x1c2/0x4d0 [18088.061400] ? watchdog+0x40/0x40 [18088.061405] ? hrtimer_interrupt+0x128/0x2c0 [18088.061410] ? smp_apic_timer_interrupt+0xa6/0x270 [18088.061419] ? apic_timer_interrupt+0xf/0x20 [18088.061423] [18088.061426] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18088.061433] do_raw_spin_lock+0x177/0x190 [18088.061438] _raw_spin_lock+0x21/0x30 [18088.061443] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18088.062053] ? do_raw_spin_unlock+0x75/0x190 [18088.062064] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18088.062678] ? do_wait_intr+0xf0/0xf0 [18088.062687] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18088.062904] mdc_close+0x53b/0xd70 [mdc] [18088.062946] lmv_close+0x18d/0x490 [lmv] [18088.062972] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [18088.063154] ll_close_inode_openhandle+0x493/0xea0 [lustre] [18088.063269] ll_md_real_close+0xa5/0x2a0 [lustre] [18088.063355] ll_md_close+0x274/0x800 [lustre] [18088.063451] ll_file_release+0x128/0xc50 [lustre] [18088.063542] __fput+0x10d/0x440 [18088.063550] ____fput+0x12/0x20 [18088.063554] task_work_run+0xc6/0x110 [18088.063566] exit_to_usermode_loop+0x1dd/0x1f0 [18088.063578] do_syscall_64+0x3d6/0x440 [18088.063582] entry_SYSCALL_64_after_hwframe+0x49/0xae [18088.063590] RIP: 0033:0x7f151bee3cd8 [18088.063598] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [18088.063602] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [18088.063606] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [18088.063609] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [18088.063611] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18088.063613] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18088.063615] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18088.456296] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18088.501499] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18088.509384] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18088.520510] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18088.528333] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18088.542851] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18088.548970] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [18088.552219] RDX: 0000000000007c87 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [18088.561490] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [18088.566873] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [18088.572242] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18088.574682] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [18088.586770] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18088.591877] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18088.595220] Call Trace: [18088.597234] [18088.598538] ? show_regs.cold.9+0x22/0x2f [18088.602296] ? watchdog_timer_fn.cold.6+0x62/0x10a [18088.605005] ? __hrtimer_run_queues+0x1c2/0x4d0 [18088.607783] ? watchdog+0x40/0x40 [18088.611643] ? hrtimer_interrupt+0x128/0x2c0 [18088.616408] ? smp_apic_timer_interrupt+0xa6/0x270 [18088.621561] ? apic_timer_interrupt+0xf/0x20 [18088.626249] [18088.628004] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18088.635430] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18088.644827] do_raw_spin_lock+0x177/0x190 [18088.646390] _raw_spin_lock+0x21/0x30 [18088.657194] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18088.661227] obd_mod_free+0x46/0x140 [ptlrpc] [18088.668096] mdc_commit_open+0x91/0xb0 [mdc] [18088.670319] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18088.674086] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18088.679370] after_reply+0xe2c/0x1780 [ptlrpc] [18088.686058] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18088.690558] ? __next_timer_interrupt+0x160/0x160 [18088.695021] ptlrpcd+0xa82/0xe40 [ptlrpc] [18088.698217] ? do_wait_intr+0xf0/0xf0 [18088.702015] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18088.704405] kthread+0x1d1/0x200 [18088.706891] ? set_kthread_struct+0x70/0x70 [18088.709993] ret_from_fork+0x1f/0x30 [18104.057499] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [18104.073730] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18104.136433] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18104.155083] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18104.161773] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18104.168529] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18104.172399] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18104.192328] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18104.213242] RAX: 0000000000003e8d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18104.225439] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [18104.238203] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [18104.247549] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [18104.256920] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [18104.267654] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [18104.272260] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18104.289315] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18104.297644] Call Trace: [18104.305405] [18104.312528] ? show_regs.cold.9+0x22/0x2f [18104.323877] ? watchdog_timer_fn.cold.6+0x62/0x10a [18104.336046] ? __hrtimer_run_queues+0x1c2/0x4d0 [18104.346852] ? watchdog+0x40/0x40 [18104.350100] ? hrtimer_interrupt+0x128/0x2c0 [18104.356291] ? smp_apic_timer_interrupt+0xa6/0x270 [18104.363880] ? apic_timer_interrupt+0xf/0x20 [18104.372397] [18104.379334] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18104.385541] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18104.396786] do_raw_spin_lock+0x177/0x190 [18104.399829] _raw_spin_lock+0x21/0x30 [18104.408002] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18104.410093] process_one_work+0x2c8/0x700 [18104.425876] worker_thread+0x296/0x6e0 [18104.429336] ? rescuer_thread+0x570/0x570 [18104.436302] kthread+0x1d1/0x200 [18104.438572] ? set_kthread_struct+0x70/0x70 [18104.442318] ret_from_fork+0x1f/0x30 [18112.052336] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [18112.061942] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18112.124974] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18112.140990] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18112.143393] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18112.149547] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18112.183277] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18112.203024] RAX: 0000000000003108 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18112.213284] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [18112.236555] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [18112.246875] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [18112.259883] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [18112.273302] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [18112.280835] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18112.289059] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [18112.304645] Call Trace: [18112.309891] [18112.313853] ? show_regs.cold.9+0x22/0x2f [18112.319145] ? watchdog_timer_fn.cold.6+0x62/0x10a [18112.321759] ? __hrtimer_run_queues+0x1c2/0x4d0 [18112.329047] ? watchdog+0x40/0x40 [18112.331842] ? hrtimer_interrupt+0x128/0x2c0 [18112.333411] ? smp_apic_timer_interrupt+0xa6/0x270 [18112.349626] ? apic_timer_interrupt+0xf/0x20 [18112.354774] [18112.359670] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18112.366642] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18112.370012] do_raw_spin_lock+0x177/0x190 [18112.374214] _raw_spin_lock+0x21/0x30 [18112.375604] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18112.381766] ? do_raw_spin_unlock+0x75/0x190 [18112.386770] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18112.401089] ? do_wait_intr+0xf0/0xf0 [18112.411056] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18112.412929] mdc_reint+0x2d/0x110 [mdc] [18112.424633] mdc_unlink+0x658/0x870 [mdc] [18112.426164] lmv_unlink+0x34d/0xbd0 [lmv] [18112.432511] ll_unlink+0x24d/0x7f0 [lustre] [18112.436124] vfs_unlink+0x202/0x330 [18112.439269] do_unlinkat+0x2fa/0x460 [18112.441717] __x64_sys_unlink+0x37/0x50 [18112.446332] do_syscall_64+0xc1/0x440 [18112.449720] entry_SYSCALL_64_after_hwframe+0x49/0xae [18112.462036] RIP: 0033:0x7f151bdee10b [18112.470547] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [18112.501421] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [18112.523001] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [18112.540987] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [18112.555930] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18112.565733] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18112.572967] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18115.042627] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16184s! [18115.053331] Showing busy workqueues and worker pools: [18115.061288] workqueue events: flags=0x0 [18115.069974] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18115.075348] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [18115.086274] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18115.096473] pending: kfree_rcu_monitor [18115.102044] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18115.109846] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [18115.119205] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18115.124392] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [18115.139529] workqueue events_long: flags=0x0 [18115.144153] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18115.152156] pending: lu_objects_flush [obdclass] [18115.157210] workqueue events_unbound: flags=0x2 [18115.159995] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [18115.163633] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [18115.174951] workqueue events_power_efficient: flags=0x80 [18115.180054] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18115.187124] pending: do_cache_clean, neigh_periodic_work, check_lifetime [18115.199758] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18115.206683] pending: neigh_periodic_work [18115.209823] workqueue mm_percpu_wq: flags=0x8 [18115.213046] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18115.219298] pending: vmstat_update, lru_add_drain_per_cpu [18115.225194] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18115.233556] pending: vmstat_update [18115.237535] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18115.246305] pending: lru_add_drain_per_cpu [18115.252865] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [18115.257388] pending: lru_add_drain_per_cpu BAR(48) [18115.265383] workqueue writeback: flags=0x4a [18115.269654] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [18115.277170] pending: wb_workfn [18115.278931] workqueue xprtiod: flags=0xa [18115.285162] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [18115.289935] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [18115.297438] workqueue ipv6_addrconf: flags=0x40008 [18115.302879] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18115.309096] pending: addrconf_verify_work [18115.312469] workqueue knbd0-recv: flags=0x1a [18115.315014] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18115.321104] in-flight: 658:recv_work [18115.330655] workqueue ptlrpc_pinger: flags=0x2 [18115.335908] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18115.345248] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [18115.351014] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16184s workers=4 idle: 32984 37 8 [18115.358673] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18100s workers=2 idle: 661 [18116.055203] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [18116.057538] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [18116.070460] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) [18116.072435] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18116.072529] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18116.072536] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18116.072538] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18116.072549] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18116.072553] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18116.072558] RAX: 0000000000003eef RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18116.072560] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [18116.072563] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [18116.072564] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [18116.072572] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [18116.072575] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [18116.072578] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18116.072581] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [18116.072589] Call Trace: [18116.072595] [18116.072601] ? show_regs.cold.9+0x22/0x2f [18116.072609] ? watchdog_timer_fn.cold.6+0x62/0x10a [18116.072613] ? __hrtimer_run_queues+0x1c2/0x4d0 [18116.072616] ? watchdog+0x40/0x40 [18116.072619] ? hrtimer_interrupt+0x128/0x2c0 [18116.072622] ? smp_apic_timer_interrupt+0xa6/0x270 [18116.072628] ? apic_timer_interrupt+0xf/0x20 [18116.072631] [18116.072633] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18116.072638] do_raw_spin_lock+0x177/0x190 [18116.072641] _raw_spin_lock+0x21/0x30 [18116.072644] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18116.072867] ? do_raw_spin_unlock+0x75/0x190 [18116.072872] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18116.073078] ? do_wait_intr+0xf0/0xf0 [18116.073085] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18116.073392] mdc_close+0x53b/0xd70 [mdc] [18116.073439] lmv_close+0x18d/0x490 [lmv] [18116.073478] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [18116.073639] ll_close_inode_openhandle+0x493/0xea0 [lustre] [18116.073723] ll_md_real_close+0xa5/0x2a0 [lustre] [18116.073775] ll_md_close+0x274/0x800 [lustre] [18116.073923] ll_file_release+0x128/0xc50 [lustre] [18116.079389] lquota(O) [18116.091598] __fput+0x10d/0x440 [18116.104057] lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18116.104122] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18116.104128] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18116.104131] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18116.104143] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18116.104146] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18116.104150] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [18116.104153] RDX: 0000000000003348 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [18116.104155] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [18116.104157] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [18116.104159] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18116.104162] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [18116.104165] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18116.104167] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18116.104174] Call Trace: [18116.104178] [18116.104183] ? show_regs.cold.9+0x22/0x2f [18116.104191] ? watchdog_timer_fn.cold.6+0x62/0x10a [18116.104197] ? __hrtimer_run_queues+0x1c2/0x4d0 [18116.104201] ? watchdog+0x40/0x40 [18116.104205] ? hrtimer_interrupt+0x128/0x2c0 [18116.104208] ? smp_apic_timer_interrupt+0xa6/0x270 [18116.104216] ? apic_timer_interrupt+0xf/0x20 [18116.104220] [18116.104223] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18116.104227] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18116.104231] do_raw_spin_lock+0x177/0x190 [18116.104235] _raw_spin_lock+0x21/0x30 [18116.104240] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18116.104515] obd_mod_free+0x46/0x140 [ptlrpc] [18116.109774] ____fput+0x12/0x20 [18116.109790] task_work_run+0xc6/0x110 [18116.109797] exit_to_usermode_loop+0x1dd/0x1f0 [18116.109801] do_syscall_64+0x3d6/0x440 [18116.109804] entry_SYSCALL_64_after_hwframe+0x49/0xae [18116.109811] RIP: 0033:0x7f151bee3cd8 [18116.109818] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [18116.109821] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [18116.109825] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [18116.109826] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [18116.109828] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18116.109829] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18116.109831] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18116.271866] mdc_commit_open+0x91/0xb0 [mdc] [18116.273002] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18116.275744] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18116.277566] after_reply+0xe2c/0x1780 [ptlrpc] [18116.279119] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18116.280747] ? __next_timer_interrupt+0x160/0x160 [18116.282427] ptlrpcd+0xa82/0xe40 [ptlrpc] [18116.283712] ? do_wait_intr+0xf0/0xf0 [18116.284748] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18116.286350] kthread+0x1d1/0x200 [18116.287089] ? set_kthread_struct+0x70/0x70 [18116.288251] ret_from_fork+0x1f/0x30 [18132.061572] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [18132.069318] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18132.107720] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18132.115363] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18132.124933] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18132.129146] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18132.136956] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18132.154056] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18132.160972] RAX: 000000000000460c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18132.165277] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [18132.170774] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [18132.173835] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [18132.182077] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [18132.189770] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [18132.196938] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18132.200891] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18132.206851] Call Trace: [18132.208153] [18132.209626] ? show_regs.cold.9+0x22/0x2f [18132.213959] ? watchdog_timer_fn.cold.6+0x62/0x10a [18132.220709] ? __hrtimer_run_queues+0x1c2/0x4d0 [18132.223250] ? watchdog+0x40/0x40 [18132.225062] ? hrtimer_interrupt+0x128/0x2c0 [18132.228553] ? smp_apic_timer_interrupt+0xa6/0x270 [18132.235924] ? apic_timer_interrupt+0xf/0x20 [18132.250729] [18132.251429] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18132.256127] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18132.278037] do_raw_spin_lock+0x177/0x190 [18132.283087] _raw_spin_lock+0x21/0x30 [18132.287612] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18132.296284] process_one_work+0x2c8/0x700 [18132.298778] worker_thread+0x296/0x6e0 [18132.303971] ? rescuer_thread+0x570/0x570 [18132.308010] kthread+0x1d1/0x200 [18132.317987] ? set_kthread_struct+0x70/0x70 [18132.324085] ret_from_fork+0x1f/0x30 [18140.052168] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [18140.058627] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18140.136662] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18140.149169] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18140.161521] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18140.166339] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18140.193712] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18140.199388] RAX: 0000000000005475 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18140.204831] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [18140.211600] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [18140.217377] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [18140.224042] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [18140.228587] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [18140.240990] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18140.245758] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [18140.252481] Call Trace: [18140.253302] [18140.254009] ? show_regs.cold.9+0x22/0x2f [18140.255137] ? watchdog_timer_fn.cold.6+0x62/0x10a [18140.272068] ? __hrtimer_run_queues+0x1c2/0x4d0 [18140.281754] ? watchdog+0x40/0x40 [18140.284511] ? hrtimer_interrupt+0x128/0x2c0 [18140.290932] ? smp_apic_timer_interrupt+0xa6/0x270 [18140.296841] ? apic_timer_interrupt+0xf/0x20 [18140.302808] [18140.304527] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18140.318509] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18140.324838] do_raw_spin_lock+0x177/0x190 [18140.327486] _raw_spin_lock+0x21/0x30 [18140.330900] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18140.339680] ? do_raw_spin_unlock+0x75/0x190 [18140.343628] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18140.351389] ? do_wait_intr+0xf0/0xf0 [18140.360873] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18140.367836] mdc_reint+0x2d/0x110 [mdc] [18140.373837] mdc_unlink+0x658/0x870 [mdc] [18140.378200] lmv_unlink+0x34d/0xbd0 [lmv] [18140.380995] ll_unlink+0x24d/0x7f0 [lustre] [18140.386134] vfs_unlink+0x202/0x330 [18140.391198] do_unlinkat+0x2fa/0x460 [18140.396142] __x64_sys_unlink+0x37/0x50 [18140.402514] do_syscall_64+0xc1/0x440 [18140.406172] entry_SYSCALL_64_after_hwframe+0x49/0xae [18140.411285] RIP: 0033:0x7f151bdee10b [18140.414077] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [18140.424965] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [18140.428864] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [18140.432769] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [18140.436930] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18140.440842] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18140.445941] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18144.055692] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [18144.057028] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [18144.060945] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18144.061036] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18144.061055] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18144.061058] RIP: 0010:native_safe_halt+0x2e/0x50 [18144.061074] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18144.061079] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18144.061083] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [18144.061086] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [18144.061088] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [18144.061090] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [18144.061093] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18144.061095] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [18144.061099] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18144.061101] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18144.061110] Call Trace: [18144.061115] [18144.061121] ? show_regs.cold.9+0x22/0x2f [18144.061131] ? watchdog_timer_fn.cold.6+0x62/0x10a [18144.061137] ? __hrtimer_run_queues+0x1c2/0x4d0 [18144.061142] ? watchdog+0x40/0x40 [18144.061146] ? hrtimer_interrupt+0x128/0x2c0 [18144.061151] ? smp_apic_timer_interrupt+0xa6/0x270 [18144.061157] ? apic_timer_interrupt+0xf/0x20 [18144.061162] [18144.061165] ? native_safe_halt+0x2e/0x50 [18144.061169] kvm_wait+0xcd/0xe0 [18144.061210] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18144.061218] do_raw_spin_lock+0x177/0x190 [18144.061223] _raw_spin_lock+0x21/0x30 [18144.061228] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18144.061505] obd_mod_free+0x46/0x140 [ptlrpc] [18144.066655] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18144.066733] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18144.066741] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18144.066744] RIP: 0010:native_safe_halt+0x2e/0x50 [18144.066756] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18144.066760] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18144.066764] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18144.066766] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [18144.066768] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [18144.066770] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [18144.066772] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [18144.066775] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [18144.066777] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18144.066780] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [18144.066789] Call Trace: [18144.066794] [18144.066799] ? show_regs.cold.9+0x22/0x2f [18144.066809] ? watchdog_timer_fn.cold.6+0x62/0x10a [18144.066816] ? __hrtimer_run_queues+0x1c2/0x4d0 [18144.066821] ? watchdog+0x40/0x40 [18144.066824] ? hrtimer_interrupt+0x128/0x2c0 [18144.066828] ? smp_apic_timer_interrupt+0xa6/0x270 [18144.066834] ? apic_timer_interrupt+0xf/0x20 [18144.066838] [18144.066840] ? native_safe_halt+0x2e/0x50 [18144.066843] kvm_wait+0xcd/0xe0 [18144.066852] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18144.066861] do_raw_spin_lock+0x177/0x190 [18144.066865] _raw_spin_lock+0x21/0x30 [18144.066869] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18144.067166] ? do_raw_spin_unlock+0x75/0x190 [18144.067171] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18144.106827] mdc_commit_open+0x91/0xb0 [mdc] [18144.122714] ? do_wait_intr+0xf0/0xf0 [18144.127091] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18144.129366] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18144.137799] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18144.138222] after_reply+0xe2c/0x1780 [ptlrpc] [18144.138448] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18144.138732] ? __next_timer_interrupt+0x160/0x160 [18144.138744] ptlrpcd+0xa82/0xe40 [ptlrpc] [18144.138968] ? do_wait_intr+0xf0/0xf0 [18144.138977] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18144.139274] kthread+0x1d1/0x200 [18144.139280] ? set_kthread_struct+0x70/0x70 [18144.139284] ret_from_fork+0x1f/0x30 [18144.483343] mdc_close+0x53b/0xd70 [mdc] [18144.485236] lmv_close+0x18d/0x490 [lmv] [18144.490034] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [18144.491972] ll_close_inode_openhandle+0x493/0xea0 [lustre] [18144.508376] ll_md_real_close+0xa5/0x2a0 [lustre] [18144.514597] ll_md_close+0x274/0x800 [lustre] [18144.524305] ll_file_release+0x128/0xc50 [lustre] [18144.526046] __fput+0x10d/0x440 [18144.527147] ____fput+0x12/0x20 [18144.534407] task_work_run+0xc6/0x110 [18144.536599] exit_to_usermode_loop+0x1dd/0x1f0 [18144.540580] do_syscall_64+0x3d6/0x440 [18144.542491] entry_SYSCALL_64_after_hwframe+0x49/0xae [18144.554014] RIP: 0033:0x7f151bee3cd8 [18144.556313] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [18144.573677] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [18144.577787] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [18144.581574] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [18144.586787] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18144.594743] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18144.602909] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18145.764435] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16215s! [18145.770255] Showing busy workqueues and worker pools: [18145.775677] workqueue events: flags=0x0 [18145.777101] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18145.784493] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [18145.793483] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18145.809688] pending: kfree_rcu_monitor [18145.821737] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18145.826810] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [18145.836742] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18145.843858] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [18145.847244] workqueue events_long: flags=0x0 [18145.848920] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18145.851970] pending: lu_objects_flush [obdclass] [18145.854250] workqueue events_unbound: flags=0x2 [18145.855915] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [18145.861340] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [18145.874809] workqueue events_power_efficient: flags=0x80 [18145.880913] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18145.887885] pending: do_cache_clean, neigh_periodic_work, check_lifetime [18145.893361] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18145.901949] pending: neigh_periodic_work [18145.905724] workqueue mm_percpu_wq: flags=0x8 [18145.909780] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18145.919039] pending: vmstat_update, lru_add_drain_per_cpu [18145.923908] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18145.930349] pending: vmstat_update [18145.933709] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18145.943993] pending: lru_add_drain_per_cpu [18145.946876] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [18145.952453] pending: lru_add_drain_per_cpu BAR(48) [18145.955186] workqueue writeback: flags=0x4a [18145.956734] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [18145.959300] pending: wb_workfn [18145.964011] workqueue xprtiod: flags=0xa [18145.967509] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [18145.971346] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [18145.978975] workqueue ipv6_addrconf: flags=0x40008 [18145.981261] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18145.994744] pending: addrconf_verify_work [18145.997617] workqueue knbd0-recv: flags=0x1a [18146.003606] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18146.010934] in-flight: 658:recv_work [18146.012692] workqueue ptlrpc_pinger: flags=0x2 [18146.015194] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18146.018189] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [18146.021261] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16215s workers=4 idle: 32984 37 8 [18146.025738] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18130s workers=2 idle: 661 [18160.056066] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [18160.058548] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18160.108414] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18160.120665] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18160.130747] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18160.134563] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18160.139618] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18160.157730] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18160.166855] RAX: 0000000000006996 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18160.180907] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [18160.186805] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [18160.189734] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [18160.193333] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [18160.199497] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [18160.205284] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18160.210814] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18160.217767] Call Trace: [18160.218753] [18160.219542] ? show_regs.cold.9+0x22/0x2f [18160.221184] ? watchdog_timer_fn.cold.6+0x62/0x10a [18160.223527] ? __hrtimer_run_queues+0x1c2/0x4d0 [18160.230200] ? watchdog+0x40/0x40 [18160.231834] ? hrtimer_interrupt+0x128/0x2c0 [18160.238544] ? smp_apic_timer_interrupt+0xa6/0x270 [18160.240535] ? apic_timer_interrupt+0xf/0x20 [18160.243379] [18160.245422] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18160.247729] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18160.251595] do_raw_spin_lock+0x177/0x190 [18160.253625] _raw_spin_lock+0x21/0x30 [18160.255198] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18160.257375] process_one_work+0x2c8/0x700 [18160.259618] worker_thread+0x296/0x6e0 [18160.261116] ? rescuer_thread+0x570/0x570 [18160.262775] kthread+0x1d1/0x200 [18160.264258] ? set_kthread_struct+0x70/0x70 [18160.266009] ret_from_fork+0x1f/0x30 [18172.052986] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [18172.055049] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [18172.057048] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [18172.057059] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18172.057148] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18172.057153] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18172.057156] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18172.057172] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18172.057177] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18172.057182] RAX: 000000000000741a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18172.057185] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [18172.057193] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [18172.057196] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [18172.057199] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [18172.057202] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [18172.057206] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18172.057209] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [18172.057220] Call Trace: [18172.057226] [18172.057236] ? show_regs.cold.9+0x22/0x2f [18172.057250] ? watchdog_timer_fn.cold.6+0x62/0x10a [18172.057258] ? __hrtimer_run_queues+0x1c2/0x4d0 [18172.057262] ? watchdog+0x40/0x40 [18172.057266] ? hrtimer_interrupt+0x128/0x2c0 [18172.057269] ? smp_apic_timer_interrupt+0xa6/0x270 [18172.057282] ? apic_timer_interrupt+0xf/0x20 [18172.057287] [18172.057290] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18172.057298] do_raw_spin_lock+0x177/0x190 [18172.057325] _raw_spin_lock+0x21/0x30 [18172.057332] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18172.057847] ? do_raw_spin_unlock+0x75/0x190 [18172.057854] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18172.058089] ? do_wait_intr+0xf0/0xf0 [18172.058097] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18172.058307] mdc_close+0x53b/0xd70 [mdc] [18172.058346] lmv_close+0x18d/0x490 [lmv] [18172.058375] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [18172.058556] ll_close_inode_openhandle+0x493/0xea0 [lustre] [18172.058801] ll_md_real_close+0xa5/0x2a0 [lustre] [18172.058881] ll_md_close+0x274/0x800 [lustre] [18172.058971] ll_file_release+0x128/0xc50 [lustre] [18172.059050] __fput+0x10d/0x440 [18172.059058] ____fput+0x12/0x20 [18172.059062] task_work_run+0xc6/0x110 [18172.059070] exit_to_usermode_loop+0x1dd/0x1f0 [18172.059077] do_syscall_64+0x3d6/0x440 [18172.059081] entry_SYSCALL_64_after_hwframe+0x49/0xae [18172.059088] RIP: 0033:0x7f151bee3cd8 [18172.059096] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [18172.059101] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [18172.059106] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [18172.059109] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [18172.059111] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18172.059113] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18172.059115] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18172.062395] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18172.062484] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18172.062492] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18172.062495] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18172.062509] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18172.062513] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [18172.062517] RAX: 000000000000601c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18172.062520] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [18172.062522] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [18172.062525] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [18172.062527] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [18172.062530] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [18172.062534] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18172.062536] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [18172.062544] Call Trace: [18172.062550] [18172.062555] ? show_regs.cold.9+0x22/0x2f [18172.062565] ? watchdog_timer_fn.cold.6+0x62/0x10a [18172.062572] ? __hrtimer_run_queues+0x1c2/0x4d0 [18172.062576] ? watchdog+0x40/0x40 [18172.062581] ? hrtimer_interrupt+0x128/0x2c0 [18172.062584] ? smp_apic_timer_interrupt+0xa6/0x270 [18172.062594] ? apic_timer_interrupt+0xf/0x20 [18172.062599] [18172.062602] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18172.062607] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18172.062612] do_raw_spin_lock+0x177/0x190 [18172.062618] _raw_spin_lock+0x21/0x30 [18172.062624] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18172.063260] ? do_raw_spin_unlock+0x75/0x190 [18172.063267] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18172.074428] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18172.074512] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18172.074521] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18172.074524] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18172.074540] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18172.074544] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18172.074548] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [18172.074550] RDX: 0000000000003ed6 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [18172.074552] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [18172.074554] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [18172.074560] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18172.074563] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [18172.074566] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18172.074568] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18172.074575] Call Trace: [18172.074580] [18172.074586] ? show_regs.cold.9+0x22/0x2f [18172.074595] ? watchdog_timer_fn.cold.6+0x62/0x10a [18172.074600] ? __hrtimer_run_queues+0x1c2/0x4d0 [18172.074609] ? watchdog+0x40/0x40 [18172.074613] ? hrtimer_interrupt+0x128/0x2c0 [18172.074616] ? smp_apic_timer_interrupt+0xa6/0x270 [18172.074623] ? apic_timer_interrupt+0xf/0x20 [18172.074628] [18172.074631] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18172.074635] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18172.074639] do_raw_spin_lock+0x177/0x190 [18172.074643] _raw_spin_lock+0x21/0x30 [18172.074648] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18172.081185] ? do_wait_intr+0xf0/0xf0 [18172.081209] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18172.081551] mdc_reint+0x2d/0x110 [mdc] [18172.081589] mdc_unlink+0x658/0x870 [mdc] [18172.081614] lmv_unlink+0x34d/0xbd0 [lmv] [18172.081644] ll_unlink+0x24d/0x7f0 [lustre] [18172.082103] vfs_unlink+0x202/0x330 [18172.082220] do_unlinkat+0x2fa/0x460 [18172.082227] __x64_sys_unlink+0x37/0x50 [18172.082230] do_syscall_64+0xc1/0x440 [18172.082241] entry_SYSCALL_64_after_hwframe+0x49/0xae [18172.082251] RIP: 0033:0x7f151bdee10b [18172.082260] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [18172.082264] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [18172.082270] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [18172.082273] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [18172.082280] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18172.082282] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18172.082285] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18172.730286] obd_mod_free+0x46/0x140 [ptlrpc] [18172.738576] mdc_commit_open+0x91/0xb0 [mdc] [18172.747237] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18172.754072] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18172.758912] after_reply+0xe2c/0x1780 [ptlrpc] [18172.772259] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18172.777980] ? __next_timer_interrupt+0x160/0x160 [18172.784720] ptlrpcd+0xa82/0xe40 [ptlrpc] [18172.794797] ? do_wait_intr+0xf0/0xf0 [18172.796095] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18172.806045] kthread+0x1d1/0x200 [18172.811543] ? set_kthread_struct+0x70/0x70 [18172.817093] ret_from_fork+0x1f/0x30 [18176.482043] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16245s! [18176.486069] Showing busy workqueues and worker pools: [18176.489142] workqueue events: flags=0x0 [18176.491291] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18176.495219] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [18176.500388] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18176.503088] pending: kfree_rcu_monitor [18176.505029] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18176.507476] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [18176.511193] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18176.513494] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [18176.517228] workqueue events_long: flags=0x0 [18176.518949] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18176.524358] pending: lu_objects_flush [obdclass] [18176.526196] workqueue events_unbound: flags=0x2 [18176.527515] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [18176.531080] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [18176.538327] workqueue events_power_efficient: flags=0x80 [18176.545907] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18176.551512] pending: do_cache_clean, neigh_periodic_work, check_lifetime [18176.555889] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18176.559713] pending: neigh_periodic_work [18176.561849] workqueue mm_percpu_wq: flags=0x8 [18176.567425] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18176.574104] pending: vmstat_update, lru_add_drain_per_cpu [18176.584408] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18176.595971] pending: vmstat_update [18176.598184] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18176.607104] pending: lru_add_drain_per_cpu [18176.614506] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [18176.625416] pending: lru_add_drain_per_cpu BAR(48) [18176.630893] workqueue writeback: flags=0x4a [18176.632457] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [18176.637467] pending: wb_workfn [18176.641125] workqueue xprtiod: flags=0xa [18176.645175] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [18176.649179] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [18176.656224] workqueue ipv6_addrconf: flags=0x40008 [18176.659446] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18176.664374] pending: addrconf_verify_work [18176.667793] workqueue knbd0-recv: flags=0x1a [18176.670348] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18176.676600] in-flight: 658:recv_work [18176.679084] workqueue ptlrpc_pinger: flags=0x2 [18176.683341] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18176.692432] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [18176.694758] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16246s workers=4 idle: 32984 37 8 [18176.710967] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18161s workers=2 idle: 661 [18184.984034] rcu: INFO: rcu_sched self-detected stall on CPU [18184.988438] rcu: 1-...!: (9784492 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [18185.000636] (t=16260337 jiffies g=249657 q=7473) [18185.009150] rcu: rcu_sched kthread starved for 16255289 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [18185.017216] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [18185.030739] rcu: RCU grace-period kthread stack dump: [18185.036885] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [18185.048428] Call Trace: [18185.049269] __schedule+0x351/0xcb0 [18185.052470] schedule+0xc0/0x180 [18185.053570] schedule_timeout+0xb4/0x190 [18185.058310] ? do_raw_spin_unlock+0x75/0x190 [18185.060550] ? __next_timer_interrupt+0x160/0x160 [18185.064612] ? prepare_to_swait_event+0xf3/0x140 [18185.066935] rcu_gp_kthread+0x8dd/0x15b0 [18185.069765] ? force_qs_rnp+0x2c0/0x2c0 [18185.071248] kthread+0x1d1/0x200 [18185.072748] ? set_kthread_struct+0x70/0x70 [18185.074452] ret_from_fork+0x1f/0x30 [18185.076001] rcu: Stack dump where RCU GP kthread last ran: [18185.078155] Sending NMI from CPU 1 to CPUs 2: [18185.080323] NMI backtrace for cpu 2 skipped: idling at native_safe_halt+0x2e/0x50 [18185.083248] NMI backtrace for cpu 1 [18185.084902] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18185.090247] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18185.094230] Call Trace: [18185.095288] [18185.096298] ? dump_stack+0xbb/0x10e [18185.097921] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [18185.100153] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [18185.102205] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [18185.104370] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [18185.106436] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [18185.108571] ? rcu_dump_cpu_stacks+0x11b/0x171 [18185.110655] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [18185.112995] ? update_process_times+0x6d/0xc0 [18185.115027] ? tick_sched_handle.isra.18+0x27/0x90 [18185.117156] ? tick_sched_timer+0x53/0xb0 [18185.119442] ? __hrtimer_run_queues+0x1c2/0x4d0 [18185.121101] ? tick_sched_handle.isra.18+0x90/0x90 [18185.122488] ? hrtimer_interrupt+0x128/0x2c0 [18185.124388] ? smp_apic_timer_interrupt+0xa6/0x270 [18185.126487] ? apic_timer_interrupt+0xf/0x20 [18185.128424] [18185.129519] ? native_safe_halt+0x2e/0x50 [18185.131165] ? kvm_wait+0xcd/0xe0 [18185.135425] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18185.137505] ? do_raw_spin_lock+0x177/0x190 [18185.140500] ? _raw_spin_lock+0x21/0x30 [18185.143577] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18185.146865] ? obd_mod_free+0x46/0x140 [ptlrpc] [18185.150444] ? mdc_commit_open+0x91/0xb0 [mdc] [18185.153514] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18185.157932] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18185.162249] ? after_reply+0xe2c/0x1780 [ptlrpc] [18185.164889] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18185.167251] ? __next_timer_interrupt+0x160/0x160 [18185.170318] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [18185.172437] ? do_wait_intr+0xf0/0xf0 [18185.174082] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18185.176065] ? kthread+0x1d1/0x200 [18185.177965] ? set_kthread_struct+0x70/0x70 [18185.179632] ? ret_from_fork+0x1f/0x30 [18200.057027] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [18200.060108] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18200.080662] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18200.086213] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18200.090514] RIP: 0010:native_safe_halt+0x2e/0x50 [18200.092405] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18200.099775] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [18200.103672] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18200.106914] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [18200.110232] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [18200.113236] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [18200.116305] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [18200.120052] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [18200.122936] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18200.125867] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [18200.128976] Call Trace: [18200.130038] [18200.131717] ? show_regs.cold.9+0x22/0x2f [18200.133275] ? watchdog_timer_fn.cold.6+0x62/0x10a [18200.135922] ? __hrtimer_run_queues+0x1c2/0x4d0 [18200.137930] ? watchdog+0x40/0x40 [18200.139479] ? hrtimer_interrupt+0x128/0x2c0 [18200.141461] ? smp_apic_timer_interrupt+0xa6/0x270 [18200.143943] ? apic_timer_interrupt+0xf/0x20 [18200.146052] [18200.147070] ? native_safe_halt+0x2e/0x50 [18200.149126] kvm_wait+0xcd/0xe0 [18200.150706] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18200.152415] do_raw_spin_lock+0x177/0x190 [18200.154721] _raw_spin_lock+0x21/0x30 [18200.156154] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18200.159426] ? do_raw_spin_unlock+0x75/0x190 [18200.161706] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18200.164068] ? do_wait_intr+0xf0/0xf0 [18200.165941] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18200.168366] mdc_close+0x53b/0xd70 [mdc] [18200.170792] lmv_close+0x18d/0x490 [lmv] [18200.173088] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [18200.176001] ll_close_inode_openhandle+0x493/0xea0 [lustre] [18200.178432] ll_md_real_close+0xa5/0x2a0 [lustre] [18200.180894] ll_md_close+0x274/0x800 [lustre] [18200.183008] ll_file_release+0x128/0xc50 [lustre] [18200.185626] __fput+0x10d/0x440 [18200.187811] ____fput+0x12/0x20 [18200.189113] task_work_run+0xc6/0x110 [18200.191454] exit_to_usermode_loop+0x1dd/0x1f0 [18200.193457] do_syscall_64+0x3d6/0x440 [18200.195231] entry_SYSCALL_64_after_hwframe+0x49/0xae [18200.197676] RIP: 0033:0x7f151bee3cd8 [18200.198808] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [18200.207683] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [18200.211044] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [18200.214370] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [18200.217425] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18200.221059] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18200.224654] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18204.052308] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [18204.057450] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18204.082891] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18204.091204] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18204.094972] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18204.097766] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18204.106358] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18204.109604] RAX: 0000000000007ef8 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18204.112324] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [18204.114962] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [18204.117391] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [18204.120372] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [18204.123201] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [18204.126766] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18204.129212] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [18204.132122] Call Trace: [18204.133507] [18204.134168] ? show_regs.cold.9+0x22/0x2f [18204.135847] ? watchdog_timer_fn.cold.6+0x62/0x10a [18204.137801] ? __hrtimer_run_queues+0x1c2/0x4d0 [18204.139617] ? watchdog+0x40/0x40 [18204.141139] ? hrtimer_interrupt+0x128/0x2c0 [18204.143040] ? smp_apic_timer_interrupt+0xa6/0x270 [18204.144860] ? apic_timer_interrupt+0xf/0x20 [18204.146266] [18204.147572] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18204.149301] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18204.151517] do_raw_spin_lock+0x177/0x190 [18204.152614] _raw_spin_lock+0x21/0x30 [18204.154200] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18204.156662] ? do_raw_spin_unlock+0x75/0x190 [18204.158647] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18204.161023] ? do_wait_intr+0xf0/0xf0 [18204.162166] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18204.164561] mdc_reint+0x2d/0x110 [mdc] [18204.166412] mdc_unlink+0x658/0x870 [mdc] [18204.168322] lmv_unlink+0x34d/0xbd0 [lmv] [18204.169986] ll_unlink+0x24d/0x7f0 [lustre] [18204.171890] vfs_unlink+0x202/0x330 [18204.173235] do_unlinkat+0x2fa/0x460 [18204.174750] __x64_sys_unlink+0x37/0x50 [18204.176143] do_syscall_64+0xc1/0x440 [18204.177829] entry_SYSCALL_64_after_hwframe+0x49/0xae [18204.179880] RIP: 0033:0x7f151bdee10b [18204.181354] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [18204.188719] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [18204.191946] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [18204.195117] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [18204.199011] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18204.202260] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18204.205271] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18207.200283] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16276s! [18207.206118] Showing busy workqueues and worker pools: [18207.209324] workqueue events: flags=0x0 [18207.213159] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18207.216308] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [18207.222240] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18207.225762] pending: kfree_rcu_monitor [18207.228314] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18207.232634] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [18207.239118] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18207.242248] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [18207.246037] workqueue events_long: flags=0x0 [18207.247937] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18207.251642] pending: lu_objects_flush [obdclass] [18207.256273] workqueue events_unbound: flags=0x2 [18207.258090] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [18207.260521] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [18207.265267] workqueue events_power_efficient: flags=0x80 [18207.267358] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18207.270859] pending: do_cache_clean, neigh_periodic_work, check_lifetime [18207.274377] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18207.281986] pending: neigh_periodic_work [18207.287372] workqueue mm_percpu_wq: flags=0x8 [18207.290090] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18207.292031] pending: vmstat_update, lru_add_drain_per_cpu [18207.294763] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18207.301138] pending: vmstat_update [18207.306150] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18207.313269] pending: lru_add_drain_per_cpu [18207.315801] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [18207.318799] pending: lru_add_drain_per_cpu BAR(48) [18207.323256] workqueue writeback: flags=0x4a [18207.327927] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [18207.334885] pending: wb_workfn [18207.336119] workqueue xprtiod: flags=0xa [18207.338185] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [18207.342657] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [18207.346617] workqueue ipv6_addrconf: flags=0x40008 [18207.349025] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18207.357247] pending: addrconf_verify_work [18207.361610] workqueue knbd0-recv: flags=0x1a [18207.364614] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18207.369701] in-flight: 658:recv_work [18207.374207] workqueue ptlrpc_pinger: flags=0x2 [18207.379221] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18207.387190] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [18207.391484] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16276s workers=4 idle: 32984 37 8 [18207.402354] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18192s workers=2 idle: 661 [18212.058614] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [18212.058821] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [18212.063841] Modules linked in: lustre(O) [18212.067604] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18212.067793] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18212.067801] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18212.067811] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18212.068265] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18212.068278] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18212.068283] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18212.068287] RAX: 0000000000001302 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18212.068290] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [18212.068293] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [18212.068295] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [18212.068298] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [18212.068302] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [18212.068377] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18212.068446] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18212.068455] Call Trace: [18212.068460] [18212.068466] ? show_regs.cold.9+0x22/0x2f [18212.068476] ? watchdog_timer_fn.cold.6+0x62/0x10a [18212.068482] ? __hrtimer_run_queues+0x1c2/0x4d0 [18212.068487] ? watchdog+0x40/0x40 [18212.068491] ? hrtimer_interrupt+0x128/0x2c0 [18212.068495] ? smp_apic_timer_interrupt+0xa6/0x270 [18212.068503] ? apic_timer_interrupt+0xf/0x20 [18212.068508] [18212.068510] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18212.068524] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18212.068529] do_raw_spin_lock+0x177/0x190 [18212.068534] _raw_spin_lock+0x21/0x30 [18212.068538] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18212.069125] process_one_work+0x2c8/0x700 [18212.069208] worker_thread+0x296/0x6e0 [18212.069214] ? rescuer_thread+0x570/0x570 [18212.069218] kthread+0x1d1/0x200 [18212.069222] ? set_kthread_struct+0x70/0x70 [18212.069226] ret_from_fork+0x1f/0x30 [18212.345462] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18212.390344] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18212.408623] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18212.423411] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18212.425875] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18212.434673] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18212.438180] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [18212.445488] RDX: 0000000000000202 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [18212.457987] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [18212.463543] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [18212.468282] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18212.474201] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [18212.479158] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18212.481745] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18212.484069] Call Trace: [18212.484990] [18212.485698] ? show_regs.cold.9+0x22/0x2f [18212.487235] ? watchdog_timer_fn.cold.6+0x62/0x10a [18212.489004] ? __hrtimer_run_queues+0x1c2/0x4d0 [18212.490785] ? watchdog+0x40/0x40 [18212.491985] ? hrtimer_interrupt+0x128/0x2c0 [18212.493656] ? smp_apic_timer_interrupt+0xa6/0x270 [18212.495487] ? apic_timer_interrupt+0xf/0x20 [18212.497713] [18212.498881] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18212.501630] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18212.503418] do_raw_spin_lock+0x177/0x190 [18212.507807] _raw_spin_lock+0x21/0x30 [18212.509120] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18212.512042] obd_mod_free+0x46/0x140 [ptlrpc] [18212.515095] mdc_commit_open+0x91/0xb0 [mdc] [18212.522728] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18212.531365] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18212.535818] after_reply+0xe2c/0x1780 [ptlrpc] [18212.546938] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18212.552175] ? __next_timer_interrupt+0x160/0x160 [18212.554709] ptlrpcd+0xa82/0xe40 [ptlrpc] [18212.558939] ? do_wait_intr+0xf0/0xf0 [18212.565227] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18212.567318] kthread+0x1d1/0x200 [18212.568360] ? set_kthread_struct+0x70/0x70 [18212.569726] ret_from_fork+0x1f/0x30 [18228.057236] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [18228.065217] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18228.116163] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18228.125770] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18228.138011] RIP: 0010:native_safe_halt+0x2e/0x50 [18228.140946] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18228.155510] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18228.161304] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18228.167194] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [18228.174006] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [18228.181256] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [18228.186899] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [18228.191552] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [18228.198373] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18228.205058] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [18228.209562] Call Trace: [18228.210973] [18228.212597] ? show_regs.cold.9+0x22/0x2f [18228.214325] ? watchdog_timer_fn.cold.6+0x62/0x10a [18228.218155] ? __hrtimer_run_queues+0x1c2/0x4d0 [18228.220807] ? watchdog+0x40/0x40 [18228.222390] ? hrtimer_interrupt+0x128/0x2c0 [18228.224384] ? smp_apic_timer_interrupt+0xa6/0x270 [18228.226070] ? apic_timer_interrupt+0xf/0x20 [18228.227582] [18228.228448] ? native_safe_halt+0x2e/0x50 [18228.230197] kvm_wait+0xcd/0xe0 [18228.231632] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18228.234177] do_raw_spin_lock+0x177/0x190 [18228.235799] _raw_spin_lock+0x21/0x30 [18228.237923] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18228.242652] ? do_raw_spin_unlock+0x75/0x190 [18228.244617] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18228.247323] ? do_wait_intr+0xf0/0xf0 [18228.248962] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18228.253795] mdc_close+0x53b/0xd70 [mdc] [18228.256094] lmv_close+0x18d/0x490 [lmv] [18228.258101] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [18228.268094] ll_close_inode_openhandle+0x493/0xea0 [lustre] [18228.276825] ll_md_real_close+0xa5/0x2a0 [lustre] [18228.282125] ll_md_close+0x274/0x800 [lustre] [18228.287027] ll_file_release+0x128/0xc50 [lustre] [18228.289134] __fput+0x10d/0x440 [18228.291033] ____fput+0x12/0x20 [18228.292345] task_work_run+0xc6/0x110 [18228.295825] exit_to_usermode_loop+0x1dd/0x1f0 [18228.297799] do_syscall_64+0x3d6/0x440 [18228.299865] entry_SYSCALL_64_after_hwframe+0x49/0xae [18228.302312] RIP: 0033:0x7f151bee3cd8 [18228.304711] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [18228.315583] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [18228.318344] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [18228.321956] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [18228.325988] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18228.330693] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18228.334569] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18232.052113] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [18232.062997] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18232.087509] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18232.093381] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18232.098541] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18232.102295] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18232.112441] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18232.116516] RAX: 0000000000007a8c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18232.120693] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [18232.124987] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [18232.129657] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [18232.133823] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [18232.145687] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [18232.158539] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18232.167264] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [18232.172455] Call Trace: [18232.173848] [18232.175067] ? show_regs.cold.9+0x22/0x2f [18232.177076] ? watchdog_timer_fn.cold.6+0x62/0x10a [18232.179567] ? __hrtimer_run_queues+0x1c2/0x4d0 [18232.185535] ? watchdog+0x40/0x40 [18232.187250] ? hrtimer_interrupt+0x128/0x2c0 [18232.192401] ? smp_apic_timer_interrupt+0xa6/0x270 [18232.194953] ? apic_timer_interrupt+0xf/0x20 [18232.201304] [18232.202719] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18232.206982] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18232.209928] do_raw_spin_lock+0x177/0x190 [18232.212822] _raw_spin_lock+0x21/0x30 [18232.215428] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18232.218491] ? do_raw_spin_unlock+0x75/0x190 [18232.220743] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18232.225454] ? do_wait_intr+0xf0/0xf0 [18232.232831] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18232.239898] mdc_reint+0x2d/0x110 [mdc] [18232.242669] mdc_unlink+0x658/0x870 [mdc] [18232.244494] lmv_unlink+0x34d/0xbd0 [lmv] [18232.247315] ll_unlink+0x24d/0x7f0 [lustre] [18232.250287] vfs_unlink+0x202/0x330 [18232.253067] do_unlinkat+0x2fa/0x460 [18232.255639] __x64_sys_unlink+0x37/0x50 [18232.258105] do_syscall_64+0xc1/0x440 [18232.271737] entry_SYSCALL_64_after_hwframe+0x49/0xae [18232.279020] RIP: 0033:0x7f151bdee10b [18232.287066] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [18232.321859] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [18232.334534] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [18232.347685] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [18232.356270] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18232.363570] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18232.369901] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18237.924359] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16307s! [18237.933875] Showing busy workqueues and worker pools: [18237.941125] workqueue events: flags=0x0 [18237.944400] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18237.956255] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [18237.978509] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18237.986354] pending: kfree_rcu_monitor [18237.991770] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18238.012760] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [18238.021716] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18238.029772] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [18238.040567] workqueue events_long: flags=0x0 [18238.044557] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18238.053595] pending: lu_objects_flush [obdclass] [18238.060114] workqueue events_unbound: flags=0x2 [18238.067417] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [18238.074963] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [18238.104578] workqueue events_power_efficient: flags=0x80 [18238.119867] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18238.132895] pending: do_cache_clean, neigh_periodic_work, check_lifetime [18238.151988] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18238.166079] pending: neigh_periodic_work [18238.172242] workqueue mm_percpu_wq: flags=0x8 [18238.181604] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18238.188168] pending: vmstat_update, lru_add_drain_per_cpu [18238.198899] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18238.203320] pending: vmstat_update [18238.206409] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18238.221965] pending: lru_add_drain_per_cpu [18238.227096] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [18238.236940] pending: lru_add_drain_per_cpu BAR(48) [18238.239701] workqueue writeback: flags=0x4a [18238.248844] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [18238.255368] pending: wb_workfn [18238.257204] workqueue xprtiod: flags=0xa [18238.262970] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [18238.266322] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [18238.277832] workqueue ipv6_addrconf: flags=0x40008 [18238.284642] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18238.292798] pending: addrconf_verify_work [18238.304785] workqueue knbd0-recv: flags=0x1a [18238.309096] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18238.314988] in-flight: 658:recv_work [18238.319843] workqueue ptlrpc_pinger: flags=0x2 [18238.322151] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18238.340918] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [18238.350518] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16307s workers=4 idle: 32984 37 8 [18238.364347] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18223s workers=2 idle: 661 [18240.057367] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [18240.060361] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [18240.063204] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18240.063291] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18240.063298] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18240.063304] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18240.065063] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18240.065080] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18240.065084] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18240.065089] RAX: 0000000000007288 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18240.065092] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [18240.065094] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [18240.065096] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [18240.065098] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [18240.065102] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [18240.065105] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18240.065107] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18240.065116] Call Trace: [18240.065121] [18240.065127] ? show_regs.cold.9+0x22/0x2f [18240.065137] ? watchdog_timer_fn.cold.6+0x62/0x10a [18240.065144] ? __hrtimer_run_queues+0x1c2/0x4d0 [18240.065148] ? watchdog+0x40/0x40 [18240.065153] ? hrtimer_interrupt+0x128/0x2c0 [18240.065157] ? smp_apic_timer_interrupt+0xa6/0x270 [18240.065164] ? apic_timer_interrupt+0xf/0x20 [18240.065169] [18240.065171] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18240.065176] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18240.065182] do_raw_spin_lock+0x177/0x190 [18240.065188] _raw_spin_lock+0x21/0x30 [18240.065193] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18240.066837] process_one_work+0x2c8/0x700 [18240.066848] worker_thread+0x296/0x6e0 [18240.066853] ? rescuer_thread+0x570/0x570 [18240.066856] kthread+0x1d1/0x200 [18240.066862] ? set_kthread_struct+0x70/0x70 [18240.066865] ret_from_fork+0x1f/0x30 [18240.306220] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18240.355693] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18240.361214] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18240.366973] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18240.371292] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18240.384615] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18240.388771] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [18240.394152] RDX: 0000000000007c85 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [18240.400687] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [18240.407352] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [18240.413287] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18240.418369] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [18240.428700] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18240.434120] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18240.438596] Call Trace: [18240.441660] [18240.445722] ? show_regs.cold.9+0x22/0x2f [18240.449231] ? watchdog_timer_fn.cold.6+0x62/0x10a [18240.451812] ? __hrtimer_run_queues+0x1c2/0x4d0 [18240.458196] ? watchdog+0x40/0x40 [18240.460999] ? hrtimer_interrupt+0x128/0x2c0 [18240.464888] ? smp_apic_timer_interrupt+0xa6/0x270 [18240.472701] ? apic_timer_interrupt+0xf/0x20 [18240.477722] [18240.478995] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18240.481722] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18240.489026] do_raw_spin_lock+0x177/0x190 [18240.491712] _raw_spin_lock+0x21/0x30 [18240.495886] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18240.501734] obd_mod_free+0x46/0x140 [ptlrpc] [18240.507783] mdc_commit_open+0x91/0xb0 [mdc] [18240.512952] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18240.515379] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18240.517846] after_reply+0xe2c/0x1780 [ptlrpc] [18240.526766] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18240.533833] ? __next_timer_interrupt+0x160/0x160 [18240.537488] ptlrpcd+0xa82/0xe40 [ptlrpc] [18240.543014] ? do_wait_intr+0xf0/0xf0 [18240.546420] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18240.551612] kthread+0x1d1/0x200 [18240.557021] ? set_kthread_struct+0x70/0x70 [18240.559681] ret_from_fork+0x1f/0x30 [18256.057207] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [18256.062215] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18256.084101] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18256.088812] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18256.093194] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18256.095739] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18256.103020] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18256.106194] RAX: 00000000000057b8 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18256.110353] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [18256.114374] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [18256.117449] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [18256.121058] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [18256.125387] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [18256.129268] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18256.132290] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [18256.135895] Call Trace: [18256.136880] [18256.137492] ? show_regs.cold.9+0x22/0x2f [18256.138938] ? watchdog_timer_fn.cold.6+0x62/0x10a [18256.140351] ? __hrtimer_run_queues+0x1c2/0x4d0 [18256.142162] ? watchdog+0x40/0x40 [18256.143528] ? hrtimer_interrupt+0x128/0x2c0 [18256.146279] ? smp_apic_timer_interrupt+0xa6/0x270 [18256.149749] ? apic_timer_interrupt+0xf/0x20 [18256.152459] [18256.153474] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18256.156877] do_raw_spin_lock+0x177/0x190 [18256.159811] _raw_spin_lock+0x21/0x30 [18256.161495] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18256.164457] ? do_raw_spin_unlock+0x75/0x190 [18256.165944] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18256.168248] ? do_wait_intr+0xf0/0xf0 [18256.170980] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18256.175042] mdc_close+0x53b/0xd70 [mdc] [18256.177409] lmv_close+0x18d/0x490 [lmv] [18256.179794] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [18256.184211] ll_close_inode_openhandle+0x493/0xea0 [lustre] [18256.188005] ll_md_real_close+0xa5/0x2a0 [lustre] [18256.190786] ll_md_close+0x274/0x800 [lustre] [18256.192716] ll_file_release+0x128/0xc50 [lustre] [18256.195223] __fput+0x10d/0x440 [18256.196288] ____fput+0x12/0x20 [18256.202485] task_work_run+0xc6/0x110 [18256.204019] exit_to_usermode_loop+0x1dd/0x1f0 [18256.207158] do_syscall_64+0x3d6/0x440 [18256.209511] entry_SYSCALL_64_after_hwframe+0x49/0xae [18256.213602] RIP: 0033:0x7f151bee3cd8 [18256.219826] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [18256.238361] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [18256.242823] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [18256.250147] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [18256.255776] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18256.265141] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18256.274340] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18264.052073] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [18264.059318] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18264.080799] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18264.085344] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18264.088700] RIP: 0010:native_safe_halt+0x2e/0x50 [18264.090753] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18264.098674] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18264.101609] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18264.104765] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [18264.107294] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [18264.110538] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [18264.113630] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [18264.116839] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [18264.119584] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18264.121687] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [18264.124599] Call Trace: [18264.125603] [18264.126446] ? show_regs.cold.9+0x22/0x2f [18264.128557] ? watchdog_timer_fn.cold.6+0x62/0x10a [18264.130761] ? __hrtimer_run_queues+0x1c2/0x4d0 [18264.133174] ? watchdog+0x40/0x40 [18264.134603] ? hrtimer_interrupt+0x128/0x2c0 [18264.136316] ? smp_apic_timer_interrupt+0xa6/0x270 [18264.138716] ? apic_timer_interrupt+0xf/0x20 [18264.140659] [18264.141696] ? native_safe_halt+0x2e/0x50 [18264.143570] kvm_wait+0xcd/0xe0 [18264.145202] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18264.147487] do_raw_spin_lock+0x177/0x190 [18264.149557] _raw_spin_lock+0x21/0x30 [18264.151125] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18264.153648] ? do_raw_spin_unlock+0x75/0x190 [18264.155708] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18264.158201] ? do_wait_intr+0xf0/0xf0 [18264.159760] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18264.162321] mdc_reint+0x2d/0x110 [mdc] [18264.163989] mdc_unlink+0x658/0x870 [mdc] [18264.165902] lmv_unlink+0x34d/0xbd0 [lmv] [18264.167590] ll_unlink+0x24d/0x7f0 [lustre] [18264.169789] vfs_unlink+0x202/0x330 [18264.171495] do_unlinkat+0x2fa/0x460 [18264.173109] __x64_sys_unlink+0x37/0x50 [18264.174966] do_syscall_64+0xc1/0x440 [18264.176594] entry_SYSCALL_64_after_hwframe+0x49/0xae [18264.178647] RIP: 0033:0x7f151bdee10b [18264.179678] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [18264.186976] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [18264.190261] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [18264.193377] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [18264.196529] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18264.199903] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18264.202999] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18268.056019] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [18268.056351] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [18268.061810] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18268.061898] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18268.061906] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18268.061912] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18268.062384] RIP: 0010:native_safe_halt+0x2e/0x50 [18268.068090] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18268.068317] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18268.068324] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18268.068327] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18268.104220] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18268.104234] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18268.104240] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18268.104243] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [18268.104245] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [18268.104247] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [18268.104250] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [18268.104254] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [18268.104257] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18268.104259] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18268.104268] Call Trace: [18268.104273] [18268.104278] ? show_regs.cold.9+0x22/0x2f [18268.104289] ? watchdog_timer_fn.cold.6+0x62/0x10a [18268.104297] ? __hrtimer_run_queues+0x1c2/0x4d0 [18268.104301] ? watchdog+0x40/0x40 [18268.104305] ? hrtimer_interrupt+0x128/0x2c0 [18268.104310] ? smp_apic_timer_interrupt+0xa6/0x270 [18268.104318] ? apic_timer_interrupt+0xf/0x20 [18268.104323] [18268.104326] ? native_safe_halt+0x2e/0x50 [18268.104331] kvm_wait+0xcd/0xe0 [18268.104338] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18268.104348] do_raw_spin_lock+0x177/0x190 [18268.104352] _raw_spin_lock+0x21/0x30 [18268.104356] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18268.113277] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18268.113290] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18268.113296] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [18268.113298] RDX: 00000000000015a7 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [18268.113300] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [18268.113302] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [18268.113304] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18268.113306] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [18268.113308] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18268.113310] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18268.113317] Call Trace: [18268.113462] [18268.113469] ? show_regs.cold.9+0x22/0x2f [18268.113482] ? watchdog_timer_fn.cold.6+0x62/0x10a [18268.113488] ? __hrtimer_run_queues+0x1c2/0x4d0 [18268.113493] ? watchdog+0x40/0x40 [18268.113502] ? hrtimer_interrupt+0x128/0x2c0 [18268.113505] ? smp_apic_timer_interrupt+0xa6/0x270 [18268.113516] ? apic_timer_interrupt+0xf/0x20 [18268.113520] [18268.113523] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18268.113529] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18268.113533] do_raw_spin_lock+0x177/0x190 [18268.113537] _raw_spin_lock+0x21/0x30 [18268.113544] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18268.114307] obd_mod_free+0x46/0x140 [ptlrpc] [18268.114528] mdc_commit_open+0x91/0xb0 [mdc] [18268.114564] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18268.115016] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18268.115288] after_reply+0xe2c/0x1780 [ptlrpc] [18268.115748] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18268.116068] ? __next_timer_interrupt+0x160/0x160 [18268.116078] ptlrpcd+0xa82/0xe40 [ptlrpc] [18268.116350] ? do_wait_intr+0xf0/0xf0 [18268.116364] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18268.116787] kthread+0x1d1/0x200 [18268.116793] ? set_kthread_struct+0x70/0x70 [18268.116797] ret_from_fork+0x1f/0x30 [18268.454008] process_one_work+0x2c8/0x700 [18268.455658] worker_thread+0x296/0x6e0 [18268.457978] ? rescuer_thread+0x570/0x570 [18268.460336] kthread+0x1d1/0x200 [18268.461620] ? set_kthread_struct+0x70/0x70 [18268.463220] ret_from_fork+0x1f/0x30 [18268.640160] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16338s! [18268.647508] Showing busy workqueues and worker pools: [18268.651475] workqueue events: flags=0x0 [18268.654206] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18268.664239] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [18268.673353] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18268.675774] pending: kfree_rcu_monitor [18268.687009] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18268.692321] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [18268.698367] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18268.700621] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [18268.707451] workqueue events_long: flags=0x0 [18268.712118] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18268.719033] pending: lu_objects_flush [obdclass] [18268.722438] workqueue events_unbound: flags=0x2 [18268.724461] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [18268.727217] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [18268.735011] workqueue events_power_efficient: flags=0x80 [18268.736963] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18268.746467] pending: do_cache_clean, neigh_periodic_work, check_lifetime [18268.749784] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18268.756299] pending: neigh_periodic_work [18268.758783] workqueue mm_percpu_wq: flags=0x8 [18268.765219] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18268.772618] pending: vmstat_update, lru_add_drain_per_cpu [18268.777794] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18268.783787] pending: vmstat_update [18268.786386] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18268.790279] pending: lru_add_drain_per_cpu [18268.791635] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [18268.798468] pending: lru_add_drain_per_cpu BAR(48) [18268.807217] workqueue writeback: flags=0x4a [18268.809599] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [18268.813490] pending: wb_workfn [18268.819700] workqueue xprtiod: flags=0xa [18268.822223] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [18268.829403] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [18268.839599] workqueue ipv6_addrconf: flags=0x40008 [18268.843296] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18268.848685] pending: addrconf_verify_work [18268.853209] workqueue knbd0-recv: flags=0x1a [18268.854802] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18268.862486] in-flight: 658:recv_work [18268.864389] workqueue ptlrpc_pinger: flags=0x2 [18268.868119] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18268.877327] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [18268.884656] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16338s workers=4 idle: 32984 37 8 [18268.895668] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18253s workers=2 idle: 661 [18284.057167] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [18284.059645] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18284.089751] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18284.097281] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18284.101769] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18284.104606] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18284.113211] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18284.116417] RAX: 00000000000037d6 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18284.119913] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [18284.123138] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [18284.126222] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [18284.129922] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [18284.132652] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [18284.135839] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18284.139252] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [18284.142887] Call Trace: [18284.144284] [18284.145753] ? show_regs.cold.9+0x22/0x2f [18284.147814] ? watchdog_timer_fn.cold.6+0x62/0x10a [18284.153455] ? __hrtimer_run_queues+0x1c2/0x4d0 [18284.156231] ? watchdog+0x40/0x40 [18284.157549] ? hrtimer_interrupt+0x128/0x2c0 [18284.159288] ? smp_apic_timer_interrupt+0xa6/0x270 [18284.161580] ? apic_timer_interrupt+0xf/0x20 [18284.164616] [18284.165455] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18284.168105] do_raw_spin_lock+0x177/0x190 [18284.170510] _raw_spin_lock+0x21/0x30 [18284.172619] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18284.176650] ? do_raw_spin_unlock+0x75/0x190 [18284.179288] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18284.182378] ? do_wait_intr+0xf0/0xf0 [18284.185340] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18284.188531] mdc_close+0x53b/0xd70 [mdc] [18284.190548] lmv_close+0x18d/0x490 [lmv] [18284.194887] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [18284.198158] ll_close_inode_openhandle+0x493/0xea0 [lustre] [18284.200886] ll_md_real_close+0xa5/0x2a0 [lustre] [18284.203372] ll_md_close+0x274/0x800 [lustre] [18284.205629] ll_file_release+0x128/0xc50 [lustre] [18284.207841] __fput+0x10d/0x440 [18284.209558] ____fput+0x12/0x20 [18284.210909] task_work_run+0xc6/0x110 [18284.212917] exit_to_usermode_loop+0x1dd/0x1f0 [18284.215270] do_syscall_64+0x3d6/0x440 [18284.217105] entry_SYSCALL_64_after_hwframe+0x49/0xae [18284.219847] RIP: 0033:0x7f151bee3cd8 [18284.221567] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [18284.230291] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [18284.233605] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [18284.237120] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [18284.240327] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18284.243268] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18284.249127] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18296.055053] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [18296.057722] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [18296.057784] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [18296.057790] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18296.057861] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18296.057866] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18296.057869] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18296.057882] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18296.057886] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18296.057891] RAX: 0000000000002692 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18296.057893] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [18296.057895] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [18296.057897] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [18296.057900] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [18296.057903] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [18296.057906] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18296.057908] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [18296.057916] Call Trace: [18296.057920] [18296.057925] ? show_regs.cold.9+0x22/0x2f [18296.057934] ? watchdog_timer_fn.cold.6+0x62/0x10a [18296.057940] ? __hrtimer_run_queues+0x1c2/0x4d0 [18296.057944] ? watchdog+0x40/0x40 [18296.057948] ? hrtimer_interrupt+0x128/0x2c0 [18296.057952] ? smp_apic_timer_interrupt+0xa6/0x270 [18296.057960] ? apic_timer_interrupt+0xf/0x20 [18296.057965] [18296.057968] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18296.057972] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18296.057977] do_raw_spin_lock+0x177/0x190 [18296.057981] _raw_spin_lock+0x21/0x30 [18296.057987] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18296.058266] ? do_raw_spin_unlock+0x75/0x190 [18296.058272] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18296.059883] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) [18296.060268] ? do_wait_intr+0xf0/0xf0 [18296.059917] ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18296.061732] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18296.061740] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18296.061743] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18296.061756] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18296.061760] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18296.061765] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [18296.061768] RDX: 0000000000007668 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [18296.061771] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [18296.061774] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [18296.061776] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18296.061780] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [18296.061784] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18296.061786] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18296.061795] Call Trace: [18296.061800] [18296.061804] ? show_regs.cold.9+0x22/0x2f [18296.061814] ? watchdog_timer_fn.cold.6+0x62/0x10a [18296.061820] ? __hrtimer_run_queues+0x1c2/0x4d0 [18296.061825] ? watchdog+0x40/0x40 [18296.061830] ? hrtimer_interrupt+0x128/0x2c0 [18296.061835] ? smp_apic_timer_interrupt+0xa6/0x270 [18296.061843] ? apic_timer_interrupt+0xf/0x20 [18296.061849] [18296.061852] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18296.061857] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18296.061862] do_raw_spin_lock+0x177/0x190 [18296.061867] _raw_spin_lock+0x21/0x30 [18296.061873] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18296.062149] obd_mod_free+0x46/0x140 [ptlrpc] [18296.062347] mdc_commit_open+0x91/0xb0 [mdc] [18296.060279] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18296.063787] mdc_reint+0x2d/0x110 [mdc] [18296.063825] mdc_unlink+0x658/0x870 [mdc] [18296.063852] lmv_unlink+0x34d/0xbd0 [lmv] [18296.063882] ll_unlink+0x24d/0x7f0 [lustre] [18296.063994] vfs_unlink+0x202/0x330 [18296.064004] do_unlinkat+0x2fa/0x460 [18296.064009] __x64_sys_unlink+0x37/0x50 [18296.064013] do_syscall_64+0xc1/0x440 [18296.064020] entry_SYSCALL_64_after_hwframe+0x49/0xae [18296.064029] RIP: 0033:0x7f151bdee10b [18296.064038] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [18296.064041] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [18296.064046] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [18296.064048] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [18296.064050] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18296.064052] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18296.064055] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18296.062385] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18296.069736] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18296.070689] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18296.070698] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18296.070714] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18296.076034] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18296.076053] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18296.076058] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18296.076063] RAX: 0000000000007e18 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18296.076068] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [18296.076071] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [18296.076073] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [18296.076075] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [18296.076079] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [18296.076085] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18296.076087] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18296.076095] Call Trace: [18296.076103] [18296.077221] ? show_regs.cold.9+0x22/0x2f [18296.083892] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18296.143769] ? watchdog_timer_fn.cold.6+0x62/0x10a [18296.143789] ? __hrtimer_run_queues+0x1c2/0x4d0 [18296.143794] ? watchdog+0x40/0x40 [18296.143798] ? hrtimer_interrupt+0x128/0x2c0 [18296.143802] ? smp_apic_timer_interrupt+0xa6/0x270 [18296.143810] ? apic_timer_interrupt+0xf/0x20 [18296.143814] [18296.143819] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18296.143825] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18296.143832] do_raw_spin_lock+0x177/0x190 [18296.143838] _raw_spin_lock+0x21/0x30 [18296.143843] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18296.144430] process_one_work+0x2c8/0x700 [18296.144438] worker_thread+0x296/0x6e0 [18296.144442] ? rescuer_thread+0x570/0x570 [18296.144446] kthread+0x1d1/0x200 [18296.144451] ? set_kthread_struct+0x70/0x70 [18296.144455] ret_from_fork+0x1f/0x30 [18297.056507] after_reply+0xe2c/0x1780 [ptlrpc] [18297.064313] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18297.070001] ? __next_timer_interrupt+0x160/0x160 [18297.072473] ptlrpcd+0xa82/0xe40 [ptlrpc] [18297.080023] ? do_wait_intr+0xf0/0xf0 [18297.084732] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18297.092703] kthread+0x1d1/0x200 [18297.101437] ? set_kthread_struct+0x70/0x70 [18297.105349] ret_from_fork+0x1f/0x30 [18299.367625] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16368s! [18299.372177] Showing busy workqueues and worker pools: [18299.376817] workqueue events: flags=0x0 [18299.378625] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18299.381973] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [18299.392931] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18299.396969] pending: kfree_rcu_monitor [18299.401890] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18299.405268] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [18299.409189] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18299.411405] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [18299.415620] workqueue events_long: flags=0x0 [18299.417837] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18299.421811] pending: lu_objects_flush [obdclass] [18299.427821] workqueue events_unbound: flags=0x2 [18299.432403] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [18299.436943] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [18299.449884] workqueue events_power_efficient: flags=0x80 [18299.453378] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18299.459614] pending: do_cache_clean, neigh_periodic_work, check_lifetime [18299.462922] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18299.466327] pending: neigh_periodic_work [18299.468646] workqueue mm_percpu_wq: flags=0x8 [18299.470356] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18299.473514] pending: vmstat_update, lru_add_drain_per_cpu [18299.476241] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18299.478862] pending: vmstat_update [18299.480149] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18299.483026] pending: lru_add_drain_per_cpu [18299.484915] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [18299.487848] pending: lru_add_drain_per_cpu BAR(48) [18299.489718] workqueue writeback: flags=0x4a [18299.491263] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [18299.493889] pending: wb_workfn [18299.495365] workqueue xprtiod: flags=0xa [18299.497310] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [18299.500077] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [18299.503340] workqueue ipv6_addrconf: flags=0x40008 [18299.505488] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18299.508201] pending: addrconf_verify_work [18299.510100] workqueue knbd0-recv: flags=0x1a [18299.511988] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18299.514314] in-flight: 658:recv_work [18299.515683] workqueue ptlrpc_pinger: flags=0x2 [18299.517702] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18299.520263] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [18299.522937] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16368s workers=4 idle: 32984 37 8 [18299.525189] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18284s workers=2 idle: 661 [18312.057364] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [18312.063658] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18312.120137] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18312.132881] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18312.138048] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18312.147383] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18312.166098] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18312.172762] RAX: 0000000000004f8b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18312.188817] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [18312.193800] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [18312.200161] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [18312.211916] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [18312.218770] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [18312.223711] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18312.227945] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [18312.233642] Call Trace: [18312.234409] [18312.235034] ? show_regs.cold.9+0x22/0x2f [18312.246071] ? watchdog_timer_fn.cold.6+0x62/0x10a [18312.251694] ? __hrtimer_run_queues+0x1c2/0x4d0 [18312.254993] ? watchdog+0x40/0x40 [18312.261831] ? hrtimer_interrupt+0x128/0x2c0 [18312.264388] ? smp_apic_timer_interrupt+0xa6/0x270 [18312.266107] ? apic_timer_interrupt+0xf/0x20 [18312.276661] [18312.278575] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18312.283633] do_raw_spin_lock+0x177/0x190 [18312.285665] _raw_spin_lock+0x21/0x30 [18312.287738] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18312.291043] ? do_raw_spin_unlock+0x75/0x190 [18312.296809] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18312.300629] ? do_wait_intr+0xf0/0xf0 [18312.303844] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18312.313944] mdc_close+0x53b/0xd70 [mdc] [18312.316828] lmv_close+0x18d/0x490 [lmv] [18312.322235] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [18312.324194] ll_close_inode_openhandle+0x493/0xea0 [lustre] [18312.330857] ll_md_real_close+0xa5/0x2a0 [lustre] [18312.337109] ll_md_close+0x274/0x800 [lustre] [18312.345708] ll_file_release+0x128/0xc50 [lustre] [18312.351212] __fput+0x10d/0x440 [18312.355169] ____fput+0x12/0x20 [18312.360911] task_work_run+0xc6/0x110 [18312.366081] exit_to_usermode_loop+0x1dd/0x1f0 [18312.375707] do_syscall_64+0x3d6/0x440 [18312.379000] entry_SYSCALL_64_after_hwframe+0x49/0xae [18312.392890] RIP: 0033:0x7f151bee3cd8 [18312.394226] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [18312.422060] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [18312.427817] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [18312.434855] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [18312.444600] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18312.447170] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18312.458854] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18324.052048] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [18324.055019] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [18324.056758] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18324.056831] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18324.056839] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18324.056842] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18324.056856] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18324.056860] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18324.056865] RAX: 0000000000001f07 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18324.056867] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [18324.056870] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [18324.056872] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [18324.056874] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [18324.056878] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [18324.056881] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18324.056883] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [18324.056892] Call Trace: [18324.056897] [18324.056902] ? show_regs.cold.9+0x22/0x2f [18324.056911] ? watchdog_timer_fn.cold.6+0x62/0x10a [18324.056918] ? __hrtimer_run_queues+0x1c2/0x4d0 [18324.056922] ? watchdog+0x40/0x40 [18324.056927] ? hrtimer_interrupt+0x128/0x2c0 [18324.056931] ? smp_apic_timer_interrupt+0xa6/0x270 [18324.056939] ? apic_timer_interrupt+0xf/0x20 [18324.056944] [18324.056946] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18324.056951] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18324.056955] do_raw_spin_lock+0x177/0x190 [18324.056960] _raw_spin_lock+0x21/0x30 [18324.056965] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18324.057252] ? do_raw_spin_unlock+0x75/0x190 [18324.057445] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [18324.057450] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18324.057512] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18324.057516] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18324.057257] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18324.058138] ? do_wait_intr+0xf0/0xf0 [18324.058146] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18324.058410] mdc_reint+0x2d/0x110 [mdc] [18324.058480] mdc_unlink+0x658/0x870 [mdc] [18324.058504] lmv_unlink+0x34d/0xbd0 [lmv] [18324.058527] ll_unlink+0x24d/0x7f0 [lustre] [18324.059827] vfs_unlink+0x202/0x330 [18324.059835] do_unlinkat+0x2fa/0x460 [18324.059840] __x64_sys_unlink+0x37/0x50 [18324.059844] do_syscall_64+0xc1/0x440 [18324.059851] entry_SYSCALL_64_after_hwframe+0x49/0xae [18324.059860] RIP: 0033:0x7f151bdee10b [18324.059866] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [18324.059870] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [18324.059876] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [18324.059879] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [18324.059881] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18324.059884] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18324.059886] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18324.057521] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18324.064074] Modules linked in: lustre(O) [18324.113231] [18324.113240] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18324.113259] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18324.119610] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18324.119766] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18324.119775] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18324.119777] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18324.119793] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18324.119798] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18324.119803] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [18324.119806] RDX: 0000000000003d9c RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [18324.119808] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [18324.119811] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [18324.119814] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18324.119817] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [18324.119820] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18324.119823] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18324.119833] Call Trace: [18324.119838] [18324.119845] ? show_regs.cold.9+0x22/0x2f [18324.119855] ? watchdog_timer_fn.cold.6+0x62/0x10a [18324.119862] ? __hrtimer_run_queues+0x1c2/0x4d0 [18324.119867] ? watchdog+0x40/0x40 [18324.119872] ? hrtimer_interrupt+0x128/0x2c0 [18324.119877] ? smp_apic_timer_interrupt+0xa6/0x270 [18324.119885] ? apic_timer_interrupt+0xf/0x20 [18324.119890] [18324.119893] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18324.119899] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18324.119903] do_raw_spin_lock+0x177/0x190 [18324.119909] _raw_spin_lock+0x21/0x30 [18324.119916] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18324.137263] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18324.139299] obd_mod_free+0x46/0x140 [ptlrpc] [18324.139780] mdc_commit_open+0x91/0xb0 [mdc] [18324.139824] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18324.140030] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18324.140221] after_reply+0xe2c/0x1780 [ptlrpc] [18324.166842] RAX: 000000000000331b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18324.171239] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18324.180885] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [18324.180897] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [18324.180899] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [18324.180901] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [18324.180904] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [18324.180908] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18324.180909] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18324.180919] Call Trace: [18324.184545] ? __next_timer_interrupt+0x160/0x160 [18324.184567] ptlrpcd+0xa82/0xe40 [ptlrpc] [18324.185068] ? do_wait_intr+0xf0/0xf0 [18324.185079] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18324.185348] kthread+0x1d1/0x200 [18324.185355] ? set_kthread_struct+0x70/0x70 [18324.185364] ret_from_fork+0x1f/0x30 [18324.799560] [18324.801573] ? show_regs.cold.9+0x22/0x2f [18324.811648] ? watchdog_timer_fn.cold.6+0x62/0x10a [18324.815443] ? __hrtimer_run_queues+0x1c2/0x4d0 [18324.818736] ? watchdog+0x40/0x40 [18324.822934] ? hrtimer_interrupt+0x128/0x2c0 [18324.828528] ? smp_apic_timer_interrupt+0xa6/0x270 [18324.838608] ? apic_timer_interrupt+0xf/0x20 [18324.840637] [18324.841703] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18324.845161] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18324.848521] do_raw_spin_lock+0x177/0x190 [18324.856354] _raw_spin_lock+0x21/0x30 [18324.857618] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18324.862521] process_one_work+0x2c8/0x700 [18324.864506] worker_thread+0x296/0x6e0 [18324.867180] ? rescuer_thread+0x570/0x570 [18324.870706] kthread+0x1d1/0x200 [18324.872860] ? set_kthread_struct+0x70/0x70 [18324.881552] ret_from_fork+0x1f/0x30 [18329.570054] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16398s! [18329.574015] Showing busy workqueues and worker pools: [18329.576547] workqueue events: flags=0x0 [18329.578860] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18329.581465] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [18329.585466] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18329.588877] pending: kfree_rcu_monitor [18329.590868] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18329.594156] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [18329.599438] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18329.602754] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [18329.606446] workqueue events_long: flags=0x0 [18329.608733] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18329.611876] pending: lu_objects_flush [obdclass] [18329.615480] workqueue events_unbound: flags=0x2 [18329.617720] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [18329.620862] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [18329.625910] workqueue events_power_efficient: flags=0x80 [18329.628412] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18329.631922] pending: do_cache_clean, neigh_periodic_work, check_lifetime [18329.635854] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18329.639770] pending: neigh_periodic_work [18329.642474] workqueue mm_percpu_wq: flags=0x8 [18329.644719] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18329.648005] pending: vmstat_update, lru_add_drain_per_cpu [18329.651553] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18329.654820] pending: vmstat_update [18329.656606] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18329.659921] pending: lru_add_drain_per_cpu [18329.662121] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [18329.666179] pending: lru_add_drain_per_cpu BAR(48) [18329.668498] workqueue writeback: flags=0x4a [18329.670849] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [18329.673954] pending: wb_workfn [18329.675533] workqueue xprtiod: flags=0xa [18329.681556] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [18329.686932] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [18329.694203] workqueue ipv6_addrconf: flags=0x40008 [18329.699639] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18329.705720] pending: addrconf_verify_work [18329.708391] workqueue knbd0-recv: flags=0x1a [18329.712894] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18329.718247] in-flight: 658:recv_work [18329.722889] workqueue ptlrpc_pinger: flags=0x2 [18329.725748] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18329.738060] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [18329.748404] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16399s workers=4 idle: 32984 37 8 [18329.758204] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18314s workers=2 idle: 661 [18340.058798] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [18340.062450] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18340.109076] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18340.121789] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18340.127313] RIP: 0010:native_safe_halt+0x2e/0x50 [18340.129755] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18340.140154] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18340.147851] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18340.157870] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [18340.164763] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [18340.168872] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [18340.176034] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [18340.184864] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [18340.197801] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18340.201709] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [18340.212305] Call Trace: [18340.213454] [18340.214795] ? show_regs.cold.9+0x22/0x2f [18340.216198] ? watchdog_timer_fn.cold.6+0x62/0x10a [18340.218640] ? __hrtimer_run_queues+0x1c2/0x4d0 [18340.220406] ? watchdog+0x40/0x40 [18340.222467] ? hrtimer_interrupt+0x128/0x2c0 [18340.224617] ? smp_apic_timer_interrupt+0xa6/0x270 [18340.226680] ? apic_timer_interrupt+0xf/0x20 [18340.229903] [18340.231176] ? native_safe_halt+0x2e/0x50 [18340.233622] kvm_wait+0xcd/0xe0 [18340.234799] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18340.239889] do_raw_spin_lock+0x177/0x190 [18340.243343] _raw_spin_lock+0x21/0x30 [18340.247336] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18340.253389] ? do_raw_spin_unlock+0x75/0x190 [18340.256792] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18340.260197] ? do_wait_intr+0xf0/0xf0 [18340.264945] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18340.270535] mdc_close+0x53b/0xd70 [mdc] [18340.272102] lmv_close+0x18d/0x490 [lmv] [18340.274345] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [18340.280200] ll_close_inode_openhandle+0x493/0xea0 [lustre] [18340.283631] ll_md_real_close+0xa5/0x2a0 [lustre] [18340.293452] ll_md_close+0x274/0x800 [lustre] [18340.298333] ll_file_release+0x128/0xc50 [lustre] [18340.300197] __fput+0x10d/0x440 [18340.304282] ____fput+0x12/0x20 [18340.307374] task_work_run+0xc6/0x110 [18340.312360] exit_to_usermode_loop+0x1dd/0x1f0 [18340.317990] do_syscall_64+0x3d6/0x440 [18340.319625] entry_SYSCALL_64_after_hwframe+0x49/0xae [18340.321681] RIP: 0033:0x7f151bee3cd8 [18340.326634] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [18340.341969] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [18340.346910] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [18340.349511] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [18340.358418] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18340.363125] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18340.369962] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18352.055061] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [18352.056333] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [18352.057962] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18352.066384] Modules linked in: [18352.082287] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18352.082304] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18352.082307] RIP: 0010:native_safe_halt+0x2e/0x50 [18352.082321] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18352.082325] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18352.082330] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [18352.082332] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [18352.082334] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [18352.082336] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [18352.082339] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18352.082341] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [18352.082344] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18352.082347] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18352.082356] Call Trace: [18352.082362] [18352.082368] ? show_regs.cold.9+0x22/0x2f [18352.082378] ? watchdog_timer_fn.cold.6+0x62/0x10a [18352.082384] ? __hrtimer_run_queues+0x1c2/0x4d0 [18352.082388] ? watchdog+0x40/0x40 [18352.082392] ? hrtimer_interrupt+0x128/0x2c0 [18352.082396] ? smp_apic_timer_interrupt+0xa6/0x270 [18352.082402] ? apic_timer_interrupt+0xf/0x20 [18352.082407] [18352.082409] ? native_safe_halt+0x2e/0x50 [18352.082414] kvm_wait+0xcd/0xe0 [18352.082422] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18352.082429] do_raw_spin_lock+0x177/0x190 [18352.082433] _raw_spin_lock+0x21/0x30 [18352.082440] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18352.082716] obd_mod_free+0x46/0x140 [ptlrpc] [18352.082914] mdc_commit_open+0x91/0xb0 [mdc] [18352.082953] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18352.083151] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18352.085798] lustre(O) [18352.089910] after_reply+0xe2c/0x1780 [ptlrpc] [18352.098614] osp(O) [18352.100131] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18352.114282] ofd(O) [18352.117007] ? __next_timer_interrupt+0x160/0x160 [18352.119977] lod(O) [18352.122901] ptlrpcd+0xa82/0xe40 [ptlrpc] [18352.123232] ? do_wait_intr+0xf0/0xf0 [18352.123244] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18352.123446] kthread+0x1d1/0x200 [18352.123452] ? set_kthread_struct+0x70/0x70 [18352.123456] ret_from_fork+0x1f/0x30 [18352.310690] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18352.371975] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18352.384074] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18352.393978] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18352.399243] RIP: 0010:native_safe_halt+0x2e/0x50 [18352.400890] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18352.410520] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18352.414005] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18352.419023] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [18352.423996] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [18352.434879] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [18352.438835] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [18352.447647] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [18352.452739] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18352.466808] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18352.473894] Call Trace: [18352.480561] [18352.482713] ? show_regs.cold.9+0x22/0x2f [18352.488477] ? watchdog_timer_fn.cold.6+0x62/0x10a [18352.494347] ? __hrtimer_run_queues+0x1c2/0x4d0 [18352.499039] ? watchdog+0x40/0x40 [18352.506750] ? hrtimer_interrupt+0x128/0x2c0 [18352.510266] ? smp_apic_timer_interrupt+0xa6/0x270 [18352.520857] ? apic_timer_interrupt+0xf/0x20 [18352.530432] [18352.533581] ? native_safe_halt+0x2e/0x50 [18352.537696] kvm_wait+0xcd/0xe0 [18352.540939] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18352.542822] do_raw_spin_lock+0x177/0x190 [18352.553200] _raw_spin_lock+0x21/0x30 [18352.559905] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18352.567718] process_one_work+0x2c8/0x700 [18352.569613] worker_thread+0x296/0x6e0 [18352.570806] ? rescuer_thread+0x570/0x570 [18352.575481] kthread+0x1d1/0x200 [18352.578641] ? set_kthread_struct+0x70/0x70 [18352.581380] ret_from_fork+0x1f/0x30 [18356.059458] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [18356.071660] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18356.106346] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18356.114485] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18356.120960] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18356.123731] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18356.140885] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18356.181819] RAX: 0000000000002c1a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18356.196822] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [18356.202766] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [18356.207689] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [18356.215531] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [18356.221794] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [18356.224663] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18356.226794] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [18356.230236] Call Trace: [18356.231446] [18356.232352] ? show_regs.cold.9+0x22/0x2f [18356.234826] ? watchdog_timer_fn.cold.6+0x62/0x10a [18356.236979] ? __hrtimer_run_queues+0x1c2/0x4d0 [18356.239322] ? watchdog+0x40/0x40 [18356.240622] ? hrtimer_interrupt+0x128/0x2c0 [18356.242956] ? smp_apic_timer_interrupt+0xa6/0x270 [18356.250007] ? apic_timer_interrupt+0xf/0x20 [18356.251655] [18356.256067] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18356.258557] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18356.260868] do_raw_spin_lock+0x177/0x190 [18356.262771] _raw_spin_lock+0x21/0x30 [18356.264053] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18356.265930] ? do_raw_spin_unlock+0x75/0x190 [18356.267049] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18356.269291] ? do_wait_intr+0xf0/0xf0 [18356.270857] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18356.273307] mdc_reint+0x2d/0x110 [mdc] [18356.275002] mdc_unlink+0x658/0x870 [mdc] [18356.276374] lmv_unlink+0x34d/0xbd0 [lmv] [18356.277824] ll_unlink+0x24d/0x7f0 [lustre] [18356.279962] vfs_unlink+0x202/0x330 [18356.281479] do_unlinkat+0x2fa/0x460 [18356.283262] __x64_sys_unlink+0x37/0x50 [18356.284852] do_syscall_64+0xc1/0x440 [18356.286504] entry_SYSCALL_64_after_hwframe+0x49/0xae [18356.288638] RIP: 0033:0x7f151bdee10b [18356.290141] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [18356.297776] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [18356.300947] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [18356.304291] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [18356.307565] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18356.310477] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18356.313850] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18359.790549] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16429s! [18359.806797] Showing busy workqueues and worker pools: [18359.811860] workqueue events: flags=0x0 [18359.816623] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18359.830932] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [18359.838370] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18359.842622] pending: kfree_rcu_monitor [18359.849530] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18359.854691] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [18359.864669] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18359.873177] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [18359.879791] workqueue events_long: flags=0x0 [18359.885607] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18359.891653] pending: lu_objects_flush [obdclass] [18359.894588] workqueue events_unbound: flags=0x2 [18359.901213] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [18359.910064] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [18359.925823] workqueue events_power_efficient: flags=0x80 [18359.939899] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18359.942400] pending: do_cache_clean, neigh_periodic_work, check_lifetime [18359.948550] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18359.955620] pending: neigh_periodic_work [18359.963073] workqueue mm_percpu_wq: flags=0x8 [18359.965382] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18359.972203] pending: vmstat_update, lru_add_drain_per_cpu [18359.979227] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18359.983564] pending: vmstat_update [18359.986412] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18359.994472] pending: lru_add_drain_per_cpu [18359.996769] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [18360.015977] pending: lru_add_drain_per_cpu BAR(48) [18360.024715] workqueue writeback: flags=0x4a [18360.036673] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [18360.045591] pending: wb_workfn [18360.048430] workqueue xprtiod: flags=0xa [18360.055898] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [18360.065076] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [18360.072401] workqueue ipv6_addrconf: flags=0x40008 [18360.082057] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18360.097884] pending: addrconf_verify_work [18360.103569] workqueue knbd0-recv: flags=0x1a [18360.112455] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18360.119966] in-flight: 658:recv_work [18360.127978] workqueue ptlrpc_pinger: flags=0x2 [18360.134254] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18360.146950] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [18360.159512] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16429s workers=4 idle: 32984 37 8 [18360.170565] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18344s workers=2 idle: 661 [18364.987572] rcu: INFO: rcu_sched self-detected stall on CPU [18365.007447] rcu: 1-...!: (9906324 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [18365.015314] (t=16440340 jiffies g=249657 q=7500) [18365.017104] rcu: rcu_sched kthread starved for 16435292 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [18365.024571] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [18365.032330] rcu: RCU grace-period kthread stack dump: [18365.036718] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [18365.044704] Call Trace: [18365.047945] __schedule+0x351/0xcb0 [18365.052394] schedule+0xc0/0x180 [18365.054741] schedule_timeout+0xb4/0x190 [18365.056945] ? do_raw_spin_unlock+0x75/0x190 [18365.059336] ? __next_timer_interrupt+0x160/0x160 [18365.061577] ? prepare_to_swait_event+0xf3/0x140 [18365.063752] rcu_gp_kthread+0x8dd/0x15b0 [18365.067605] ? force_qs_rnp+0x2c0/0x2c0 [18365.069716] kthread+0x1d1/0x200 [18365.071615] ? set_kthread_struct+0x70/0x70 [18365.077811] ret_from_fork+0x1f/0x30 [18365.079136] rcu: Stack dump where RCU GP kthread last ran: [18365.088406] Sending NMI from CPU 1 to CPUs 0: [18365.091458] NMI backtrace for cpu 0 [18365.091465] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18365.091467] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18365.091469] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18365.091471] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18365.091473] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 [18365.091475] RAX: 0000000000005491 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18365.091477] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [18365.091478] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [18365.091479] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [18365.091480] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [18365.091481] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [18365.091483] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18365.091484] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [18365.091485] Call Trace: [18365.091486] [18365.091487] ? show_regs.cold.9+0x22/0x2f [18365.091488] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [18365.091489] ? nmi_cpu_backtrace_handler+0x15/0x40 [18365.091491] ? nmi_handle+0xbd/0x210 [18365.091492] ? default_do_nmi+0x4b/0x1b0 [18365.091493] ? do_nmi+0x280/0x380 [18365.091494] ? end_repeat_nmi+0x16/0x1f [18365.091495] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18365.091497] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18365.091498] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18365.091499] [18365.091500] do_raw_spin_lock+0x177/0x190 [18365.091501] _raw_spin_lock+0x21/0x30 [18365.091503] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18365.091504] ? do_raw_spin_unlock+0x75/0x190 [18365.091504] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18365.091506] ? do_wait_intr+0xf0/0xf0 [18365.091507] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18365.091508] mdc_reint+0x2d/0x110 [mdc] [18365.091509] mdc_unlink+0x658/0x870 [mdc] [18365.091510] lmv_unlink+0x34d/0xbd0 [lmv] [18365.091511] ll_unlink+0x24d/0x7f0 [lustre] [18365.091512] vfs_unlink+0x202/0x330 [18365.091513] do_unlinkat+0x2fa/0x460 [18365.091514] __x64_sys_unlink+0x37/0x50 [18365.091515] do_syscall_64+0xc1/0x440 [18365.091516] entry_SYSCALL_64_after_hwframe+0x49/0xae [18365.091517] RIP: 0033:0x7f151bdee10b [18365.091518] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [18365.091520] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [18365.091523] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [18365.091524] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [18365.091537] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18365.091539] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18365.091540] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18365.091792] NMI backtrace for cpu 1 [18365.424943] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18365.437178] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18365.441520] Call Trace: [18365.442698] [18365.443709] ? dump_stack+0xbb/0x10e [18365.445561] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [18365.447856] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [18365.450224] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [18365.463132] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [18365.465726] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [18365.468326] ? rcu_dump_cpu_stacks+0x11b/0x171 [18365.473879] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [18365.482771] ? update_process_times+0x6d/0xc0 [18365.490207] ? tick_sched_handle.isra.18+0x27/0x90 [18365.493548] ? tick_sched_timer+0x53/0xb0 [18365.498990] ? __hrtimer_run_queues+0x1c2/0x4d0 [18365.507262] ? tick_sched_handle.isra.18+0x90/0x90 [18365.510997] ? hrtimer_interrupt+0x128/0x2c0 [18365.515475] ? smp_apic_timer_interrupt+0xa6/0x270 [18365.517187] ? apic_timer_interrupt+0xf/0x20 [18365.533996] [18365.536534] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18365.540673] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18365.553530] ? do_raw_spin_lock+0x177/0x190 [18365.555649] ? _raw_spin_lock+0x21/0x30 [18365.564002] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18365.570138] ? obd_mod_free+0x46/0x140 [ptlrpc] [18365.581560] ? mdc_commit_open+0x91/0xb0 [mdc] [18365.589506] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18365.594292] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18365.600287] ? after_reply+0xe2c/0x1780 [ptlrpc] [18365.605223] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18365.610505] ? __next_timer_interrupt+0x160/0x160 [18365.613338] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [18365.616227] ? do_wait_intr+0xf0/0xf0 [18365.618210] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18365.624713] ? kthread+0x1d1/0x200 [18365.626477] ? set_kthread_struct+0x70/0x70 [18365.632819] ? ret_from_fork+0x1f/0x30 [18368.064282] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [18368.079722] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18368.153613] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18368.170306] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18368.177129] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18368.179251] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18368.202330] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18368.210591] RAX: 0000000000005a68 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18368.222579] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [18368.235365] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [18368.241441] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [18368.254138] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [18368.265208] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [18368.275951] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18368.285196] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [18368.291664] Call Trace: [18368.293171] [18368.294620] ? show_regs.cold.9+0x22/0x2f [18368.296958] ? watchdog_timer_fn.cold.6+0x62/0x10a [18368.308121] ? __hrtimer_run_queues+0x1c2/0x4d0 [18368.316892] ? watchdog+0x40/0x40 [18368.328317] ? hrtimer_interrupt+0x128/0x2c0 [18368.333651] ? smp_apic_timer_interrupt+0xa6/0x270 [18368.340941] ? apic_timer_interrupt+0xf/0x20 [18368.343584] [18368.345637] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18368.347936] do_raw_spin_lock+0x177/0x190 [18368.355015] _raw_spin_lock+0x21/0x30 [18368.356652] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18368.363501] ? do_raw_spin_unlock+0x75/0x190 [18368.373247] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18368.377615] ? do_wait_intr+0xf0/0xf0 [18368.379766] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18368.385245] mdc_close+0x53b/0xd70 [mdc] [18368.387388] lmv_close+0x18d/0x490 [lmv] [18368.396519] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [18368.402794] ll_close_inode_openhandle+0x493/0xea0 [lustre] [18368.411379] ll_md_real_close+0xa5/0x2a0 [lustre] [18368.414152] ll_md_close+0x274/0x800 [lustre] [18368.418745] ll_file_release+0x128/0xc50 [lustre] [18368.420606] __fput+0x10d/0x440 [18368.423162] ____fput+0x12/0x20 [18368.424448] task_work_run+0xc6/0x110 [18368.432551] exit_to_usermode_loop+0x1dd/0x1f0 [18368.437491] do_syscall_64+0x3d6/0x440 [18368.440200] entry_SYSCALL_64_after_hwframe+0x49/0xae [18368.446527] RIP: 0033:0x7f151bee3cd8 [18368.456508] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [18368.490808] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [18368.497508] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [18368.504731] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [18368.510357] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18368.516390] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18368.524270] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18380.056049] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [18380.062518] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18380.085762] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18380.091420] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18380.095937] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18380.099333] RIP: 0010:native_safe_halt+0x2e/0x50 [18380.101595] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18380.111290] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18380.115233] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18380.119284] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [18380.122974] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [18380.126690] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [18380.130425] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [18380.135027] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [18380.139063] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18380.142039] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18380.145840] Call Trace: [18380.148478] [18380.149510] ? show_regs.cold.9+0x22/0x2f [18380.151635] ? watchdog_timer_fn.cold.6+0x62/0x10a [18380.154440] ? __hrtimer_run_queues+0x1c2/0x4d0 [18380.156395] ? watchdog+0x40/0x40 [18380.157573] ? hrtimer_interrupt+0x128/0x2c0 [18380.159125] ? smp_apic_timer_interrupt+0xa6/0x270 [18380.161518] ? apic_timer_interrupt+0xf/0x20 [18380.163683] [18380.164678] ? native_safe_halt+0x2e/0x50 [18380.166963] kvm_wait+0xcd/0xe0 [18380.168604] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18380.171469] do_raw_spin_lock+0x177/0x190 [18380.173287] _raw_spin_lock+0x21/0x30 [18380.175255] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18380.178359] process_one_work+0x2c8/0x700 [18380.180309] worker_thread+0x296/0x6e0 [18380.182274] ? rescuer_thread+0x570/0x570 [18380.184476] kthread+0x1d1/0x200 [18380.186074] ? set_kthread_struct+0x70/0x70 [18380.188060] ret_from_fork+0x1f/0x30 [18390.496073] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16459s! [18390.500037] Showing busy workqueues and worker pools: [18390.502607] workqueue events: flags=0x0 [18390.504484] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18390.508144] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [18390.513190] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18390.517406] pending: kfree_rcu_monitor [18390.519947] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18390.523320] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [18390.527918] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18390.531454] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [18390.535860] workqueue events_long: flags=0x0 [18390.538123] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18390.541937] pending: lu_objects_flush [obdclass] [18390.544958] workqueue events_unbound: flags=0x2 [18390.547418] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [18390.551208] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [18390.556528] workqueue events_power_efficient: flags=0x80 [18390.559428] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18390.563024] pending: do_cache_clean, neigh_periodic_work, check_lifetime [18390.566676] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18390.570346] pending: neigh_periodic_work [18390.572353] workqueue mm_percpu_wq: flags=0x8 [18390.574257] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18390.577867] pending: vmstat_update, lru_add_drain_per_cpu [18390.580432] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18390.583691] pending: vmstat_update [18390.585684] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18390.588816] pending: lru_add_drain_per_cpu [18390.590972] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [18390.594072] pending: lru_add_drain_per_cpu BAR(48) [18390.596310] workqueue writeback: flags=0x4a [18390.598405] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [18390.601158] pending: wb_workfn [18390.602559] workqueue xprtiod: flags=0xa [18390.604312] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [18390.607094] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [18390.610624] workqueue ipv6_addrconf: flags=0x40008 [18390.612613] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18390.615840] pending: addrconf_verify_work [18390.617932] workqueue knbd0-recv: flags=0x1a [18390.620026] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18390.623083] in-flight: 658:recv_work [18390.624965] workqueue ptlrpc_pinger: flags=0x2 [18390.626750] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18390.629822] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [18390.632578] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16460s workers=4 idle: 32984 37 8 [18390.636342] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18375s workers=2 idle: 661 [18392.056247] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [18392.069034] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18392.114687] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18392.121655] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18392.132321] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18392.135431] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18392.156113] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18392.163794] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [18392.168631] RDX: 0000000000007b47 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [18392.174207] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [18392.180215] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [18392.189406] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18392.196134] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [18392.199820] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18392.205194] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18392.212080] Call Trace: [18392.213259] [18392.213869] ? show_regs.cold.9+0x22/0x2f [18392.215426] ? watchdog_timer_fn.cold.6+0x62/0x10a [18392.221295] ? __hrtimer_run_queues+0x1c2/0x4d0 [18392.223407] ? watchdog+0x40/0x40 [18392.224975] ? hrtimer_interrupt+0x128/0x2c0 [18392.227035] ? smp_apic_timer_interrupt+0xa6/0x270 [18392.228854] ? apic_timer_interrupt+0xf/0x20 [18392.230977] [18392.232260] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18392.234115] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18392.236761] do_raw_spin_lock+0x177/0x190 [18392.238056] _raw_spin_lock+0x21/0x30 [18392.239430] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18392.242154] obd_mod_free+0x46/0x140 [ptlrpc] [18392.244192] mdc_commit_open+0x91/0xb0 [mdc] [18392.246217] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18392.248814] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18392.251372] after_reply+0xe2c/0x1780 [ptlrpc] [18392.253965] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18392.256531] ? __next_timer_interrupt+0x160/0x160 [18392.258446] ptlrpcd+0xa82/0xe40 [ptlrpc] [18392.260447] ? do_wait_intr+0xf0/0xf0 [18392.262059] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18392.264345] kthread+0x1d1/0x200 [18392.265978] ? set_kthread_struct+0x70/0x70 [18392.267575] ret_from_fork+0x1f/0x30 [18396.057585] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [18396.064825] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18396.112863] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18396.123483] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18396.128057] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18396.138147] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18396.168692] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18396.171214] RAX: 0000000000002d98 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18396.180935] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [18396.185026] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [18396.188469] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [18396.191346] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [18396.194912] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [18396.198472] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18396.201134] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [18396.205484] Call Trace: [18396.206507] [18396.207352] ? show_regs.cold.9+0x22/0x2f [18396.209223] ? watchdog_timer_fn.cold.6+0x62/0x10a [18396.210727] ? __hrtimer_run_queues+0x1c2/0x4d0 [18396.212019] ? watchdog+0x40/0x40 [18396.213450] ? hrtimer_interrupt+0x128/0x2c0 [18396.215156] ? smp_apic_timer_interrupt+0xa6/0x270 [18396.217254] ? apic_timer_interrupt+0xf/0x20 [18396.219011] [18396.220177] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18396.222434] do_raw_spin_lock+0x177/0x190 [18396.224047] _raw_spin_lock+0x21/0x30 [18396.225493] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18396.227926] ? do_raw_spin_unlock+0x75/0x190 [18396.229481] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18396.231580] ? do_wait_intr+0xf0/0xf0 [18396.233157] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18396.235369] mdc_close+0x53b/0xd70 [mdc] [18396.236913] lmv_close+0x18d/0x490 [lmv] [18396.238415] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [18396.240758] ll_close_inode_openhandle+0x493/0xea0 [lustre] [18396.243884] ll_md_real_close+0xa5/0x2a0 [lustre] [18396.245489] ll_md_close+0x274/0x800 [lustre] [18396.247377] ll_file_release+0x128/0xc50 [lustre] [18396.249505] __fput+0x10d/0x440 [18396.250881] ____fput+0x12/0x20 [18396.252109] task_work_run+0xc6/0x110 [18396.253671] exit_to_usermode_loop+0x1dd/0x1f0 [18396.255219] do_syscall_64+0x3d6/0x440 [18396.256054] entry_SYSCALL_64_after_hwframe+0x49/0xae [18396.257326] RIP: 0033:0x7f151bee3cd8 [18396.258419] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [18396.265754] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [18396.268819] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [18396.271946] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [18396.274722] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18396.276667] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18396.279398] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18408.057071] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [18408.065853] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18408.132412] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18408.146353] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18408.157264] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18408.166306] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18408.171708] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18408.188464] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18408.193329] RAX: 0000000000006c15 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18408.198064] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [18408.213952] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [18408.230876] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [18408.240268] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [18408.247033] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [18408.251178] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18408.257588] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18408.266238] Call Trace: [18408.267217] [18408.271341] ? show_regs.cold.9+0x22/0x2f [18408.274161] ? watchdog_timer_fn.cold.6+0x62/0x10a [18408.279538] ? __hrtimer_run_queues+0x1c2/0x4d0 [18408.287749] ? watchdog+0x40/0x40 [18408.289001] ? hrtimer_interrupt+0x128/0x2c0 [18408.290579] ? smp_apic_timer_interrupt+0xa6/0x270 [18408.302239] ? apic_timer_interrupt+0xf/0x20 [18408.306932] [18408.307971] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18408.318790] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18408.321376] do_raw_spin_lock+0x177/0x190 [18408.327874] _raw_spin_lock+0x21/0x30 [18408.331479] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18408.335812] process_one_work+0x2c8/0x700 [18408.340439] worker_thread+0x296/0x6e0 [18408.346357] ? rescuer_thread+0x570/0x570 [18408.352014] kthread+0x1d1/0x200 [18408.356540] ? set_kthread_struct+0x70/0x70 [18408.362336] ret_from_fork+0x1f/0x30 [18416.052021] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [18416.060629] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18416.102298] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18416.111916] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18416.122297] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18416.125323] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18416.134328] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [18416.140386] RAX: 000000000000578a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18416.144868] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [18416.149846] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [18416.154438] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [18416.158384] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [18416.162225] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [18416.166803] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18416.170133] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [18416.172758] Call Trace: [18416.173675] [18416.174407] ? show_regs.cold.9+0x22/0x2f [18416.177605] ? watchdog_timer_fn.cold.6+0x62/0x10a [18416.180663] ? __hrtimer_run_queues+0x1c2/0x4d0 [18416.183040] ? watchdog+0x40/0x40 [18416.185748] ? hrtimer_interrupt+0x128/0x2c0 [18416.188659] ? smp_apic_timer_interrupt+0xa6/0x270 [18416.191775] ? apic_timer_interrupt+0xf/0x20 [18416.195073] [18416.196599] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18416.199781] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18416.202743] do_raw_spin_lock+0x177/0x190 [18416.206994] _raw_spin_lock+0x21/0x30 [18416.212641] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18416.217468] ? do_raw_spin_unlock+0x75/0x190 [18416.220748] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18416.224970] ? do_wait_intr+0xf0/0xf0 [18416.227829] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18416.234029] mdc_reint+0x2d/0x110 [mdc] [18416.236386] mdc_unlink+0x658/0x870 [mdc] [18416.239992] lmv_unlink+0x34d/0xbd0 [lmv] [18416.242644] ll_unlink+0x24d/0x7f0 [lustre] [18416.247889] vfs_unlink+0x202/0x330 [18416.250275] do_unlinkat+0x2fa/0x460 [18416.257472] __x64_sys_unlink+0x37/0x50 [18416.260806] do_syscall_64+0xc1/0x440 [18416.263864] entry_SYSCALL_64_after_hwframe+0x49/0xae [18416.269120] RIP: 0033:0x7f151bdee10b [18416.272965] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [18416.287671] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [18416.293220] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [18416.298790] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [18416.303920] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18416.310622] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18416.315727] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18420.055743] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [18420.068245] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18420.141056] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18420.153183] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18420.159078] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18420.166290] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18420.186999] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18420.191431] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [18420.196102] RDX: 0000000000001090 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [18420.200206] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [18420.233165] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [18420.249944] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18420.265141] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [18420.276402] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18420.287031] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18420.293225] Call Trace: [18420.296798] [18420.299196] ? show_regs.cold.9+0x22/0x2f [18420.300183] ? watchdog_timer_fn.cold.6+0x62/0x10a [18420.307945] ? __hrtimer_run_queues+0x1c2/0x4d0 [18420.318347] ? watchdog+0x40/0x40 [18420.322475] ? hrtimer_interrupt+0x128/0x2c0 [18420.327241] ? smp_apic_timer_interrupt+0xa6/0x270 [18420.337196] ? apic_timer_interrupt+0xf/0x20 [18420.338505] [18420.339129] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18420.355424] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18420.357059] do_raw_spin_lock+0x177/0x190 [18420.368579] _raw_spin_lock+0x21/0x30 [18420.369737] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18420.371459] obd_mod_free+0x46/0x140 [ptlrpc] [18420.389720] mdc_commit_open+0x91/0xb0 [mdc] [18420.392989] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18420.401667] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18420.408918] after_reply+0xe2c/0x1780 [ptlrpc] [18420.412986] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18420.420110] ? __next_timer_interrupt+0x160/0x160 [18420.423223] ptlrpcd+0xa82/0xe40 [ptlrpc] [18420.426064] ? do_wait_intr+0xf0/0xf0 [18420.427864] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18420.432468] kthread+0x1d1/0x200 [18420.434934] ? set_kthread_struct+0x70/0x70 [18420.442449] ret_from_fork+0x1f/0x30 [18420.704081] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16490s! [18420.708250] Showing busy workqueues and worker pools: [18420.711688] workqueue events: flags=0x0 [18420.714674] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18420.717891] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [18420.725753] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18420.731810] pending: kfree_rcu_monitor [18420.736789] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18420.740678] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [18420.748855] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18420.757823] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [18420.774645] workqueue events_long: flags=0x0 [18420.780680] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18420.790503] pending: lu_objects_flush [obdclass] [18420.796274] workqueue events_unbound: flags=0x2 [18420.800194] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [18420.804284] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [18420.818613] workqueue events_power_efficient: flags=0x80 [18420.823750] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18420.829364] pending: do_cache_clean, neigh_periodic_work, check_lifetime [18420.837335] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18420.844255] pending: neigh_periodic_work [18420.851481] workqueue mm_percpu_wq: flags=0x8 [18420.859816] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18420.870288] pending: vmstat_update, lru_add_drain_per_cpu [18420.882166] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18420.886202] pending: vmstat_update [18420.892302] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18420.900789] pending: lru_add_drain_per_cpu [18420.906740] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [18420.927501] pending: lru_add_drain_per_cpu BAR(48) [18420.933258] workqueue writeback: flags=0x4a [18420.939981] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [18420.947627] pending: wb_workfn [18420.950677] workqueue xprtiod: flags=0xa [18420.967307] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [18420.985275] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [18420.998544] workqueue ipv6_addrconf: flags=0x40008 [18421.007816] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18421.014822] pending: addrconf_verify_work [18421.023585] workqueue knbd0-recv: flags=0x1a [18421.027810] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18421.043321] in-flight: 658:recv_work [18421.053973] workqueue ptlrpc_pinger: flags=0x2 [18421.065618] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18421.075850] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [18421.088151] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16490s workers=4 idle: 32984 37 8 [18421.101148] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18405s workers=2 idle: 661 [18424.057024] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [18424.059898] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18424.090692] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18424.099162] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18424.102724] RIP: 0010:native_safe_halt+0x2e/0x50 [18424.106663] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18424.113920] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18424.118708] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18424.122301] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [18424.126253] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [18424.130327] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [18424.137758] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [18424.143913] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [18424.149293] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18424.155129] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [18424.160341] Call Trace: [18424.161393] [18424.162149] ? show_regs.cold.9+0x22/0x2f [18424.164157] ? watchdog_timer_fn.cold.6+0x62/0x10a [18424.168185] ? __hrtimer_run_queues+0x1c2/0x4d0 [18424.171219] ? watchdog+0x40/0x40 [18424.172301] ? hrtimer_interrupt+0x128/0x2c0 [18424.174985] ? smp_apic_timer_interrupt+0xa6/0x270 [18424.177173] ? apic_timer_interrupt+0xf/0x20 [18424.179259] [18424.180551] ? native_safe_halt+0x2e/0x50 [18424.182779] kvm_wait+0xcd/0xe0 [18424.184266] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18424.187263] do_raw_spin_lock+0x177/0x190 [18424.189262] _raw_spin_lock+0x21/0x30 [18424.191077] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18424.194245] ? do_raw_spin_unlock+0x75/0x190 [18424.196580] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18424.199162] ? do_wait_intr+0xf0/0xf0 [18424.201577] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18424.204022] mdc_close+0x53b/0xd70 [mdc] [18424.206051] lmv_close+0x18d/0x490 [lmv] [18424.208053] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [18424.211533] ll_close_inode_openhandle+0x493/0xea0 [lustre] [18424.214922] ll_md_real_close+0xa5/0x2a0 [lustre] [18424.220183] ll_md_close+0x274/0x800 [lustre] [18424.225195] ll_file_release+0x128/0xc50 [lustre] [18424.227976] __fput+0x10d/0x440 [18424.230435] ____fput+0x12/0x20 [18424.232560] task_work_run+0xc6/0x110 [18424.234706] exit_to_usermode_loop+0x1dd/0x1f0 [18424.236769] do_syscall_64+0x3d6/0x440 [18424.238987] entry_SYSCALL_64_after_hwframe+0x49/0xae [18424.241219] RIP: 0033:0x7f151bee3cd8 [18424.243101] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [18424.259330] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [18424.263389] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [18424.266856] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [18424.271167] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18424.273071] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18424.276232] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18436.057983] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [18436.065623] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18436.097883] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18436.108649] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18436.116543] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18436.122611] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18436.127028] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18436.137431] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18436.141965] RAX: 0000000000006691 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18436.147667] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [18436.153669] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [18436.160088] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [18436.166676] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [18436.172695] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [18436.180079] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18436.184640] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18436.190177] Call Trace: [18436.191070] [18436.192415] ? show_regs.cold.9+0x22/0x2f [18436.194088] ? watchdog_timer_fn.cold.6+0x62/0x10a [18436.196658] ? __hrtimer_run_queues+0x1c2/0x4d0 [18436.198112] ? watchdog+0x40/0x40 [18436.199452] ? hrtimer_interrupt+0x128/0x2c0 [18436.201013] ? smp_apic_timer_interrupt+0xa6/0x270 [18436.208520] ? apic_timer_interrupt+0xf/0x20 [18436.213406] [18436.215154] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18436.217648] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18436.222412] do_raw_spin_lock+0x177/0x190 [18436.224965] _raw_spin_lock+0x21/0x30 [18436.227106] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18436.233061] process_one_work+0x2c8/0x700 [18436.235983] worker_thread+0x296/0x6e0 [18436.239490] ? rescuer_thread+0x570/0x570 [18436.241426] kthread+0x1d1/0x200 [18436.242887] ? set_kthread_struct+0x70/0x70 [18436.244204] ret_from_fork+0x1f/0x30 [18448.054662] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [18448.058518] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [18448.062673] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18448.064360] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18448.064374] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18448.064377] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18448.064400] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18448.064405] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18448.064409] RAX: 0000000000003e17 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18448.064411] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [18448.064414] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [18448.064415] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [18448.064418] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [18448.064420] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [18448.064424] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18448.064426] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [18448.064434] Call Trace: [18448.064440] [18448.064446] ? show_regs.cold.9+0x22/0x2f [18448.064456] ? watchdog_timer_fn.cold.6+0x62/0x10a [18448.064468] ? __hrtimer_run_queues+0x1c2/0x4d0 [18448.064473] ? watchdog+0x40/0x40 [18448.064477] ? hrtimer_interrupt+0x128/0x2c0 [18448.064481] ? smp_apic_timer_interrupt+0xa6/0x270 [18448.064490] ? apic_timer_interrupt+0xf/0x20 [18448.064495] [18448.064497] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18448.064502] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18448.064507] do_raw_spin_lock+0x177/0x190 [18448.064512] _raw_spin_lock+0x21/0x30 [18448.064518] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18448.074321] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18448.074396] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18448.074403] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18448.074406] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18448.074419] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18448.074423] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18448.074428] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [18448.074431] RDX: 00000000000042f0 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [18448.074433] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [18448.074436] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [18448.074439] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18448.074442] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [18448.074445] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18448.074447] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18448.074455] Call Trace: [18448.074461] [18448.074466] ? show_regs.cold.9+0x22/0x2f [18448.074476] ? watchdog_timer_fn.cold.6+0x62/0x10a [18448.074483] ? __hrtimer_run_queues+0x1c2/0x4d0 [18448.074488] ? watchdog+0x40/0x40 [18448.074493] ? hrtimer_interrupt+0x128/0x2c0 [18448.074496] ? smp_apic_timer_interrupt+0xa6/0x270 [18448.074506] ? apic_timer_interrupt+0xf/0x20 [18448.074511] [18448.074514] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18448.074520] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18448.074525] do_raw_spin_lock+0x177/0x190 [18448.074530] _raw_spin_lock+0x21/0x30 [18448.074537] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18448.116609] ? do_raw_spin_unlock+0x75/0x190 [18448.116690] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18448.119180] ? do_wait_intr+0xf0/0xf0 [18448.119194] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18448.119419] mdc_reint+0x2d/0x110 [mdc] [18448.119460] mdc_unlink+0x658/0x870 [mdc] [18448.119486] lmv_unlink+0x34d/0xbd0 [lmv] [18448.119522] ll_unlink+0x24d/0x7f0 [lustre] [18448.119639] vfs_unlink+0x202/0x330 [18448.119648] do_unlinkat+0x2fa/0x460 [18448.119652] __x64_sys_unlink+0x37/0x50 [18448.119655] do_syscall_64+0xc1/0x440 [18448.119662] entry_SYSCALL_64_after_hwframe+0x49/0xae [18448.119670] RIP: 0033:0x7f151bdee10b [18448.119677] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [18448.119681] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [18448.119686] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [18448.119688] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [18448.119691] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18448.119693] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18448.119695] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18448.454009] obd_mod_free+0x46/0x140 [ptlrpc] [18448.456194] mdc_commit_open+0x91/0xb0 [mdc] [18448.458616] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18448.460655] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18448.464411] after_reply+0xe2c/0x1780 [ptlrpc] [18448.466531] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18448.468637] ? __next_timer_interrupt+0x160/0x160 [18448.470613] ptlrpcd+0xa82/0xe40 [ptlrpc] [18448.472201] ? do_wait_intr+0xf0/0xf0 [18448.476890] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18448.480356] kthread+0x1d1/0x200 [18448.481674] ? set_kthread_struct+0x70/0x70 [18448.484513] ret_from_fork+0x1f/0x30 [18451.426045] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16520s! [18451.434889] Showing busy workqueues and worker pools: [18451.438857] workqueue events: flags=0x0 [18451.439770] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18451.441500] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [18451.449916] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18451.456642] pending: kfree_rcu_monitor [18451.458924] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18451.464458] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [18451.471564] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18451.475852] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [18451.482042] workqueue events_long: flags=0x0 [18451.484819] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18451.488453] pending: lu_objects_flush [obdclass] [18451.490638] workqueue events_unbound: flags=0x2 [18451.491923] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [18451.495556] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [18451.498906] workqueue events_power_efficient: flags=0x80 [18451.501546] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18451.504585] pending: do_cache_clean, neigh_periodic_work, check_lifetime [18451.507864] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18451.510061] pending: neigh_periodic_work [18451.511468] workqueue mm_percpu_wq: flags=0x8 [18451.512737] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18451.516540] pending: vmstat_update, lru_add_drain_per_cpu [18451.519992] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18451.526837] pending: vmstat_update [18451.528897] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18451.534037] pending: lru_add_drain_per_cpu [18451.536798] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [18451.543717] pending: lru_add_drain_per_cpu BAR(48) [18451.545971] workqueue writeback: flags=0x4a [18451.547391] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [18451.551647] pending: wb_workfn [18451.554371] workqueue xprtiod: flags=0xa [18451.560887] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [18451.565479] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [18451.569640] workqueue ipv6_addrconf: flags=0x40008 [18451.574586] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18451.576453] pending: addrconf_verify_work [18451.579641] workqueue knbd0-recv: flags=0x1a [18451.581660] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18451.585687] in-flight: 658:recv_work [18451.588647] workqueue ptlrpc_pinger: flags=0x2 [18451.594500] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18451.597781] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [18451.601397] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16521s workers=4 idle: 32984 37 8 [18451.608903] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18436s workers=2 idle: 661 [18452.058804] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [18452.068800] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18452.116549] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18452.123243] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18452.131823] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18452.139340] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18452.151128] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18452.155617] RAX: 0000000000000f31 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18452.158631] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [18452.161631] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [18452.164814] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [18452.168371] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [18452.171304] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [18452.174700] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18452.177019] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [18452.180467] Call Trace: [18452.183615] [18452.184722] ? show_regs.cold.9+0x22/0x2f [18452.186943] ? watchdog_timer_fn.cold.6+0x62/0x10a [18452.191682] ? __hrtimer_run_queues+0x1c2/0x4d0 [18452.194213] ? watchdog+0x40/0x40 [18452.195661] ? hrtimer_interrupt+0x128/0x2c0 [18452.197285] ? smp_apic_timer_interrupt+0xa6/0x270 [18452.206276] ? apic_timer_interrupt+0xf/0x20 [18452.209204] [18452.210703] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18452.212799] do_raw_spin_lock+0x177/0x190 [18452.214984] _raw_spin_lock+0x21/0x30 [18452.217419] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18452.222399] ? do_raw_spin_unlock+0x75/0x190 [18452.223753] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18452.231216] ? do_wait_intr+0xf0/0xf0 [18452.232992] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18452.238934] mdc_close+0x53b/0xd70 [mdc] [18452.241480] lmv_close+0x18d/0x490 [lmv] [18452.245013] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [18452.248636] ll_close_inode_openhandle+0x493/0xea0 [lustre] [18452.253856] ll_md_real_close+0xa5/0x2a0 [lustre] [18452.258484] ll_md_close+0x274/0x800 [lustre] [18452.262951] ll_file_release+0x128/0xc50 [lustre] [18452.266280] __fput+0x10d/0x440 [18452.268306] ____fput+0x12/0x20 [18452.269347] task_work_run+0xc6/0x110 [18452.271298] exit_to_usermode_loop+0x1dd/0x1f0 [18452.273402] do_syscall_64+0x3d6/0x440 [18452.276473] entry_SYSCALL_64_after_hwframe+0x49/0xae [18452.280976] RIP: 0033:0x7f151bee3cd8 [18452.286442] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [18452.301606] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [18452.308684] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [18452.314659] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [18452.320772] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18452.325926] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18452.330699] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18464.057162] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [18464.065470] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18464.106474] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18464.116737] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18464.125984] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18464.133654] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18464.137771] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18464.162281] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18464.167402] RAX: 0000000000006f81 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18464.176860] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [18464.188749] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [18464.197286] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [18464.201899] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [18464.207646] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [18464.222974] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18464.232654] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18464.236467] Call Trace: [18464.238415] [18464.244135] ? show_regs.cold.9+0x22/0x2f [18464.248486] ? watchdog_timer_fn.cold.6+0x62/0x10a [18464.255877] ? __hrtimer_run_queues+0x1c2/0x4d0 [18464.260303] ? watchdog+0x40/0x40 [18464.262205] ? hrtimer_interrupt+0x128/0x2c0 [18464.265379] ? smp_apic_timer_interrupt+0xa6/0x270 [18464.267882] ? apic_timer_interrupt+0xf/0x20 [18464.270517] [18464.271297] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18464.279798] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18464.294009] do_raw_spin_lock+0x177/0x190 [18464.297985] _raw_spin_lock+0x21/0x30 [18464.302307] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18464.308190] process_one_work+0x2c8/0x700 [18464.310387] worker_thread+0x296/0x6e0 [18464.312597] ? rescuer_thread+0x570/0x570 [18464.315259] kthread+0x1d1/0x200 [18464.320309] ? set_kthread_struct+0x70/0x70 [18464.323802] ret_from_fork+0x1f/0x30 [18476.060123] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [18476.066356] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [18476.072834] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18476.072908] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18476.072916] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18476.072919] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18476.072933] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18476.072937] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18476.072942] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [18476.072944] RDX: 0000000000007f87 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [18476.072946] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [18476.072949] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [18476.072951] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18476.072964] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [18476.072967] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18476.072970] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18476.072978] Call Trace: [18476.072983] [18476.072988] ? show_regs.cold.9+0x22/0x2f [18476.072996] ? watchdog_timer_fn.cold.6+0x62/0x10a [18476.073003] ? __hrtimer_run_queues+0x1c2/0x4d0 [18476.073008] ? watchdog+0x40/0x40 [18476.073012] ? hrtimer_interrupt+0x128/0x2c0 [18476.073016] ? smp_apic_timer_interrupt+0xa6/0x270 [18476.073025] ? apic_timer_interrupt+0xf/0x20 [18476.073030] [18476.073033] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18476.073038] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18476.073042] do_raw_spin_lock+0x177/0x190 [18476.073047] _raw_spin_lock+0x21/0x30 [18476.073052] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18476.073331] obd_mod_free+0x46/0x140 [ptlrpc] [18476.073540] mdc_commit_open+0x91/0xb0 [mdc] [18476.073585] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18476.073781] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18476.073993] after_reply+0xe2c/0x1780 [ptlrpc] [18476.074193] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18476.074391] ? __next_timer_interrupt+0x160/0x160 [18476.074400] ptlrpcd+0xa82/0xe40 [ptlrpc] [18476.074623] ? do_wait_intr+0xf0/0xf0 [18476.074632] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18476.074828] kthread+0x1d1/0x200 [18476.074834] ? set_kthread_struct+0x70/0x70 [18476.074838] ret_from_fork+0x1f/0x30 [18476.751142] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18476.823011] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18476.851117] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18476.879816] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18476.898815] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18476.941667] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18476.967761] RAX: 0000000000007d88 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18476.982708] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [18476.999698] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [18477.018754] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [18477.033142] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [18477.035687] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [18477.076284] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18477.083463] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [18477.097217] Call Trace: [18477.098102] [18477.098767] ? show_regs.cold.9+0x22/0x2f [18477.114018] ? watchdog_timer_fn.cold.6+0x62/0x10a [18477.117485] ? __hrtimer_run_queues+0x1c2/0x4d0 [18477.122310] ? watchdog+0x40/0x40 [18477.124699] ? hrtimer_interrupt+0x128/0x2c0 [18477.134785] ? smp_apic_timer_interrupt+0xa6/0x270 [18477.143330] ? apic_timer_interrupt+0xf/0x20 [18477.154118] [18477.154777] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18477.156102] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18477.170142] do_raw_spin_lock+0x177/0x190 [18477.177066] _raw_spin_lock+0x21/0x30 [18477.182307] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18477.188762] ? do_raw_spin_unlock+0x75/0x190 [18477.198269] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18477.206490] ? do_wait_intr+0xf0/0xf0 [18477.209660] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18477.231145] mdc_reint+0x2d/0x110 [mdc] [18477.241239] mdc_unlink+0x658/0x870 [mdc] [18477.245538] lmv_unlink+0x34d/0xbd0 [lmv] [18477.252118] ll_unlink+0x24d/0x7f0 [lustre] [18477.257259] vfs_unlink+0x202/0x330 [18477.260795] do_unlinkat+0x2fa/0x460 [18477.265107] __x64_sys_unlink+0x37/0x50 [18477.268450] do_syscall_64+0xc1/0x440 [18477.277167] entry_SYSCALL_64_after_hwframe+0x49/0xae [18477.284191] RIP: 0033:0x7f151bdee10b [18477.294928] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [18477.346172] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [18477.359768] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [18477.374674] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [18477.391394] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18477.401413] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18477.416358] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18480.064143] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [18480.076220] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18480.121781] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18480.127317] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18480.134702] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18480.137213] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18480.146345] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18480.149448] RAX: 000000000000322b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18480.153394] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [18480.157154] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [18480.161268] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [18480.164679] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [18480.168772] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [18480.173095] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18480.176108] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [18480.179379] Call Trace: [18480.180479] [18480.181445] ? show_regs.cold.9+0x22/0x2f [18480.184107] ? watchdog_timer_fn.cold.6+0x62/0x10a [18480.186396] ? __hrtimer_run_queues+0x1c2/0x4d0 [18480.188708] ? watchdog+0x40/0x40 [18480.190436] ? hrtimer_interrupt+0x128/0x2c0 [18480.192514] ? smp_apic_timer_interrupt+0xa6/0x270 [18480.195322] ? apic_timer_interrupt+0xf/0x20 [18480.197466] [18480.199014] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18480.201668] do_raw_spin_lock+0x177/0x190 [18480.204018] _raw_spin_lock+0x21/0x30 [18480.205716] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18480.208770] ? do_raw_spin_unlock+0x75/0x190 [18480.211209] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18480.214234] ? do_wait_intr+0xf0/0xf0 [18480.216265] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18480.219511] mdc_close+0x53b/0xd70 [mdc] [18480.221554] lmv_close+0x18d/0x490 [lmv] [18480.223824] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [18480.229274] ll_close_inode_openhandle+0x493/0xea0 [lustre] [18480.232808] ll_md_real_close+0xa5/0x2a0 [lustre] [18480.235600] ll_md_close+0x274/0x800 [lustre] [18480.238181] ll_file_release+0x128/0xc50 [lustre] [18480.240378] __fput+0x10d/0x440 [18480.242327] ____fput+0x12/0x20 [18480.243715] task_work_run+0xc6/0x110 [18480.245643] exit_to_usermode_loop+0x1dd/0x1f0 [18480.247816] do_syscall_64+0x3d6/0x440 [18480.249827] entry_SYSCALL_64_after_hwframe+0x49/0xae [18480.252714] RIP: 0033:0x7f151bee3cd8 [18480.254676] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [18480.263862] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [18480.267778] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [18480.271243] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [18480.274692] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18480.278163] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18480.282021] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18481.647735] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16551s! [18481.650713] Showing busy workqueues and worker pools: [18481.676165] workqueue events: flags=0x0 [18481.682076] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18481.693871] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [18481.709631] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18481.723163] pending: kfree_rcu_monitor [18481.726702] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18481.739693] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [18481.751677] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18481.758304] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [18481.767404] workqueue events_long: flags=0x0 [18481.770835] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18481.774573] pending: lu_objects_flush [obdclass] [18481.777396] workqueue events_unbound: flags=0x2 [18481.779458] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [18481.785584] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [18481.796630] workqueue events_power_efficient: flags=0x80 [18481.800953] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18481.806070] pending: do_cache_clean, neigh_periodic_work, check_lifetime [18481.815302] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18481.821265] pending: neigh_periodic_work [18481.824419] workqueue mm_percpu_wq: flags=0x8 [18481.833311] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18481.842460] pending: vmstat_update, lru_add_drain_per_cpu [18481.847318] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18481.854568] pending: vmstat_update [18481.856684] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18481.867287] pending: lru_add_drain_per_cpu [18481.877959] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [18481.885348] pending: lru_add_drain_per_cpu BAR(48) [18481.893889] workqueue writeback: flags=0x4a [18481.895307] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [18481.912516] pending: wb_workfn [18481.917678] workqueue xprtiod: flags=0xa [18481.924127] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [18481.940982] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [18481.951105] workqueue ipv6_addrconf: flags=0x40008 [18481.952708] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18481.961316] pending: addrconf_verify_work [18481.964130] workqueue knbd0-recv: flags=0x1a [18481.971667] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18481.975410] in-flight: 658:recv_work [18481.979557] workqueue ptlrpc_pinger: flags=0x2 [18481.982157] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18481.991770] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [18481.996363] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16551s workers=4 idle: 32984 37 8 [18482.012165] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18466s workers=2 idle: 661 [18492.063499] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [18492.080817] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18492.174600] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18492.188855] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18492.198935] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18492.204678] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18492.221580] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18492.255307] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18492.270577] RAX: 000000000000718e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18492.284377] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [18492.291076] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [18492.297162] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [18492.315201] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [18492.326925] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [18492.345352] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18492.349329] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18492.360554] Call Trace: [18492.363939] [18492.369182] ? show_regs.cold.9+0x22/0x2f [18492.371434] ? watchdog_timer_fn.cold.6+0x62/0x10a [18492.382347] ? __hrtimer_run_queues+0x1c2/0x4d0 [18492.398336] ? watchdog+0x40/0x40 [18492.403238] ? hrtimer_interrupt+0x128/0x2c0 [18492.426904] ? smp_apic_timer_interrupt+0xa6/0x270 [18492.428602] ? apic_timer_interrupt+0xf/0x20 [18492.443867] [18492.444652] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18492.446424] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18492.462197] do_raw_spin_lock+0x177/0x190 [18492.463679] _raw_spin_lock+0x21/0x30 [18492.483652] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18492.492963] process_one_work+0x2c8/0x700 [18492.496330] worker_thread+0x296/0x6e0 [18492.503954] ? rescuer_thread+0x570/0x570 [18492.507364] kthread+0x1d1/0x200 [18492.513253] ? set_kthread_struct+0x70/0x70 [18492.520318] ret_from_fork+0x1f/0x30 [18504.055109] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [18504.059986] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18504.104285] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18504.111991] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18504.115480] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18504.118281] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18504.130560] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18504.135050] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [18504.137759] RDX: 0000000000007e8d RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [18504.143411] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [18504.147877] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [18504.151045] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18504.153171] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [18504.158375] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18504.163574] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18504.170923] Call Trace: [18504.173264] [18504.175060] ? show_regs.cold.9+0x22/0x2f [18504.178695] ? watchdog_timer_fn.cold.6+0x62/0x10a [18504.181142] ? __hrtimer_run_queues+0x1c2/0x4d0 [18504.183106] ? watchdog+0x40/0x40 [18504.184490] ? hrtimer_interrupt+0x128/0x2c0 [18504.186488] ? smp_apic_timer_interrupt+0xa6/0x270 [18504.188935] ? apic_timer_interrupt+0xf/0x20 [18504.191366] [18504.192451] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18504.195339] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18504.197908] do_raw_spin_lock+0x177/0x190 [18504.201107] _raw_spin_lock+0x21/0x30 [18504.202179] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18504.206570] obd_mod_free+0x46/0x140 [ptlrpc] [18504.214221] mdc_commit_open+0x91/0xb0 [mdc] [18504.217068] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18504.222448] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18504.226212] after_reply+0xe2c/0x1780 [ptlrpc] [18504.229005] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18504.236270] ? __next_timer_interrupt+0x160/0x160 [18504.239059] ptlrpcd+0xa82/0xe40 [ptlrpc] [18504.243513] ? do_wait_intr+0xf0/0xf0 [18504.245367] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18504.256759] kthread+0x1d1/0x200 [18504.260911] ? set_kthread_struct+0x70/0x70 [18504.262917] ret_from_fork+0x1f/0x30 [18508.054426] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [18508.061311] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18508.063339] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [18508.103919] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18508.103935] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18508.103939] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18508.103954] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18508.103958] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18508.103967] RAX: 0000000000006f0e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18508.103970] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [18508.103972] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [18508.103975] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [18508.103977] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [18508.103983] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [18508.103987] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18508.103989] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [18508.103999] Call Trace: [18508.104009] [18508.104016] ? show_regs.cold.9+0x22/0x2f [18508.104030] ? watchdog_timer_fn.cold.6+0x62/0x10a [18508.104038] ? __hrtimer_run_queues+0x1c2/0x4d0 [18508.104042] ? watchdog+0x40/0x40 [18508.104112] ? hrtimer_interrupt+0x128/0x2c0 [18508.104118] ? smp_apic_timer_interrupt+0xa6/0x270 [18508.104131] ? apic_timer_interrupt+0xf/0x20 [18508.104139] [18508.104142] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18508.104147] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18508.104151] do_raw_spin_lock+0x177/0x190 [18508.104159] _raw_spin_lock+0x21/0x30 [18508.104168] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18508.104604] ? do_raw_spin_unlock+0x75/0x190 [18508.104611] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18508.114461] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18508.114557] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18508.114566] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18508.114569] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18508.114587] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18508.114592] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18508.114597] RAX: 0000000000007107 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18508.114600] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [18508.114602] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [18508.114604] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [18508.114606] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [18508.114614] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [18508.114617] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18508.114620] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [18508.114628] Call Trace: [18508.114636] [18508.114650] ? show_regs.cold.9+0x22/0x2f [18508.114670] ? watchdog_timer_fn.cold.6+0x62/0x10a [18508.114678] ? __hrtimer_run_queues+0x1c2/0x4d0 [18508.114682] ? watchdog+0x40/0x40 [18508.114686] ? hrtimer_interrupt+0x128/0x2c0 [18508.114694] ? smp_apic_timer_interrupt+0xa6/0x270 [18508.114703] ? apic_timer_interrupt+0xf/0x20 [18508.114768] [18508.114771] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18508.114779] do_raw_spin_lock+0x177/0x190 [18508.114820] _raw_spin_lock+0x21/0x30 [18508.114836] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18508.125327] ? do_wait_intr+0xf0/0xf0 [18508.125352] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18508.125661] mdc_reint+0x2d/0x110 [mdc] [18508.125701] mdc_unlink+0x658/0x870 [mdc] [18508.125731] lmv_unlink+0x34d/0xbd0 [lmv] [18508.125765] ll_unlink+0x24d/0x7f0 [lustre] [18508.125883] vfs_unlink+0x202/0x330 [18508.125893] do_unlinkat+0x2fa/0x460 [18508.125898] __x64_sys_unlink+0x37/0x50 [18508.125903] do_syscall_64+0xc1/0x440 [18508.125909] entry_SYSCALL_64_after_hwframe+0x49/0xae [18508.125918] RIP: 0033:0x7f151bdee10b [18508.125926] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [18508.125931] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [18508.125936] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [18508.125939] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [18508.125941] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18508.125943] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18508.125946] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18508.614615] ? do_raw_spin_unlock+0x75/0x190 [18508.615948] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18508.626009] ? do_wait_intr+0xf0/0xf0 [18508.629037] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18508.635576] mdc_close+0x53b/0xd70 [mdc] [18508.641931] lmv_close+0x18d/0x490 [lmv] [18508.643435] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [18508.652486] ll_close_inode_openhandle+0x493/0xea0 [lustre] [18508.654622] ll_md_real_close+0xa5/0x2a0 [lustre] [18508.667635] ll_md_close+0x274/0x800 [lustre] [18508.675129] ll_file_release+0x128/0xc50 [lustre] [18508.679080] __fput+0x10d/0x440 [18508.683191] ____fput+0x12/0x20 [18508.684341] task_work_run+0xc6/0x110 [18508.691353] exit_to_usermode_loop+0x1dd/0x1f0 [18508.695817] do_syscall_64+0x3d6/0x440 [18508.700350] entry_SYSCALL_64_after_hwframe+0x49/0xae [18508.703181] RIP: 0033:0x7f151bee3cd8 [18508.705377] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [18508.727806] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [18508.732692] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [18508.739967] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [18508.747997] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18508.757239] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18508.759926] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18512.352394] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16581s! [18512.360213] Showing busy workqueues and worker pools: [18512.365500] workqueue events: flags=0x0 [18512.367151] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18512.373211] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [18512.384286] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18512.387686] pending: kfree_rcu_monitor [18512.393840] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18512.399684] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [18512.410261] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18512.417779] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [18512.420985] workqueue events_long: flags=0x0 [18512.425164] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18512.435693] pending: lu_objects_flush [obdclass] [18512.441710] workqueue events_unbound: flags=0x2 [18512.444929] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [18512.451789] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [18512.461191] workqueue events_power_efficient: flags=0x80 [18512.469843] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18512.474044] pending: do_cache_clean, neigh_periodic_work, check_lifetime [18512.481264] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18512.487979] pending: neigh_periodic_work [18512.490519] workqueue mm_percpu_wq: flags=0x8 [18512.492582] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18512.496373] pending: vmstat_update, lru_add_drain_per_cpu [18512.501794] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18512.508205] pending: vmstat_update [18512.512159] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18512.518025] pending: lru_add_drain_per_cpu [18512.521347] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [18512.527258] pending: lru_add_drain_per_cpu BAR(48) [18512.532269] workqueue writeback: flags=0x4a [18512.535075] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [18512.540902] pending: wb_workfn [18512.543928] workqueue xprtiod: flags=0xa [18512.547695] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [18512.553500] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [18512.560236] workqueue ipv6_addrconf: flags=0x40008 [18512.562174] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18512.568649] pending: addrconf_verify_work [18512.572181] workqueue knbd0-recv: flags=0x1a [18512.575203] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18512.580892] in-flight: 658:recv_work [18512.587164] workqueue ptlrpc_pinger: flags=0x2 [18512.590943] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18512.596830] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [18512.605459] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16582s workers=4 idle: 32984 37 8 [18512.611862] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18497s workers=2 idle: 661 [18520.056030] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [18520.071648] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18520.131360] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18520.136055] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18520.140854] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18520.151617] RIP: 0010:native_safe_halt+0x2e/0x50 [18520.158820] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18520.171212] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18520.174485] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18520.185057] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [18520.203880] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [18520.214748] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [18520.234817] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [18520.252741] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [18520.261572] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18520.268642] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18520.274208] Call Trace: [18520.274995] [18520.278230] ? show_regs.cold.9+0x22/0x2f [18520.283752] ? watchdog_timer_fn.cold.6+0x62/0x10a [18520.289264] ? __hrtimer_run_queues+0x1c2/0x4d0 [18520.295146] ? watchdog+0x40/0x40 [18520.298360] ? hrtimer_interrupt+0x128/0x2c0 [18520.299989] ? smp_apic_timer_interrupt+0xa6/0x270 [18520.305135] ? apic_timer_interrupt+0xf/0x20 [18520.313050] [18520.314787] ? native_safe_halt+0x2e/0x50 [18520.325590] kvm_wait+0xcd/0xe0 [18520.331837] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18520.338499] do_raw_spin_lock+0x177/0x190 [18520.341853] _raw_spin_lock+0x21/0x30 [18520.346902] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18520.359904] process_one_work+0x2c8/0x700 [18520.367096] worker_thread+0x296/0x6e0 [18520.371756] ? rescuer_thread+0x570/0x570 [18520.377752] kthread+0x1d1/0x200 [18520.380996] ? set_kthread_struct+0x70/0x70 [18520.387756] ret_from_fork+0x1f/0x30 [18532.055027] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [18532.058347] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18532.078496] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18532.083752] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18532.088027] RIP: 0010:native_safe_halt+0x2e/0x50 [18532.090163] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18532.099400] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18532.103428] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [18532.106758] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [18532.109953] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [18532.113566] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [18532.116849] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18532.120112] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [18532.123912] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18532.126813] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18532.129967] Call Trace: [18532.131095] [18532.132045] ? show_regs.cold.9+0x22/0x2f [18532.133917] ? watchdog_timer_fn.cold.6+0x62/0x10a [18532.135991] ? __hrtimer_run_queues+0x1c2/0x4d0 [18532.138037] ? watchdog+0x40/0x40 [18532.139634] ? hrtimer_interrupt+0x128/0x2c0 [18532.141350] ? smp_apic_timer_interrupt+0xa6/0x270 [18532.143556] ? apic_timer_interrupt+0xf/0x20 [18532.145054] [18532.146481] ? native_safe_halt+0x2e/0x50 [18532.147939] kvm_wait+0xcd/0xe0 [18532.149126] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18532.151480] do_raw_spin_lock+0x177/0x190 [18532.153495] _raw_spin_lock+0x21/0x30 [18532.154983] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18532.157511] obd_mod_free+0x46/0x140 [ptlrpc] [18532.159855] mdc_commit_open+0x91/0xb0 [mdc] [18532.161806] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18532.164170] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18532.166710] after_reply+0xe2c/0x1780 [ptlrpc] [18532.169001] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18532.171425] ? __next_timer_interrupt+0x160/0x160 [18532.173592] ptlrpcd+0xa82/0xe40 [ptlrpc] [18532.175750] ? do_wait_intr+0xf0/0xf0 [18532.177141] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18532.179627] kthread+0x1d1/0x200 [18532.180864] ? set_kthread_struct+0x70/0x70 [18532.182732] ret_from_fork+0x1f/0x30 [18536.057286] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [18536.068028] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18536.100916] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18536.112069] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18536.119146] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18536.121131] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18536.137797] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18536.145303] RAX: 0000000000006b5d RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18536.151050] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [18536.155182] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [18536.158646] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [18536.161559] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [18536.164011] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [18536.169130] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18536.173121] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [18536.176243] Call Trace: [18536.177315] [18536.177996] ? show_regs.cold.9+0x22/0x2f [18536.183004] ? watchdog_timer_fn.cold.6+0x62/0x10a [18536.187925] ? __hrtimer_run_queues+0x1c2/0x4d0 [18536.191541] ? watchdog+0x40/0x40 [18536.194932] ? hrtimer_interrupt+0x128/0x2c0 [18536.198115] ? smp_apic_timer_interrupt+0xa6/0x270 [18536.203555] ? apic_timer_interrupt+0xf/0x20 [18536.208761] [18536.209766] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18536.212107] do_raw_spin_lock+0x177/0x190 [18536.217809] _raw_spin_lock+0x21/0x30 [18536.220621] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18536.227065] ? do_raw_spin_unlock+0x75/0x190 [18536.229856] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18536.233892] ? do_wait_intr+0xf0/0xf0 [18536.236472] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18536.241818] mdc_close+0x53b/0xd70 [mdc] [18536.243686] lmv_close+0x18d/0x490 [lmv] [18536.245519] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [18536.248452] ll_close_inode_openhandle+0x493/0xea0 [lustre] [18536.252406] ll_md_real_close+0xa5/0x2a0 [lustre] [18536.255550] ll_md_close+0x274/0x800 [lustre] [18536.257524] ll_file_release+0x128/0xc50 [lustre] [18536.260058] __fput+0x10d/0x440 [18536.261818] ____fput+0x12/0x20 [18536.263571] task_work_run+0xc6/0x110 [18536.266930] exit_to_usermode_loop+0x1dd/0x1f0 [18536.270498] do_syscall_64+0x3d6/0x440 [18536.274550] entry_SYSCALL_64_after_hwframe+0x49/0xae [18536.276525] RIP: 0033:0x7f151bee3cd8 [18536.281021] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [18536.291207] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [18536.296659] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [18536.305302] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [18536.311717] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18536.319253] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18536.324401] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18540.057927] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [18540.061883] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18540.090476] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18540.102466] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18540.113742] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18540.124435] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18540.139683] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18540.148717] RAX: 00000000000077af RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18540.154534] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [18540.159803] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [18540.171865] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [18540.177106] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [18540.180544] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [18540.190091] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18540.193539] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [18540.199832] Call Trace: [18540.201783] [18540.203834] ? show_regs.cold.9+0x22/0x2f [18540.208771] ? watchdog_timer_fn.cold.6+0x62/0x10a [18540.211646] ? __hrtimer_run_queues+0x1c2/0x4d0 [18540.214437] ? watchdog+0x40/0x40 [18540.217759] ? hrtimer_interrupt+0x128/0x2c0 [18540.220654] ? smp_apic_timer_interrupt+0xa6/0x270 [18540.226778] ? apic_timer_interrupt+0xf/0x20 [18540.228787] [18540.230747] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18540.233578] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18540.240154] do_raw_spin_lock+0x177/0x190 [18540.243839] _raw_spin_lock+0x21/0x30 [18540.247725] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18540.254435] ? do_raw_spin_unlock+0x75/0x190 [18540.256415] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18540.263844] ? do_wait_intr+0xf0/0xf0 [18540.266776] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18540.270955] mdc_reint+0x2d/0x110 [mdc] [18540.274440] mdc_unlink+0x658/0x870 [mdc] [18540.278674] lmv_unlink+0x34d/0xbd0 [lmv] [18540.282546] ll_unlink+0x24d/0x7f0 [lustre] [18540.287078] vfs_unlink+0x202/0x330 [18540.288685] do_unlinkat+0x2fa/0x460 [18540.291656] __x64_sys_unlink+0x37/0x50 [18540.296495] do_syscall_64+0xc1/0x440 [18540.304305] entry_SYSCALL_64_after_hwframe+0x49/0xae [18540.310976] RIP: 0033:0x7f151bdee10b [18540.313056] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [18540.329941] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [18540.336163] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [18540.343094] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [18540.347771] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18540.351734] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18540.357656] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18543.072031] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16612s! [18543.075805] Showing busy workqueues and worker pools: [18543.078376] workqueue events: flags=0x0 [18543.079913] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18543.085703] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [18543.089976] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18543.093741] pending: kfree_rcu_monitor [18543.096811] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18543.099296] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [18543.105723] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18543.108951] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [18543.113609] workqueue events_long: flags=0x0 [18543.118985] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18543.123796] pending: lu_objects_flush [obdclass] [18543.127764] workqueue events_unbound: flags=0x2 [18543.132080] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [18543.136783] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [18543.142690] workqueue events_power_efficient: flags=0x80 [18543.145516] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18543.147872] pending: do_cache_clean, neigh_periodic_work, check_lifetime [18543.151691] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18543.153763] pending: neigh_periodic_work [18543.155561] workqueue mm_percpu_wq: flags=0x8 [18543.157813] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18543.160706] pending: vmstat_update, lru_add_drain_per_cpu [18543.163338] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18543.165619] pending: vmstat_update [18543.166747] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18543.169035] pending: lru_add_drain_per_cpu [18543.170885] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [18543.178067] pending: lru_add_drain_per_cpu BAR(48) [18543.184867] workqueue writeback: flags=0x4a [18543.188726] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [18543.195144] pending: wb_workfn [18543.199649] workqueue xprtiod: flags=0xa [18543.201907] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [18543.214152] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [18543.226570] workqueue ipv6_addrconf: flags=0x40008 [18543.231517] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18543.238206] pending: addrconf_verify_work [18543.243775] workqueue knbd0-recv: flags=0x1a [18543.245807] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18543.252657] in-flight: 658:recv_work [18543.256642] workqueue ptlrpc_pinger: flags=0x2 [18543.259082] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18543.265603] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [18543.272639] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16612s workers=4 idle: 32984 37 8 [18543.282499] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18528s workers=2 idle: 661 [18544.992090] rcu: INFO: rcu_sched self-detected stall on CPU [18545.002200] rcu: 1-...!: (9991492 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [18545.011600] (t=16620345 jiffies g=249657 q=7623) [18545.015235] rcu: rcu_sched kthread starved for 16615297 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [18545.019681] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [18545.028553] rcu: RCU grace-period kthread stack dump: [18545.035860] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [18545.044912] Call Trace: [18545.046141] __schedule+0x351/0xcb0 [18545.047713] schedule+0xc0/0x180 [18545.049141] schedule_timeout+0xb4/0x190 [18545.050090] ? do_raw_spin_unlock+0x75/0x190 [18545.054661] ? __next_timer_interrupt+0x160/0x160 [18545.057217] ? prepare_to_swait_event+0xf3/0x140 [18545.060827] rcu_gp_kthread+0x8dd/0x15b0 [18545.063852] ? force_qs_rnp+0x2c0/0x2c0 [18545.065649] kthread+0x1d1/0x200 [18545.067252] ? set_kthread_struct+0x70/0x70 [18545.073731] ret_from_fork+0x1f/0x30 [18545.075914] rcu: Stack dump where RCU GP kthread last ran: [18545.079620] Sending NMI from CPU 1 to CPUs 0: [18545.084390] NMI backtrace for cpu 0 [18545.084397] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18545.084399] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18545.084401] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18545.084403] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18545.084405] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 [18545.084408] RAX: 000000000000739a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18545.084410] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [18545.084411] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [18545.084413] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [18545.084414] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [18545.084416] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [18545.084417] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18545.084419] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [18545.084420] Call Trace: [18545.084421] [18545.084422] ? show_regs.cold.9+0x22/0x2f [18545.084424] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [18545.084425] ? nmi_cpu_backtrace_handler+0x15/0x40 [18545.084484] ? nmi_handle+0xbd/0x210 [18545.084486] ? default_do_nmi+0x4b/0x1b0 [18545.084487] ? do_nmi+0x280/0x380 [18545.084488] ? end_repeat_nmi+0x16/0x1f [18545.084489] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18545.084490] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18545.084492] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18545.084493] [18545.084494] do_raw_spin_lock+0x177/0x190 [18545.084495] _raw_spin_lock+0x21/0x30 [18545.084496] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18545.084497] ? do_raw_spin_unlock+0x75/0x190 [18545.084498] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18545.084499] ? do_wait_intr+0xf0/0xf0 [18545.084500] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18545.084502] mdc_reint+0x2d/0x110 [mdc] [18545.084502] mdc_unlink+0x658/0x870 [mdc] [18545.084503] lmv_unlink+0x34d/0xbd0 [lmv] [18545.084505] ll_unlink+0x24d/0x7f0 [lustre] [18545.084506] vfs_unlink+0x202/0x330 [18545.084507] do_unlinkat+0x2fa/0x460 [18545.084508] __x64_sys_unlink+0x37/0x50 [18545.084510] do_syscall_64+0xc1/0x440 [18545.084511] entry_SYSCALL_64_after_hwframe+0x49/0xae [18545.084512] RIP: 0033:0x7f151bdee10b [18545.084514] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [18545.084517] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [18545.084519] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [18545.084520] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [18545.084522] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18545.084523] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18545.084525] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18545.085668] NMI backtrace for cpu 1 [18545.392094] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18545.405967] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18545.415764] Call Trace: [18545.417747] [18545.418735] ? dump_stack+0xbb/0x10e [18545.420216] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [18545.422833] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [18545.425161] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [18545.428248] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [18545.431069] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [18545.440352] ? rcu_dump_cpu_stacks+0x11b/0x171 [18545.446017] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [18545.452808] ? update_process_times+0x6d/0xc0 [18545.457774] ? tick_sched_handle.isra.18+0x27/0x90 [18545.461730] ? tick_sched_timer+0x53/0xb0 [18545.467096] ? __hrtimer_run_queues+0x1c2/0x4d0 [18545.472179] ? tick_sched_handle.isra.18+0x90/0x90 [18545.477752] ? hrtimer_interrupt+0x128/0x2c0 [18545.482553] ? smp_apic_timer_interrupt+0xa6/0x270 [18545.488755] ? apic_timer_interrupt+0xf/0x20 [18545.490227] [18545.491015] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18545.494064] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18545.498492] ? do_raw_spin_lock+0x177/0x190 [18545.500988] ? _raw_spin_lock+0x21/0x30 [18545.505711] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18545.507960] ? obd_mod_free+0x46/0x140 [ptlrpc] [18545.510238] ? mdc_commit_open+0x91/0xb0 [mdc] [18545.512208] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18545.515127] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18545.518411] ? after_reply+0xe2c/0x1780 [ptlrpc] [18545.524808] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18545.528004] ? __next_timer_interrupt+0x160/0x160 [18545.532439] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [18545.534844] ? do_wait_intr+0xf0/0xf0 [18545.536061] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18545.538496] ? kthread+0x1d1/0x200 [18545.539783] ? set_kthread_struct+0x70/0x70 [18545.541353] ? ret_from_fork+0x1f/0x30 [18548.059141] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [18548.066640] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18548.146608] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18548.158672] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18548.189059] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18548.197399] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18548.204201] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18548.221545] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18548.237153] RAX: 000000000000010c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18548.243066] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [18548.253176] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [18548.257773] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [18548.263487] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [18548.269000] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [18548.275620] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18548.280650] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18548.287961] Call Trace: [18548.292065] [18548.295757] ? show_regs.cold.9+0x22/0x2f [18548.301134] ? watchdog_timer_fn.cold.6+0x62/0x10a [18548.308822] ? __hrtimer_run_queues+0x1c2/0x4d0 [18548.311629] ? watchdog+0x40/0x40 [18548.316274] ? hrtimer_interrupt+0x128/0x2c0 [18548.318956] ? smp_apic_timer_interrupt+0xa6/0x270 [18548.322490] ? apic_timer_interrupt+0xf/0x20 [18548.325993] [18548.328049] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18548.332908] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18548.336477] do_raw_spin_lock+0x177/0x190 [18548.339088] _raw_spin_lock+0x21/0x30 [18548.341971] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18548.345498] process_one_work+0x2c8/0x700 [18548.350374] worker_thread+0x296/0x6e0 [18548.360282] ? rescuer_thread+0x570/0x570 [18548.361740] kthread+0x1d1/0x200 [18548.368627] ? set_kthread_struct+0x70/0x70 [18548.377104] ret_from_fork+0x1f/0x30 [18564.057047] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [18564.059114] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18564.102754] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18564.115935] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18564.122784] RIP: 0010:native_safe_halt+0x2e/0x50 [18564.124451] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18564.132820] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18564.138797] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18564.150840] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [18564.154816] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [18564.170015] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [18564.172878] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [18564.182178] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [18564.194585] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18564.198757] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [18564.206635] Call Trace: [18564.207487] [18564.208755] ? show_regs.cold.9+0x22/0x2f [18564.211753] ? watchdog_timer_fn.cold.6+0x62/0x10a [18564.214843] ? __hrtimer_run_queues+0x1c2/0x4d0 [18564.220434] ? watchdog+0x40/0x40 [18564.221873] ? hrtimer_interrupt+0x128/0x2c0 [18564.230913] ? smp_apic_timer_interrupt+0xa6/0x270 [18564.235195] ? apic_timer_interrupt+0xf/0x20 [18564.236825] [18564.237833] ? native_safe_halt+0x2e/0x50 [18564.242939] kvm_wait+0xcd/0xe0 [18564.243954] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18564.247515] do_raw_spin_lock+0x177/0x190 [18564.256320] _raw_spin_lock+0x21/0x30 [18564.259695] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18564.270917] ? do_raw_spin_unlock+0x75/0x190 [18564.275865] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18564.281908] ? do_wait_intr+0xf0/0xf0 [18564.288780] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18564.291723] mdc_close+0x53b/0xd70 [mdc] [18564.294681] lmv_close+0x18d/0x490 [lmv] [18564.298755] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [18564.304694] ll_close_inode_openhandle+0x493/0xea0 [lustre] [18564.313467] ll_md_real_close+0xa5/0x2a0 [lustre] [18564.319840] ll_md_close+0x274/0x800 [lustre] [18564.323970] ll_file_release+0x128/0xc50 [lustre] [18564.327505] __fput+0x10d/0x440 [18564.328871] ____fput+0x12/0x20 [18564.330677] task_work_run+0xc6/0x110 [18564.333482] exit_to_usermode_loop+0x1dd/0x1f0 [18564.336535] do_syscall_64+0x3d6/0x440 [18564.341275] entry_SYSCALL_64_after_hwframe+0x49/0xae [18564.345994] RIP: 0033:0x7f151bee3cd8 [18564.347010] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [18564.366770] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [18564.374525] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [18564.377721] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [18564.391996] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18564.395527] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18564.400277] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18572.054020] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [18572.055332] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [18572.059558] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18572.059702] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18572.059710] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18572.059712] RIP: 0010:native_safe_halt+0x2e/0x50 [18572.059812] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18572.059818] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [18572.059823] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18572.059826] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [18572.059829] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [18572.059831] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [18572.059834] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [18572.059838] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [18572.059841] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18572.059844] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [18572.059853] Call Trace: [18572.059858] [18572.059865] ? show_regs.cold.9+0x22/0x2f [18572.059875] ? watchdog_timer_fn.cold.6+0x62/0x10a [18572.059882] ? __hrtimer_run_queues+0x1c2/0x4d0 [18572.059886] ? watchdog+0x40/0x40 [18572.059891] ? hrtimer_interrupt+0x128/0x2c0 [18572.059895] ? smp_apic_timer_interrupt+0xa6/0x270 [18572.059903] ? apic_timer_interrupt+0xf/0x20 [18572.059908] [18572.059911] ? native_safe_halt+0x2e/0x50 [18572.059915] kvm_wait+0xcd/0xe0 [18572.059921] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18572.059928] do_raw_spin_lock+0x177/0x190 [18572.059932] _raw_spin_lock+0x21/0x30 [18572.059938] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18572.060235] ? do_raw_spin_unlock+0x75/0x190 [18572.060241] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18572.060459] ? do_wait_intr+0xf0/0xf0 [18572.060467] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18572.072394] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18572.072478] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18572.072486] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18572.072489] RIP: 0010:native_safe_halt+0x2e/0x50 [18572.072503] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18572.072507] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18572.072511] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [18572.072514] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [18572.072516] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [18572.072518] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [18572.072521] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18572.072524] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [18572.072527] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18572.072529] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18572.072538] Call Trace: [18572.072543] [18572.072548] ? show_regs.cold.9+0x22/0x2f [18572.072559] ? watchdog_timer_fn.cold.6+0x62/0x10a [18572.072566] ? __hrtimer_run_queues+0x1c2/0x4d0 [18572.072570] ? watchdog+0x40/0x40 [18572.072574] ? hrtimer_interrupt+0x128/0x2c0 [18572.072578] ? smp_apic_timer_interrupt+0xa6/0x270 [18572.072585] ? apic_timer_interrupt+0xf/0x20 [18572.072590] [18572.072593] ? native_safe_halt+0x2e/0x50 [18572.097230] mdc_reint+0x2d/0x110 [mdc] [18572.097286] mdc_unlink+0x658/0x870 [mdc] [18572.097317] lmv_unlink+0x34d/0xbd0 [lmv] [18572.097357] ll_unlink+0x24d/0x7f0 [lustre] [18572.097493] vfs_unlink+0x202/0x330 [18572.097503] do_unlinkat+0x2fa/0x460 [18572.097509] __x64_sys_unlink+0x37/0x50 [18572.097513] do_syscall_64+0xc1/0x440 [18572.097520] entry_SYSCALL_64_after_hwframe+0x49/0xae [18572.097529] RIP: 0033:0x7f151bdee10b [18572.097537] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [18572.097541] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [18572.097547] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [18572.097550] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [18572.097552] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18572.097554] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18572.097557] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18572.515849] kvm_wait+0xcd/0xe0 [18572.518653] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18572.524497] do_raw_spin_lock+0x177/0x190 [18572.528793] _raw_spin_lock+0x21/0x30 [18572.531642] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18572.538704] obd_mod_free+0x46/0x140 [ptlrpc] [18572.545098] mdc_commit_open+0x91/0xb0 [mdc] [18572.549246] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18572.553070] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18572.555394] after_reply+0xe2c/0x1780 [ptlrpc] [18572.557588] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18572.561796] ? __next_timer_interrupt+0x160/0x160 [18572.564936] ptlrpcd+0xa82/0xe40 [ptlrpc] [18572.566831] ? do_wait_intr+0xf0/0xf0 [18572.572039] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18572.574112] kthread+0x1d1/0x200 [18572.575853] ? set_kthread_struct+0x70/0x70 [18572.577886] ret_from_fork+0x1f/0x30 [18573.792150] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16643s! [18573.809183] Showing busy workqueues and worker pools: [18573.813088] workqueue events: flags=0x0 [18573.817202] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18573.823256] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [18573.829346] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18573.833399] pending: kfree_rcu_monitor [18573.836691] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18573.841121] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [18573.846767] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18573.855790] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [18573.864674] workqueue events_long: flags=0x0 [18573.869667] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18573.876808] pending: lu_objects_flush [obdclass] [18573.883020] workqueue events_unbound: flags=0x2 [18573.887150] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [18573.893834] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [18573.900356] workqueue events_power_efficient: flags=0x80 [18573.905122] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18573.909431] pending: do_cache_clean, neigh_periodic_work, check_lifetime [18573.912812] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18573.919393] pending: neigh_periodic_work [18573.922322] workqueue mm_percpu_wq: flags=0x8 [18573.926414] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18573.933505] pending: vmstat_update, lru_add_drain_per_cpu [18573.935598] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18573.939101] pending: vmstat_update [18573.940436] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18573.948359] pending: lru_add_drain_per_cpu [18573.955052] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [18573.964139] pending: lru_add_drain_per_cpu BAR(48) [18573.971589] workqueue writeback: flags=0x4a [18573.975324] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [18573.987900] pending: wb_workfn [18573.989574] workqueue xprtiod: flags=0xa [18573.991437] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [18573.995190] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [18574.000441] workqueue ipv6_addrconf: flags=0x40008 [18574.004187] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18574.008150] pending: addrconf_verify_work [18574.012418] workqueue knbd0-recv: flags=0x1a [18574.016449] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18574.021427] in-flight: 658:recv_work [18574.027199] workqueue ptlrpc_pinger: flags=0x2 [18574.030835] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18574.036610] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [18574.046564] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16643s workers=4 idle: 32984 37 8 [18574.050181] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18558s workers=2 idle: 661 [18576.060125] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [18576.075106] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18576.128118] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18576.150616] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18576.166894] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18576.170455] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18576.174101] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18576.191337] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18576.195111] RAX: 000000000000657f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18576.198758] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [18576.201917] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [18576.204092] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [18576.207400] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [18576.214010] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [18576.221068] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18576.223774] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18576.226902] Call Trace: [18576.228038] [18576.230528] ? show_regs.cold.9+0x22/0x2f [18576.231866] ? watchdog_timer_fn.cold.6+0x62/0x10a [18576.237228] ? __hrtimer_run_queues+0x1c2/0x4d0 [18576.238911] ? watchdog+0x40/0x40 [18576.241792] ? hrtimer_interrupt+0x128/0x2c0 [18576.243751] ? smp_apic_timer_interrupt+0xa6/0x270 [18576.246176] ? apic_timer_interrupt+0xf/0x20 [18576.248305] [18576.249268] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18576.252042] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18576.256011] do_raw_spin_lock+0x177/0x190 [18576.259074] _raw_spin_lock+0x21/0x30 [18576.262013] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18576.267224] process_one_work+0x2c8/0x700 [18576.271109] worker_thread+0x296/0x6e0 [18576.274865] ? rescuer_thread+0x570/0x570 [18576.277048] kthread+0x1d1/0x200 [18576.279256] ? set_kthread_struct+0x70/0x70 [18576.281840] ret_from_fork+0x1f/0x30 [18592.061322] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [18592.078914] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18592.189644] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18592.225297] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18592.250754] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18592.276394] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18592.309628] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18592.327968] RAX: 0000000000003403 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18592.348936] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [18592.359435] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [18592.365396] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [18592.373854] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [18592.378052] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [18592.386795] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18592.391299] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [18592.399377] Call Trace: [18592.403717] [18592.405222] ? show_regs.cold.9+0x22/0x2f [18592.409617] ? watchdog_timer_fn.cold.6+0x62/0x10a [18592.416331] ? __hrtimer_run_queues+0x1c2/0x4d0 [18592.421117] ? watchdog+0x40/0x40 [18592.423341] ? hrtimer_interrupt+0x128/0x2c0 [18592.436964] ? smp_apic_timer_interrupt+0xa6/0x270 [18592.453408] ? apic_timer_interrupt+0xf/0x20 [18592.459184] [18592.459984] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18592.467632] do_raw_spin_lock+0x177/0x190 [18592.474869] _raw_spin_lock+0x21/0x30 [18592.477062] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18592.479063] ? do_raw_spin_unlock+0x75/0x190 [18592.489440] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18592.511423] ? do_wait_intr+0xf0/0xf0 [18592.513663] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18592.534452] mdc_close+0x53b/0xd70 [mdc] [18592.545629] lmv_close+0x18d/0x490 [lmv] [18592.556999] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [18592.571975] ll_close_inode_openhandle+0x493/0xea0 [lustre] [18592.574030] ll_md_real_close+0xa5/0x2a0 [lustre] [18592.585174] ll_md_close+0x274/0x800 [lustre] [18592.589335] ll_file_release+0x128/0xc50 [lustre] [18592.607613] __fput+0x10d/0x440 [18592.610476] ____fput+0x12/0x20 [18592.618412] task_work_run+0xc6/0x110 [18592.624611] exit_to_usermode_loop+0x1dd/0x1f0 [18592.637093] do_syscall_64+0x3d6/0x440 [18592.640122] entry_SYSCALL_64_after_hwframe+0x49/0xae [18592.650739] RIP: 0033:0x7f151bee3cd8 [18592.658289] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [18592.691308] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [18592.701417] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [18592.713373] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [18592.731548] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18592.753253] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18592.755469] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18600.052341] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [18600.057159] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18600.059771] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [18600.089946] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18600.089961] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18600.089964] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18600.090050] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18600.090054] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [18600.090060] RAX: 0000000000002aa7 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18600.090062] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [18600.090065] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [18600.090071] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [18600.090077] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [18600.090082] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [18600.090085] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18600.090087] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [18600.090095] Call Trace: [18600.090101] [18600.090106] ? show_regs.cold.9+0x22/0x2f [18600.090115] ? watchdog_timer_fn.cold.6+0x62/0x10a [18600.090121] ? __hrtimer_run_queues+0x1c2/0x4d0 [18600.090125] ? watchdog+0x40/0x40 [18600.090128] ? hrtimer_interrupt+0x128/0x2c0 [18600.090132] ? smp_apic_timer_interrupt+0xa6/0x270 [18600.090140] ? apic_timer_interrupt+0xf/0x20 [18600.090144] [18600.090146] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18600.090151] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18600.090155] do_raw_spin_lock+0x177/0x190 [18600.090159] _raw_spin_lock+0x21/0x30 [18600.090166] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18600.090453] ? do_raw_spin_unlock+0x75/0x190 [18600.090459] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18600.090909] ? do_wait_intr+0xf0/0xf0 [18600.090918] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18600.091425] mdc_reint+0x2d/0x110 [mdc] [18600.091461] mdc_unlink+0x658/0x870 [mdc] [18600.091486] lmv_unlink+0x34d/0xbd0 [lmv] [18600.091517] ll_unlink+0x24d/0x7f0 [lustre] [18600.091815] vfs_unlink+0x202/0x330 [18600.091824] do_unlinkat+0x2fa/0x460 [18600.091828] __x64_sys_unlink+0x37/0x50 [18600.091831] do_syscall_64+0xc1/0x440 [18600.091836] entry_SYSCALL_64_after_hwframe+0x49/0xae [18600.091844] RIP: 0033:0x7f151bdee10b [18600.091851] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [18600.091854] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [18600.091859] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [18600.091861] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [18600.091863] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18600.091865] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18600.091867] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18600.232201] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18600.250151] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18600.254526] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18600.258531] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18600.261135] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18600.269537] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18600.274021] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [18600.277371] RDX: 0000000000003145 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [18600.280283] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [18600.283510] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [18600.287010] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18600.289284] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [18600.291459] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18600.293176] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18600.296119] Call Trace: [18600.297226] [18600.298188] ? show_regs.cold.9+0x22/0x2f [18600.300092] ? watchdog_timer_fn.cold.6+0x62/0x10a [18600.302002] ? __hrtimer_run_queues+0x1c2/0x4d0 [18600.304236] ? watchdog+0x40/0x40 [18600.305786] ? hrtimer_interrupt+0x128/0x2c0 [18600.307477] ? smp_apic_timer_interrupt+0xa6/0x270 [18600.309761] ? apic_timer_interrupt+0xf/0x20 [18600.311692] [18600.312789] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18600.315224] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18600.317766] do_raw_spin_lock+0x177/0x190 [18600.319343] _raw_spin_lock+0x21/0x30 [18600.321483] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18600.323897] obd_mod_free+0x46/0x140 [ptlrpc] [18600.326353] mdc_commit_open+0x91/0xb0 [mdc] [18600.328195] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18600.330535] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18600.333232] after_reply+0xe2c/0x1780 [ptlrpc] [18600.335426] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18600.338054] ? __next_timer_interrupt+0x160/0x160 [18600.340323] ptlrpcd+0xa82/0xe40 [ptlrpc] [18600.342427] ? do_wait_intr+0xf0/0xf0 [18600.344052] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18600.346834] kthread+0x1d1/0x200 [18600.348408] ? set_kthread_struct+0x70/0x70 [18600.350275] ret_from_fork+0x1f/0x30 [18604.057825] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [18604.071170] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18604.143438] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18604.158066] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18604.168474] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18604.177504] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18604.185028] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18604.213947] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18604.225860] RAX: 0000000000005d0a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18604.237583] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [18604.248571] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [18604.259115] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [18604.267238] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [18604.277201] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [18604.290988] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18604.301472] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18604.309415] Call Trace: [18604.311354] [18604.312555] ? show_regs.cold.9+0x22/0x2f [18604.315070] ? watchdog_timer_fn.cold.6+0x62/0x10a [18604.318288] ? __hrtimer_run_queues+0x1c2/0x4d0 [18604.322089] ? watchdog+0x40/0x40 [18604.331093] ? hrtimer_interrupt+0x128/0x2c0 [18604.335082] ? smp_apic_timer_interrupt+0xa6/0x270 [18604.339188] ? apic_timer_interrupt+0xf/0x20 [18604.342436] [18604.346667] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18604.351286] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18604.354067] do_raw_spin_lock+0x177/0x190 [18604.356443] _raw_spin_lock+0x21/0x30 [18604.359153] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18604.364373] process_one_work+0x2c8/0x700 [18604.367119] worker_thread+0x296/0x6e0 [18604.369414] ? rescuer_thread+0x570/0x570 [18604.371422] kthread+0x1d1/0x200 [18604.373471] ? set_kthread_struct+0x70/0x70 [18604.375838] ret_from_fork+0x1f/0x30 [18604.513219] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16673s! [18604.520378] Showing busy workqueues and worker pools: [18604.525538] workqueue events: flags=0x0 [18604.526962] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18604.545052] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [18604.561988] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18604.569046] pending: kfree_rcu_monitor [18604.571779] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18604.577158] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [18604.586204] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18604.594320] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [18604.598634] workqueue events_long: flags=0x0 [18604.600148] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18604.603197] pending: lu_objects_flush [obdclass] [18604.605460] workqueue events_unbound: flags=0x2 [18604.607424] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [18604.615937] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [18604.621930] workqueue events_power_efficient: flags=0x80 [18604.624789] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18604.635820] pending: do_cache_clean, neigh_periodic_work, check_lifetime [18604.641241] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18604.654158] pending: neigh_periodic_work [18604.656443] workqueue mm_percpu_wq: flags=0x8 [18604.662428] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18604.669119] pending: vmstat_update, lru_add_drain_per_cpu [18604.673215] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18604.678035] pending: vmstat_update [18604.680114] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18604.686148] pending: lru_add_drain_per_cpu [18604.690764] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [18604.696324] pending: lru_add_drain_per_cpu BAR(48) [18604.698996] workqueue writeback: flags=0x4a [18604.703592] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [18604.715991] pending: wb_workfn [18604.717268] workqueue xprtiod: flags=0xa [18604.718664] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [18604.731528] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [18604.738866] workqueue ipv6_addrconf: flags=0x40008 [18604.747411] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18604.756210] pending: addrconf_verify_work [18604.760217] workqueue knbd0-recv: flags=0x1a [18604.763457] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18604.768351] in-flight: 658:recv_work [18604.771145] workqueue ptlrpc_pinger: flags=0x2 [18604.779486] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18604.787468] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [18604.794077] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16674s workers=4 idle: 32984 37 8 [18604.802200] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18589s workers=2 idle: 661 [18620.062751] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [18620.079366] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18620.139737] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18620.157409] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18620.170459] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18620.174356] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18620.189298] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [18620.193428] RAX: 0000000000007486 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18620.197525] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [18620.204705] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [18620.217323] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [18620.227747] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [18620.238518] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [18620.255465] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18620.261783] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [18620.270213] Call Trace: [18620.272882] [18620.273909] ? show_regs.cold.9+0x22/0x2f [18620.277005] ? watchdog_timer_fn.cold.6+0x62/0x10a [18620.282950] ? __hrtimer_run_queues+0x1c2/0x4d0 [18620.286482] ? watchdog+0x40/0x40 [18620.287600] ? hrtimer_interrupt+0x128/0x2c0 [18620.293219] ? smp_apic_timer_interrupt+0xa6/0x270 [18620.300189] ? apic_timer_interrupt+0xf/0x20 [18620.302655] [18620.303795] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18620.306275] do_raw_spin_lock+0x177/0x190 [18620.310233] _raw_spin_lock+0x21/0x30 [18620.314113] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18620.319066] ? do_raw_spin_unlock+0x75/0x190 [18620.321915] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18620.324366] ? do_wait_intr+0xf0/0xf0 [18620.325895] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18620.333381] mdc_close+0x53b/0xd70 [mdc] [18620.344908] lmv_close+0x18d/0x490 [lmv] [18620.356493] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [18620.364942] ll_close_inode_openhandle+0x493/0xea0 [lustre] [18620.376776] ll_md_real_close+0xa5/0x2a0 [lustre] [18620.381330] ll_md_close+0x274/0x800 [lustre] [18620.391451] ll_file_release+0x128/0xc50 [lustre] [18620.395255] __fput+0x10d/0x440 [18620.401219] ____fput+0x12/0x20 [18620.402265] task_work_run+0xc6/0x110 [18620.406899] exit_to_usermode_loop+0x1dd/0x1f0 [18620.411160] do_syscall_64+0x3d6/0x440 [18620.419528] entry_SYSCALL_64_after_hwframe+0x49/0xae [18620.425392] RIP: 0033:0x7f151bee3cd8 [18620.431272] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [18620.462869] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [18620.467842] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [18620.474861] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [18620.482871] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18620.491360] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18620.500916] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18628.055061] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [18628.057898] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18628.075182] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18628.078339] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18628.081906] RIP: 0010:native_safe_halt+0x2e/0x50 [18628.083852] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18628.090959] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18628.093845] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [18628.097055] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [18628.099730] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [18628.102287] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [18628.105017] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18628.107948] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [18628.111097] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18628.113913] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18628.119998] Call Trace: [18628.121994] [18628.123192] ? show_regs.cold.9+0x22/0x2f [18628.129225] ? watchdog_timer_fn.cold.6+0x62/0x10a [18628.133610] ? __hrtimer_run_queues+0x1c2/0x4d0 [18628.137320] ? watchdog+0x40/0x40 [18628.147166] ? hrtimer_interrupt+0x128/0x2c0 [18628.152339] ? smp_apic_timer_interrupt+0xa6/0x270 [18628.155888] ? apic_timer_interrupt+0xf/0x20 [18628.157793] [18628.158917] ? native_safe_halt+0x2e/0x50 [18628.166415] kvm_wait+0xcd/0xe0 [18628.174454] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18628.176307] do_raw_spin_lock+0x177/0x190 [18628.182497] _raw_spin_lock+0x21/0x30 [18628.185169] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18628.191468] obd_mod_free+0x46/0x140 [ptlrpc] [18628.193306] mdc_commit_open+0x91/0xb0 [mdc] [18628.204757] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18628.218980] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18628.228175] after_reply+0xe2c/0x1780 [ptlrpc] [18628.233620] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18628.240239] ? __next_timer_interrupt+0x160/0x160 [18628.246611] ptlrpcd+0xa82/0xe40 [ptlrpc] [18628.249922] ? do_wait_intr+0xf0/0xf0 [18628.254075] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18628.259037] kthread+0x1d1/0x200 [18628.263840] ? set_kthread_struct+0x70/0x70 [18628.266923] ret_from_fork+0x1f/0x30 [18632.053522] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [18632.056011] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [18632.071832] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18632.071923] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18632.071937] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18632.071940] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18632.071954] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18632.071962] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18632.071978] RAX: 00000000000077ad RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18632.071981] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [18632.071983] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [18632.071986] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [18632.071988] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [18632.071992] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [18632.071995] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18632.072001] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [18632.072009] Call Trace: [18632.072014] [18632.072020] ? show_regs.cold.9+0x22/0x2f [18632.072029] ? watchdog_timer_fn.cold.6+0x62/0x10a [18632.072036] ? __hrtimer_run_queues+0x1c2/0x4d0 [18632.072046] ? watchdog+0x40/0x40 [18632.072051] ? hrtimer_interrupt+0x128/0x2c0 [18632.072055] ? smp_apic_timer_interrupt+0xa6/0x270 [18632.072064] ? apic_timer_interrupt+0xf/0x20 [18632.072068] [18632.072071] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18632.072081] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18632.072085] do_raw_spin_lock+0x177/0x190 [18632.072089] _raw_spin_lock+0x21/0x30 [18632.072095] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18632.072600] ? do_raw_spin_unlock+0x75/0x190 [18632.072606] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18632.073063] ? do_wait_intr+0xf0/0xf0 [18632.073072] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18632.073403] mdc_reint+0x2d/0x110 [mdc] [18632.073856] mdc_unlink+0x658/0x870 [mdc] [18632.073896] lmv_unlink+0x34d/0xbd0 [lmv] [18632.073934] ll_unlink+0x24d/0x7f0 [lustre] [18632.074098] vfs_unlink+0x202/0x330 [18632.074112] do_unlinkat+0x2fa/0x460 [18632.074117] __x64_sys_unlink+0x37/0x50 [18632.074121] do_syscall_64+0xc1/0x440 [18632.074128] entry_SYSCALL_64_after_hwframe+0x49/0xae [18632.074139] RIP: 0033:0x7f151bdee10b [18632.074405] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [18632.074412] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [18632.074417] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [18632.074419] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [18632.074421] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18632.074424] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18632.074426] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18632.326355] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18632.349946] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18632.356786] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18632.362549] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18632.367240] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18632.371479] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18632.385206] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18632.390058] RAX: 0000000000005f3f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18632.394787] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [18632.399229] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [18632.403958] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [18632.408708] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [18632.413412] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [18632.418253] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18632.420745] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18632.425262] Call Trace: [18632.427265] [18632.429379] ? show_regs.cold.9+0x22/0x2f [18632.432282] ? watchdog_timer_fn.cold.6+0x62/0x10a [18632.437397] ? __hrtimer_run_queues+0x1c2/0x4d0 [18632.440157] ? watchdog+0x40/0x40 [18632.442926] ? hrtimer_interrupt+0x128/0x2c0 [18632.447217] ? smp_apic_timer_interrupt+0xa6/0x270 [18632.450389] ? apic_timer_interrupt+0xf/0x20 [18632.453864] [18632.454696] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18632.458944] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18632.460885] do_raw_spin_lock+0x177/0x190 [18632.462112] _raw_spin_lock+0x21/0x30 [18632.463420] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18632.469661] process_one_work+0x2c8/0x700 [18632.471048] worker_thread+0x296/0x6e0 [18632.474037] ? rescuer_thread+0x570/0x570 [18632.480228] kthread+0x1d1/0x200 [18632.484214] ? set_kthread_struct+0x70/0x70 [18632.485681] ret_from_fork+0x1f/0x30 [18635.232222] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16704s! [18635.237107] Showing busy workqueues and worker pools: [18635.239992] workqueue events: flags=0x0 [18635.241615] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18635.245216] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [18635.255927] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18635.260711] pending: kfree_rcu_monitor [18635.261872] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18635.265701] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [18635.269201] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18635.272557] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [18635.277038] workqueue events_long: flags=0x0 [18635.279189] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18635.282072] pending: lu_objects_flush [obdclass] [18635.285024] workqueue events_unbound: flags=0x2 [18635.287089] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [18635.291919] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [18635.298073] workqueue events_power_efficient: flags=0x80 [18635.300487] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18635.303447] pending: do_cache_clean, neigh_periodic_work, check_lifetime [18635.306217] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18635.309230] pending: neigh_periodic_work [18635.311445] workqueue mm_percpu_wq: flags=0x8 [18635.313438] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18635.321982] pending: vmstat_update, lru_add_drain_per_cpu [18635.326191] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18635.331434] pending: vmstat_update [18635.335152] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18635.338603] pending: lru_add_drain_per_cpu [18635.342472] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [18635.345809] pending: lru_add_drain_per_cpu BAR(48) [18635.347994] workqueue writeback: flags=0x4a [18635.350116] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [18635.354219] pending: wb_workfn [18635.355839] workqueue xprtiod: flags=0xa [18635.357133] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [18635.359492] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [18635.362547] workqueue ipv6_addrconf: flags=0x40008 [18635.363871] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18635.366108] pending: addrconf_verify_work [18635.367925] workqueue knbd0-recv: flags=0x1a [18635.369544] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18635.371633] in-flight: 658:recv_work [18635.372713] workqueue ptlrpc_pinger: flags=0x2 [18635.374144] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18635.376111] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [18635.378930] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16704s workers=4 idle: 32984 37 8 [18635.381906] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18620s workers=2 idle: 661 [18648.057336] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [18648.061363] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18648.086437] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18648.091211] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18648.095899] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18648.099180] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18648.108136] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18648.112247] RAX: 0000000000007a86 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18648.116392] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [18648.119748] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [18648.123514] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [18648.127184] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [18648.130805] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [18648.136260] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18648.139370] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [18648.143005] Call Trace: [18648.144296] [18648.145408] ? show_regs.cold.9+0x22/0x2f [18648.147679] ? watchdog_timer_fn.cold.6+0x62/0x10a [18648.150423] ? __hrtimer_run_queues+0x1c2/0x4d0 [18648.152970] ? watchdog+0x40/0x40 [18648.154216] ? hrtimer_interrupt+0x128/0x2c0 [18648.156465] ? smp_apic_timer_interrupt+0xa6/0x270 [18648.159578] ? apic_timer_interrupt+0xf/0x20 [18648.168760] [18648.170951] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18648.174200] do_raw_spin_lock+0x177/0x190 [18648.178793] _raw_spin_lock+0x21/0x30 [18648.181738] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18648.184296] ? do_raw_spin_unlock+0x75/0x190 [18648.186309] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18648.189690] ? do_wait_intr+0xf0/0xf0 [18648.192785] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18648.196918] mdc_close+0x53b/0xd70 [mdc] [18648.201199] lmv_close+0x18d/0x490 [lmv] [18648.202589] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [18648.208620] ll_close_inode_openhandle+0x493/0xea0 [lustre] [18648.214017] ll_md_real_close+0xa5/0x2a0 [lustre] [18648.216862] ll_md_close+0x274/0x800 [lustre] [18648.219709] ll_file_release+0x128/0xc50 [lustre] [18648.226484] __fput+0x10d/0x440 [18648.228039] ____fput+0x12/0x20 [18648.232123] task_work_run+0xc6/0x110 [18648.235072] exit_to_usermode_loop+0x1dd/0x1f0 [18648.238162] do_syscall_64+0x3d6/0x440 [18648.240933] entry_SYSCALL_64_after_hwframe+0x49/0xae [18648.249477] RIP: 0033:0x7f151bee3cd8 [18648.253547] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [18648.268088] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [18648.272840] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [18648.275893] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [18648.278990] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18648.281800] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18648.284559] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18656.055075] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [18656.058076] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18656.079536] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18656.085151] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18656.089513] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18656.092759] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18656.102287] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18656.105872] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [18656.109422] RDX: 0000000000004cb3 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [18656.112912] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [18656.116442] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [18656.119777] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18656.123025] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [18656.127114] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18656.129131] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18656.132400] Call Trace: [18656.133776] [18656.135042] ? show_regs.cold.9+0x22/0x2f [18656.136991] ? watchdog_timer_fn.cold.6+0x62/0x10a [18656.139652] ? __hrtimer_run_queues+0x1c2/0x4d0 [18656.142236] ? watchdog+0x40/0x40 [18656.144014] ? hrtimer_interrupt+0x128/0x2c0 [18656.145844] ? smp_apic_timer_interrupt+0xa6/0x270 [18656.148393] ? apic_timer_interrupt+0xf/0x20 [18656.150832] [18656.151913] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18656.154482] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18656.156345] do_raw_spin_lock+0x177/0x190 [18656.157778] _raw_spin_lock+0x21/0x30 [18656.159188] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18656.161753] obd_mod_free+0x46/0x140 [ptlrpc] [18656.163594] mdc_commit_open+0x91/0xb0 [mdc] [18656.165298] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18656.170554] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18656.173518] after_reply+0xe2c/0x1780 [ptlrpc] [18656.175316] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18656.177637] ? __next_timer_interrupt+0x160/0x160 [18656.179708] ptlrpcd+0xa82/0xe40 [ptlrpc] [18656.181994] ? do_wait_intr+0xf0/0xf0 [18656.185949] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18656.191264] kthread+0x1d1/0x200 [18656.193058] ? set_kthread_struct+0x70/0x70 [18656.195370] ret_from_fork+0x1f/0x30 [18660.052280] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [18660.055772] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18660.056682] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [18660.122798] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18660.122811] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18660.122814] RIP: 0010:native_safe_halt+0x2e/0x50 [18660.122827] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18660.122831] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18660.122837] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18660.122839] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [18660.122842] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [18660.126223] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18660.126308] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18660.126316] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18660.126395] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18660.136881] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [18660.136894] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [18660.140412] [18660.140422] RIP: 0010:native_safe_halt+0x2e/0x50 [18660.140439] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18660.140443] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18660.140448] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18660.140450] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [18660.140452] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [18660.140454] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [18660.140457] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [18660.140460] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [18660.140463] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18660.140465] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18660.140474] Call Trace: [18660.140480] [18660.140486] ? show_regs.cold.9+0x22/0x2f [18660.140497] ? watchdog_timer_fn.cold.6+0x62/0x10a [18660.140504] ? __hrtimer_run_queues+0x1c2/0x4d0 [18660.140508] ? watchdog+0x40/0x40 [18660.140513] ? hrtimer_interrupt+0x128/0x2c0 [18660.140517] ? smp_apic_timer_interrupt+0xa6/0x270 [18660.140604] ? apic_timer_interrupt+0xf/0x20 [18660.140612] [18660.140615] ? native_safe_halt+0x2e/0x50 [18660.140619] kvm_wait+0xcd/0xe0 [18660.140626] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18660.140636] do_raw_spin_lock+0x177/0x190 [18660.140640] _raw_spin_lock+0x21/0x30 [18660.140644] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18660.140937] process_one_work+0x2c8/0x700 [18660.140943] worker_thread+0x296/0x6e0 [18660.140947] ? rescuer_thread+0x570/0x570 [18660.140951] kthread+0x1d1/0x200 [18660.140955] ? set_kthread_struct+0x70/0x70 [18660.140958] ret_from_fork+0x1f/0x30 [18660.415248] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [18660.417142] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18660.420771] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [18660.424757] Call Trace: [18660.426218] [18660.427318] ? show_regs.cold.9+0x22/0x2f [18660.429461] ? watchdog_timer_fn.cold.6+0x62/0x10a [18660.433361] ? __hrtimer_run_queues+0x1c2/0x4d0 [18660.435406] ? watchdog+0x40/0x40 [18660.436806] ? hrtimer_interrupt+0x128/0x2c0 [18660.438727] ? smp_apic_timer_interrupt+0xa6/0x270 [18660.440919] ? apic_timer_interrupt+0xf/0x20 [18660.442928] [18660.444241] ? native_safe_halt+0x2e/0x50 [18660.446268] kvm_wait+0xcd/0xe0 [18660.447626] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18660.449967] do_raw_spin_lock+0x177/0x190 [18660.451678] _raw_spin_lock+0x21/0x30 [18660.454511] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18660.457518] ? do_raw_spin_unlock+0x75/0x190 [18660.459656] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18660.462433] ? do_wait_intr+0xf0/0xf0 [18660.464210] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18660.466896] mdc_reint+0x2d/0x110 [mdc] [18660.468815] mdc_unlink+0x658/0x870 [mdc] [18660.470599] lmv_unlink+0x34d/0xbd0 [lmv] [18660.472491] ll_unlink+0x24d/0x7f0 [lustre] [18660.474409] vfs_unlink+0x202/0x330 [18660.475835] do_unlinkat+0x2fa/0x460 [18660.477489] __x64_sys_unlink+0x37/0x50 [18660.478914] do_syscall_64+0xc1/0x440 [18660.480074] entry_SYSCALL_64_after_hwframe+0x49/0xae [18660.482429] RIP: 0033:0x7f151bdee10b [18660.484734] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [18660.495453] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [18660.498964] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [18660.503848] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [18660.507660] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18660.510855] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18660.514545] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18665.441050] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16734s! [18665.445783] Showing busy workqueues and worker pools: [18665.448563] workqueue events: flags=0x0 [18665.449759] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18665.456411] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [18665.475811] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18665.486184] pending: kfree_rcu_monitor [18665.493151] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18665.496299] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [18665.508589] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18665.512208] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [18665.524914] workqueue events_long: flags=0x0 [18665.530874] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18665.537001] pending: lu_objects_flush [obdclass] [18665.539185] workqueue events_unbound: flags=0x2 [18665.543266] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [18665.548442] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [18665.562010] workqueue events_power_efficient: flags=0x80 [18665.566540] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18665.583226] pending: do_cache_clean, neigh_periodic_work, check_lifetime [18665.596615] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18665.602864] pending: neigh_periodic_work [18665.604344] workqueue mm_percpu_wq: flags=0x8 [18665.614508] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18665.626484] pending: vmstat_update, lru_add_drain_per_cpu [18665.635432] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18665.643597] pending: vmstat_update [18665.648227] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18665.659872] pending: lru_add_drain_per_cpu [18665.661791] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [18665.668141] pending: lru_add_drain_per_cpu BAR(48) [18665.674578] workqueue writeback: flags=0x4a [18665.680498] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [18665.694872] pending: wb_workfn [18665.703965] workqueue xprtiod: flags=0xa [18665.711737] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [18665.714371] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [18665.725208] workqueue ipv6_addrconf: flags=0x40008 [18665.734298] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18665.745487] pending: addrconf_verify_work [18665.756197] workqueue knbd0-recv: flags=0x1a [18665.765282] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18665.780507] in-flight: 658:recv_work [18665.786371] workqueue ptlrpc_pinger: flags=0x2 [18665.796019] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18665.804508] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [18665.815355] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16735s workers=4 idle: 32984 37 8 [18665.823142] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18650s workers=2 idle: 661 [18676.057066] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [18676.059672] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18676.086866] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18676.094776] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18676.100548] RIP: 0010:native_safe_halt+0x2e/0x50 [18676.102302] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18676.112338] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18676.116416] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18676.120176] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [18676.123635] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [18676.126856] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [18676.130046] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [18676.133276] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [18676.136790] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18676.139492] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [18676.142547] Call Trace: [18676.143466] [18676.144246] ? show_regs.cold.9+0x22/0x2f [18676.146054] ? watchdog_timer_fn.cold.6+0x62/0x10a [18676.149033] ? __hrtimer_run_queues+0x1c2/0x4d0 [18676.150874] ? watchdog+0x40/0x40 [18676.152481] ? hrtimer_interrupt+0x128/0x2c0 [18676.154387] ? smp_apic_timer_interrupt+0xa6/0x270 [18676.156359] ? apic_timer_interrupt+0xf/0x20 [18676.158474] [18676.159675] ? native_safe_halt+0x2e/0x50 [18676.161607] kvm_wait+0xcd/0xe0 [18676.163076] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18676.166515] do_raw_spin_lock+0x177/0x190 [18676.168376] _raw_spin_lock+0x21/0x30 [18676.171074] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18676.173358] ? do_raw_spin_unlock+0x75/0x190 [18676.176322] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18676.178691] ? do_wait_intr+0xf0/0xf0 [18676.184283] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18676.187147] mdc_close+0x53b/0xd70 [mdc] [18676.189593] lmv_close+0x18d/0x490 [lmv] [18676.192530] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [18676.195434] ll_close_inode_openhandle+0x493/0xea0 [lustre] [18676.198770] ll_md_real_close+0xa5/0x2a0 [lustre] [18676.203821] ll_md_close+0x274/0x800 [lustre] [18676.206421] ll_file_release+0x128/0xc50 [lustre] [18676.207867] __fput+0x10d/0x440 [18676.209429] ____fput+0x12/0x20 [18676.211295] task_work_run+0xc6/0x110 [18676.216789] exit_to_usermode_loop+0x1dd/0x1f0 [18676.220981] do_syscall_64+0x3d6/0x440 [18676.223169] entry_SYSCALL_64_after_hwframe+0x49/0xae [18676.227650] RIP: 0033:0x7f151bee3cd8 [18676.235680] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [18676.245368] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [18676.249337] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [18676.251680] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [18676.258754] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18676.262514] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18676.266466] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18684.055020] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [18684.065004] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18684.107767] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18684.116550] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18684.124772] RIP: 0010:native_safe_halt+0x2e/0x50 [18684.128985] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18684.149584] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18684.157765] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [18684.164757] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [18684.170202] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [18684.177396] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [18684.182250] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18684.188191] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [18684.197395] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18684.205435] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18684.208423] Call Trace: [18684.212502] [18684.213275] ? show_regs.cold.9+0x22/0x2f [18684.215362] ? watchdog_timer_fn.cold.6+0x62/0x10a [18684.217664] ? __hrtimer_run_queues+0x1c2/0x4d0 [18684.225510] ? watchdog+0x40/0x40 [18684.228787] ? hrtimer_interrupt+0x128/0x2c0 [18684.236237] ? smp_apic_timer_interrupt+0xa6/0x270 [18684.237671] ? apic_timer_interrupt+0xf/0x20 [18684.243382] [18684.245138] ? native_safe_halt+0x2e/0x50 [18684.247384] kvm_wait+0xcd/0xe0 [18684.252600] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18684.255779] do_raw_spin_lock+0x177/0x190 [18684.261545] _raw_spin_lock+0x21/0x30 [18684.265050] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18684.268344] obd_mod_free+0x46/0x140 [ptlrpc] [18684.271243] mdc_commit_open+0x91/0xb0 [mdc] [18684.278256] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18684.281874] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18684.287860] after_reply+0xe2c/0x1780 [ptlrpc] [18684.292954] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18684.297684] ? __next_timer_interrupt+0x160/0x160 [18684.302164] ptlrpcd+0xa82/0xe40 [ptlrpc] [18684.306404] ? do_wait_intr+0xf0/0xf0 [18684.307797] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18684.312055] kthread+0x1d1/0x200 [18684.317105] ? set_kthread_struct+0x70/0x70 [18684.320050] ret_from_fork+0x1f/0x30 [18688.056022] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [18688.058860] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18688.083079] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18688.089637] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18688.094430] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18688.098183] RIP: 0010:native_safe_halt+0x2e/0x50 [18688.100446] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18688.109397] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18688.113367] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18688.116790] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [18688.119976] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [18688.123876] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [18688.127637] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [18688.131864] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [18688.136425] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18688.139236] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18688.143175] Call Trace: [18688.144425] [18688.145478] ? show_regs.cold.9+0x22/0x2f [18688.147472] ? watchdog_timer_fn.cold.6+0x62/0x10a [18688.150036] ? __hrtimer_run_queues+0x1c2/0x4d0 [18688.152443] ? watchdog+0x40/0x40 [18688.154420] ? hrtimer_interrupt+0x128/0x2c0 [18688.156728] ? smp_apic_timer_interrupt+0xa6/0x270 [18688.159209] ? apic_timer_interrupt+0xf/0x20 [18688.161583] [18688.162711] ? native_safe_halt+0x2e/0x50 [18688.165296] kvm_wait+0xcd/0xe0 [18688.167369] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18688.171299] do_raw_spin_lock+0x177/0x190 [18688.173965] _raw_spin_lock+0x21/0x30 [18688.176054] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18688.180883] process_one_work+0x2c8/0x700 [18688.183475] worker_thread+0x296/0x6e0 [18688.186088] ? rescuer_thread+0x570/0x570 [18688.188814] kthread+0x1d1/0x200 [18688.190726] ? set_kthread_struct+0x70/0x70 [18688.193438] ret_from_fork+0x1f/0x30 [18692.055357] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [18692.059378] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18692.095653] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18692.102094] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18692.114603] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18692.124255] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18692.146073] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18692.151323] RAX: 000000000000559e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18692.155990] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [18692.159991] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [18692.167031] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [18692.170549] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [18692.179150] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [18692.185511] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18692.196026] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [18692.210732] Call Trace: [18692.212094] [18692.212708] ? show_regs.cold.9+0x22/0x2f [18692.220572] ? watchdog_timer_fn.cold.6+0x62/0x10a [18692.225469] ? __hrtimer_run_queues+0x1c2/0x4d0 [18692.231605] ? watchdog+0x40/0x40 [18692.239376] ? hrtimer_interrupt+0x128/0x2c0 [18692.244006] ? smp_apic_timer_interrupt+0xa6/0x270 [18692.245709] ? apic_timer_interrupt+0xf/0x20 [18692.254596] [18692.257306] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18692.270779] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18692.282270] do_raw_spin_lock+0x177/0x190 [18692.287264] _raw_spin_lock+0x21/0x30 [18692.290589] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18692.293974] ? do_raw_spin_unlock+0x75/0x190 [18692.296490] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18692.305258] ? do_wait_intr+0xf0/0xf0 [18692.308885] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18692.311677] mdc_reint+0x2d/0x110 [mdc] [18692.313596] mdc_unlink+0x658/0x870 [mdc] [18692.315133] lmv_unlink+0x34d/0xbd0 [lmv] [18692.316920] ll_unlink+0x24d/0x7f0 [lustre] [18692.320382] vfs_unlink+0x202/0x330 [18692.322704] do_unlinkat+0x2fa/0x460 [18692.323832] __x64_sys_unlink+0x37/0x50 [18692.329824] do_syscall_64+0xc1/0x440 [18692.332137] entry_SYSCALL_64_after_hwframe+0x49/0xae [18692.337316] RIP: 0033:0x7f151bdee10b [18692.338629] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [18692.349329] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [18692.356025] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [18692.360172] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [18692.366332] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18692.370145] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18692.373149] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18696.161176] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16765s! [18696.170360] Showing busy workqueues and worker pools: [18696.173642] workqueue events: flags=0x0 [18696.177597] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18696.182553] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [18696.189350] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18696.193966] pending: kfree_rcu_monitor [18696.196757] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18696.200808] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [18696.206486] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18696.210469] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [18696.215498] workqueue events_long: flags=0x0 [18696.218383] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18696.222707] pending: lu_objects_flush [obdclass] [18696.226864] workqueue events_unbound: flags=0x2 [18696.229635] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [18696.233915] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [18696.240522] workqueue events_power_efficient: flags=0x80 [18696.243554] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18696.247559] pending: do_cache_clean, neigh_periodic_work, check_lifetime [18696.251612] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18696.255856] pending: neigh_periodic_work [18696.258564] workqueue mm_percpu_wq: flags=0x8 [18696.261827] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18696.265489] pending: vmstat_update, lru_add_drain_per_cpu [18696.268602] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18696.271960] pending: vmstat_update [18696.274196] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18696.278419] pending: lru_add_drain_per_cpu [18696.280812] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [18696.284968] pending: lru_add_drain_per_cpu BAR(48) [18696.288284] workqueue writeback: flags=0x4a [18696.290356] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [18696.293844] pending: wb_workfn [18696.295912] workqueue xprtiod: flags=0xa [18696.297928] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [18696.301552] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [18696.306198] workqueue ipv6_addrconf: flags=0x40008 [18696.309123] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18696.312528] pending: addrconf_verify_work [18696.314926] workqueue knbd0-recv: flags=0x1a [18696.317447] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18696.320972] in-flight: 658:recv_work [18696.323492] workqueue ptlrpc_pinger: flags=0x2 [18696.326317] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18696.329604] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [18696.332940] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16765s workers=4 idle: 32984 37 8 [18696.337139] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18681s workers=2 idle: 661 [18704.057272] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [18704.061905] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18704.120956] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18704.131959] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18704.142294] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18704.148420] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18704.171629] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [18704.175483] RAX: 0000000000003190 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18704.190935] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [18704.205844] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [18704.222759] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [18704.235695] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [18704.238297] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [18704.254242] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18704.261365] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [18704.273498] Call Trace: [18704.279545] [18704.284325] ? show_regs.cold.9+0x22/0x2f [18704.288555] ? watchdog_timer_fn.cold.6+0x62/0x10a [18704.293456] ? __hrtimer_run_queues+0x1c2/0x4d0 [18704.305801] ? watchdog+0x40/0x40 [18704.308030] ? hrtimer_interrupt+0x128/0x2c0 [18704.311552] ? smp_apic_timer_interrupt+0xa6/0x270 [18704.319927] ? apic_timer_interrupt+0xf/0x20 [18704.321340] [18704.323292] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18704.326575] do_raw_spin_lock+0x177/0x190 [18704.335356] _raw_spin_lock+0x21/0x30 [18704.337407] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18704.341750] ? do_raw_spin_unlock+0x75/0x190 [18704.349554] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18704.351546] ? do_wait_intr+0xf0/0xf0 [18704.357452] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18704.370127] mdc_close+0x53b/0xd70 [mdc] [18704.373307] lmv_close+0x18d/0x490 [lmv] [18704.374817] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [18704.388465] ll_close_inode_openhandle+0x493/0xea0 [lustre] [18704.395735] ll_md_real_close+0xa5/0x2a0 [lustre] [18704.397669] ll_md_close+0x274/0x800 [lustre] [18704.402588] ll_file_release+0x128/0xc50 [lustre] [18704.413442] __fput+0x10d/0x440 [18704.419395] ____fput+0x12/0x20 [18704.424289] task_work_run+0xc6/0x110 [18704.426966] exit_to_usermode_loop+0x1dd/0x1f0 [18704.428984] do_syscall_64+0x3d6/0x440 [18704.430865] entry_SYSCALL_64_after_hwframe+0x49/0xae [18704.433257] RIP: 0033:0x7f151bee3cd8 [18704.434548] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [18704.465823] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [18704.478421] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [18704.480553] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [18704.497017] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18704.511264] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18704.522383] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18712.055006] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [18712.066499] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18712.100059] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18712.114879] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18712.122212] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18712.130429] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18712.152166] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18712.166222] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [18712.172161] RDX: 0000000000007e08 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [18712.179069] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [18712.183216] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [18712.188775] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18712.196483] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [18712.202777] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18712.209248] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18712.214413] Call Trace: [18712.216437] [18712.217888] ? show_regs.cold.9+0x22/0x2f [18712.219243] ? watchdog_timer_fn.cold.6+0x62/0x10a [18712.227753] ? __hrtimer_run_queues+0x1c2/0x4d0 [18712.229395] ? watchdog+0x40/0x40 [18712.230630] ? hrtimer_interrupt+0x128/0x2c0 [18712.248181] ? smp_apic_timer_interrupt+0xa6/0x270 [18712.251879] ? apic_timer_interrupt+0xf/0x20 [18712.253404] [18712.254179] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18712.265460] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18712.273146] do_raw_spin_lock+0x177/0x190 [18712.276054] _raw_spin_lock+0x21/0x30 [18712.280349] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18712.289646] obd_mod_free+0x46/0x140 [ptlrpc] [18712.295477] mdc_commit_open+0x91/0xb0 [mdc] [18712.300108] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18712.309197] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18712.316808] after_reply+0xe2c/0x1780 [ptlrpc] [18712.320922] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18712.331490] ? __next_timer_interrupt+0x160/0x160 [18712.333808] ptlrpcd+0xa82/0xe40 [ptlrpc] [18712.336633] ? do_wait_intr+0xf0/0xf0 [18712.338321] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18712.340845] kthread+0x1d1/0x200 [18712.342484] ? set_kthread_struct+0x70/0x70 [18712.344311] ret_from_fork+0x1f/0x30 [18716.056107] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [18716.059501] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18716.107302] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18716.123139] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18716.134030] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18716.139367] RIP: 0010:native_safe_halt+0x2e/0x50 [18716.141573] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18716.156387] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18716.161550] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18716.166467] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [18716.170381] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [18716.176899] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [18716.180345] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [18716.183228] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [18716.187090] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18716.196136] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18716.200444] Call Trace: [18716.201749] [18716.202969] ? show_regs.cold.9+0x22/0x2f [18716.205509] ? watchdog_timer_fn.cold.6+0x62/0x10a [18716.208290] ? __hrtimer_run_queues+0x1c2/0x4d0 [18716.210704] ? watchdog+0x40/0x40 [18716.212797] ? hrtimer_interrupt+0x128/0x2c0 [18716.215357] ? smp_apic_timer_interrupt+0xa6/0x270 [18716.226365] ? apic_timer_interrupt+0xf/0x20 [18716.230507] [18716.231642] ? native_safe_halt+0x2e/0x50 [18716.233442] kvm_wait+0xcd/0xe0 [18716.235166] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18716.237571] do_raw_spin_lock+0x177/0x190 [18716.239924] _raw_spin_lock+0x21/0x30 [18716.241621] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18716.245620] process_one_work+0x2c8/0x700 [18716.247895] worker_thread+0x296/0x6e0 [18716.249560] ? rescuer_thread+0x570/0x570 [18716.251698] kthread+0x1d1/0x200 [18716.253384] ? set_kthread_struct+0x70/0x70 [18716.254954] ret_from_fork+0x1f/0x30 [18724.053708] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [18724.056731] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18724.102976] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18724.111981] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18724.115616] RIP: 0010:native_safe_halt+0x2e/0x50 [18724.118255] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18724.131880] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18724.143732] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18724.150802] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [18724.159872] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [18724.174828] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [18724.182737] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [18724.190892] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [18724.199312] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18724.209618] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [18724.212583] Call Trace: [18724.213706] [18724.214655] ? show_regs.cold.9+0x22/0x2f [18724.216387] ? watchdog_timer_fn.cold.6+0x62/0x10a [18724.218304] ? __hrtimer_run_queues+0x1c2/0x4d0 [18724.224679] ? watchdog+0x40/0x40 [18724.230305] ? hrtimer_interrupt+0x128/0x2c0 [18724.237809] ? smp_apic_timer_interrupt+0xa6/0x270 [18724.247660] ? apic_timer_interrupt+0xf/0x20 [18724.250319] [18724.252803] ? native_safe_halt+0x2e/0x50 [18724.256802] kvm_wait+0xcd/0xe0 [18724.257972] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18724.266356] do_raw_spin_lock+0x177/0x190 [18724.268788] _raw_spin_lock+0x21/0x30 [18724.271532] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18724.275615] ? do_raw_spin_unlock+0x75/0x190 [18724.278667] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18724.293640] ? do_wait_intr+0xf0/0xf0 [18724.302220] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18724.310620] mdc_reint+0x2d/0x110 [mdc] [18724.315625] mdc_unlink+0x658/0x870 [mdc] [18724.323344] lmv_unlink+0x34d/0xbd0 [lmv] [18724.330572] ll_unlink+0x24d/0x7f0 [lustre] [18724.332774] vfs_unlink+0x202/0x330 [18724.339295] do_unlinkat+0x2fa/0x460 [18724.346173] __x64_sys_unlink+0x37/0x50 [18724.356244] do_syscall_64+0xc1/0x440 [18724.361619] entry_SYSCALL_64_after_hwframe+0x49/0xae [18724.369534] RIP: 0033:0x7f151bdee10b [18724.371820] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [18724.415943] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [18724.425636] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [18724.438244] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [18724.447532] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18724.456865] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18724.464161] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18724.995114] rcu: INFO: rcu_sched self-detected stall on CPU [18724.996688] rcu: 1-...!: (10098623 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [18725.003365] (t=16800348 jiffies g=249657 q=7672) [18725.013845] rcu: rcu_sched kthread starved for 16795300 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [18725.017906] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [18725.023909] rcu: RCU grace-period kthread stack dump: [18725.025696] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [18725.039519] Call Trace: [18725.041105] __schedule+0x351/0xcb0 [18725.042499] schedule+0xc0/0x180 [18725.044219] schedule_timeout+0xb4/0x190 [18725.047346] ? do_raw_spin_unlock+0x75/0x190 [18725.049596] ? __next_timer_interrupt+0x160/0x160 [18725.052265] ? prepare_to_swait_event+0xf3/0x140 [18725.056213] rcu_gp_kthread+0x8dd/0x15b0 [18725.057323] ? force_qs_rnp+0x2c0/0x2c0 [18725.060128] kthread+0x1d1/0x200 [18725.064918] ? set_kthread_struct+0x70/0x70 [18725.068356] ret_from_fork+0x1f/0x30 [18725.069732] rcu: Stack dump where RCU GP kthread last ran: [18725.070988] NMI backtrace for cpu 1 [18725.075154] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18725.085802] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18725.092993] Call Trace: [18725.095102] [18725.099646] ? dump_stack+0xbb/0x10e [18725.102191] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [18725.104860] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [18725.106140] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [18725.108122] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [18725.112595] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [18725.115085] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [18725.119102] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [18725.120338] ? update_process_times+0x6d/0xc0 [18725.122266] ? tick_sched_handle.isra.18+0x27/0x90 [18725.123830] ? tick_sched_timer+0x53/0xb0 [18725.128975] ? __hrtimer_run_queues+0x1c2/0x4d0 [18725.131801] ? tick_sched_handle.isra.18+0x90/0x90 [18725.134282] ? hrtimer_interrupt+0x128/0x2c0 [18725.136110] ? smp_apic_timer_interrupt+0xa6/0x270 [18725.138919] ? apic_timer_interrupt+0xf/0x20 [18725.140359] [18725.141324] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18725.143968] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18725.145395] ? do_raw_spin_lock+0x177/0x190 [18725.148636] ? _raw_spin_lock+0x21/0x30 [18725.150096] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18725.155786] ? obd_mod_free+0x46/0x140 [ptlrpc] [18725.157104] ? mdc_commit_open+0x91/0xb0 [mdc] [18725.158957] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18725.163064] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18725.169677] ? after_reply+0xe2c/0x1780 [ptlrpc] [18725.171134] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18725.175946] ? __next_timer_interrupt+0x160/0x160 [18725.180317] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [18725.182302] ? do_wait_intr+0xf0/0xf0 [18725.183194] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18725.186359] ? kthread+0x1d1/0x200 [18725.187309] ? set_kthread_struct+0x70/0x70 [18725.191486] ? ret_from_fork+0x1f/0x30 [18725.194150] NMI backtrace for cpu 1 [18725.195829] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18725.202771] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18725.206970] Call Trace: [18725.208128] [18725.209017] ? dump_stack+0xbb/0x10e [18725.211057] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [18725.212642] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [18725.214109] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [18725.223223] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [18725.228575] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [18725.234898] ? rcu_dump_cpu_stacks+0x11b/0x171 [18725.236449] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [18725.240100] ? update_process_times+0x6d/0xc0 [18725.242932] ? tick_sched_handle.isra.18+0x27/0x90 [18725.250444] ? tick_sched_timer+0x53/0xb0 [18725.251447] ? __hrtimer_run_queues+0x1c2/0x4d0 [18725.253824] ? tick_sched_handle.isra.18+0x90/0x90 [18725.255192] ? hrtimer_interrupt+0x128/0x2c0 [18725.257335] ? smp_apic_timer_interrupt+0xa6/0x270 [18725.262389] ? apic_timer_interrupt+0xf/0x20 [18725.264291] [18725.265121] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18725.267081] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18725.268987] ? do_raw_spin_lock+0x177/0x190 [18725.272185] ? _raw_spin_lock+0x21/0x30 [18725.276638] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18725.279027] ? obd_mod_free+0x46/0x140 [ptlrpc] [18725.281547] ? mdc_commit_open+0x91/0xb0 [mdc] [18725.283321] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18725.285392] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18725.289288] ? after_reply+0xe2c/0x1780 [ptlrpc] [18725.291349] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18725.294897] ? __next_timer_interrupt+0x160/0x160 [18725.299225] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [18725.301903] ? do_wait_intr+0xf0/0xf0 [18725.303679] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18725.308322] ? kthread+0x1d1/0x200 [18725.312281] ? set_kthread_struct+0x70/0x70 [18725.314285] ? ret_from_fork+0x1f/0x30 [18726.387311] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16795s! [18726.399311] Showing busy workqueues and worker pools: [18726.408161] workqueue events: flags=0x0 [18726.411291] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18726.429705] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [18726.441409] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18726.453924] pending: kfree_rcu_monitor [18726.456082] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18726.468196] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [18726.487331] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18726.489367] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [18726.499129] workqueue events_long: flags=0x0 [18726.505550] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18726.514049] pending: lu_objects_flush [obdclass] [18726.516291] workqueue events_unbound: flags=0x2 [18726.522209] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [18726.533167] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [18726.549312] workqueue events_power_efficient: flags=0x80 [18726.561185] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18726.564475] pending: do_cache_clean, neigh_periodic_work, check_lifetime [18726.578483] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18726.589147] pending: neigh_periodic_work [18726.595384] workqueue mm_percpu_wq: flags=0x8 [18726.606180] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18726.614207] pending: vmstat_update, lru_add_drain_per_cpu [18726.617763] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18726.623310] pending: vmstat_update [18726.626281] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18726.633791] pending: lru_add_drain_per_cpu [18726.636869] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [18726.642308] pending: lru_add_drain_per_cpu BAR(48) [18726.647510] workqueue writeback: flags=0x4a [18726.652864] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [18726.657880] pending: wb_workfn [18726.659351] workqueue xprtiod: flags=0xa [18726.663040] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [18726.669917] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [18726.676650] workqueue ipv6_addrconf: flags=0x40008 [18726.686179] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18726.689287] pending: addrconf_verify_work [18726.696170] workqueue knbd0-recv: flags=0x1a [18726.702857] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18726.708351] in-flight: 658:recv_work [18726.716006] workqueue ptlrpc_pinger: flags=0x2 [18726.720168] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18726.727103] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [18726.733382] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16796s workers=4 idle: 32984 37 8 [18726.737498] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18711s workers=2 idle: 661 [18732.057010] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [18732.060856] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18732.099613] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18732.106421] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18732.110747] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [18732.113680] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [18732.122770] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18732.125915] RAX: 0000000000006650 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18732.129170] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [18732.132247] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [18732.136582] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [18732.140187] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [18732.143708] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [18732.147566] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18732.150571] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [18732.153658] Call Trace: [18732.154514] [18732.155074] ? show_regs.cold.9+0x22/0x2f [18732.156494] ? watchdog_timer_fn.cold.6+0x62/0x10a [18732.158213] ? __hrtimer_run_queues+0x1c2/0x4d0 [18732.159560] ? watchdog+0x40/0x40 [18732.160517] ? hrtimer_interrupt+0x128/0x2c0 [18732.161787] ? smp_apic_timer_interrupt+0xa6/0x270 [18732.163125] ? apic_timer_interrupt+0xf/0x20 [18732.164183] [18732.164701] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [18732.166098] do_raw_spin_lock+0x177/0x190 [18732.167120] _raw_spin_lock+0x21/0x30 [18732.168059] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18732.170649] ? do_raw_spin_unlock+0x75/0x190 [18732.171853] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18732.174197] ? do_wait_intr+0xf0/0xf0 [18732.175154] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18732.176675] mdc_close+0x53b/0xd70 [mdc] [18732.177789] lmv_close+0x18d/0x490 [lmv] [18732.179166] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [18732.180812] ll_close_inode_openhandle+0x493/0xea0 [lustre] [18732.182513] ll_md_real_close+0xa5/0x2a0 [lustre] [18732.183857] ll_md_close+0x274/0x800 [lustre] [18732.185112] ll_file_release+0x128/0xc50 [lustre] [18732.186679] __fput+0x10d/0x440 [18732.187584] ____fput+0x12/0x20 [18732.188675] task_work_run+0xc6/0x110 [18732.189849] exit_to_usermode_loop+0x1dd/0x1f0 [18732.191148] do_syscall_64+0x3d6/0x440 [18732.192260] entry_SYSCALL_64_after_hwframe+0x49/0xae [18732.193709] RIP: 0033:0x7f151bee3cd8 [18732.194827] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [18732.201311] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [18732.203793] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [18732.206202] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [18732.208289] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18732.210300] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18732.212613] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18744.056567] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [18744.060211] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18744.085588] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18744.090122] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18744.093777] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18744.096475] RIP: 0010:native_safe_halt+0x2e/0x50 [18744.098505] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18744.104091] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18744.106455] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18744.108892] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [18744.111444] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [18744.114174] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [18744.116733] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [18744.119416] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [18744.122092] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18744.124394] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18744.127055] Call Trace: [18744.128017] [18744.128786] ? show_regs.cold.9+0x22/0x2f [18744.130143] ? watchdog_timer_fn.cold.6+0x62/0x10a [18744.132042] ? __hrtimer_run_queues+0x1c2/0x4d0 [18744.133563] ? watchdog+0x40/0x40 [18744.134907] ? hrtimer_interrupt+0x128/0x2c0 [18744.136519] ? smp_apic_timer_interrupt+0xa6/0x270 [18744.137995] ? apic_timer_interrupt+0xf/0x20 [18744.139607] [18744.140467] ? native_safe_halt+0x2e/0x50 [18744.141970] kvm_wait+0xcd/0xe0 [18744.143131] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18744.144958] do_raw_spin_lock+0x177/0x190 [18744.146041] _raw_spin_lock+0x21/0x30 [18744.147550] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18744.149731] process_one_work+0x2c8/0x700 [18744.151213] worker_thread+0x296/0x6e0 [18744.152751] ? rescuer_thread+0x570/0x570 [18744.154360] kthread+0x1d1/0x200 [18744.155558] ? set_kthread_struct+0x70/0x70 [18744.157115] ret_from_fork+0x1f/0x30 [18752.052028] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [18752.055054] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [18752.056607] Modules linked in: lustre(O) [18752.063943] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18752.064125] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18752.064134] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18752.064137] RIP: 0010:native_safe_halt+0x2e/0x50 [18752.064207] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18752.064213] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [18752.064218] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [18752.064220] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [18752.064222] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [18752.064224] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [18752.064226] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18752.064232] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [18752.064235] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18752.064237] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18752.064245] Call Trace: [18752.064321] [18752.064327] ? show_regs.cold.9+0x22/0x2f [18752.064337] ? watchdog_timer_fn.cold.6+0x62/0x10a [18752.064344] ? __hrtimer_run_queues+0x1c2/0x4d0 [18752.064348] ? watchdog+0x40/0x40 [18752.064353] ? hrtimer_interrupt+0x128/0x2c0 [18752.064361] ? smp_apic_timer_interrupt+0xa6/0x270 [18752.064368] ? apic_timer_interrupt+0xf/0x20 [18752.064372] [18752.064443] ? native_safe_halt+0x2e/0x50 [18752.064448] kvm_wait+0xcd/0xe0 [18752.064461] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18752.064469] do_raw_spin_lock+0x177/0x190 [18752.064473] _raw_spin_lock+0x21/0x30 [18752.064479] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18752.065005] obd_mod_free+0x46/0x140 [ptlrpc] [18752.065364] mdc_commit_open+0x91/0xb0 [mdc] [18752.065489] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18752.065784] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18752.071153] osp(O) ofd(O) [18752.106200] after_reply+0xe2c/0x1780 [ptlrpc] [18752.108064] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18752.108398] ? __next_timer_interrupt+0x160/0x160 [18752.108409] ptlrpcd+0xa82/0xe40 [ptlrpc] [18752.108761] ? do_wait_intr+0xf0/0xf0 [18752.108772] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18752.109038] kthread+0x1d1/0x200 [18752.109044] ? set_kthread_struct+0x70/0x70 [18752.109048] ret_from_fork+0x1f/0x30 [18752.312202] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18752.343700] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18752.354116] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18752.360212] RIP: 0010:native_safe_halt+0x2e/0x50 [18752.363778] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18752.378961] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18752.382419] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18752.387338] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [18752.393034] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [18752.396063] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [18752.398396] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [18752.400920] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [18752.408843] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18752.412494] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [18752.417981] Call Trace: [18752.420256] [18752.421091] ? show_regs.cold.9+0x22/0x2f [18752.422803] ? watchdog_timer_fn.cold.6+0x62/0x10a [18752.425182] ? __hrtimer_run_queues+0x1c2/0x4d0 [18752.428241] ? watchdog+0x40/0x40 [18752.433643] ? hrtimer_interrupt+0x128/0x2c0 [18752.436864] ? smp_apic_timer_interrupt+0xa6/0x270 [18752.439311] ? apic_timer_interrupt+0xf/0x20 [18752.441649] [18752.442792] ? native_safe_halt+0x2e/0x50 [18752.444864] kvm_wait+0xcd/0xe0 [18752.446359] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18752.450038] do_raw_spin_lock+0x177/0x190 [18752.452235] _raw_spin_lock+0x21/0x30 [18752.455359] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18752.459732] ? do_raw_spin_unlock+0x75/0x190 [18752.463106] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18752.464706] ? do_wait_intr+0xf0/0xf0 [18752.466637] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18752.469367] mdc_reint+0x2d/0x110 [mdc] [18752.473168] mdc_unlink+0x658/0x870 [mdc] [18752.475969] lmv_unlink+0x34d/0xbd0 [lmv] [18752.477829] ll_unlink+0x24d/0x7f0 [lustre] [18752.482451] vfs_unlink+0x202/0x330 [18752.483563] do_unlinkat+0x2fa/0x460 [18752.484754] __x64_sys_unlink+0x37/0x50 [18752.485997] do_syscall_64+0xc1/0x440 [18752.492505] entry_SYSCALL_64_after_hwframe+0x49/0xae [18752.497301] RIP: 0033:0x7f151bdee10b [18752.499474] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [18752.516269] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [18752.521923] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [18752.527088] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [18752.534084] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18752.538447] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18752.540658] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18757.096048] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16826s! [18757.098871] Showing busy workqueues and worker pools: [18757.101247] workqueue events: flags=0x0 [18757.102949] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18757.106793] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [18757.110555] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18757.112452] pending: kfree_rcu_monitor [18757.113987] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18757.117687] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [18757.120785] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18757.123572] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [18757.126424] workqueue events_long: flags=0x0 [18757.128267] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18757.130688] pending: lu_objects_flush [obdclass] [18757.133870] workqueue events_unbound: flags=0x2 [18757.135839] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [18757.138443] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [18757.142586] workqueue events_power_efficient: flags=0x80 [18757.144639] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18757.147503] pending: do_cache_clean, neigh_periodic_work, check_lifetime [18757.150354] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18757.153174] pending: neigh_periodic_work [18757.154886] workqueue mm_percpu_wq: flags=0x8 [18757.156747] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18757.159400] pending: vmstat_update, lru_add_drain_per_cpu [18757.161934] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18757.164767] pending: vmstat_update [18757.166680] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18757.168938] pending: lru_add_drain_per_cpu [18757.171383] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [18757.173752] pending: lru_add_drain_per_cpu BAR(48) [18757.175478] workqueue writeback: flags=0x4a [18757.176679] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [18757.178656] pending: wb_workfn [18757.179768] workqueue xprtiod: flags=0xa [18757.181083] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [18757.183693] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [18757.186574] workqueue ipv6_addrconf: flags=0x40008 [18757.188412] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18757.190621] pending: addrconf_verify_work [18757.192241] workqueue knbd0-recv: flags=0x1a [18757.193611] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18757.195802] in-flight: 658:recv_work [18757.197158] workqueue ptlrpc_pinger: flags=0x2 [18757.198647] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18757.200628] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [18757.202810] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16826s workers=4 idle: 32984 37 8 [18757.205339] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18741s workers=2 idle: 661 [18760.057061] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [18760.061340] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18760.097799] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18760.109403] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18760.114266] RIP: 0010:native_safe_halt+0x2e/0x50 [18760.116551] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18760.129699] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18760.133294] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18760.136279] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [18760.144613] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [18760.149675] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [18760.152497] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [18760.155847] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [18760.160914] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18760.163626] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [18760.172564] Call Trace: [18760.177307] [18760.177763] ? show_regs.cold.9+0x22/0x2f [18760.182324] ? watchdog_timer_fn.cold.6+0x62/0x10a [18760.187611] ? __hrtimer_run_queues+0x1c2/0x4d0 [18760.190167] ? watchdog+0x40/0x40 [18760.194755] ? hrtimer_interrupt+0x128/0x2c0 [18760.199567] ? smp_apic_timer_interrupt+0xa6/0x270 [18760.207110] ? apic_timer_interrupt+0xf/0x20 [18760.210593] [18760.214638] ? native_safe_halt+0x2e/0x50 [18760.219226] kvm_wait+0xcd/0xe0 [18760.222021] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18760.231952] do_raw_spin_lock+0x177/0x190 [18760.233394] _raw_spin_lock+0x21/0x30 [18760.238086] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18760.242736] ? do_raw_spin_unlock+0x75/0x190 [18760.246238] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18760.254815] ? do_wait_intr+0xf0/0xf0 [18760.256117] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18760.262505] mdc_close+0x53b/0xd70 [mdc] [18760.263683] lmv_close+0x18d/0x490 [lmv] [18760.265095] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [18760.273129] ll_close_inode_openhandle+0x493/0xea0 [lustre] [18760.277695] ll_md_real_close+0xa5/0x2a0 [lustre] [18760.282269] ll_md_close+0x274/0x800 [lustre] [18760.286336] ll_file_release+0x128/0xc50 [lustre] [18760.289515] __fput+0x10d/0x440 [18760.292144] ____fput+0x12/0x20 [18760.295232] task_work_run+0xc6/0x110 [18760.296767] exit_to_usermode_loop+0x1dd/0x1f0 [18760.302179] do_syscall_64+0x3d6/0x440 [18760.304229] entry_SYSCALL_64_after_hwframe+0x49/0xae [18760.308392] RIP: 0033:0x7f151bee3cd8 [18760.309715] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [18760.326465] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [18760.329168] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [18760.332707] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [18760.337994] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18760.342455] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18760.346680] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18772.056226] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [18772.061726] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18772.102818] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18772.110110] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18772.115663] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18772.124277] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18772.128549] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18772.151387] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18772.154802] RAX: 0000000000007213 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18772.159112] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [18772.163357] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [18772.169890] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [18772.179012] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [18772.182856] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [18772.189734] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18772.195912] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18772.198027] Call Trace: [18772.198764] [18772.200277] ? show_regs.cold.9+0x22/0x2f [18772.202720] ? watchdog_timer_fn.cold.6+0x62/0x10a [18772.204774] ? __hrtimer_run_queues+0x1c2/0x4d0 [18772.209307] ? watchdog+0x40/0x40 [18772.218476] ? hrtimer_interrupt+0x128/0x2c0 [18772.223042] ? smp_apic_timer_interrupt+0xa6/0x270 [18772.224464] ? apic_timer_interrupt+0xf/0x20 [18772.225615] [18772.237218] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18772.245014] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18772.253937] do_raw_spin_lock+0x177/0x190 [18772.262929] _raw_spin_lock+0x21/0x30 [18772.264937] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18772.277677] process_one_work+0x2c8/0x700 [18772.289133] worker_thread+0x296/0x6e0 [18772.291825] ? rescuer_thread+0x570/0x570 [18772.295912] kthread+0x1d1/0x200 [18772.298387] ? set_kthread_struct+0x70/0x70 [18772.302474] ret_from_fork+0x1f/0x30 [18780.055268] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [18780.059436] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18780.107408] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18780.120414] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18780.126128] RIP: 0010:native_safe_halt+0x2e/0x50 [18780.128058] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18780.143155] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18780.146395] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [18780.154157] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [18780.157473] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [18780.160577] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [18780.162976] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18780.164719] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [18780.170166] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18780.176321] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18780.184795] Call Trace: [18780.186490] [18780.187211] ? show_regs.cold.9+0x22/0x2f [18780.190675] ? watchdog_timer_fn.cold.6+0x62/0x10a [18780.196064] ? __hrtimer_run_queues+0x1c2/0x4d0 [18780.197508] ? watchdog+0x40/0x40 [18780.199339] ? hrtimer_interrupt+0x128/0x2c0 [18780.201207] ? smp_apic_timer_interrupt+0xa6/0x270 [18780.203673] ? apic_timer_interrupt+0xf/0x20 [18780.205728] [18780.206721] ? native_safe_halt+0x2e/0x50 [18780.208285] kvm_wait+0xcd/0xe0 [18780.209710] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18780.211697] do_raw_spin_lock+0x177/0x190 [18780.213335] _raw_spin_lock+0x21/0x30 [18780.215289] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18780.219296] obd_mod_free+0x46/0x140 [ptlrpc] [18780.222077] mdc_commit_open+0x91/0xb0 [mdc] [18780.224688] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18780.232412] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18780.242958] after_reply+0xe2c/0x1780 [ptlrpc] [18780.246150] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18780.248907] ? __next_timer_interrupt+0x160/0x160 [18780.252155] ptlrpcd+0xa82/0xe40 [ptlrpc] [18780.254395] ? do_wait_intr+0xf0/0xf0 [18780.256459] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18780.258774] kthread+0x1d1/0x200 [18780.259981] ? set_kthread_struct+0x70/0x70 [18780.263315] ret_from_fork+0x1f/0x30 [18784.052045] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [18784.057656] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18784.087527] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18784.099390] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18784.112134] RIP: 0010:native_safe_halt+0x2e/0x50 [18784.115329] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18784.128366] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18784.137348] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18784.140710] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [18784.145580] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [18784.150214] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [18784.154290] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [18784.160929] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [18784.165092] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18784.168032] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [18784.173424] Call Trace: [18784.174686] [18784.175613] ? show_regs.cold.9+0x22/0x2f [18784.177402] ? watchdog_timer_fn.cold.6+0x62/0x10a [18784.180853] ? __hrtimer_run_queues+0x1c2/0x4d0 [18784.183161] ? watchdog+0x40/0x40 [18784.185119] ? hrtimer_interrupt+0x128/0x2c0 [18784.187123] ? smp_apic_timer_interrupt+0xa6/0x270 [18784.189720] ? apic_timer_interrupt+0xf/0x20 [18784.192285] [18784.193532] ? native_safe_halt+0x2e/0x50 [18784.198004] kvm_wait+0xcd/0xe0 [18784.199845] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18784.203584] do_raw_spin_lock+0x177/0x190 [18784.205555] _raw_spin_lock+0x21/0x30 [18784.207232] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18784.211821] ? do_raw_spin_unlock+0x75/0x190 [18784.214170] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18784.217406] ? do_wait_intr+0xf0/0xf0 [18784.219813] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18784.222294] mdc_reint+0x2d/0x110 [mdc] [18784.223696] mdc_unlink+0x658/0x870 [mdc] [18784.224475] lmv_unlink+0x34d/0xbd0 [lmv] [18784.225469] ll_unlink+0x24d/0x7f0 [lustre] [18784.226616] vfs_unlink+0x202/0x330 [18784.227768] do_unlinkat+0x2fa/0x460 [18784.229004] __x64_sys_unlink+0x37/0x50 [18784.230818] do_syscall_64+0xc1/0x440 [18784.232111] entry_SYSCALL_64_after_hwframe+0x49/0xae [18784.234025] RIP: 0033:0x7f151bdee10b [18784.235642] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [18784.247437] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [18784.250751] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [18784.254127] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [18784.257273] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18784.260150] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18784.263142] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18787.297034] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16856s! [18787.309083] Showing busy workqueues and worker pools: [18787.312495] workqueue events: flags=0x0 [18787.316248] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18787.322170] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [18787.331501] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18787.338369] pending: kfree_rcu_monitor [18787.343190] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18787.348874] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [18787.361119] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18787.365751] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [18787.375496] workqueue events_long: flags=0x0 [18787.382966] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18787.393560] pending: lu_objects_flush [obdclass] [18787.400159] workqueue events_unbound: flags=0x2 [18787.403413] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [18787.407586] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [18787.417847] workqueue events_power_efficient: flags=0x80 [18787.425504] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18787.432988] pending: do_cache_clean, neigh_periodic_work, check_lifetime [18787.444057] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18787.456733] pending: neigh_periodic_work [18787.458708] workqueue mm_percpu_wq: flags=0x8 [18787.462675] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18787.469444] pending: vmstat_update, lru_add_drain_per_cpu [18787.473153] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18787.476469] pending: vmstat_update [18787.479321] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18787.483368] pending: lru_add_drain_per_cpu [18787.486102] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [18787.489449] pending: lru_add_drain_per_cpu BAR(48) [18787.493136] workqueue writeback: flags=0x4a [18787.499081] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [18787.516356] pending: wb_workfn [18787.517445] workqueue xprtiod: flags=0xa [18787.518808] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [18787.521768] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [18787.526121] workqueue ipv6_addrconf: flags=0x40008 [18787.528285] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18787.530613] pending: addrconf_verify_work [18787.532502] workqueue knbd0-recv: flags=0x1a [18787.534369] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18787.537382] in-flight: 658:recv_work [18787.538887] workqueue ptlrpc_pinger: flags=0x2 [18787.540731] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18787.543548] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [18787.546343] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16856s workers=4 idle: 32984 37 8 [18787.549784] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18772s workers=2 idle: 661 [18788.058397] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [18788.067394] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18788.114974] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18788.122750] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18788.127750] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18788.130438] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18788.138434] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18788.141689] RAX: 0000000000001113 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18788.145303] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [18788.150541] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [18788.154015] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [18788.160113] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [18788.165231] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [18788.170559] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18788.172993] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [18788.176227] Call Trace: [18788.182286] [18788.183210] ? show_regs.cold.9+0x22/0x2f [18788.184940] ? watchdog_timer_fn.cold.6+0x62/0x10a [18788.187606] ? __hrtimer_run_queues+0x1c2/0x4d0 [18788.189775] ? watchdog+0x40/0x40 [18788.191527] ? hrtimer_interrupt+0x128/0x2c0 [18788.197771] ? smp_apic_timer_interrupt+0xa6/0x270 [18788.206238] ? apic_timer_interrupt+0xf/0x20 [18788.208443] [18788.209960] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18788.212949] do_raw_spin_lock+0x177/0x190 [18788.215129] _raw_spin_lock+0x21/0x30 [18788.217070] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18788.220086] ? do_raw_spin_unlock+0x75/0x190 [18788.222204] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18788.225238] ? do_wait_intr+0xf0/0xf0 [18788.226919] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18788.229508] mdc_close+0x53b/0xd70 [mdc] [18788.231445] lmv_close+0x18d/0x490 [lmv] [18788.233247] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [18788.236079] ll_close_inode_openhandle+0x493/0xea0 [lustre] [18788.239324] ll_md_real_close+0xa5/0x2a0 [lustre] [18788.241727] ll_md_close+0x274/0x800 [lustre] [18788.243773] ll_file_release+0x128/0xc50 [lustre] [18788.245546] __fput+0x10d/0x440 [18788.246887] ____fput+0x12/0x20 [18788.248105] task_work_run+0xc6/0x110 [18788.249148] exit_to_usermode_loop+0x1dd/0x1f0 [18788.250422] do_syscall_64+0x3d6/0x440 [18788.252159] entry_SYSCALL_64_after_hwframe+0x49/0xae [18788.253749] RIP: 0033:0x7f151bee3cd8 [18788.255635] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [18788.263676] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [18788.267706] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [18788.271427] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [18788.274939] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18788.278245] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18788.281208] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18800.056081] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [18800.060338] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18800.083796] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18800.088672] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18800.092125] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18800.095119] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18800.098036] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18800.105992] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18800.109161] RAX: 000000000000170a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18800.112554] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [18800.115669] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [18800.118628] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [18800.121637] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [18800.124549] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [18800.127795] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18800.130349] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18800.133426] Call Trace: [18800.134295] [18800.135377] ? show_regs.cold.9+0x22/0x2f [18800.136942] ? watchdog_timer_fn.cold.6+0x62/0x10a [18800.138964] ? __hrtimer_run_queues+0x1c2/0x4d0 [18800.140955] ? watchdog+0x40/0x40 [18800.142323] ? hrtimer_interrupt+0x128/0x2c0 [18800.144180] ? smp_apic_timer_interrupt+0xa6/0x270 [18800.146151] ? apic_timer_interrupt+0xf/0x20 [18800.148008] [18800.149073] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18800.151282] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18800.154867] do_raw_spin_lock+0x177/0x190 [18800.156540] _raw_spin_lock+0x21/0x30 [18800.158003] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18800.160495] process_one_work+0x2c8/0x700 [18800.162207] worker_thread+0x296/0x6e0 [18800.163826] ? rescuer_thread+0x570/0x570 [18800.165353] kthread+0x1d1/0x200 [18800.166818] ? set_kthread_struct+0x70/0x70 [18800.168608] ret_from_fork+0x1f/0x30 [18808.055026] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [18808.057512] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18808.075766] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18808.081871] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18808.089145] RIP: 0010:native_safe_halt+0x2e/0x50 [18808.092854] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18808.105558] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18808.113537] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [18808.119234] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [18808.125441] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [18808.132041] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [18808.140363] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18808.147027] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [18808.150261] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18808.152762] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18808.156252] Call Trace: [18808.157333] [18808.158751] ? show_regs.cold.9+0x22/0x2f [18808.160493] ? watchdog_timer_fn.cold.6+0x62/0x10a [18808.163162] ? __hrtimer_run_queues+0x1c2/0x4d0 [18808.164747] ? watchdog+0x40/0x40 [18808.166785] ? hrtimer_interrupt+0x128/0x2c0 [18808.170399] ? smp_apic_timer_interrupt+0xa6/0x270 [18808.171848] ? apic_timer_interrupt+0xf/0x20 [18808.176518] [18808.177325] ? native_safe_halt+0x2e/0x50 [18808.180233] kvm_wait+0xcd/0xe0 [18808.183271] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18808.188125] do_raw_spin_lock+0x177/0x190 [18808.190499] _raw_spin_lock+0x21/0x30 [18808.195140] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18808.197399] obd_mod_free+0x46/0x140 [ptlrpc] [18808.200219] mdc_commit_open+0x91/0xb0 [mdc] [18808.202604] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18808.204461] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18808.209591] after_reply+0xe2c/0x1780 [ptlrpc] [18808.213382] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18808.215217] ? __next_timer_interrupt+0x160/0x160 [18808.216705] ptlrpcd+0xa82/0xe40 [ptlrpc] [18808.223961] ? do_wait_intr+0xf0/0xf0 [18808.226089] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18808.228152] kthread+0x1d1/0x200 [18808.230992] ? set_kthread_struct+0x70/0x70 [18808.234958] ret_from_fork+0x1f/0x30 [18812.055382] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [18812.062539] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18812.085326] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18812.089992] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18812.094284] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18812.097552] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18812.105681] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18812.109266] RAX: 0000000000007f8b RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18812.113157] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [18812.117053] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [18812.120837] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [18812.124658] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [18812.128598] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [18812.132091] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18812.135960] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [18812.139290] Call Trace: [18812.140469] [18812.141377] ? show_regs.cold.9+0x22/0x2f [18812.143354] ? watchdog_timer_fn.cold.6+0x62/0x10a [18812.145443] ? __hrtimer_run_queues+0x1c2/0x4d0 [18812.147818] ? watchdog+0x40/0x40 [18812.149955] ? hrtimer_interrupt+0x128/0x2c0 [18812.152410] ? smp_apic_timer_interrupt+0xa6/0x270 [18812.154606] ? apic_timer_interrupt+0xf/0x20 [18812.156538] [18812.157612] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18812.160888] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18812.163723] do_raw_spin_lock+0x177/0x190 [18812.166457] _raw_spin_lock+0x21/0x30 [18812.168348] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18812.171157] ? do_raw_spin_unlock+0x75/0x190 [18812.173585] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18812.176840] ? do_wait_intr+0xf0/0xf0 [18812.179400] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18812.182491] mdc_reint+0x2d/0x110 [mdc] [18812.184248] mdc_unlink+0x658/0x870 [mdc] [18812.186088] lmv_unlink+0x34d/0xbd0 [lmv] [18812.188233] ll_unlink+0x24d/0x7f0 [lustre] [18812.190609] vfs_unlink+0x202/0x330 [18812.192405] do_unlinkat+0x2fa/0x460 [18812.193814] __x64_sys_unlink+0x37/0x50 [18812.195216] do_syscall_64+0xc1/0x440 [18812.197412] entry_SYSCALL_64_after_hwframe+0x49/0xae [18812.199621] RIP: 0033:0x7f151bdee10b [18812.200871] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [18812.207683] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [18812.210764] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [18812.213974] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [18812.216896] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18812.219489] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18812.223033] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18816.057116] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [18816.062380] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18816.110870] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18816.121481] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18816.126889] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18816.130859] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18816.145878] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18816.155668] RAX: 0000000000007e8f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18816.162868] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [18816.166171] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [18816.170248] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [18816.173468] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [18816.177285] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [18816.185930] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18816.189951] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [18816.203491] Call Trace: [18816.204401] [18816.205174] ? show_regs.cold.9+0x22/0x2f [18816.216890] ? watchdog_timer_fn.cold.6+0x62/0x10a [18816.228874] ? __hrtimer_run_queues+0x1c2/0x4d0 [18816.237604] ? watchdog+0x40/0x40 [18816.238847] ? hrtimer_interrupt+0x128/0x2c0 [18816.244463] ? smp_apic_timer_interrupt+0xa6/0x270 [18816.250224] ? apic_timer_interrupt+0xf/0x20 [18816.252610] [18816.253429] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18816.261603] do_raw_spin_lock+0x177/0x190 [18816.263826] _raw_spin_lock+0x21/0x30 [18816.268869] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18816.278415] ? do_raw_spin_unlock+0x75/0x190 [18816.281923] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18816.285155] ? do_wait_intr+0xf0/0xf0 [18816.287194] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18816.290087] mdc_close+0x53b/0xd70 [mdc] [18816.291587] lmv_close+0x18d/0x490 [lmv] [18816.296769] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [18816.298607] ll_close_inode_openhandle+0x493/0xea0 [lustre] [18816.302660] ll_md_real_close+0xa5/0x2a0 [lustre] [18816.310080] ll_md_close+0x274/0x800 [lustre] [18816.315397] ll_file_release+0x128/0xc50 [lustre] [18816.317302] __fput+0x10d/0x440 [18816.318465] ____fput+0x12/0x20 [18816.320354] task_work_run+0xc6/0x110 [18816.321851] exit_to_usermode_loop+0x1dd/0x1f0 [18816.323797] do_syscall_64+0x3d6/0x440 [18816.324951] entry_SYSCALL_64_after_hwframe+0x49/0xae [18816.326176] RIP: 0033:0x7f151bee3cd8 [18816.331090] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [18816.347509] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [18816.353846] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [18816.364481] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [18816.369410] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18816.379141] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18816.384431] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18818.018160] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16887s! [18818.022452] Showing busy workqueues and worker pools: [18818.024620] workqueue events: flags=0x0 [18818.025971] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18818.028443] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [18818.032208] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18818.034477] pending: kfree_rcu_monitor [18818.035967] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18818.039106] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [18818.042970] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18818.045171] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [18818.049064] workqueue events_long: flags=0x0 [18818.050915] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18818.053907] pending: lu_objects_flush [obdclass] [18818.056961] workqueue events_unbound: flags=0x2 [18818.058746] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [18818.061711] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [18818.067386] workqueue events_power_efficient: flags=0x80 [18818.072766] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18818.076327] pending: do_cache_clean, neigh_periodic_work, check_lifetime [18818.079260] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18818.082766] pending: neigh_periodic_work [18818.085293] workqueue mm_percpu_wq: flags=0x8 [18818.087472] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18818.091026] pending: vmstat_update, lru_add_drain_per_cpu [18818.094159] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18818.097229] pending: vmstat_update [18818.099337] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18818.102782] pending: lru_add_drain_per_cpu [18818.105116] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [18818.108378] pending: lru_add_drain_per_cpu BAR(48) [18818.111769] workqueue writeback: flags=0x4a [18818.113781] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [18818.117256] pending: wb_workfn [18818.118848] workqueue xprtiod: flags=0xa [18818.120913] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [18818.124613] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [18818.129271] workqueue ipv6_addrconf: flags=0x40008 [18818.131590] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18818.134594] pending: addrconf_verify_work [18818.136749] workqueue knbd0-recv: flags=0x1a [18818.139055] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18818.142094] in-flight: 658:recv_work [18818.144164] workqueue ptlrpc_pinger: flags=0x2 [18818.146223] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18818.149143] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [18818.152725] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16887s workers=4 idle: 32984 37 8 [18818.156647] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18802s workers=2 idle: 661 [18828.056026] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [18828.061464] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18828.093845] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18828.101214] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18828.105087] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18828.108424] RIP: 0010:native_safe_halt+0x2e/0x50 [18828.111184] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18828.119968] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18828.123442] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18828.126897] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [18828.135597] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [18828.140335] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [18828.146019] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [18828.151050] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [18828.154727] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18828.157194] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18828.160219] Call Trace: [18828.161259] [18828.162460] ? show_regs.cold.9+0x22/0x2f [18828.164018] ? watchdog_timer_fn.cold.6+0x62/0x10a [18828.165898] ? __hrtimer_run_queues+0x1c2/0x4d0 [18828.167525] ? watchdog+0x40/0x40 [18828.169236] ? hrtimer_interrupt+0x128/0x2c0 [18828.170413] ? smp_apic_timer_interrupt+0xa6/0x270 [18828.172027] ? apic_timer_interrupt+0xf/0x20 [18828.173334] [18828.173954] ? native_safe_halt+0x2e/0x50 [18828.175142] kvm_wait+0xcd/0xe0 [18828.176047] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18828.177292] do_raw_spin_lock+0x177/0x190 [18828.178713] _raw_spin_lock+0x21/0x30 [18828.179846] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18828.182008] process_one_work+0x2c8/0x700 [18828.183193] worker_thread+0x296/0x6e0 [18828.184721] ? rescuer_thread+0x570/0x570 [18828.186304] kthread+0x1d1/0x200 [18828.188086] ? set_kthread_struct+0x70/0x70 [18828.189690] ret_from_fork+0x1f/0x30 [18836.055052] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [18836.057321] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18836.074026] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18836.078684] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18836.081733] RIP: 0010:native_safe_halt+0x2e/0x50 [18836.083292] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18836.089603] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18836.091869] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [18836.094233] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [18836.096739] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [18836.099214] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [18836.101549] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18836.103830] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [18836.106663] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18836.109269] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18836.112155] Call Trace: [18836.113010] [18836.113772] ? show_regs.cold.9+0x22/0x2f [18836.115225] ? watchdog_timer_fn.cold.6+0x62/0x10a [18836.116830] ? __hrtimer_run_queues+0x1c2/0x4d0 [18836.118825] ? watchdog+0x40/0x40 [18836.119944] ? hrtimer_interrupt+0x128/0x2c0 [18836.121611] ? smp_apic_timer_interrupt+0xa6/0x270 [18836.124824] ? apic_timer_interrupt+0xf/0x20 [18836.126584] [18836.127279] ? native_safe_halt+0x2e/0x50 [18836.128774] kvm_wait+0xcd/0xe0 [18836.129815] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18836.132138] do_raw_spin_lock+0x177/0x190 [18836.133808] _raw_spin_lock+0x21/0x30 [18836.135138] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18836.136926] obd_mod_free+0x46/0x140 [ptlrpc] [18836.140478] mdc_commit_open+0x91/0xb0 [mdc] [18836.142601] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18836.144594] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18836.147804] after_reply+0xe2c/0x1780 [ptlrpc] [18836.150121] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18836.153418] ? __next_timer_interrupt+0x160/0x160 [18836.156234] ptlrpcd+0xa82/0xe40 [ptlrpc] [18836.159420] ? do_wait_intr+0xf0/0xf0 [18836.162495] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18836.166311] kthread+0x1d1/0x200 [18836.168310] ? set_kthread_struct+0x70/0x70 [18836.170604] ret_from_fork+0x1f/0x30 [18844.059081] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [18844.060463] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [18844.078598] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18844.078687] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18844.078695] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18844.078698] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18844.078713] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18844.078718] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18844.078722] RAX: 0000000000007306 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18844.078724] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [18844.078727] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [18844.078729] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [18844.078732] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [18844.078735] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [18844.078738] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18844.078740] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [18844.078749] Call Trace: [18844.078754] [18844.078760] ? show_regs.cold.9+0x22/0x2f [18844.078770] ? watchdog_timer_fn.cold.6+0x62/0x10a [18844.078777] ? __hrtimer_run_queues+0x1c2/0x4d0 [18844.078782] ? watchdog+0x40/0x40 [18844.078787] ? hrtimer_interrupt+0x128/0x2c0 [18844.078791] ? smp_apic_timer_interrupt+0xa6/0x270 [18844.078808] ? apic_timer_interrupt+0xf/0x20 [18844.078813] [18844.078816] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18844.078820] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18844.078825] do_raw_spin_lock+0x177/0x190 [18844.078829] _raw_spin_lock+0x21/0x30 [18844.078836] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18844.079126] ? do_raw_spin_unlock+0x75/0x190 [18844.079132] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18844.079485] ? do_wait_intr+0xf0/0xf0 [18844.079494] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18844.084961] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18844.085039] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18844.085046] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18844.085050] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18844.085066] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18844.085071] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18844.085075] RAX: 0000000000002b10 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18844.085077] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [18844.085080] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [18844.085082] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [18844.085084] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [18844.085087] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [18844.085090] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18844.085092] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [18844.085102] Call Trace: [18844.085107] [18844.085113] ? show_regs.cold.9+0x22/0x2f [18844.085122] ? watchdog_timer_fn.cold.6+0x62/0x10a [18844.085829] ? __hrtimer_run_queues+0x1c2/0x4d0 [18844.085833] ? watchdog+0x40/0x40 [18844.085837] ? hrtimer_interrupt+0x128/0x2c0 [18844.085841] ? smp_apic_timer_interrupt+0xa6/0x270 [18844.085848] ? apic_timer_interrupt+0xf/0x20 [18844.085853] [18844.085856] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18844.085863] do_raw_spin_lock+0x177/0x190 [18844.085868] _raw_spin_lock+0x21/0x30 [18844.085873] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18844.086248] ? do_raw_spin_unlock+0x75/0x190 [18844.086254] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18844.086465] ? do_wait_intr+0xf0/0xf0 [18844.086473] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18844.086669] mdc_close+0x53b/0xd70 [mdc] [18844.086706] lmv_close+0x18d/0x490 [lmv] [18844.086734] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [18844.086902] ll_close_inode_openhandle+0x493/0xea0 [lustre] [18844.087029] ll_md_real_close+0xa5/0x2a0 [lustre] [18844.087111] ll_md_close+0x274/0x800 [lustre] [18844.087423] ll_file_release+0x128/0xc50 [lustre] [18844.087505] __fput+0x10d/0x440 [18844.087516] ____fput+0x12/0x20 [18844.087521] task_work_run+0xc6/0x110 [18844.087529] exit_to_usermode_loop+0x1dd/0x1f0 [18844.087538] do_syscall_64+0x3d6/0x440 [18844.087542] entry_SYSCALL_64_after_hwframe+0x49/0xae [18844.087550] RIP: 0033:0x7f151bee3cd8 [18844.087557] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [18844.087561] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [18844.087567] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [18844.087569] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [18844.087572] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18844.087574] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18844.087575] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18844.535857] mdc_reint+0x2d/0x110 [mdc] [18844.541664] mdc_unlink+0x658/0x870 [mdc] [18844.543468] lmv_unlink+0x34d/0xbd0 [lmv] [18844.546695] ll_unlink+0x24d/0x7f0 [lustre] [18844.548897] vfs_unlink+0x202/0x330 [18844.551570] do_unlinkat+0x2fa/0x460 [18844.553304] __x64_sys_unlink+0x37/0x50 [18844.555556] do_syscall_64+0xc1/0x440 [18844.559414] entry_SYSCALL_64_after_hwframe+0x49/0xae [18844.562339] RIP: 0033:0x7f151bdee10b [18844.566929] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [18844.579048] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [18844.582644] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [18844.586396] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [18844.591136] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18844.595154] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18844.599877] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18848.225370] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16917s! [18848.228646] Showing busy workqueues and worker pools: [18848.230238] workqueue events: flags=0x0 [18848.231654] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18848.233887] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [18848.237121] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18848.239675] pending: kfree_rcu_monitor [18848.241298] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18848.243680] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [18848.247308] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18848.249970] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [18848.252782] workqueue events_long: flags=0x0 [18848.254505] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18848.256986] pending: lu_objects_flush [obdclass] [18848.258895] workqueue events_unbound: flags=0x2 [18848.260570] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [18848.263006] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [18848.268026] workqueue events_power_efficient: flags=0x80 [18848.270635] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18848.273523] pending: do_cache_clean, neigh_periodic_work, check_lifetime [18848.276071] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18848.278312] pending: neigh_periodic_work [18848.279782] workqueue mm_percpu_wq: flags=0x8 [18848.281378] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18848.283899] pending: vmstat_update, lru_add_drain_per_cpu [18848.286082] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18848.288722] pending: vmstat_update [18848.290125] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18848.292605] pending: lru_add_drain_per_cpu [18848.294423] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [18848.296642] pending: lru_add_drain_per_cpu BAR(48) [18848.298449] workqueue writeback: flags=0x4a [18848.299841] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [18848.302128] pending: wb_workfn [18848.303532] workqueue xprtiod: flags=0xa [18848.305090] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [18848.307484] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [18848.310142] workqueue ipv6_addrconf: flags=0x40008 [18848.311908] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18848.314170] pending: addrconf_verify_work [18848.315955] workqueue knbd0-recv: flags=0x1a [18848.317407] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18848.319792] in-flight: 658:recv_work [18848.321124] workqueue ptlrpc_pinger: flags=0x2 [18848.322805] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18848.325361] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [18848.327523] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16917s workers=4 idle: 32984 37 8 [18848.330450] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18833s workers=2 idle: 661 [18856.056046] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [18856.060331] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18856.097044] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18856.103437] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18856.109463] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18856.114590] RIP: 0010:native_safe_halt+0x2e/0x50 [18856.119372] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18856.136414] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18856.142666] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18856.150378] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [18856.156918] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [18856.161242] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [18856.165112] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [18856.175595] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [18856.179614] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18856.185889] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18856.191580] Call Trace: [18856.195213] [18856.196877] ? show_regs.cold.9+0x22/0x2f [18856.203689] ? watchdog_timer_fn.cold.6+0x62/0x10a [18856.215246] ? __hrtimer_run_queues+0x1c2/0x4d0 [18856.218101] ? watchdog+0x40/0x40 [18856.219814] ? hrtimer_interrupt+0x128/0x2c0 [18856.222450] ? smp_apic_timer_interrupt+0xa6/0x270 [18856.224961] ? apic_timer_interrupt+0xf/0x20 [18856.230924] [18856.232416] ? native_safe_halt+0x2e/0x50 [18856.234958] kvm_wait+0xcd/0xe0 [18856.236685] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18856.239196] do_raw_spin_lock+0x177/0x190 [18856.240698] _raw_spin_lock+0x21/0x30 [18856.244684] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18856.252306] process_one_work+0x2c8/0x700 [18856.258731] worker_thread+0x296/0x6e0 [18856.263891] ? rescuer_thread+0x570/0x570 [18856.274285] kthread+0x1d1/0x200 [18856.278871] ? set_kthread_struct+0x70/0x70 [18856.282904] ret_from_fork+0x1f/0x30 [18864.055309] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [18864.063711] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18864.100869] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18864.108400] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18864.133050] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18864.135326] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18864.161120] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18864.168201] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [18864.170816] RDX: 0000000000003ec6 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [18864.173698] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [18864.176580] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [18864.179247] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18864.182028] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [18864.185262] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18864.187345] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18864.189986] Call Trace: [18864.190957] [18864.191686] ? show_regs.cold.9+0x22/0x2f [18864.193305] ? watchdog_timer_fn.cold.6+0x62/0x10a [18864.195120] ? __hrtimer_run_queues+0x1c2/0x4d0 [18864.196820] ? watchdog+0x40/0x40 [18864.198165] ? hrtimer_interrupt+0x128/0x2c0 [18864.199788] ? smp_apic_timer_interrupt+0xa6/0x270 [18864.201564] ? apic_timer_interrupt+0xf/0x20 [18864.203202] [18864.203867] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18864.205824] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18864.207926] do_raw_spin_lock+0x177/0x190 [18864.209562] _raw_spin_lock+0x21/0x30 [18864.210975] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18864.213240] obd_mod_free+0x46/0x140 [ptlrpc] [18864.215200] mdc_commit_open+0x91/0xb0 [mdc] [18864.216926] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18864.219204] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18864.221479] after_reply+0xe2c/0x1780 [ptlrpc] [18864.223406] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18864.231720] ? __next_timer_interrupt+0x160/0x160 [18864.235012] ptlrpcd+0xa82/0xe40 [ptlrpc] [18864.239276] ? do_wait_intr+0xf0/0xf0 [18864.242265] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18864.248602] kthread+0x1d1/0x200 [18864.250352] ? set_kthread_struct+0x70/0x70 [18864.252819] ret_from_fork+0x1f/0x30 [18872.057397] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [18872.065193] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18872.088972] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18872.094487] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18872.098120] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18872.100711] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18872.108438] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18872.111572] RAX: 000000000000477f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18872.114454] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [18872.117431] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [18872.120365] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [18872.123263] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [18872.126122] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [18872.129493] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18872.131963] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [18872.135738] Call Trace: [18872.136855] [18872.137817] ? show_regs.cold.9+0x22/0x2f [18872.141511] ? watchdog_timer_fn.cold.6+0x62/0x10a [18872.145912] ? __hrtimer_run_queues+0x1c2/0x4d0 [18872.151521] ? watchdog+0x40/0x40 [18872.153485] ? hrtimer_interrupt+0x128/0x2c0 [18872.156312] ? smp_apic_timer_interrupt+0xa6/0x270 [18872.160922] ? apic_timer_interrupt+0xf/0x20 [18872.164440] [18872.164946] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18872.168825] do_raw_spin_lock+0x177/0x190 [18872.171150] _raw_spin_lock+0x21/0x30 [18872.173745] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18872.177581] ? do_raw_spin_unlock+0x75/0x190 [18872.178752] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18872.182669] ? do_wait_intr+0xf0/0xf0 [18872.185839] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18872.191651] mdc_close+0x53b/0xd70 [mdc] [18872.193882] lmv_close+0x18d/0x490 [lmv] [18872.196157] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [18872.198466] ll_close_inode_openhandle+0x493/0xea0 [lustre] [18872.200738] ll_md_real_close+0xa5/0x2a0 [lustre] [18872.202893] ll_md_close+0x274/0x800 [lustre] [18872.204354] ll_file_release+0x128/0xc50 [lustre] [18872.206652] __fput+0x10d/0x440 [18872.207881] ____fput+0x12/0x20 [18872.210898] task_work_run+0xc6/0x110 [18872.214688] exit_to_usermode_loop+0x1dd/0x1f0 [18872.216068] do_syscall_64+0x3d6/0x440 [18872.217625] entry_SYSCALL_64_after_hwframe+0x49/0xae [18872.219163] RIP: 0033:0x7f151bee3cd8 [18872.224074] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [18872.233907] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [18872.237430] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [18872.241652] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [18872.246744] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18872.254115] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18872.258499] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18876.059137] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [18876.065853] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18876.081293] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18876.094504] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18876.106669] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18876.115906] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18876.132043] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18876.139573] RAX: 0000000000004483 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18876.143726] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [18876.153629] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [18876.157852] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [18876.163735] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [18876.167837] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [18876.177562] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18876.182850] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [18876.190658] Call Trace: [18876.191569] [18876.192335] ? show_regs.cold.9+0x22/0x2f [18876.194802] ? watchdog_timer_fn.cold.6+0x62/0x10a [18876.196658] ? __hrtimer_run_queues+0x1c2/0x4d0 [18876.199346] ? watchdog+0x40/0x40 [18876.200768] ? hrtimer_interrupt+0x128/0x2c0 [18876.202893] ? smp_apic_timer_interrupt+0xa6/0x270 [18876.205316] ? apic_timer_interrupt+0xf/0x20 [18876.207934] [18876.208653] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18876.210125] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18876.213064] do_raw_spin_lock+0x177/0x190 [18876.214846] _raw_spin_lock+0x21/0x30 [18876.216631] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18876.226473] ? do_raw_spin_unlock+0x75/0x190 [18876.228731] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18876.233825] ? do_wait_intr+0xf0/0xf0 [18876.236548] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18876.238193] mdc_reint+0x2d/0x110 [mdc] [18876.244829] mdc_unlink+0x658/0x870 [mdc] [18876.247113] lmv_unlink+0x34d/0xbd0 [lmv] [18876.249128] ll_unlink+0x24d/0x7f0 [lustre] [18876.251665] vfs_unlink+0x202/0x330 [18876.253703] do_unlinkat+0x2fa/0x460 [18876.256822] __x64_sys_unlink+0x37/0x50 [18876.261703] do_syscall_64+0xc1/0x440 [18876.264801] entry_SYSCALL_64_after_hwframe+0x49/0xae [18876.270320] RIP: 0033:0x7f151bdee10b [18876.272055] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [18876.293207] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [18876.295326] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [18876.299738] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [18876.303498] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18876.308881] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18876.312306] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18878.436030] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16947s! [18878.440838] Showing busy workqueues and worker pools: [18878.443644] workqueue events: flags=0x0 [18878.445845] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18878.450536] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [18878.456558] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18878.460228] pending: kfree_rcu_monitor [18878.462690] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18878.466524] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [18878.471773] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18878.475572] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [18878.481470] workqueue events_long: flags=0x0 [18878.483851] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18878.489697] pending: lu_objects_flush [obdclass] [18878.492926] workqueue events_unbound: flags=0x2 [18878.496607] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [18878.499736] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [18878.505550] workqueue events_power_efficient: flags=0x80 [18878.508421] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18878.511783] pending: do_cache_clean, neigh_periodic_work, check_lifetime [18878.515700] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18878.519467] pending: neigh_periodic_work [18878.521837] workqueue mm_percpu_wq: flags=0x8 [18878.524450] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18878.527854] pending: vmstat_update, lru_add_drain_per_cpu [18878.531672] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18878.535580] pending: vmstat_update [18878.538243] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18878.542436] pending: lru_add_drain_per_cpu [18878.544818] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [18878.548187] pending: lru_add_drain_per_cpu BAR(48) [18878.550837] workqueue writeback: flags=0x4a [18878.553974] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [18878.557933] pending: wb_workfn [18878.559691] workqueue xprtiod: flags=0xa [18878.561402] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [18878.563804] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [18878.567785] workqueue ipv6_addrconf: flags=0x40008 [18878.570703] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18878.576542] pending: addrconf_verify_work [18878.579924] workqueue knbd0-recv: flags=0x1a [18878.583204] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18878.592944] in-flight: 658:recv_work [18878.603863] workqueue ptlrpc_pinger: flags=0x2 [18878.612828] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18878.621528] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [18878.623812] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16948s workers=4 idle: 32984 37 8 [18878.639879] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18863s workers=2 idle: 661 [18884.056529] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [18884.060474] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18884.107574] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18884.120477] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18884.123893] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18884.127398] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18884.128814] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18884.135190] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [18884.139016] RAX: 000000000000679c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18884.140748] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [18884.142848] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [18884.144978] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [18884.152028] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [18884.158068] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [18884.164259] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18884.167387] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18884.169519] Call Trace: [18884.172612] [18884.174629] ? show_regs.cold.9+0x22/0x2f [18884.176022] ? watchdog_timer_fn.cold.6+0x62/0x10a [18884.180247] ? __hrtimer_run_queues+0x1c2/0x4d0 [18884.184038] ? watchdog+0x40/0x40 [18884.188787] ? hrtimer_interrupt+0x128/0x2c0 [18884.190645] ? smp_apic_timer_interrupt+0xa6/0x270 [18884.193059] ? apic_timer_interrupt+0xf/0x20 [18884.194672] [18884.195723] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18884.199032] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18884.202259] do_raw_spin_lock+0x177/0x190 [18884.203746] _raw_spin_lock+0x21/0x30 [18884.205195] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18884.207732] process_one_work+0x2c8/0x700 [18884.210796] worker_thread+0x296/0x6e0 [18884.213110] ? rescuer_thread+0x570/0x570 [18884.215977] kthread+0x1d1/0x200 [18884.217169] ? set_kthread_struct+0x70/0x70 [18884.218476] ret_from_fork+0x1f/0x30 [18892.055075] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [18892.060123] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18892.091730] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18892.103742] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18892.109856] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18892.114029] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18892.130687] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18892.136942] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [18892.141902] RDX: 000000000000256b RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [18892.147502] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [18892.151901] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [18892.158527] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18892.164267] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [18892.169964] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18892.174276] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18892.178483] Call Trace: [18892.179818] [18892.181290] ? show_regs.cold.9+0x22/0x2f [18892.184647] ? watchdog_timer_fn.cold.6+0x62/0x10a [18892.187674] ? __hrtimer_run_queues+0x1c2/0x4d0 [18892.190733] ? watchdog+0x40/0x40 [18892.192751] ? hrtimer_interrupt+0x128/0x2c0 [18892.195572] ? smp_apic_timer_interrupt+0xa6/0x270 [18892.199337] ? apic_timer_interrupt+0xf/0x20 [18892.201820] [18892.202861] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18892.205898] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18892.209284] do_raw_spin_lock+0x177/0x190 [18892.211985] _raw_spin_lock+0x21/0x30 [18892.215024] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18892.220514] obd_mod_free+0x46/0x140 [ptlrpc] [18892.226354] mdc_commit_open+0x91/0xb0 [mdc] [18892.229800] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18892.232466] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18892.236293] after_reply+0xe2c/0x1780 [ptlrpc] [18892.239490] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18892.242687] ? __next_timer_interrupt+0x160/0x160 [18892.244833] ptlrpcd+0xa82/0xe40 [ptlrpc] [18892.247573] ? do_wait_intr+0xf0/0xf0 [18892.250965] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18892.254593] kthread+0x1d1/0x200 [18892.256424] ? set_kthread_struct+0x70/0x70 [18892.258759] ret_from_fork+0x1f/0x30 [18900.057353] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [18900.059837] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18900.099678] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18900.112245] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18900.119350] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18900.127372] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18900.149806] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18900.158438] RAX: 00000000000079f1 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18900.167246] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [18900.175252] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [18900.179058] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [18900.184411] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [18900.194244] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [18900.198434] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18900.208175] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [18900.213388] Call Trace: [18900.215035] [18900.217105] ? show_regs.cold.9+0x22/0x2f [18900.219374] ? watchdog_timer_fn.cold.6+0x62/0x10a [18900.224337] ? __hrtimer_run_queues+0x1c2/0x4d0 [18900.226346] ? watchdog+0x40/0x40 [18900.230062] ? hrtimer_interrupt+0x128/0x2c0 [18900.236076] ? smp_apic_timer_interrupt+0xa6/0x270 [18900.239392] ? apic_timer_interrupt+0xf/0x20 [18900.241446] [18900.244915] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18900.248007] do_raw_spin_lock+0x177/0x190 [18900.251795] _raw_spin_lock+0x21/0x30 [18900.254260] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18900.263019] ? do_raw_spin_unlock+0x75/0x190 [18900.268667] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18900.270730] ? do_wait_intr+0xf0/0xf0 [18900.280204] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18900.283507] mdc_close+0x53b/0xd70 [mdc] [18900.285053] lmv_close+0x18d/0x490 [lmv] [18900.290029] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [18900.293080] ll_close_inode_openhandle+0x493/0xea0 [lustre] [18900.299969] ll_md_real_close+0xa5/0x2a0 [lustre] [18900.301819] ll_md_close+0x274/0x800 [lustre] [18900.315934] ll_file_release+0x128/0xc50 [lustre] [18900.323700] __fput+0x10d/0x440 [18900.326126] ____fput+0x12/0x20 [18900.327922] task_work_run+0xc6/0x110 [18900.333117] exit_to_usermode_loop+0x1dd/0x1f0 [18900.340191] do_syscall_64+0x3d6/0x440 [18900.342417] entry_SYSCALL_64_after_hwframe+0x49/0xae [18900.347647] RIP: 0033:0x7f151bee3cd8 [18900.352126] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [18900.374544] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [18900.380541] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [18900.387821] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [18900.393442] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18900.398303] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18900.405490] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18904.061079] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [18904.071852] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18904.116589] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18904.126404] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18904.131757] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18904.135618] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18904.156532] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18904.164636] RAX: 000000000000280e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18904.169051] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [18904.175390] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [18904.181811] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [18904.195601] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [18904.209556] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [18904.213420] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18904.218276] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [18904.221882] Call Trace: [18904.223768] [18904.225096] ? show_regs.cold.9+0x22/0x2f [18904.229135] ? watchdog_timer_fn.cold.6+0x62/0x10a [18904.235375] ? __hrtimer_run_queues+0x1c2/0x4d0 [18904.238880] ? watchdog+0x40/0x40 [18904.240900] ? hrtimer_interrupt+0x128/0x2c0 [18904.244456] ? smp_apic_timer_interrupt+0xa6/0x270 [18904.252045] ? apic_timer_interrupt+0xf/0x20 [18904.258031] [18904.259095] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18904.264552] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18904.267649] do_raw_spin_lock+0x177/0x190 [18904.270008] _raw_spin_lock+0x21/0x30 [18904.272869] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18904.279910] ? do_raw_spin_unlock+0x75/0x190 [18904.282810] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18904.289024] ? do_wait_intr+0xf0/0xf0 [18904.291045] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18904.296384] mdc_reint+0x2d/0x110 [mdc] [18904.299507] mdc_unlink+0x658/0x870 [mdc] [18904.305932] lmv_unlink+0x34d/0xbd0 [lmv] [18904.307156] ll_unlink+0x24d/0x7f0 [lustre] [18904.308428] vfs_unlink+0x202/0x330 [18904.312321] do_unlinkat+0x2fa/0x460 [18904.315333] __x64_sys_unlink+0x37/0x50 [18904.319632] do_syscall_64+0xc1/0x440 [18904.322929] entry_SYSCALL_64_after_hwframe+0x49/0xae [18904.325515] RIP: 0033:0x7f151bdee10b [18904.327090] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [18904.342889] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [18904.354315] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [18904.360203] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [18904.365237] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18904.370186] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18904.382264] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18904.998005] rcu: INFO: rcu_sched self-detected stall on CPU [18905.004392] rcu: 1-...!: (10213967 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [18905.015457] (t=16980351 jiffies g=249657 q=7673) [18905.017534] rcu: rcu_sched kthread starved for 16975303 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [18905.028108] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [18905.034343] rcu: RCU grace-period kthread stack dump: [18905.036465] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [18905.046362] Call Trace: [18905.047362] __schedule+0x351/0xcb0 [18905.052211] schedule+0xc0/0x180 [18905.053543] schedule_timeout+0xb4/0x190 [18905.055920] ? do_raw_spin_unlock+0x75/0x190 [18905.060840] ? __next_timer_interrupt+0x160/0x160 [18905.066315] ? prepare_to_swait_event+0xf3/0x140 [18905.067705] rcu_gp_kthread+0x8dd/0x15b0 [18905.071630] ? force_qs_rnp+0x2c0/0x2c0 [18905.074008] kthread+0x1d1/0x200 [18905.078715] ? set_kthread_struct+0x70/0x70 [18905.080709] ret_from_fork+0x1f/0x30 [18905.082224] rcu: Stack dump where RCU GP kthread last ran: [18905.084772] Sending NMI from CPU 1 to CPUs 3: [18905.096328] NMI backtrace for cpu 3 [18905.096339] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18905.096342] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18905.096346] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18905.096348] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18905.096350] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 [18905.096353] RAX: 0000000000006d8a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18905.096354] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [18905.096356] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [18905.096357] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [18905.096359] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [18905.096360] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [18905.096362] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18905.096363] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [18905.096364] Call Trace: [18905.096369] [18905.096370] ? show_regs.cold.9+0x22/0x2f [18905.096371] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [18905.096372] ? nmi_cpu_backtrace_handler+0x15/0x40 [18905.096373] ? nmi_handle+0xbd/0x210 [18905.096374] ? default_do_nmi+0x4b/0x1b0 [18905.096375] ? do_nmi+0x280/0x380 [18905.096375] ? end_repeat_nmi+0x16/0x1f [18905.096377] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18905.096378] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18905.096379] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18905.096380] [18905.096381] do_raw_spin_lock+0x177/0x190 [18905.096382] _raw_spin_lock+0x21/0x30 [18905.096383] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18905.096384] ? do_raw_spin_unlock+0x75/0x190 [18905.096385] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18905.096389] ? do_wait_intr+0xf0/0xf0 [18905.096390] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18905.096392] mdc_close+0x53b/0xd70 [mdc] [18905.096393] lmv_close+0x18d/0x490 [lmv] [18905.096394] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [18905.096395] ll_close_inode_openhandle+0x493/0xea0 [lustre] [18905.096396] ll_md_real_close+0xa5/0x2a0 [lustre] [18905.096397] ll_md_close+0x274/0x800 [lustre] [18905.096398] ll_file_release+0x128/0xc50 [lustre] [18905.096399] __fput+0x10d/0x440 [18905.096400] ____fput+0x12/0x20 [18905.096401] task_work_run+0xc6/0x110 [18905.096402] exit_to_usermode_loop+0x1dd/0x1f0 [18905.096403] do_syscall_64+0x3d6/0x440 [18905.096405] entry_SYSCALL_64_after_hwframe+0x49/0xae [18905.096406] RIP: 0033:0x7f151bee3cd8 [18905.096407] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [18905.096409] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [18905.096411] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [18905.096413] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [18905.096414] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18905.096415] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18905.096417] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18905.100005] NMI backtrace for cpu 1 [18905.304898] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18905.314569] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18905.321938] Call Trace: [18905.322885] [18905.323991] ? dump_stack+0xbb/0x10e [18905.328448] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [18905.334244] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [18905.339329] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [18905.347037] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [18905.353375] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [18905.361846] ? rcu_dump_cpu_stacks+0x11b/0x171 [18905.364289] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [18905.370066] ? update_process_times+0x6d/0xc0 [18905.372981] ? tick_sched_handle.isra.18+0x27/0x90 [18905.377483] ? tick_sched_timer+0x53/0xb0 [18905.382323] ? __hrtimer_run_queues+0x1c2/0x4d0 [18905.385397] ? tick_sched_handle.isra.18+0x90/0x90 [18905.389567] ? hrtimer_interrupt+0x128/0x2c0 [18905.397469] ? smp_apic_timer_interrupt+0xa6/0x270 [18905.406386] ? apic_timer_interrupt+0xf/0x20 [18905.407945] [18905.408724] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18905.414248] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18905.419562] ? do_raw_spin_lock+0x177/0x190 [18905.422761] ? _raw_spin_lock+0x21/0x30 [18905.426751] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18905.432032] ? obd_mod_free+0x46/0x140 [ptlrpc] [18905.435229] ? mdc_commit_open+0x91/0xb0 [mdc] [18905.440338] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18905.442507] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18905.449436] ? after_reply+0xe2c/0x1780 [ptlrpc] [18905.454781] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18905.460405] ? __next_timer_interrupt+0x160/0x160 [18905.463311] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [18905.467874] ? do_wait_intr+0xf0/0xf0 [18905.472501] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18905.478009] ? kthread+0x1d1/0x200 [18905.479776] ? set_kthread_struct+0x70/0x70 [18905.484001] ? ret_from_fork+0x1f/0x30 [18909.160487] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16978s! [18909.170622] Showing busy workqueues and worker pools: [18909.177649] workqueue events: flags=0x0 [18909.179047] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18909.192315] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [18909.203341] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18909.209412] pending: kfree_rcu_monitor [18909.215395] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18909.223958] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [18909.228869] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18909.241694] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [18909.252164] workqueue events_long: flags=0x0 [18909.257094] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18909.263743] pending: lu_objects_flush [obdclass] [18909.265983] workqueue events_unbound: flags=0x2 [18909.271087] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [18909.278250] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [18909.292342] workqueue events_power_efficient: flags=0x80 [18909.297382] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18909.301483] pending: do_cache_clean, neigh_periodic_work, check_lifetime [18909.307211] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18909.312242] pending: neigh_periodic_work [18909.319098] workqueue mm_percpu_wq: flags=0x8 [18909.323062] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18909.329135] pending: vmstat_update, lru_add_drain_per_cpu [18909.335155] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18909.340932] pending: vmstat_update [18909.343329] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18909.349069] pending: lru_add_drain_per_cpu [18909.354480] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [18909.360850] pending: lru_add_drain_per_cpu BAR(48) [18909.364350] workqueue writeback: flags=0x4a [18909.366533] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [18909.373597] pending: wb_workfn [18909.379957] workqueue xprtiod: flags=0xa [18909.386602] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [18909.394138] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [18909.399783] workqueue ipv6_addrconf: flags=0x40008 [18909.401667] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18909.408390] pending: addrconf_verify_work [18909.410483] workqueue knbd0-recv: flags=0x1a [18909.413087] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18909.420280] in-flight: 658:recv_work [18909.422586] workqueue ptlrpc_pinger: flags=0x2 [18909.428797] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18909.432552] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [18909.436607] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16978s workers=4 idle: 32984 37 8 [18909.446384] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18894s workers=2 idle: 661 [18912.058779] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [18912.063899] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18912.105772] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18912.111786] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18912.119481] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18912.121467] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18912.127764] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18912.141367] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18912.153282] RAX: 0000000000002cc5 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18912.164890] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [18912.171970] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [18912.175924] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [18912.185163] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [18912.196799] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [18912.205133] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18912.209341] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18912.222221] Call Trace: [18912.224352] [18912.229086] ? show_regs.cold.9+0x22/0x2f [18912.236057] ? watchdog_timer_fn.cold.6+0x62/0x10a [18912.241868] ? __hrtimer_run_queues+0x1c2/0x4d0 [18912.245236] ? watchdog+0x40/0x40 [18912.246391] ? hrtimer_interrupt+0x128/0x2c0 [18912.250671] ? smp_apic_timer_interrupt+0xa6/0x270 [18912.254051] ? apic_timer_interrupt+0xf/0x20 [18912.256702] [18912.261681] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18912.265698] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18912.271418] do_raw_spin_lock+0x177/0x190 [18912.277505] _raw_spin_lock+0x21/0x30 [18912.282000] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18912.287259] process_one_work+0x2c8/0x700 [18912.289163] worker_thread+0x296/0x6e0 [18912.290591] ? rescuer_thread+0x570/0x570 [18912.294179] kthread+0x1d1/0x200 [18912.297355] ? set_kthread_struct+0x70/0x70 [18912.300688] ret_from_fork+0x1f/0x30 [18932.057065] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [18932.057393] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [18932.059236] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18932.059320] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18932.059327] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18932.059331] RIP: 0010:native_safe_halt+0x2e/0x50 [18932.059343] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18932.059348] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18932.059355] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18932.059357] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [18932.059359] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [18932.059364] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [18932.059368] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [18932.059371] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [18932.059375] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18932.059377] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [18932.059385] Call Trace: [18932.059392] [18932.059575] ? show_regs.cold.9+0x22/0x2f [18932.059590] ? watchdog_timer_fn.cold.6+0x62/0x10a [18932.059597] ? __hrtimer_run_queues+0x1c2/0x4d0 [18932.059604] ? watchdog+0x40/0x40 [18932.059609] ? hrtimer_interrupt+0x128/0x2c0 [18932.059614] ? smp_apic_timer_interrupt+0xa6/0x270 [18932.059624] ? apic_timer_interrupt+0xf/0x20 [18932.059631] [18932.059635] ? native_safe_halt+0x2e/0x50 [18932.059639] kvm_wait+0xcd/0xe0 [18932.059647] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18932.059658] do_raw_spin_lock+0x177/0x190 [18932.059663] _raw_spin_lock+0x21/0x30 [18932.059667] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18932.070454] Modules linked in: [18932.105917] ? do_raw_spin_unlock+0x75/0x190 [18932.105947] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18932.126199] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18932.126338] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18932.126346] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18932.126349] RIP: 0010:native_safe_halt+0x2e/0x50 [18932.126363] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18932.126366] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18932.126371] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [18932.126373] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [18932.126375] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [18932.126377] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [18932.126380] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18932.126383] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [18932.126385] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18932.126387] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18932.126593] Call Trace: [18932.126600] [18932.126607] ? show_regs.cold.9+0x22/0x2f [18932.126618] ? watchdog_timer_fn.cold.6+0x62/0x10a [18932.126624] ? __hrtimer_run_queues+0x1c2/0x4d0 [18932.126629] ? watchdog+0x40/0x40 [18932.126634] ? hrtimer_interrupt+0x128/0x2c0 [18932.126637] ? smp_apic_timer_interrupt+0xa6/0x270 [18932.126644] ? apic_timer_interrupt+0xf/0x20 [18932.126649] [18932.126651] ? native_safe_halt+0x2e/0x50 [18932.126655] kvm_wait+0xcd/0xe0 [18932.126663] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18932.126670] do_raw_spin_lock+0x177/0x190 [18932.126674] _raw_spin_lock+0x21/0x30 [18932.126680] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18932.132650] ? do_wait_intr+0xf0/0xf0 [18932.132677] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18932.133128] mdc_close+0x53b/0xd70 [mdc] [18932.133181] lmv_close+0x18d/0x490 [lmv] [18932.138919] obd_mod_free+0x46/0x140 [ptlrpc] [18932.147120] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [18932.150592] mdc_commit_open+0x91/0xb0 [mdc] [18932.153354] ll_close_inode_openhandle+0x493/0xea0 [lustre] [18932.155496] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18932.158455] ll_md_real_close+0xa5/0x2a0 [lustre] [18932.163272] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18932.165337] ll_md_close+0x274/0x800 [lustre] [18932.175115] after_reply+0xe2c/0x1780 [ptlrpc] [18932.176994] ll_file_release+0x128/0xc50 [lustre] [18932.182077] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18932.182867] __fput+0x10d/0x440 [18932.183836] ? __next_timer_interrupt+0x160/0x160 [18932.184909] ____fput+0x12/0x20 [18932.186245] ptlrpcd+0xa82/0xe40 [ptlrpc] [18932.191827] task_work_run+0xc6/0x110 [18932.191852] exit_to_usermode_loop+0x1dd/0x1f0 [18932.191860] do_syscall_64+0x3d6/0x440 [18932.191865] entry_SYSCALL_64_after_hwframe+0x49/0xae [18932.191874] RIP: 0033:0x7f151bee3cd8 [18932.191975] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [18932.191982] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [18932.191988] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [18932.191990] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [18932.191993] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18932.191995] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18932.191998] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18932.494759] ? do_wait_intr+0xf0/0xf0 [18932.496592] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18932.501217] kthread+0x1d1/0x200 [18932.504108] ? set_kthread_struct+0x70/0x70 [18932.506215] ret_from_fork+0x1f/0x30 [18936.057193] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [18936.063388] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18936.097132] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18936.107761] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18936.113755] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18936.116849] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18936.133524] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18936.139210] RAX: 0000000000007d86 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18936.143867] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [18936.149639] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [18936.153930] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [18936.157537] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [18936.164808] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [18936.193691] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18936.205193] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [18936.220411] Call Trace: [18936.221388] [18936.222074] ? show_regs.cold.9+0x22/0x2f [18936.227864] ? watchdog_timer_fn.cold.6+0x62/0x10a [18936.229395] ? __hrtimer_run_queues+0x1c2/0x4d0 [18936.234086] ? watchdog+0x40/0x40 [18936.240621] ? hrtimer_interrupt+0x128/0x2c0 [18936.247036] ? smp_apic_timer_interrupt+0xa6/0x270 [18936.261936] ? apic_timer_interrupt+0xf/0x20 [18936.266918] [18936.267665] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18936.275506] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18936.280516] do_raw_spin_lock+0x177/0x190 [18936.282026] _raw_spin_lock+0x21/0x30 [18936.286747] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18936.289254] ? do_raw_spin_unlock+0x75/0x190 [18936.292965] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18936.299758] ? do_wait_intr+0xf0/0xf0 [18936.301132] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18936.303967] mdc_reint+0x2d/0x110 [mdc] [18936.305286] mdc_unlink+0x658/0x870 [mdc] [18936.311617] lmv_unlink+0x34d/0xbd0 [lmv] [18936.313469] ll_unlink+0x24d/0x7f0 [lustre] [18936.317216] vfs_unlink+0x202/0x330 [18936.319218] do_unlinkat+0x2fa/0x460 [18936.323949] __x64_sys_unlink+0x37/0x50 [18936.326192] do_syscall_64+0xc1/0x440 [18936.329013] entry_SYSCALL_64_after_hwframe+0x49/0xae [18936.334255] RIP: 0033:0x7f151bdee10b [18936.336040] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [18936.355727] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [18936.360577] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [18936.365596] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [18936.369191] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18936.371998] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18936.374708] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18939.872184] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17009s! [18939.878447] Showing busy workqueues and worker pools: [18939.880261] workqueue events: flags=0x0 [18939.881784] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18939.884336] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [18939.887606] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18939.890510] pending: kfree_rcu_monitor [18939.892373] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18939.895139] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [18939.899300] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18939.902063] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [18939.904870] workqueue events_long: flags=0x0 [18939.906742] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18939.909951] pending: lu_objects_flush [obdclass] [18939.912331] workqueue events_unbound: flags=0x2 [18939.914019] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [18939.916646] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [18939.921144] workqueue events_power_efficient: flags=0x80 [18939.923134] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18939.926040] pending: do_cache_clean, neigh_periodic_work, check_lifetime [18939.928434] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18939.931572] pending: neigh_periodic_work [18939.933153] workqueue mm_percpu_wq: flags=0x8 [18939.935020] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18939.937587] pending: vmstat_update, lru_add_drain_per_cpu [18939.939827] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18939.942633] pending: vmstat_update [18939.944129] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18939.946900] pending: lru_add_drain_per_cpu [18939.948629] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [18939.951238] pending: lru_add_drain_per_cpu BAR(48) [18939.953310] workqueue writeback: flags=0x4a [18939.955016] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [18939.957696] pending: wb_workfn [18939.959095] workqueue xprtiod: flags=0xa [18939.960463] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [18939.963013] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [18939.966151] workqueue ipv6_addrconf: flags=0x40008 [18939.967555] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18939.970313] pending: addrconf_verify_work [18939.972019] workqueue knbd0-recv: flags=0x1a [18939.973786] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18939.976134] in-flight: 658:recv_work [18939.977680] workqueue ptlrpc_pinger: flags=0x2 [18939.979430] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18939.981868] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [18939.984586] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17009s workers=4 idle: 32984 37 8 [18939.987731] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18924s workers=2 idle: 661 [18940.057998] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [18940.061353] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18940.082063] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18940.085647] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18940.089246] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18940.092002] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18940.094381] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18940.101512] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18940.104447] RAX: 0000000000007686 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18940.107297] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [18940.110074] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [18940.112901] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [18940.115647] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [18940.118408] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [18940.121462] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18940.122917] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18940.125024] Call Trace: [18940.126202] [18940.127191] ? show_regs.cold.9+0x22/0x2f [18940.130149] ? watchdog_timer_fn.cold.6+0x62/0x10a [18940.133208] ? __hrtimer_run_queues+0x1c2/0x4d0 [18940.135502] ? watchdog+0x40/0x40 [18940.137163] ? hrtimer_interrupt+0x128/0x2c0 [18940.139561] ? smp_apic_timer_interrupt+0xa6/0x270 [18940.141705] ? apic_timer_interrupt+0xf/0x20 [18940.144213] [18940.145296] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18940.148073] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18940.151256] do_raw_spin_lock+0x177/0x190 [18940.154071] _raw_spin_lock+0x21/0x30 [18940.156212] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18940.158754] process_one_work+0x2c8/0x700 [18940.160622] worker_thread+0x296/0x6e0 [18940.162624] ? rescuer_thread+0x570/0x570 [18940.164525] kthread+0x1d1/0x200 [18940.166496] ? set_kthread_struct+0x70/0x70 [18940.168812] ret_from_fork+0x1f/0x30 [18960.055755] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [18960.057070] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [18960.059595] Modules linked in: [18960.062716] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18960.062792] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18960.062800] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18960.062803] RIP: 0010:native_safe_halt+0x2e/0x50 [18960.062816] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18960.062820] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18960.062825] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18960.062827] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [18960.062830] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [18960.062832] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [18960.062834] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [18960.062837] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [18960.062840] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18960.062842] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [18960.062851] Call Trace: [18960.062857] [18960.062863] ? show_regs.cold.9+0x22/0x2f [18960.062873] ? watchdog_timer_fn.cold.6+0x62/0x10a [18960.062880] ? __hrtimer_run_queues+0x1c2/0x4d0 [18960.062887] ? watchdog+0x40/0x40 [18960.062891] ? hrtimer_interrupt+0x128/0x2c0 [18960.062895] ? smp_apic_timer_interrupt+0xa6/0x270 [18960.062901] ? apic_timer_interrupt+0xf/0x20 [18960.062905] [18960.062908] ? native_safe_halt+0x2e/0x50 [18960.062912] kvm_wait+0xcd/0xe0 [18960.063089] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18960.063101] do_raw_spin_lock+0x177/0x190 [18960.063106] _raw_spin_lock+0x21/0x30 [18960.063111] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18960.064677] ? do_raw_spin_unlock+0x75/0x190 [18960.068190] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18960.068285] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18960.068293] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18960.068296] RIP: 0010:native_safe_halt+0x2e/0x50 [18960.068311] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18960.068315] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [18960.068320] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [18960.068323] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [18960.068325] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [18960.068328] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [18960.068330] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18960.068334] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [18960.068336] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18960.095890] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18960.105971] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18960.111895] ? do_wait_intr+0xf0/0xf0 [18960.111922] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18960.113415] mdc_close+0x53b/0xd70 [mdc] [18960.113452] lmv_close+0x18d/0x490 [lmv] [18960.113480] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [18960.113647] ll_close_inode_openhandle+0x493/0xea0 [lustre] [18960.113752] ll_md_real_close+0xa5/0x2a0 [lustre] [18960.113831] ll_md_close+0x274/0x800 [lustre] [18960.113915] ll_file_release+0x128/0xc50 [lustre] [18960.118855] Call Trace: [18960.127639] __fput+0x10d/0x440 [18960.127658] ____fput+0x12/0x20 [18960.127663] task_work_run+0xc6/0x110 [18960.127670] exit_to_usermode_loop+0x1dd/0x1f0 [18960.127676] do_syscall_64+0x3d6/0x440 [18960.127680] entry_SYSCALL_64_after_hwframe+0x49/0xae [18960.127688] RIP: 0033:0x7f151bee3cd8 [18960.127696] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [18960.127699] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [18960.127704] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [18960.127707] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [18960.127709] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18960.127711] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18960.127713] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18960.554324] [18960.555793] ? show_regs.cold.9+0x22/0x2f [18960.557876] ? watchdog_timer_fn.cold.6+0x62/0x10a [18960.560645] ? __hrtimer_run_queues+0x1c2/0x4d0 [18960.565314] ? watchdog+0x40/0x40 [18960.566464] ? hrtimer_interrupt+0x128/0x2c0 [18960.571829] ? smp_apic_timer_interrupt+0xa6/0x270 [18960.576553] ? apic_timer_interrupt+0xf/0x20 [18960.580895] [18960.582065] ? native_safe_halt+0x2e/0x50 [18960.584478] kvm_wait+0xcd/0xe0 [18960.588124] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18960.593652] do_raw_spin_lock+0x177/0x190 [18960.595105] _raw_spin_lock+0x21/0x30 [18960.597963] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18960.600785] obd_mod_free+0x46/0x140 [ptlrpc] [18960.605277] mdc_commit_open+0x91/0xb0 [mdc] [18960.612952] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18960.614896] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18960.621043] after_reply+0xe2c/0x1780 [ptlrpc] [18960.624276] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18960.627617] ? __next_timer_interrupt+0x160/0x160 [18960.637493] ptlrpcd+0xa82/0xe40 [ptlrpc] [18960.648080] ? do_wait_intr+0xf0/0xf0 [18960.654687] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18960.659999] kthread+0x1d1/0x200 [18960.664623] ? set_kthread_struct+0x70/0x70 [18960.666139] ret_from_fork+0x1f/0x30 [18964.052039] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [18964.059066] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18964.112369] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18964.122714] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18964.131353] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18964.135322] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18964.154449] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18964.159803] RAX: 0000000000004751 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18964.165702] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [18964.172758] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [18964.183752] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [18964.192919] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [18964.198755] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [18964.204986] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18964.206866] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [18964.220868] Call Trace: [18964.222775] [18964.223583] ? show_regs.cold.9+0x22/0x2f [18964.225361] ? watchdog_timer_fn.cold.6+0x62/0x10a [18964.226849] ? __hrtimer_run_queues+0x1c2/0x4d0 [18964.229506] ? watchdog+0x40/0x40 [18964.231866] ? hrtimer_interrupt+0x128/0x2c0 [18964.236779] ? smp_apic_timer_interrupt+0xa6/0x270 [18964.239073] ? apic_timer_interrupt+0xf/0x20 [18964.241373] [18964.242506] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18964.248192] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18964.250453] do_raw_spin_lock+0x177/0x190 [18964.252344] _raw_spin_lock+0x21/0x30 [18964.254165] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18964.259458] ? do_raw_spin_unlock+0x75/0x190 [18964.262683] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18964.265332] ? do_wait_intr+0xf0/0xf0 [18964.267198] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18964.269886] mdc_reint+0x2d/0x110 [mdc] [18964.275553] mdc_unlink+0x658/0x870 [mdc] [18964.278256] lmv_unlink+0x34d/0xbd0 [lmv] [18964.280031] ll_unlink+0x24d/0x7f0 [lustre] [18964.283513] vfs_unlink+0x202/0x330 [18964.286652] do_unlinkat+0x2fa/0x460 [18964.288269] __x64_sys_unlink+0x37/0x50 [18964.290335] do_syscall_64+0xc1/0x440 [18964.291667] entry_SYSCALL_64_after_hwframe+0x49/0xae [18964.295957] RIP: 0033:0x7f151bdee10b [18964.297035] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [18964.309459] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [18964.315175] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [18964.324430] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [18964.331984] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18964.341625] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18964.344048] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18968.056110] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [18968.059433] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18968.102446] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18968.107777] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18968.112326] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18968.115202] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18968.118190] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18968.126844] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18968.130625] RAX: 0000000000002d08 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18968.133796] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [18968.137761] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [18968.140348] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [18968.143644] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [18968.146639] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [18968.151509] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18968.153823] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18968.156710] Call Trace: [18968.157838] [18968.158534] ? show_regs.cold.9+0x22/0x2f [18968.159829] ? watchdog_timer_fn.cold.6+0x62/0x10a [18968.162286] ? __hrtimer_run_queues+0x1c2/0x4d0 [18968.163848] ? watchdog+0x40/0x40 [18968.165678] ? hrtimer_interrupt+0x128/0x2c0 [18968.166856] ? smp_apic_timer_interrupt+0xa6/0x270 [18968.169208] ? apic_timer_interrupt+0xf/0x20 [18968.171550] [18968.172295] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18968.175250] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18968.178265] do_raw_spin_lock+0x177/0x190 [18968.180168] _raw_spin_lock+0x21/0x30 [18968.182393] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18968.186135] process_one_work+0x2c8/0x700 [18968.187764] worker_thread+0x296/0x6e0 [18968.190754] ? rescuer_thread+0x570/0x570 [18968.193031] kthread+0x1d1/0x200 [18968.194633] ? set_kthread_struct+0x70/0x70 [18968.196642] ret_from_fork+0x1f/0x30 [18970.081369] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17039s! [18970.085894] Showing busy workqueues and worker pools: [18970.087822] workqueue events: flags=0x0 [18970.088974] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18970.091207] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [18970.102177] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18970.106146] pending: kfree_rcu_monitor [18970.109835] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18970.118413] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [18970.130187] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18970.141976] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [18970.146271] workqueue events_long: flags=0x0 [18970.151112] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18970.163521] pending: lu_objects_flush [obdclass] [18970.171488] workqueue events_unbound: flags=0x2 [18970.176435] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [18970.184174] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [18970.194189] workqueue events_power_efficient: flags=0x80 [18970.197893] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18970.204486] pending: do_cache_clean, neigh_periodic_work, check_lifetime [18970.210997] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18970.220286] pending: neigh_periodic_work [18970.225928] workqueue mm_percpu_wq: flags=0x8 [18970.230752] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18970.235979] pending: vmstat_update, lru_add_drain_per_cpu [18970.238950] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18970.243935] pending: vmstat_update [18970.247341] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18970.255643] pending: lru_add_drain_per_cpu [18970.260693] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [18970.268475] pending: lru_add_drain_per_cpu BAR(48) [18970.274919] workqueue writeback: flags=0x4a [18970.277379] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [18970.284647] pending: wb_workfn [18970.290825] workqueue xprtiod: flags=0xa [18970.297984] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [18970.308554] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [18970.317967] workqueue ipv6_addrconf: flags=0x40008 [18970.323344] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18970.327342] pending: addrconf_verify_work [18970.330446] workqueue knbd0-recv: flags=0x1a [18970.333929] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18970.337820] in-flight: 658:recv_work [18970.340206] workqueue ptlrpc_pinger: flags=0x2 [18970.342183] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18970.344785] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [18970.346711] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17039s workers=4 idle: 32984 37 8 [18970.356637] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18955s workers=2 idle: 661 [18988.055442] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [18988.057353] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [18988.063252] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18988.063334] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18988.063342] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18988.063345] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18988.063359] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18988.063364] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18988.063369] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [18988.063371] RDX: 0000000000007d96 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [18988.063374] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [18988.063376] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [18988.063379] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18988.063382] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [18988.063385] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18988.063387] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18988.063394] Call Trace: [18988.063399] [18988.063404] ? show_regs.cold.9+0x22/0x2f [18988.063413] ? watchdog_timer_fn.cold.6+0x62/0x10a [18988.063420] ? __hrtimer_run_queues+0x1c2/0x4d0 [18988.063424] ? watchdog+0x40/0x40 [18988.064200] ? hrtimer_interrupt+0x128/0x2c0 [18988.064205] ? smp_apic_timer_interrupt+0xa6/0x270 [18988.064214] ? apic_timer_interrupt+0xf/0x20 [18988.064218] [18988.064222] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18988.064227] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18988.064232] do_raw_spin_lock+0x177/0x190 [18988.064236] _raw_spin_lock+0x21/0x30 [18988.064243] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18988.066410] obd_mod_free+0x46/0x140 [ptlrpc] [18988.066829] mdc_commit_open+0x91/0xb0 [mdc] [18988.067188] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18988.075018] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18988.075103] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18988.075110] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18988.075113] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18988.075132] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18988.075139] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18988.075144] RAX: 00000000000072a4 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18988.075146] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [18988.075149] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [18988.075150] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [18988.075156] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [18988.075159] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [18988.075162] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18988.075165] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [18988.075183] Call Trace: [18988.075188] [18988.075194] ? show_regs.cold.9+0x22/0x2f [18988.075205] ? watchdog_timer_fn.cold.6+0x62/0x10a [18988.075218] ? __hrtimer_run_queues+0x1c2/0x4d0 [18988.075222] ? watchdog+0x40/0x40 [18988.075230] ? hrtimer_interrupt+0x128/0x2c0 [18988.075235] ? smp_apic_timer_interrupt+0xa6/0x270 [18988.075244] ? apic_timer_interrupt+0xf/0x20 [18988.075254] [18988.075257] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18988.075263] do_raw_spin_lock+0x177/0x190 [18988.075272] _raw_spin_lock+0x21/0x30 [18988.075277] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18988.075804] ? do_raw_spin_unlock+0x75/0x190 [18988.075810] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18988.076596] ? do_wait_intr+0xf0/0xf0 [18988.076606] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18988.076873] mdc_close+0x53b/0xd70 [mdc] [18988.104766] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18988.116814] lmv_close+0x18d/0x490 [lmv] [18988.116857] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [18988.117247] ll_close_inode_openhandle+0x493/0xea0 [lustre] [18988.117380] ll_md_real_close+0xa5/0x2a0 [lustre] [18988.117481] ll_md_close+0x274/0x800 [lustre] [18988.117567] ll_file_release+0x128/0xc50 [lustre] [18988.124580] after_reply+0xe2c/0x1780 [ptlrpc] [18988.137246] __fput+0x10d/0x440 [18988.137284] ____fput+0x12/0x20 [18988.137291] task_work_run+0xc6/0x110 [18988.137299] exit_to_usermode_loop+0x1dd/0x1f0 [18988.137307] do_syscall_64+0x3d6/0x440 [18988.137312] entry_SYSCALL_64_after_hwframe+0x49/0xae [18988.137322] RIP: 0033:0x7f151bee3cd8 [18988.137330] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [18988.137335] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [18988.137340] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [18988.137342] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [18988.137345] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18988.137347] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18988.137349] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [18988.564548] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18988.567375] ? __next_timer_interrupt+0x160/0x160 [18988.569655] ptlrpcd+0xa82/0xe40 [ptlrpc] [18988.572331] ? do_wait_intr+0xf0/0xf0 [18988.574013] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18988.576781] kthread+0x1d1/0x200 [18988.578485] ? set_kthread_struct+0x70/0x70 [18988.580520] ret_from_fork+0x1f/0x30 [18996.052010] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [18996.054397] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18996.058065] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [18996.072623] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18996.074654] Modules linked in: [18996.078092] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18996.078102] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18996.078117] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18996.078120] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18996.078126] RAX: 00000000000052ce RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18996.078128] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [18996.078130] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [18996.078132] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [18996.078134] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [18996.078138] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [18996.078140] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18996.078142] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [18996.078149] Call Trace: [18996.078154] [18996.078160] ? show_regs.cold.9+0x22/0x2f [18996.078168] ? watchdog_timer_fn.cold.6+0x62/0x10a [18996.078173] ? __hrtimer_run_queues+0x1c2/0x4d0 [18996.078177] ? watchdog+0x40/0x40 [18996.078180] ? hrtimer_interrupt+0x128/0x2c0 [18996.078184] ? smp_apic_timer_interrupt+0xa6/0x270 [18996.078191] ? apic_timer_interrupt+0xf/0x20 [18996.078195] [18996.078197] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18996.078201] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18996.078205] do_raw_spin_lock+0x177/0x190 [18996.078209] _raw_spin_lock+0x21/0x30 [18996.078216] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18996.079269] lustre(O) [18996.082261] ? do_raw_spin_unlock+0x75/0x190 [18996.083958] osp(O) [18996.090632] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18996.092658] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18996.092855] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18996.092861] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18996.092867] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18996.093182] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18996.093191] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18996.093195] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18996.093198] RAX: 0000000000005a38 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [18996.093200] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [18996.093202] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [18996.093204] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [18996.093206] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [18996.093209] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [18996.093211] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18996.093213] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [18996.093221] Call Trace: [18996.093226] [18996.093229] ? show_regs.cold.9+0x22/0x2f [18996.093238] ? watchdog_timer_fn.cold.6+0x62/0x10a [18996.093243] ? __hrtimer_run_queues+0x1c2/0x4d0 [18996.093247] ? watchdog+0x40/0x40 [18996.093251] ? hrtimer_interrupt+0x128/0x2c0 [18996.093256] ? smp_apic_timer_interrupt+0xa6/0x270 [18996.093264] ? apic_timer_interrupt+0xf/0x20 [18996.093269] [18996.093272] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18996.093277] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18996.093283] do_raw_spin_lock+0x177/0x190 [18996.093287] _raw_spin_lock+0x21/0x30 [18996.093292] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18996.093458] process_one_work+0x2c8/0x700 [18996.093466] worker_thread+0x296/0x6e0 [18996.093471] ? rescuer_thread+0x570/0x570 [18996.093475] kthread+0x1d1/0x200 [18996.093480] ? set_kthread_struct+0x70/0x70 [18996.093483] ret_from_fork+0x1f/0x30 [18996.312017] ? do_wait_intr+0xf0/0xf0 [18996.313340] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18996.315022] mdc_reint+0x2d/0x110 [mdc] [18996.316763] mdc_unlink+0x658/0x870 [mdc] [18996.318304] lmv_unlink+0x34d/0xbd0 [lmv] [18996.320648] ll_unlink+0x24d/0x7f0 [lustre] [18996.324096] vfs_unlink+0x202/0x330 [18996.325884] do_unlinkat+0x2fa/0x460 [18996.327243] __x64_sys_unlink+0x37/0x50 [18996.328618] do_syscall_64+0xc1/0x440 [18996.330037] entry_SYSCALL_64_after_hwframe+0x49/0xae [18996.331910] RIP: 0033:0x7f151bdee10b [18996.333280] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [18996.344385] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [18996.351059] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [18996.353245] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [18996.361381] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [18996.365861] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [18996.368391] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [19000.802170] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17070s! [19000.805974] Showing busy workqueues and worker pools: [19000.808237] workqueue events: flags=0x0 [19000.810292] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19000.813262] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [19000.819929] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19000.827340] pending: kfree_rcu_monitor [19000.829647] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19000.836250] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [19000.846961] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19000.851268] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [19000.856146] workqueue events_long: flags=0x0 [19000.858333] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19000.861218] pending: lu_objects_flush [obdclass] [19000.864049] workqueue events_unbound: flags=0x2 [19000.866665] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [19000.870061] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [19000.874960] workqueue events_power_efficient: flags=0x80 [19000.877445] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19000.880252] pending: do_cache_clean, neigh_periodic_work, check_lifetime [19000.886799] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19000.891142] pending: neigh_periodic_work [19000.899217] workqueue mm_percpu_wq: flags=0x8 [19000.902317] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19000.906702] pending: vmstat_update, lru_add_drain_per_cpu [19000.910141] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19000.913884] pending: vmstat_update [19000.915479] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19000.918915] pending: lru_add_drain_per_cpu [19000.921247] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [19000.924793] pending: lru_add_drain_per_cpu BAR(48) [19000.927332] workqueue writeback: flags=0x4a [19000.929413] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [19000.932574] pending: wb_workfn [19000.934165] workqueue xprtiod: flags=0xa [19000.936124] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [19000.939599] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [19000.943580] workqueue ipv6_addrconf: flags=0x40008 [19000.946472] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19000.949718] pending: addrconf_verify_work [19000.952081] workqueue knbd0-recv: flags=0x1a [19000.954623] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19000.958807] in-flight: 658:recv_work [19000.961768] workqueue ptlrpc_pinger: flags=0x2 [19000.964405] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19000.968125] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [19000.971163] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17070s workers=4 idle: 32984 37 8 [19000.975504] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18985s workers=2 idle: 661 [19016.055013] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [19016.057015] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [19016.057622] Modules linked in: lustre(O) [19016.061078] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19016.061155] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19016.061163] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19016.061166] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19016.061180] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19016.061185] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19016.061189] RAX: 0000000000002585 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [19016.061192] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [19016.061194] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [19016.061196] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [19016.061198] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [19016.061201] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [19016.061203] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19016.061205] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [19016.061213] Call Trace: [19016.061219] [19016.061224] ? show_regs.cold.9+0x22/0x2f [19016.061234] ? watchdog_timer_fn.cold.6+0x62/0x10a [19016.061240] ? __hrtimer_run_queues+0x1c2/0x4d0 [19016.061245] ? watchdog+0x40/0x40 [19016.061249] ? hrtimer_interrupt+0x128/0x2c0 [19016.061253] ? smp_apic_timer_interrupt+0xa6/0x270 [19016.061262] ? apic_timer_interrupt+0xf/0x20 [19016.061267] [19016.061269] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19016.061275] do_raw_spin_lock+0x177/0x190 [19016.061280] _raw_spin_lock+0x21/0x30 [19016.061285] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19016.065663] osp(O) [19016.087003] ? do_raw_spin_unlock+0x75/0x190 [19016.087025] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19016.087306] ? do_wait_intr+0xf0/0xf0 [19016.087315] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19016.094280] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) [19016.097913] mdc_close+0x53b/0xd70 [mdc] [19016.097975] lmv_close+0x18d/0x490 [lmv] [19016.098004] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [19016.098165] ll_close_inode_openhandle+0x493/0xea0 [lustre] [19016.098273] ll_md_real_close+0xa5/0x2a0 [lustre] [19016.098367] ll_md_close+0x274/0x800 [lustre] [19016.098451] ll_file_release+0x128/0xc50 [lustre] [19016.102833] mdc(O) lov(O) [19016.110460] __fput+0x10d/0x440 [19016.114386] osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) [19016.117166] ____fput+0x12/0x20 [19016.117187] task_work_run+0xc6/0x110 [19016.117195] exit_to_usermode_loop+0x1dd/0x1f0 [19016.117202] do_syscall_64+0x3d6/0x440 [19016.117207] entry_SYSCALL_64_after_hwframe+0x49/0xae [19016.117217] RIP: 0033:0x7f151bee3cd8 [19016.117228] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [19016.117238] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [19016.117245] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [19016.117248] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [19016.120478] lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19016.120526] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19016.123590] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [19016.123600] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [19016.123604] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [19016.232892] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19016.237247] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [19016.239891] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [19016.246328] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19016.248833] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [19016.251328] RDX: 0000000000004a14 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [19016.253436] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [19016.255665] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [19016.257726] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19016.260067] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [19016.262344] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19016.264177] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [19016.266265] Call Trace: [19016.266982] [19016.267835] ? show_regs.cold.9+0x22/0x2f [19016.268998] ? watchdog_timer_fn.cold.6+0x62/0x10a [19016.270348] ? __hrtimer_run_queues+0x1c2/0x4d0 [19016.271687] ? watchdog+0x40/0x40 [19016.272515] ? hrtimer_interrupt+0x128/0x2c0 [19016.273747] ? smp_apic_timer_interrupt+0xa6/0x270 [19016.275158] ? apic_timer_interrupt+0xf/0x20 [19016.276696] [19016.277225] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [19016.278578] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19016.280021] do_raw_spin_lock+0x177/0x190 [19016.281155] _raw_spin_lock+0x21/0x30 [19016.282147] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19016.283891] obd_mod_free+0x46/0x140 [ptlrpc] [19016.285379] mdc_commit_open+0x91/0xb0 [mdc] [19016.286488] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19016.288300] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19016.290080] after_reply+0xe2c/0x1780 [ptlrpc] [19016.291705] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19016.293176] ? __next_timer_interrupt+0x160/0x160 [19016.294456] ptlrpcd+0xa82/0xe40 [ptlrpc] [19016.295803] ? do_wait_intr+0xf0/0xf0 [19016.296812] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19016.298518] kthread+0x1d1/0x200 [19016.299754] ? set_kthread_struct+0x70/0x70 [19016.300822] ret_from_fork+0x1f/0x30 [19024.056022] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [19024.058987] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19024.079700] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19024.083451] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19024.086822] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19024.090743] RIP: 0010:native_safe_halt+0x2e/0x50 [19024.092765] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19024.101693] RSP: 0018:ffffb2cb80dabdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19024.104814] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [19024.108410] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582134154 [19024.111770] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [19024.114815] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [19024.117813] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [19024.120741] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [19024.123826] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19024.126303] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [19024.128938] Call Trace: [19024.129997] [19024.130921] ? show_regs.cold.9+0x22/0x2f [19024.132741] ? watchdog_timer_fn.cold.6+0x62/0x10a [19024.134698] ? __hrtimer_run_queues+0x1c2/0x4d0 [19024.136779] ? watchdog+0x40/0x40 [19024.138508] ? hrtimer_interrupt+0x128/0x2c0 [19024.140391] ? smp_apic_timer_interrupt+0xa6/0x270 [19024.142770] ? apic_timer_interrupt+0xf/0x20 [19024.144579] [19024.145733] ? native_safe_halt+0x2e/0x50 [19024.147791] kvm_wait+0xcd/0xe0 [19024.149030] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19024.151299] do_raw_spin_lock+0x177/0x190 [19024.153405] _raw_spin_lock+0x21/0x30 [19024.155212] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19024.158810] process_one_work+0x2c8/0x700 [19024.160802] worker_thread+0x296/0x6e0 [19024.162987] ? rescuer_thread+0x570/0x570 [19024.165557] kthread+0x1d1/0x200 [19024.167154] ? set_kthread_struct+0x70/0x70 [19024.169440] ret_from_fork+0x1f/0x30 [19028.052132] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [19028.055745] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19028.076415] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19028.081645] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19028.085328] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19028.088365] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19028.096705] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19028.100876] RAX: 0000000000001077 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [19028.103914] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [19028.107018] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [19028.110796] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [19028.115211] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [19028.118883] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [19028.124146] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19028.126691] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [19028.129924] Call Trace: [19028.131269] [19028.132193] ? show_regs.cold.9+0x22/0x2f [19028.134256] ? watchdog_timer_fn.cold.6+0x62/0x10a [19028.136399] ? __hrtimer_run_queues+0x1c2/0x4d0 [19028.138882] ? watchdog+0x40/0x40 [19028.140445] ? hrtimer_interrupt+0x128/0x2c0 [19028.142693] ? smp_apic_timer_interrupt+0xa6/0x270 [19028.145477] ? apic_timer_interrupt+0xf/0x20 [19028.147372] [19028.148570] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19028.151039] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19028.153868] do_raw_spin_lock+0x177/0x190 [19028.156086] _raw_spin_lock+0x21/0x30 [19028.157978] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19028.160796] ? do_raw_spin_unlock+0x75/0x190 [19028.162977] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19028.165928] ? do_wait_intr+0xf0/0xf0 [19028.167568] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19028.170640] mdc_reint+0x2d/0x110 [mdc] [19028.172024] mdc_unlink+0x658/0x870 [mdc] [19028.174491] lmv_unlink+0x34d/0xbd0 [lmv] [19028.176473] ll_unlink+0x24d/0x7f0 [lustre] [19028.178781] vfs_unlink+0x202/0x330 [19028.180431] do_unlinkat+0x2fa/0x460 [19028.182543] __x64_sys_unlink+0x37/0x50 [19028.184458] do_syscall_64+0xc1/0x440 [19028.186175] entry_SYSCALL_64_after_hwframe+0x49/0xae [19028.188303] RIP: 0033:0x7f151bdee10b [19028.190166] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [19028.198334] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [19028.201272] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [19028.203689] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [19028.206506] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [19028.209757] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [19028.212298] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [19031.015550] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17100s! [19031.029785] Showing busy workqueues and worker pools: [19031.033335] workqueue events: flags=0x0 [19031.036447] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19031.042398] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [19031.050451] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19031.054420] pending: kfree_rcu_monitor [19031.056533] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19031.060156] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [19031.065266] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19031.069070] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [19031.073025] workqueue events_long: flags=0x0 [19031.075232] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19031.079179] pending: lu_objects_flush [obdclass] [19031.085513] workqueue events_unbound: flags=0x2 [19031.091064] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [19031.095038] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [19031.104618] workqueue events_power_efficient: flags=0x80 [19031.107544] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19031.113533] pending: do_cache_clean, neigh_periodic_work, check_lifetime [19031.117768] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19031.122064] pending: neigh_periodic_work [19031.124783] workqueue mm_percpu_wq: flags=0x8 [19031.126957] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19031.131592] pending: vmstat_update, lru_add_drain_per_cpu [19031.134943] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19031.139141] pending: vmstat_update [19031.141179] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19031.144972] pending: lru_add_drain_per_cpu [19031.147327] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [19031.150268] pending: lru_add_drain_per_cpu BAR(48) [19031.152814] workqueue writeback: flags=0x4a [19031.155383] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [19031.158274] pending: wb_workfn [19031.160207] workqueue xprtiod: flags=0xa [19031.162264] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [19031.165961] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [19031.169993] workqueue ipv6_addrconf: flags=0x40008 [19031.171958] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19031.175592] pending: addrconf_verify_work [19031.177364] workqueue knbd0-recv: flags=0x1a [19031.179697] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19031.182961] in-flight: 658:recv_work [19031.185240] workqueue ptlrpc_pinger: flags=0x2 [19031.187370] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19031.190777] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [19031.194134] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17100s workers=4 idle: 32984 37 8 [19031.198214] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19015s workers=2 idle: 661 [19044.055149] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [19044.057031] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [19044.061212] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19044.061612] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19044.061621] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19044.061624] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [19044.061857] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [19044.061867] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19044.061872] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [19044.061874] RDX: 000000000000349e RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [19044.061880] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [19044.061890] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [19044.061892] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19044.061895] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [19044.061898] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19044.061900] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [19044.061914] Call Trace: [19044.061919] [19044.061925] ? show_regs.cold.9+0x22/0x2f [19044.061935] ? watchdog_timer_fn.cold.6+0x62/0x10a [19044.061942] ? __hrtimer_run_queues+0x1c2/0x4d0 [19044.061947] ? watchdog+0x40/0x40 [19044.061951] ? hrtimer_interrupt+0x128/0x2c0 [19044.061955] ? smp_apic_timer_interrupt+0xa6/0x270 [19044.061969] ? apic_timer_interrupt+0xf/0x20 [19044.061975] [19044.061978] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [19044.061985] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19044.061990] do_raw_spin_lock+0x177/0x190 [19044.061995] _raw_spin_lock+0x21/0x30 [19044.062005] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19044.062629] obd_mod_free+0x46/0x140 [ptlrpc] [19044.063188] mdc_commit_open+0x91/0xb0 [mdc] [19044.063227] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19044.063664] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19044.063902] after_reply+0xe2c/0x1780 [ptlrpc] [19044.064434] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19044.064668] ? __next_timer_interrupt+0x160/0x160 [19044.064694] ptlrpcd+0xa82/0xe40 [ptlrpc] [19044.070809] Modules linked in: [19044.099767] ? do_wait_intr+0xf0/0xf0 [19044.115645] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19044.115739] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19044.115747] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19044.115750] RIP: 0010:native_safe_halt+0x2e/0x50 [19044.115763] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19044.115768] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [19044.115772] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [19044.115774] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [19044.115776] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [19044.115778] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [19044.115781] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [19044.115785] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [19044.115787] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19044.115790] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [19044.115798] Call Trace: [19044.115803] [19044.115809] ? show_regs.cold.9+0x22/0x2f [19044.115824] ? watchdog_timer_fn.cold.6+0x62/0x10a [19044.115831] ? __hrtimer_run_queues+0x1c2/0x4d0 [19044.115836] ? watchdog+0x40/0x40 [19044.115839] ? hrtimer_interrupt+0x128/0x2c0 [19044.115843] ? smp_apic_timer_interrupt+0xa6/0x270 [19044.115850] ? apic_timer_interrupt+0xf/0x20 [19044.115854] [19044.115857] ? native_safe_halt+0x2e/0x50 [19044.115861] kvm_wait+0xcd/0xe0 [19044.115868] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19044.115876] do_raw_spin_lock+0x177/0x190 [19044.116046] _raw_spin_lock+0x21/0x30 [19044.116064] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19044.116503] ? do_raw_spin_unlock+0x75/0x190 [19044.116510] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19044.116827] ? do_wait_intr+0xf0/0xf0 [19044.116836] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19044.117062] mdc_close+0x53b/0xd70 [mdc] [19044.117100] lmv_close+0x18d/0x490 [lmv] [19044.117129] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [19044.117609] ll_close_inode_openhandle+0x493/0xea0 [lustre] [19044.117737] ll_md_real_close+0xa5/0x2a0 [lustre] [19044.117821] ll_md_close+0x274/0x800 [lustre] [19044.118039] ll_file_release+0x128/0xc50 [lustre] [19044.118131] __fput+0x10d/0x440 [19044.118141] ____fput+0x12/0x20 [19044.118145] task_work_run+0xc6/0x110 [19044.118154] exit_to_usermode_loop+0x1dd/0x1f0 [19044.118163] do_syscall_64+0x3d6/0x440 [19044.118167] entry_SYSCALL_64_after_hwframe+0x49/0xae [19044.118176] RIP: 0033:0x7f151bee3cd8 [19044.118184] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [19044.118194] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [19044.118199] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [19044.118202] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [19044.118204] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [19044.118206] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [19044.118208] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [19044.542710] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19044.545740] kthread+0x1d1/0x200 [19044.547792] ? set_kthread_struct+0x70/0x70 [19044.549570] ret_from_fork+0x1f/0x30 [19052.056011] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [19052.067147] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19052.119828] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19052.136852] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19052.149931] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19052.156071] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19052.162852] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19052.182334] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19052.192824] RAX: 00000000000050ba RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [19052.199910] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [19052.204001] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [19052.211802] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [19052.220827] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [19052.225982] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [19052.235432] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19052.242044] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [19052.251036] Call Trace: [19052.252500] [19052.255228] ? show_regs.cold.9+0x22/0x2f [19052.260620] ? watchdog_timer_fn.cold.6+0x62/0x10a [19052.262438] ? __hrtimer_run_queues+0x1c2/0x4d0 [19052.267006] ? watchdog+0x40/0x40 [19052.269115] ? hrtimer_interrupt+0x128/0x2c0 [19052.273450] ? smp_apic_timer_interrupt+0xa6/0x270 [19052.279081] ? apic_timer_interrupt+0xf/0x20 [19052.285401] [19052.288878] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19052.294922] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19052.301039] do_raw_spin_lock+0x177/0x190 [19052.307447] _raw_spin_lock+0x21/0x30 [19052.310764] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19052.317391] process_one_work+0x2c8/0x700 [19052.320163] worker_thread+0x296/0x6e0 [19052.324738] ? rescuer_thread+0x570/0x570 [19052.327699] kthread+0x1d1/0x200 [19052.330419] ? set_kthread_struct+0x70/0x70 [19052.333977] ret_from_fork+0x1f/0x30 [19056.053616] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [19056.067746] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19056.105478] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19056.114625] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19056.120730] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19056.126154] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19056.144523] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19056.148869] RAX: 0000000000005b4f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [19056.154795] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [19056.160650] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [19056.166768] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [19056.170793] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [19056.174901] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [19056.180342] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19056.182718] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [19056.187496] Call Trace: [19056.188610] [19056.190673] ? show_regs.cold.9+0x22/0x2f [19056.192796] ? watchdog_timer_fn.cold.6+0x62/0x10a [19056.195817] ? __hrtimer_run_queues+0x1c2/0x4d0 [19056.197886] ? watchdog+0x40/0x40 [19056.199442] ? hrtimer_interrupt+0x128/0x2c0 [19056.202864] ? smp_apic_timer_interrupt+0xa6/0x270 [19056.206794] ? apic_timer_interrupt+0xf/0x20 [19056.209380] [19056.210028] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19056.213618] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19056.215431] do_raw_spin_lock+0x177/0x190 [19056.218477] _raw_spin_lock+0x21/0x30 [19056.224517] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19056.229312] ? do_raw_spin_unlock+0x75/0x190 [19056.231788] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19056.236667] ? do_wait_intr+0xf0/0xf0 [19056.238035] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19056.243098] mdc_reint+0x2d/0x110 [mdc] [19056.245502] mdc_unlink+0x658/0x870 [mdc] [19056.247613] lmv_unlink+0x34d/0xbd0 [lmv] [19056.249853] ll_unlink+0x24d/0x7f0 [lustre] [19056.252475] vfs_unlink+0x202/0x330 [19056.254552] do_unlinkat+0x2fa/0x460 [19056.256444] __x64_sys_unlink+0x37/0x50 [19056.258059] do_syscall_64+0xc1/0x440 [19056.260730] entry_SYSCALL_64_after_hwframe+0x49/0xae [19056.263429] RIP: 0033:0x7f151bdee10b [19056.266534] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [19056.278330] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [19056.287157] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [19056.291707] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [19056.298750] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [19056.301489] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [19056.307621] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [19061.217164] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17130s! [19061.219302] Showing busy workqueues and worker pools: [19061.221089] workqueue events: flags=0x0 [19061.222403] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19061.224543] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [19061.228952] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19061.231447] pending: kfree_rcu_monitor [19061.232756] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19061.234980] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [19061.238136] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19061.239905] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [19061.242333] workqueue events_long: flags=0x0 [19061.243435] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19061.245873] pending: lu_objects_flush [obdclass] [19061.247576] workqueue events_unbound: flags=0x2 [19061.249142] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [19061.252450] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [19061.257115] workqueue events_power_efficient: flags=0x80 [19061.259784] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19061.262439] pending: do_cache_clean, neigh_periodic_work, check_lifetime [19061.264364] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19061.267515] pending: neigh_periodic_work [19061.269252] workqueue mm_percpu_wq: flags=0x8 [19061.270537] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19061.272539] pending: vmstat_update, lru_add_drain_per_cpu [19061.275114] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19061.277802] pending: vmstat_update [19061.279500] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19061.281688] pending: lru_add_drain_per_cpu [19061.282917] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [19061.285295] pending: lru_add_drain_per_cpu BAR(48) [19061.287864] workqueue writeback: flags=0x4a [19061.289313] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [19061.292861] pending: wb_workfn [19061.294166] workqueue xprtiod: flags=0xa [19061.295821] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [19061.299285] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [19061.301831] workqueue ipv6_addrconf: flags=0x40008 [19061.304056] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19061.305925] pending: addrconf_verify_work [19061.307509] workqueue knbd0-recv: flags=0x1a [19061.308660] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19061.310773] in-flight: 658:recv_work [19061.313101] workqueue ptlrpc_pinger: flags=0x2 [19061.315455] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19061.318110] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [19061.320049] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17130s workers=4 idle: 32984 37 8 [19061.322450] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19046s workers=2 idle: 661 [19072.055714] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ptlrpcd_rcv:2351] [19072.061284] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36052] [19072.061469] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19072.061545] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19072.061552] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19072.061626] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [19072.061639] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [19072.061643] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19072.061648] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [19072.061650] RDX: 0000000000006666 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [19072.061653] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [19072.061654] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [19072.061656] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19072.061660] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [19072.061663] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19072.061665] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [19072.061674] Call Trace: [19072.061679] [19072.061684] ? show_regs.cold.9+0x22/0x2f [19072.061693] ? watchdog_timer_fn.cold.6+0x62/0x10a [19072.061699] ? __hrtimer_run_queues+0x1c2/0x4d0 [19072.061704] ? watchdog+0x40/0x40 [19072.061708] ? hrtimer_interrupt+0x128/0x2c0 [19072.061712] ? smp_apic_timer_interrupt+0xa6/0x270 [19072.061721] ? apic_timer_interrupt+0xf/0x20 [19072.061726] [19072.061729] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [19072.061734] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19072.061739] do_raw_spin_lock+0x177/0x190 [19072.061744] _raw_spin_lock+0x21/0x30 [19072.061751] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19072.062015] obd_mod_free+0x46/0x140 [ptlrpc] [19072.062429] mdc_commit_open+0x91/0xb0 [mdc] [19072.063199] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19072.063741] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19072.063955] after_reply+0xe2c/0x1780 [ptlrpc] [19072.064167] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19072.064518] ? __next_timer_interrupt+0x160/0x160 [19072.064530] ptlrpcd+0xa82/0xe40 [ptlrpc] [19072.071544] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19072.071632] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19072.071639] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19072.071643] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19072.071658] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19072.071663] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19072.071668] RAX: 0000000000003f1a RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [19072.071671] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [19072.071674] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [19072.071676] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [19072.071678] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [19072.071682] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [19072.071685] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19072.071687] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [19072.071696] Call Trace: [19072.071701] [19072.071708] ? show_regs.cold.9+0x22/0x2f [19072.071718] ? watchdog_timer_fn.cold.6+0x62/0x10a [19072.071724] ? __hrtimer_run_queues+0x1c2/0x4d0 [19072.071728] ? watchdog+0x40/0x40 [19072.071733] ? hrtimer_interrupt+0x128/0x2c0 [19072.071737] ? smp_apic_timer_interrupt+0xa6/0x270 [19072.109680] ? do_wait_intr+0xf0/0xf0 [19072.109713] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19072.123638] ? apic_timer_interrupt+0xf/0x20 [19072.123663] [19072.123669] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19072.123682] do_raw_spin_lock+0x177/0x190 [19072.123692] _raw_spin_lock+0x21/0x30 [19072.123697] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19072.124752] ? do_raw_spin_unlock+0x75/0x190 [19072.124760] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19072.125057] ? do_wait_intr+0xf0/0xf0 [19072.125066] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19072.125410] mdc_close+0x53b/0xd70 [mdc] [19072.126267] lmv_close+0x18d/0x490 [lmv] [19072.126515] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [19072.127906] ll_close_inode_openhandle+0x493/0xea0 [lustre] [19072.128287] ll_md_real_close+0xa5/0x2a0 [lustre] [19072.128368] ll_md_close+0x274/0x800 [lustre] [19072.128684] ll_file_release+0x128/0xc50 [lustre] [19072.128764] __fput+0x10d/0x440 [19072.128773] ____fput+0x12/0x20 [19072.128778] task_work_run+0xc6/0x110 [19072.142845] kthread+0x1d1/0x200 [19072.142864] ? set_kthread_struct+0x70/0x70 [19072.142868] ret_from_fork+0x1f/0x30 [19072.544974] exit_to_usermode_loop+0x1dd/0x1f0 [19072.546720] do_syscall_64+0x3d6/0x440 [19072.547854] entry_SYSCALL_64_after_hwframe+0x49/0xae [19072.550498] RIP: 0033:0x7f151bee3cd8 [19072.551423] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [19072.558396] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [19072.561371] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [19072.564717] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [19072.568200] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [19072.572261] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [19072.577131] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [19080.056027] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [19080.061354] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19080.087276] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19080.092798] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19080.098243] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19080.102907] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19080.106514] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19080.119730] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19080.123717] RAX: 000000000000358e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [19080.126923] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [19080.130702] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [19080.133886] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [19080.137326] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [19080.140602] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [19080.144835] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19080.148398] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [19080.152223] Call Trace: [19080.154410] [19080.155740] ? show_regs.cold.9+0x22/0x2f [19080.157356] ? watchdog_timer_fn.cold.6+0x62/0x10a [19080.158953] ? __hrtimer_run_queues+0x1c2/0x4d0 [19080.162111] ? watchdog+0x40/0x40 [19080.164465] ? hrtimer_interrupt+0x128/0x2c0 [19080.166530] ? smp_apic_timer_interrupt+0xa6/0x270 [19080.169900] ? apic_timer_interrupt+0xf/0x20 [19080.173671] [19080.174298] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19080.176653] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19080.182579] do_raw_spin_lock+0x177/0x190 [19080.186652] _raw_spin_lock+0x21/0x30 [19080.188712] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19080.192407] process_one_work+0x2c8/0x700 [19080.194805] worker_thread+0x296/0x6e0 [19080.196937] ? rescuer_thread+0x570/0x570 [19080.199704] kthread+0x1d1/0x200 [19080.201824] ? set_kthread_struct+0x70/0x70 [19080.205471] ret_from_fork+0x1f/0x30 [19085.005747] rcu: INFO: rcu_sched self-detected stall on CPU [19085.021307] rcu: 1-...!: (10332070 ticks this GP) idle=096/1/0x4000000000000002 softirq=121911/121911 fqs=859 [19085.031192] (t=17160358 jiffies g=249657 q=7673) [19085.035573] rcu: rcu_sched kthread starved for 17155310 jiffies! g249657 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [19085.041347] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [19085.055311] rcu: RCU grace-period kthread stack dump: [19085.058492] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [19085.072881] Call Trace: [19085.074644] __schedule+0x351/0xcb0 [19085.078788] schedule+0xc0/0x180 [19085.080517] schedule_timeout+0xb4/0x190 [19085.085160] ? do_raw_spin_unlock+0x75/0x190 [19085.089836] ? __next_timer_interrupt+0x160/0x160 [19085.099151] ? prepare_to_swait_event+0xf3/0x140 [19085.102871] rcu_gp_kthread+0x8dd/0x15b0 [19085.104936] ? force_qs_rnp+0x2c0/0x2c0 [19085.109255] kthread+0x1d1/0x200 [19085.111652] ? set_kthread_struct+0x70/0x70 [19085.120254] ret_from_fork+0x1f/0x30 [19085.128490] rcu: Stack dump where RCU GP kthread last ran: [19085.130640] Sending NMI from CPU 1 to CPUs 3: [19085.134771] NMI backtrace for cpu 3 [19085.134778] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19085.134781] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19085.134782] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19085.134784] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19085.134787] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 [19085.134790] RAX: 0000000000007511 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [19085.134791] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [19085.134793] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [19085.134794] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [19085.134796] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [19085.134797] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [19085.134798] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19085.134799] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [19085.134800] Call Trace: [19085.134801] [19085.134802] ? show_regs.cold.9+0x22/0x2f [19085.134803] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [19085.134804] ? nmi_cpu_backtrace_handler+0x15/0x40 [19085.134805] ? nmi_handle+0xbd/0x210 [19085.134806] ? default_do_nmi+0x4b/0x1b0 [19085.134807] ? do_nmi+0x280/0x380 [19085.134808] ? end_repeat_nmi+0x16/0x1f [19085.134809] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19085.134810] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19085.134812] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19085.134813] [19085.134814] do_raw_spin_lock+0x177/0x190 [19085.134815] _raw_spin_lock+0x21/0x30 [19085.134816] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19085.134817] ? do_raw_spin_unlock+0x75/0x190 [19085.134818] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19085.134819] ? do_wait_intr+0xf0/0xf0 [19085.134820] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19085.134821] mdc_close+0x53b/0xd70 [mdc] [19085.134822] lmv_close+0x18d/0x490 [lmv] [19085.134823] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [19085.134824] ll_close_inode_openhandle+0x493/0xea0 [lustre] [19085.134826] ll_md_real_close+0xa5/0x2a0 [lustre] [19085.134826] ll_md_close+0x274/0x800 [lustre] [19085.134827] ll_file_release+0x128/0xc50 [lustre] [19085.134829] __fput+0x10d/0x440 [19085.134829] ____fput+0x12/0x20 [19085.134830] task_work_run+0xc6/0x110 [19085.134831] exit_to_usermode_loop+0x1dd/0x1f0 [19085.134833] do_syscall_64+0x3d6/0x440 [19085.134834] entry_SYSCALL_64_after_hwframe+0x49/0xae [19085.134835] RIP: 0033:0x7f151bee3cd8 [19085.134836] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [19085.134839] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [19085.134841] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [19085.134842] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [19085.134844] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [19085.134845] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [19085.134846] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [19085.135779] NMI backtrace for cpu 1 [19085.293779] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19085.298731] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19085.304104] Call Trace: [19085.305315] [19085.306298] ? dump_stack+0xbb/0x10e [19085.308301] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [19085.310455] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [19085.312833] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [19085.314661] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [19085.315910] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [19085.320240] ? rcu_dump_cpu_stacks+0x11b/0x171 [19085.323503] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [19085.326625] ? update_process_times+0x6d/0xc0 [19085.329183] ? tick_sched_handle.isra.18+0x27/0x90 [19085.332891] ? tick_sched_timer+0x53/0xb0 [19085.335259] ? __hrtimer_run_queues+0x1c2/0x4d0 [19085.338068] ? tick_sched_handle.isra.18+0x90/0x90 [19085.341134] ? hrtimer_interrupt+0x128/0x2c0 [19085.343646] ? smp_apic_timer_interrupt+0xa6/0x270 [19085.347282] ? apic_timer_interrupt+0xf/0x20 [19085.354356] [19085.355475] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [19085.366073] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19085.373256] ? do_raw_spin_lock+0x177/0x190 [19085.378491] ? _raw_spin_lock+0x21/0x30 [19085.380886] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19085.386121] ? obd_mod_free+0x46/0x140 [ptlrpc] [19085.391174] ? mdc_commit_open+0x91/0xb0 [mdc] [19085.399317] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19085.403103] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19085.408681] ? after_reply+0xe2c/0x1780 [ptlrpc] [19085.413466] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19085.421894] ? __next_timer_interrupt+0x160/0x160 [19085.425004] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [19085.429915] ? do_wait_intr+0xf0/0xf0 [19085.431319] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19085.438565] ? kthread+0x1d1/0x200 [19085.445628] ? set_kthread_struct+0x70/0x70 [19085.447273] ? ret_from_fork+0x1f/0x30 [19088.055577] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [19088.070238] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19088.129533] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19088.141249] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19088.144554] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19088.149370] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19088.162836] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19088.170736] RAX: 0000000000002e13 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [19088.178587] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [19088.185405] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [19088.191423] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [19088.199809] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [19088.208728] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [19088.220570] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19088.228240] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [19088.236479] Call Trace: [19088.242101] [19088.243355] ? show_regs.cold.9+0x22/0x2f [19088.245567] ? watchdog_timer_fn.cold.6+0x62/0x10a [19088.249405] ? __hrtimer_run_queues+0x1c2/0x4d0 [19088.257392] ? watchdog+0x40/0x40 [19088.264552] ? hrtimer_interrupt+0x128/0x2c0 [19088.271001] ? smp_apic_timer_interrupt+0xa6/0x270 [19088.274470] ? apic_timer_interrupt+0xf/0x20 [19088.277520] [19088.279261] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19088.284136] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19088.290979] do_raw_spin_lock+0x177/0x190 [19088.295035] _raw_spin_lock+0x21/0x30 [19088.298563] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19088.303219] ? do_raw_spin_unlock+0x75/0x190 [19088.308287] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19088.312740] ? do_wait_intr+0xf0/0xf0 [19088.315090] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19088.317839] mdc_reint+0x2d/0x110 [mdc] [19088.320164] mdc_unlink+0x658/0x870 [mdc] [19088.323088] lmv_unlink+0x34d/0xbd0 [lmv] [19088.328041] ll_unlink+0x24d/0x7f0 [lustre] [19088.332563] vfs_unlink+0x202/0x330 [19088.337082] do_unlinkat+0x2fa/0x460 [19088.339303] __x64_sys_unlink+0x37/0x50 [19088.341794] do_syscall_64+0xc1/0x440 [19088.345814] entry_SYSCALL_64_after_hwframe+0x49/0xae [19088.348465] RIP: 0033:0x7f151bdee10b [19088.350867] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [19088.360599] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [19088.364470] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [19088.372126] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [19088.379017] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [19088.388226] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [19088.395261] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [19091.425856] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17160s! [19091.429833] Showing busy workqueues and worker pools: [19091.431859] workqueue events: flags=0x0 [19091.433487] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19091.448343] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [19091.458083] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19091.461364] pending: kfree_rcu_monitor [19091.464655] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19091.470744] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [19091.476820] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19091.486246] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [19091.491664] workqueue events_long: flags=0x0 [19091.500026] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19091.515665] pending: lu_objects_flush [obdclass] [19091.526130] workqueue events_unbound: flags=0x2 [19091.529231] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [19091.535834] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [19091.550806] workqueue events_power_efficient: flags=0x80 [19091.553478] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19091.570147] pending: do_cache_clean, neigh_periodic_work, check_lifetime [19091.576803] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19091.585767] pending: neigh_periodic_work [19091.589135] workqueue mm_percpu_wq: flags=0x8 [19091.594840] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19091.599026] pending: vmstat_update, lru_add_drain_per_cpu [19091.605187] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19091.612769] pending: vmstat_update [19091.617184] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19091.621314] pending: lru_add_drain_per_cpu [19091.629362] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [19091.635256] pending: lru_add_drain_per_cpu BAR(48) [19091.642713] workqueue writeback: flags=0x4a [19091.645217] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [19091.650284] pending: wb_workfn [19091.652217] workqueue xprtiod: flags=0xa [19091.658254] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [19091.664068] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [19091.671075] workqueue ipv6_addrconf: flags=0x40008 [19091.679693] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19091.685002] pending: addrconf_verify_work [19091.688184] workqueue knbd0-recv: flags=0x1a [19091.694321] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19091.699182] in-flight: 658:recv_work [19091.702426] workqueue ptlrpc_pinger: flags=0x2 [19091.707287] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19091.712987] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [19091.722286] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17161s workers=4 idle: 32984 37 8 [19091.730655] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19076s workers=2 idle: 661 [19108.060518] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [19108.071063] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19108.110159] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19108.125636] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19108.145297] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19108.165411] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [19108.175382] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [19108.188964] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19108.197115] RAX: 00000000000038e8 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [19108.216458] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [19108.229613] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [19108.242591] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [19108.257449] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [19108.266657] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [19108.289182] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19108.294669] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [19108.305790] Call Trace: [19108.308147] [19108.309471] ? show_regs.cold.9+0x22/0x2f [19108.317288] ? watchdog_timer_fn.cold.6+0x62/0x10a [19108.319047] ? __hrtimer_run_queues+0x1c2/0x4d0 [19108.332733] ? watchdog+0x40/0x40 [19108.341903] ? hrtimer_interrupt+0x128/0x2c0 [19108.344490] ? smp_apic_timer_interrupt+0xa6/0x270 [19108.353703] ? apic_timer_interrupt+0xf/0x20 [19108.354951] [19108.355696] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [19108.364863] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19108.374687] do_raw_spin_lock+0x177/0x190 [19108.377223] _raw_spin_lock+0x21/0x30 [19108.380489] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19108.387520] process_one_work+0x2c8/0x700 [19108.392497] worker_thread+0x296/0x6e0 [19108.395491] ? rescuer_thread+0x570/0x570 [19108.402670] kthread+0x1d1/0x200 [19108.409451] ? set_kthread_struct+0x70/0x70 [19108.412039] ret_from_fork+0x1f/0x30 [19112.055634] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [19112.057026] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [19112.062201] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19112.062285] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19112.062292] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19112.062295] RIP: 0010:native_safe_halt+0x2e/0x50 [19112.062310] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19112.062314] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19112.062318] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [19112.062321] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [19112.062323] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [19112.062326] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [19112.062328] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19112.062331] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [19112.062334] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19112.062336] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [19112.062345] Call Trace: [19112.062350] [19112.062355] ? show_regs.cold.9+0x22/0x2f [19112.062364] ? watchdog_timer_fn.cold.6+0x62/0x10a [19112.062371] ? __hrtimer_run_queues+0x1c2/0x4d0 [19112.062375] ? watchdog+0x40/0x40 [19112.062379] ? hrtimer_interrupt+0x128/0x2c0 [19112.062383] ? smp_apic_timer_interrupt+0xa6/0x270 [19112.062389] ? apic_timer_interrupt+0xf/0x20 [19112.062393] [19112.062396] ? native_safe_halt+0x2e/0x50 [19112.062400] kvm_wait+0xcd/0xe0 [19112.062406] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19112.062413] do_raw_spin_lock+0x177/0x190 [19112.062417] _raw_spin_lock+0x21/0x30 [19112.062421] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19112.063378] obd_mod_free+0x46/0x140 [ptlrpc] [19112.063588] mdc_commit_open+0x91/0xb0 [mdc] [19112.063623] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19112.064024] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19112.064218] after_reply+0xe2c/0x1780 [ptlrpc] [19112.064412] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19112.064604] ? __next_timer_interrupt+0x160/0x160 [19112.064614] ptlrpcd+0xa82/0xe40 [ptlrpc] [19112.065247] ? do_wait_intr+0xf0/0xf0 [19112.065257] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19112.065409] kthread+0x1d1/0x200 [19112.065413] ? set_kthread_struct+0x70/0x70 [19112.065416] ret_from_fork+0x1f/0x30 [19112.249746] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19112.301300] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19112.315834] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19112.331448] RIP: 0010:native_safe_halt+0x2e/0x50 [19112.339402] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19112.367266] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19112.375067] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [19112.380309] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [19112.388445] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [19112.395387] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [19112.397734] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [19112.402381] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [19112.409563] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19112.415788] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [19112.419564] Call Trace: [19112.420639] [19112.421171] ? show_regs.cold.9+0x22/0x2f [19112.422695] ? watchdog_timer_fn.cold.6+0x62/0x10a [19112.424523] ? __hrtimer_run_queues+0x1c2/0x4d0 [19112.426731] ? watchdog+0x40/0x40 [19112.429481] ? hrtimer_interrupt+0x128/0x2c0 [19112.431340] ? smp_apic_timer_interrupt+0xa6/0x270 [19112.433507] ? apic_timer_interrupt+0xf/0x20 [19112.437607] [19112.438828] ? native_safe_halt+0x2e/0x50 [19112.441135] kvm_wait+0xcd/0xe0 [19112.443207] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19112.445607] do_raw_spin_lock+0x177/0x190 [19112.447733] _raw_spin_lock+0x21/0x30 [19112.449865] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19112.451732] ? do_raw_spin_unlock+0x75/0x190 [19112.453597] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19112.456074] ? do_wait_intr+0xf0/0xf0 [19112.457508] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19112.463189] mdc_close+0x53b/0xd70 [mdc] [19112.466311] lmv_close+0x18d/0x490 [lmv] [19112.469031] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [19112.471873] ll_close_inode_openhandle+0x493/0xea0 [lustre] [19112.475173] ll_md_real_close+0xa5/0x2a0 [lustre] [19112.476820] ll_md_close+0x274/0x800 [lustre] [19112.480592] ll_file_release+0x128/0xc50 [lustre] [19112.482175] __fput+0x10d/0x440 [19112.482987] ____fput+0x12/0x20 [19112.485725] task_work_run+0xc6/0x110 [19112.488241] exit_to_usermode_loop+0x1dd/0x1f0 [19112.489952] do_syscall_64+0x3d6/0x440 [19112.491801] entry_SYSCALL_64_after_hwframe+0x49/0xae [19112.493834] RIP: 0033:0x7f151bee3cd8 [19112.495371] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [19112.505293] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [19112.508881] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [19112.512108] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [19112.515284] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [19112.518723] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [19112.521117] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [19116.053324] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [19116.062965] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19116.103363] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19116.121346] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19116.130781] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19116.137343] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19116.149092] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19116.153575] RAX: 0000000000000e84 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [19116.165995] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [19116.168397] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [19116.173825] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [19116.177434] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [19116.182270] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [19116.188084] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19116.192364] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [19116.199484] Call Trace: [19116.202190] [19116.202883] ? show_regs.cold.9+0x22/0x2f [19116.204277] ? watchdog_timer_fn.cold.6+0x62/0x10a [19116.211380] ? __hrtimer_run_queues+0x1c2/0x4d0 [19116.215045] ? watchdog+0x40/0x40 [19116.216364] ? hrtimer_interrupt+0x128/0x2c0 [19116.219106] ? smp_apic_timer_interrupt+0xa6/0x270 [19116.221260] ? apic_timer_interrupt+0xf/0x20 [19116.223356] [19116.224755] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19116.227130] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19116.230082] do_raw_spin_lock+0x177/0x190 [19116.231759] _raw_spin_lock+0x21/0x30 [19116.233358] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19116.235855] ? do_raw_spin_unlock+0x75/0x190 [19116.237553] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19116.239994] ? do_wait_intr+0xf0/0xf0 [19116.241374] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19116.244051] mdc_reint+0x2d/0x110 [mdc] [19116.245566] mdc_unlink+0x658/0x870 [mdc] [19116.247023] lmv_unlink+0x34d/0xbd0 [lmv] [19116.248113] ll_unlink+0x24d/0x7f0 [lustre] [19116.249462] vfs_unlink+0x202/0x330 [19116.250853] do_unlinkat+0x2fa/0x460 [19116.252079] __x64_sys_unlink+0x37/0x50 [19116.253459] do_syscall_64+0xc1/0x440 [19116.254863] entry_SYSCALL_64_after_hwframe+0x49/0xae [19116.256887] RIP: 0033:0x7f151bdee10b [19116.258104] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [19116.275491] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [19116.282458] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [19116.290968] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [19116.294843] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [19116.297167] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [19116.305330] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [19122.145458] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17191s! [19122.149799] Showing busy workqueues and worker pools: [19122.152173] workqueue events: flags=0x0 [19122.155534] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19122.166775] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [19122.169279] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19122.171770] pending: kfree_rcu_monitor [19122.173169] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19122.175843] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [19122.179514] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19122.182196] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [19122.185202] workqueue events_long: flags=0x0 [19122.186798] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19122.189387] pending: lu_objects_flush [obdclass] [19122.191390] workqueue events_unbound: flags=0x2 [19122.193152] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [19122.195228] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [19122.199317] workqueue events_power_efficient: flags=0x80 [19122.201379] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19122.203166] pending: do_cache_clean, neigh_periodic_work, check_lifetime [19122.205098] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19122.207412] pending: neigh_periodic_work [19122.208911] workqueue mm_percpu_wq: flags=0x8 [19122.210662] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19122.213286] pending: vmstat_update, lru_add_drain_per_cpu [19122.215311] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19122.217759] pending: vmstat_update [19122.219053] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19122.221706] pending: lru_add_drain_per_cpu [19122.223286] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [19122.225366] pending: lru_add_drain_per_cpu BAR(48) [19122.227373] workqueue writeback: flags=0x4a [19122.228875] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [19122.234256] pending: wb_workfn [19122.235326] workqueue xprtiod: flags=0xa [19122.236774] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [19122.238895] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [19122.241693] workqueue ipv6_addrconf: flags=0x40008 [19122.243363] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19122.245828] pending: addrconf_verify_work [19122.247290] workqueue knbd0-recv: flags=0x1a [19122.248909] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19122.251274] in-flight: 658:recv_work [19122.252349] workqueue ptlrpc_pinger: flags=0x2 [19122.253461] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19122.255186] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [19122.257493] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17191s workers=4 idle: 32984 37 8 [19122.260847] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19107s workers=2 idle: 661 [19136.058977] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [19136.067657] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19136.103652] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19136.118469] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19136.123112] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19136.127614] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19136.129919] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19136.142807] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19136.151693] RAX: 0000000000007b1c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [19136.156397] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [19136.165027] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [19136.175379] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [19136.180356] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [19136.185284] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [19136.188534] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19136.190787] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [19136.193998] Call Trace: [19136.196729] [19136.197368] ? show_regs.cold.9+0x22/0x2f [19136.200776] ? watchdog_timer_fn.cold.6+0x62/0x10a [19136.203693] ? __hrtimer_run_queues+0x1c2/0x4d0 [19136.206400] ? watchdog+0x40/0x40 [19136.208792] ? hrtimer_interrupt+0x128/0x2c0 [19136.213284] ? smp_apic_timer_interrupt+0xa6/0x270 [19136.214794] ? apic_timer_interrupt+0xf/0x20 [19136.215919] [19136.217979] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19136.222670] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19136.228495] do_raw_spin_lock+0x177/0x190 [19136.229999] _raw_spin_lock+0x21/0x30 [19136.234350] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19136.236236] process_one_work+0x2c8/0x700 [19136.241664] worker_thread+0x296/0x6e0 [19136.242724] ? rescuer_thread+0x570/0x570 [19136.248716] kthread+0x1d1/0x200 [19136.249820] ? set_kthread_struct+0x70/0x70 [19136.251729] ret_from_fork+0x1f/0x30 [19140.055086] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [19140.058079] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [19140.058588] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19140.058669] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19140.058675] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19140.058678] RIP: 0010:native_safe_halt+0x2e/0x50 [19140.058692] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19140.058697] RSP: 0018:ffffb2cb81937bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19140.058702] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [19140.058704] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [19140.058706] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [19140.058708] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [19140.058710] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19140.058713] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [19140.058716] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19140.058718] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [19140.058726] Call Trace: [19140.058732] [19140.058738] ? show_regs.cold.9+0x22/0x2f [19140.058749] ? watchdog_timer_fn.cold.6+0x62/0x10a [19140.058757] ? __hrtimer_run_queues+0x1c2/0x4d0 [19140.058762] ? watchdog+0x40/0x40 [19140.058767] ? hrtimer_interrupt+0x128/0x2c0 [19140.058771] ? smp_apic_timer_interrupt+0xa6/0x270 [19140.058777] ? apic_timer_interrupt+0xf/0x20 [19140.058782] [19140.058785] ? native_safe_halt+0x2e/0x50 [19140.058790] kvm_wait+0xcd/0xe0 [19140.058797] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19140.058805] do_raw_spin_lock+0x177/0x190 [19140.058809] _raw_spin_lock+0x21/0x30 [19140.058815] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19140.059577] obd_mod_free+0x46/0x140 [ptlrpc] [19140.059788] mdc_commit_open+0x91/0xb0 [mdc] [19140.059830] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19140.060451] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19140.060656] after_reply+0xe2c/0x1780 [ptlrpc] [19140.060862] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19140.061086] ? __next_timer_interrupt+0x160/0x160 [19140.061098] ptlrpcd+0xa82/0xe40 [ptlrpc] [19140.061306] ? do_wait_intr+0xf0/0xf0 [19140.061316] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19140.061535] kthread+0x1d1/0x200 [19140.061541] ? set_kthread_struct+0x70/0x70 [19140.061545] ret_from_fork+0x1f/0x30 [19140.280923] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19140.325954] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19140.336098] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19140.347013] RIP: 0010:native_safe_halt+0x2e/0x50 [19140.351303] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19140.372999] RSP: 0018:ffffb2cb81c13ae0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19140.380265] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [19140.385400] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [19140.392716] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [19140.401009] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [19140.407182] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [19140.413189] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [19140.422106] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19140.424092] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [19140.433019] Call Trace: [19140.437839] [19140.438757] ? show_regs.cold.9+0x22/0x2f [19140.442599] ? watchdog_timer_fn.cold.6+0x62/0x10a [19140.450825] ? __hrtimer_run_queues+0x1c2/0x4d0 [19140.455143] ? watchdog+0x40/0x40 [19140.456310] ? hrtimer_interrupt+0x128/0x2c0 [19140.462454] ? smp_apic_timer_interrupt+0xa6/0x270 [19140.467076] ? apic_timer_interrupt+0xf/0x20 [19140.473058] [19140.475072] ? native_safe_halt+0x2e/0x50 [19140.476399] kvm_wait+0xcd/0xe0 [19140.481124] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19140.488156] do_raw_spin_lock+0x177/0x190 [19140.493194] _raw_spin_lock+0x21/0x30 [19140.495638] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19140.501987] ? do_raw_spin_unlock+0x75/0x190 [19140.504639] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19140.510042] ? do_wait_intr+0xf0/0xf0 [19140.511398] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19140.534906] mdc_close+0x53b/0xd70 [mdc] [19140.539085] lmv_close+0x18d/0x490 [lmv] [19140.545167] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [19140.548793] ll_close_inode_openhandle+0x493/0xea0 [lustre] [19140.557128] ll_md_real_close+0xa5/0x2a0 [lustre] [19140.558533] ll_md_close+0x274/0x800 [lustre] [19140.566295] ll_file_release+0x128/0xc50 [lustre] [19140.567940] __fput+0x10d/0x440 [19140.569086] ____fput+0x12/0x20 [19140.575380] task_work_run+0xc6/0x110 [19140.576709] exit_to_usermode_loop+0x1dd/0x1f0 [19140.578782] do_syscall_64+0x3d6/0x440 [19140.580860] entry_SYSCALL_64_after_hwframe+0x49/0xae [19140.582656] RIP: 0033:0x7f151bee3cd8 [19140.583890] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [19140.588603] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [19140.591163] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [19140.593994] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [19140.596971] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [19140.599842] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [19140.601893] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [19148.052631] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [writemany:36051] [19148.059012] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19148.088716] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19148.103543] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19148.114152] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19148.120993] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19148.135519] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19148.138725] RAX: 0000000000001f05 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [19148.143599] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [19148.148109] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [19148.152446] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [19148.156182] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [19148.164371] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [19148.171215] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19148.173302] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [19148.187044] Call Trace: [19148.189524] [19148.190135] ? show_regs.cold.9+0x22/0x2f [19148.191823] ? watchdog_timer_fn.cold.6+0x62/0x10a [19148.194571] ? __hrtimer_run_queues+0x1c2/0x4d0 [19148.197718] ? watchdog+0x40/0x40 [19148.198959] ? hrtimer_interrupt+0x128/0x2c0 [19148.209111] ? smp_apic_timer_interrupt+0xa6/0x270 [19148.217443] ? apic_timer_interrupt+0xf/0x20 [19148.221103] [19148.223462] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19148.229306] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19148.231313] do_raw_spin_lock+0x177/0x190 [19148.239132] _raw_spin_lock+0x21/0x30 [19148.244049] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19148.251289] ? do_raw_spin_unlock+0x75/0x190 [19148.254140] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19148.258082] ? do_wait_intr+0xf0/0xf0 [19148.260231] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19148.264416] mdc_reint+0x2d/0x110 [mdc] [19148.271487] mdc_unlink+0x658/0x870 [mdc] [19148.273957] lmv_unlink+0x34d/0xbd0 [lmv] [19148.276073] ll_unlink+0x24d/0x7f0 [lustre] [19148.279953] vfs_unlink+0x202/0x330 [19148.283638] do_unlinkat+0x2fa/0x460 [19148.285926] __x64_sys_unlink+0x37/0x50 [19148.287768] do_syscall_64+0xc1/0x440 [19148.290661] entry_SYSCALL_64_after_hwframe+0x49/0xae [19148.299426] RIP: 0033:0x7f151bdee10b [19148.302076] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [19148.325844] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [19148.334238] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [19148.337692] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [19148.345278] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [19148.351960] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [19148.359650] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [19152.352489] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17221s! [19152.359018] Showing busy workqueues and worker pools: [19152.360899] workqueue events: flags=0x0 [19152.362993] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19152.366832] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [19152.372557] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19152.374919] pending: kfree_rcu_monitor [19152.378038] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19152.384265] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [19152.391433] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19152.393889] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [19152.403146] workqueue events_long: flags=0x0 [19152.410835] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19152.413348] pending: lu_objects_flush [obdclass] [19152.417930] workqueue events_unbound: flags=0x2 [19152.422510] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [19152.430633] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [19152.444263] workqueue events_power_efficient: flags=0x80 [19152.452225] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19152.458949] pending: do_cache_clean, neigh_periodic_work, check_lifetime [19152.470671] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19152.474352] pending: neigh_periodic_work [19152.476014] workqueue mm_percpu_wq: flags=0x8 [19152.481405] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19152.486591] pending: vmstat_update, lru_add_drain_per_cpu [19152.489809] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19152.494849] pending: vmstat_update [19152.497831] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19152.501996] pending: lru_add_drain_per_cpu [19152.504245] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [19152.507819] pending: lru_add_drain_per_cpu BAR(48) [19152.510014] workqueue writeback: flags=0x4a [19152.512575] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [19152.515845] pending: wb_workfn [19152.518751] workqueue xprtiod: flags=0xa [19152.523208] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [19152.526024] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [19152.531044] workqueue ipv6_addrconf: flags=0x40008 [19152.534025] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19152.538157] pending: addrconf_verify_work [19152.540249] workqueue knbd0-recv: flags=0x1a [19152.543033] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19152.545424] in-flight: 658:recv_work [19152.552865] workqueue ptlrpc_pinger: flags=0x2 [19152.559820] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19152.570298] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [19152.576105] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17221s workers=4 idle: 32984 37 8 [19152.583257] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19137s workers=2 idle: 661 [19164.056017] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [19164.059034] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19164.079134] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19164.082690] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19164.095987] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19164.104023] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19164.109309] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19164.120206] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19164.124042] RAX: 0000000000005411 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [19164.129942] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [19164.136983] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [19164.142943] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [19164.150654] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [19164.156422] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [19164.162931] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19164.166872] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [19164.172123] Call Trace: [19164.176025] [19164.177396] ? show_regs.cold.9+0x22/0x2f [19164.179352] ? watchdog_timer_fn.cold.6+0x62/0x10a [19164.180877] ? __hrtimer_run_queues+0x1c2/0x4d0 [19164.182229] ? watchdog+0x40/0x40 [19164.185011] ? hrtimer_interrupt+0x128/0x2c0 [19164.187840] ? smp_apic_timer_interrupt+0xa6/0x270 [19164.189955] ? apic_timer_interrupt+0xf/0x20 [19164.192015] [19164.192905] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19164.198338] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19164.200668] do_raw_spin_lock+0x177/0x190 [19164.201864] _raw_spin_lock+0x21/0x30 [19164.204845] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19164.206561] process_one_work+0x2c8/0x700 [19164.209760] worker_thread+0x296/0x6e0 [19164.211006] ? rescuer_thread+0x570/0x570 [19164.215205] kthread+0x1d1/0x200 [19164.216200] ? set_kthread_struct+0x70/0x70 [19164.217519] ret_from_fork+0x1f/0x30 [19168.055103] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [19168.058562] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19168.060739] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [19168.084042] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19168.084116] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19168.084123] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [19168.084142] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [19168.084146] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19168.084151] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [19168.084153] RDX: 0000000000001faa RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [19168.084156] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [19168.084161] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [19168.084165] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19168.084168] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [19168.084170] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19168.084172] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [19168.084180] Call Trace: [19168.084188] [19168.084195] ? show_regs.cold.9+0x22/0x2f [19168.084204] ? watchdog_timer_fn.cold.6+0x62/0x10a [19168.084210] ? __hrtimer_run_queues+0x1c2/0x4d0 [19168.084214] ? watchdog+0x40/0x40 [19168.084219] ? hrtimer_interrupt+0x128/0x2c0 [19168.084230] ? smp_apic_timer_interrupt+0xa6/0x270 [19168.084251] ? apic_timer_interrupt+0xf/0x20 [19168.084263] [19168.084266] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [19168.084271] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19168.084276] do_raw_spin_lock+0x177/0x190 [19168.084281] _raw_spin_lock+0x21/0x30 [19168.084293] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19168.085042] obd_mod_free+0x46/0x140 [ptlrpc] [19168.085404] mdc_commit_open+0x91/0xb0 [mdc] [19168.085463] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19168.085773] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19168.086023] after_reply+0xe2c/0x1780 [ptlrpc] [19168.090612] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19168.090721] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19168.090735] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19168.090737] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19168.090753] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19168.090757] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19168.090766] RAX: 00000000000009f1 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [19168.090768] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [19168.090773] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [19168.090775] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [19168.090777] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [19168.090780] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [19168.090783] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19168.090785] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [19168.090792] Call Trace: [19168.090798] [19168.090803] ? show_regs.cold.9+0x22/0x2f [19168.090819] ? watchdog_timer_fn.cold.6+0x62/0x10a [19168.090825] ? __hrtimer_run_queues+0x1c2/0x4d0 [19168.090830] ? watchdog+0x40/0x40 [19168.090833] ? hrtimer_interrupt+0x128/0x2c0 [19168.090837] ? smp_apic_timer_interrupt+0xa6/0x270 [19168.090845] ? apic_timer_interrupt+0xf/0x20 [19168.090850] [19168.090852] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19168.090862] do_raw_spin_lock+0x177/0x190 [19168.090868] _raw_spin_lock+0x21/0x30 [19168.090873] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19168.091499] ? do_raw_spin_unlock+0x75/0x190 [19168.091505] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19168.091771] ? do_wait_intr+0xf0/0xf0 [19168.091780] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19168.092036] mdc_close+0x53b/0xd70 [mdc] [19168.092248] lmv_close+0x18d/0x490 [lmv] [19168.092296] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [19168.092712] ll_close_inode_openhandle+0x493/0xea0 [lustre] [19168.092939] ll_md_real_close+0xa5/0x2a0 [lustre] [19168.093151] ll_md_close+0x274/0x800 [lustre] [19168.093313] ll_file_release+0x128/0xc50 [lustre] [19168.093442] __fput+0x10d/0x440 [19168.093457] ____fput+0x12/0x20 [19168.093462] task_work_run+0xc6/0x110 [19168.099409] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19168.100024] ? __next_timer_interrupt+0x160/0x160 [19168.100041] ptlrpcd+0xa82/0xe40 [ptlrpc] [19168.105060] exit_to_usermode_loop+0x1dd/0x1f0 [19168.105087] do_syscall_64+0x3d6/0x440 [19168.105097] entry_SYSCALL_64_after_hwframe+0x49/0xae [19168.105178] RIP: 0033:0x7f151bee3cd8 [19168.105193] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [19168.105201] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [19168.105209] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [19168.105212] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [19168.105214] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [19168.105216] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [19168.105221] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [19168.724934] ? do_wait_intr+0xf0/0xf0 [19168.726826] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19168.730099] kthread+0x1d1/0x200 [19168.739783] ? set_kthread_struct+0x70/0x70 [19168.743402] ret_from_fork+0x1f/0x30 [19180.055115] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [19180.063209] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19180.119510] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19180.129200] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19180.133966] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19180.138848] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19180.153852] RSP: 0018:ffffb2cb80d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19180.159628] RAX: 000000000000380e RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [19180.164671] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582034154 [19180.171545] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [19180.176431] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [19180.180856] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [19180.187118] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [19180.193694] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19180.203743] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [19180.214355] Call Trace: [19180.217543] [19180.219816] ? show_regs.cold.9+0x22/0x2f [19180.229184] ? watchdog_timer_fn.cold.6+0x62/0x10a [19180.232977] ? __hrtimer_run_queues+0x1c2/0x4d0 [19180.235740] ? watchdog+0x40/0x40 [19180.243676] ? hrtimer_interrupt+0x128/0x2c0 [19180.250900] ? smp_apic_timer_interrupt+0xa6/0x270 [19180.262956] ? apic_timer_interrupt+0xf/0x20 [19180.270186] [19180.273435] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19180.277690] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19180.288896] do_raw_spin_lock+0x177/0x190 [19180.298238] _raw_spin_lock+0x21/0x30 [19180.302127] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19180.310797] ? do_raw_spin_unlock+0x75/0x190 [19180.312494] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19180.319337] ? do_wait_intr+0xf0/0xf0 [19180.322773] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19180.327367] mdc_reint+0x2d/0x110 [mdc] [19180.329948] mdc_unlink+0x658/0x870 [mdc] [19180.333592] lmv_unlink+0x34d/0xbd0 [lmv] [19180.336719] ll_unlink+0x24d/0x7f0 [lustre] [19180.339640] vfs_unlink+0x202/0x330 [19180.342004] do_unlinkat+0x2fa/0x460 [19180.347724] __x64_sys_unlink+0x37/0x50 [19180.349727] do_syscall_64+0xc1/0x440 [19180.351762] entry_SYSCALL_64_after_hwframe+0x49/0xae [19180.355966] RIP: 0033:0x7f151bdee10b [19180.358732] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [19180.376422] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [19180.383097] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [19180.386782] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [19180.391883] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [19180.398708] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [19180.407691] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [19183.078674] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17252s! [19183.089354] Showing busy workqueues and worker pools: [19183.094911] workqueue events: flags=0x0 [19183.098259] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19183.106082] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [19183.117813] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19183.125311] pending: kfree_rcu_monitor [19183.127707] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19183.133378] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [19183.140775] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19183.149244] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [19183.159141] workqueue events_long: flags=0x0 [19183.162180] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19183.173520] pending: lu_objects_flush [obdclass] [19183.183496] workqueue events_unbound: flags=0x2 [19183.192490] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [19183.202342] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [19183.214762] workqueue events_power_efficient: flags=0x80 [19183.216761] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19183.224575] pending: do_cache_clean, neigh_periodic_work, check_lifetime [19183.229217] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19183.233782] pending: neigh_periodic_work [19183.241574] workqueue mm_percpu_wq: flags=0x8 [19183.249778] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19183.260313] pending: vmstat_update, lru_add_drain_per_cpu [19183.266060] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19183.274240] pending: vmstat_update [19183.285157] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19183.293372] pending: lru_add_drain_per_cpu [19183.301924] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [19183.314899] pending: lru_add_drain_per_cpu BAR(48) [19183.320407] workqueue writeback: flags=0x4a [19183.325466] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [19183.332575] pending: wb_workfn [19183.334258] workqueue xprtiod: flags=0xa [19183.337024] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [19183.353457] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [19183.363036] workqueue ipv6_addrconf: flags=0x40008 [19183.372046] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19183.383841] pending: addrconf_verify_work [19183.394297] workqueue knbd0-recv: flags=0x1a [19183.398844] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19183.418085] in-flight: 658:recv_work [19183.422811] workqueue ptlrpc_pinger: flags=0x2 [19183.426540] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19183.437446] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [19183.445869] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17252s workers=4 idle: 32984 37 8 [19183.457390] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19168s workers=2 idle: 661 [19192.056030] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:3:38] [19192.067652] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19192.126052] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19192.133036] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19192.138181] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19192.144326] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19192.149316] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19192.176294] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19192.183637] RAX: 000000000000100c RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [19192.195169] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [19192.202309] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [19192.210203] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [19192.215713] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [19192.222304] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [19192.231191] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19192.233158] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [19192.238354] Call Trace: [19192.241661] [19192.244585] ? show_regs.cold.9+0x22/0x2f [19192.247426] ? watchdog_timer_fn.cold.6+0x62/0x10a [19192.252025] ? __hrtimer_run_queues+0x1c2/0x4d0 [19192.254221] ? watchdog+0x40/0x40 [19192.255691] ? hrtimer_interrupt+0x128/0x2c0 [19192.258124] ? smp_apic_timer_interrupt+0xa6/0x270 [19192.260378] ? apic_timer_interrupt+0xf/0x20 [19192.262010] [19192.268112] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19192.272668] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19192.276724] do_raw_spin_lock+0x177/0x190 [19192.279584] _raw_spin_lock+0x21/0x30 [19192.282807] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19192.289100] process_one_work+0x2c8/0x700 [19192.292193] worker_thread+0x296/0x6e0 [19192.293378] ? rescuer_thread+0x570/0x570 [19192.296431] kthread+0x1d1/0x200 [19192.300236] ? set_kthread_struct+0x70/0x70 [19192.301932] ret_from_fork+0x1f/0x30 [19196.057810] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [19196.061086] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [19196.066239] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19196.066722] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19196.066728] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19196.066735] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [19196.066753] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [19196.066756] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19196.066761] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [19196.066763] RDX: 0000000000007b78 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [19196.066765] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [19196.066767] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [19196.066770] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19196.066773] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [19196.066775] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19196.066777] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [19196.066786] Call Trace: [19196.066791] [19196.066797] ? show_regs.cold.9+0x22/0x2f [19196.066806] ? watchdog_timer_fn.cold.6+0x62/0x10a [19196.066813] ? __hrtimer_run_queues+0x1c2/0x4d0 [19196.066817] ? watchdog+0x40/0x40 [19196.066821] ? hrtimer_interrupt+0x128/0x2c0 [19196.066825] ? smp_apic_timer_interrupt+0xa6/0x270 [19196.066846] ? apic_timer_interrupt+0xf/0x20 [19196.066850] [19196.066853] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [19196.066858] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19196.066864] do_raw_spin_lock+0x177/0x190 [19196.066868] _raw_spin_lock+0x21/0x30 [19196.066874] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19196.067483] obd_mod_free+0x46/0x140 [ptlrpc] [19196.068445] mdc_commit_open+0x91/0xb0 [mdc] [19196.068486] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19196.068683] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19196.069125] after_reply+0xe2c/0x1780 [ptlrpc] [19196.069469] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19196.069713] ? __next_timer_interrupt+0x160/0x160 [19196.069721] ptlrpcd+0xa82/0xe40 [ptlrpc] [19196.069908] ? do_wait_intr+0xf0/0xf0 [19196.069915] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19196.070094] kthread+0x1d1/0x200 [19196.070098] ? set_kthread_struct+0x70/0x70 [19196.070100] ret_from_fork+0x1f/0x30 [19196.407943] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19196.452383] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19196.468568] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19196.475732] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19196.482360] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19196.492327] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [19196.495546] RAX: 0000000000006184 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [19196.498801] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [19196.501756] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [19196.509441] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [19196.517353] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [19196.527572] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [19196.532270] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19196.545481] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [19196.555866] Call Trace: [19196.557068] [19196.562354] ? show_regs.cold.9+0x22/0x2f [19196.565582] ? watchdog_timer_fn.cold.6+0x62/0x10a [19196.571557] ? __hrtimer_run_queues+0x1c2/0x4d0 [19196.578395] ? watchdog+0x40/0x40 [19196.582793] ? hrtimer_interrupt+0x128/0x2c0 [19196.589350] ? smp_apic_timer_interrupt+0xa6/0x270 [19196.595530] ? apic_timer_interrupt+0xf/0x20 [19196.598329] [19196.602997] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19196.610458] do_raw_spin_lock+0x177/0x190 [19196.623210] _raw_spin_lock+0x21/0x30 [19196.625552] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19196.638405] ? do_raw_spin_unlock+0x75/0x190 [19196.644706] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19196.657319] ? do_wait_intr+0xf0/0xf0 [19196.664023] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19196.673402] mdc_close+0x53b/0xd70 [mdc] [19196.679226] lmv_close+0x18d/0x490 [lmv] [19196.684293] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [19196.692314] ll_close_inode_openhandle+0x493/0xea0 [lustre] [19196.699555] ll_md_real_close+0xa5/0x2a0 [lustre] [19196.706418] ll_md_close+0x274/0x800 [lustre] [19196.712443] ll_file_release+0x128/0xc50 [lustre] [19196.715493] __fput+0x10d/0x440 [19196.719280] ____fput+0x12/0x20 [19196.721289] task_work_run+0xc6/0x110 [19196.726740] exit_to_usermode_loop+0x1dd/0x1f0 [19196.733101] do_syscall_64+0x3d6/0x440 [19196.739530] entry_SYSCALL_64_after_hwframe+0x49/0xae [19196.743517] RIP: 0033:0x7f151bee3cd8 [19196.751469] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [19196.769350] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [19196.773143] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [19196.776470] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [19196.780680] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [19196.783957] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [19196.787574] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [19208.052028] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [19208.054257] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19208.101672] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19208.106611] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19208.122055] RIP: 0010:native_safe_halt+0x2e/0x50 [19208.125007] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19208.150073] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19208.161183] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [19208.164530] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [19208.172047] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [19208.179457] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [19208.186725] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [19208.193687] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [19208.199743] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19208.203587] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [19208.208447] Call Trace: [19208.209220] [19208.209736] ? show_regs.cold.9+0x22/0x2f [19208.213668] ? watchdog_timer_fn.cold.6+0x62/0x10a [19208.215924] ? __hrtimer_run_queues+0x1c2/0x4d0 [19208.225083] ? watchdog+0x40/0x40 [19208.226308] ? hrtimer_interrupt+0x128/0x2c0 [19208.230791] ? smp_apic_timer_interrupt+0xa6/0x270 [19208.233626] ? apic_timer_interrupt+0xf/0x20 [19208.236039] [19208.236932] ? native_safe_halt+0x2e/0x50 [19208.239909] kvm_wait+0xcd/0xe0 [19208.242424] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19208.246149] do_raw_spin_lock+0x177/0x190 [19208.249832] _raw_spin_lock+0x21/0x30 [19208.253969] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19208.258831] ? do_raw_spin_unlock+0x75/0x190 [19208.262804] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19208.265980] ? do_wait_intr+0xf0/0xf0 [19208.270391] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19208.276187] mdc_reint+0x2d/0x110 [mdc] [19208.279897] mdc_unlink+0x658/0x870 [mdc] [19208.282659] lmv_unlink+0x34d/0xbd0 [lmv] [19208.284793] ll_unlink+0x24d/0x7f0 [lustre] [19208.287979] vfs_unlink+0x202/0x330 [19208.289366] do_unlinkat+0x2fa/0x460 [19208.290990] __x64_sys_unlink+0x37/0x50 [19208.298521] do_syscall_64+0xc1/0x440 [19208.299791] entry_SYSCALL_64_after_hwframe+0x49/0xae [19208.302502] RIP: 0033:0x7f151bdee10b [19208.304038] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [19208.322221] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [19208.328278] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [19208.332971] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [19208.336652] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [19208.341701] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [19208.346060] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [19213.792236] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17283s! [19213.801650] Showing busy workqueues and worker pools: [19213.810180] workqueue events: flags=0x0 [19213.812808] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19213.819423] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [19213.829953] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19213.841328] pending: kfree_rcu_monitor [19213.851721] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19213.866091] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [19213.872241] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19213.880388] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [19213.890427] workqueue events_long: flags=0x0 [19213.895021] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19213.901852] pending: lu_objects_flush [obdclass] [19213.910874] workqueue events_unbound: flags=0x2 [19213.914236] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [19213.917868] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [19213.930162] workqueue events_power_efficient: flags=0x80 [19213.931829] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19213.942167] pending: do_cache_clean, neigh_periodic_work, check_lifetime [19213.947228] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19213.949417] pending: neigh_periodic_work [19213.956069] workqueue mm_percpu_wq: flags=0x8 [19213.960703] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19213.964599] pending: vmstat_update, lru_add_drain_per_cpu [19213.970228] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19213.977039] pending: vmstat_update [19213.979716] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19213.984773] pending: lru_add_drain_per_cpu [19213.988639] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [19214.003139] pending: lru_add_drain_per_cpu BAR(48) [19214.014095] workqueue writeback: flags=0x4a [19214.016870] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [19214.021992] pending: wb_workfn [19214.025040] workqueue xprtiod: flags=0xa [19214.026972] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [19214.029040] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [19214.033914] workqueue ipv6_addrconf: flags=0x40008 [19214.036274] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19214.040518] pending: addrconf_verify_work [19214.045369] workqueue knbd0-recv: flags=0x1a [19214.047436] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19214.049775] in-flight: 658:recv_work [19214.052261] workqueue ptlrpc_pinger: flags=0x2 [19214.055408] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19214.061421] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [19214.067026] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17283s workers=4 idle: 32984 37 8 [19214.074169] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19198s workers=2 idle: 661 [19220.056907] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [19220.063065] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19220.082399] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19220.086433] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19220.090366] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19220.093312] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19220.095343] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19220.103705] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19220.106162] RAX: 000000000000680f RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [19220.108993] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [19220.112106] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [19220.115195] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [19220.117317] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [19220.119505] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [19220.122191] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19220.124567] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [19220.128306] Call Trace: [19220.130068] [19220.134891] ? show_regs.cold.9+0x22/0x2f [19220.136948] ? watchdog_timer_fn.cold.6+0x62/0x10a [19220.141323] ? __hrtimer_run_queues+0x1c2/0x4d0 [19220.146019] ? watchdog+0x40/0x40 [19220.147365] ? hrtimer_interrupt+0x128/0x2c0 [19220.156080] ? smp_apic_timer_interrupt+0xa6/0x270 [19220.159223] ? apic_timer_interrupt+0xf/0x20 [19220.161778] [19220.162397] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19220.168626] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19220.171348] do_raw_spin_lock+0x177/0x190 [19220.178559] _raw_spin_lock+0x21/0x30 [19220.183881] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19220.196882] process_one_work+0x2c8/0x700 [19220.202162] worker_thread+0x296/0x6e0 [19220.205370] ? rescuer_thread+0x570/0x570 [19220.207378] kthread+0x1d1/0x200 [19220.211353] ? set_kthread_struct+0x70/0x70 [19220.215948] ret_from_fork+0x1f/0x30 [19224.055066] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [19224.059778] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [19224.060534] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19224.060625] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19224.060633] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19224.060636] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [19224.060649] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [19224.060652] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19224.060657] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [19224.060659] RDX: 0000000000005149 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [19224.060662] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [19224.060664] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [19224.060666] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19224.060669] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [19224.060672] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19224.060674] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [19224.060682] Call Trace: [19224.060874] [19224.069093] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19224.069192] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19224.069202] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19224.069206] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19224.069224] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19224.069230] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19224.069235] RAX: 0000000000007581 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [19224.069238] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [19224.069241] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [19224.069243] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [19224.069245] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [19224.069249] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [19224.069252] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19224.069254] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [19224.069262] Call Trace: [19224.069269] [19224.069277] ? show_regs.cold.9+0x22/0x2f [19224.069290] ? watchdog_timer_fn.cold.6+0x62/0x10a [19224.069297] ? __hrtimer_run_queues+0x1c2/0x4d0 [19224.069302] ? watchdog+0x40/0x40 [19224.069307] ? hrtimer_interrupt+0x128/0x2c0 [19224.069311] ? smp_apic_timer_interrupt+0xa6/0x270 [19224.069321] ? apic_timer_interrupt+0xf/0x20 [19224.069326] [19224.069329] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19224.069336] do_raw_spin_lock+0x177/0x190 [19224.069341] _raw_spin_lock+0x21/0x30 [19224.069346] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19224.069655] ? do_raw_spin_unlock+0x75/0x190 [19224.069661] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19224.100787] ? show_regs.cold.9+0x22/0x2f [19224.100807] ? watchdog_timer_fn.cold.6+0x62/0x10a [19224.100817] ? __hrtimer_run_queues+0x1c2/0x4d0 [19224.100821] ? watchdog+0x40/0x40 [19224.100825] ? hrtimer_interrupt+0x128/0x2c0 [19224.100830] ? smp_apic_timer_interrupt+0xa6/0x270 [19224.100838] ? apic_timer_interrupt+0xf/0x20 [19224.100846] [19224.100851] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [19224.100862] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19224.100866] do_raw_spin_lock+0x177/0x190 [19224.100872] _raw_spin_lock+0x21/0x30 [19224.100879] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19224.101172] obd_mod_free+0x46/0x140 [ptlrpc] [19224.101383] mdc_commit_open+0x91/0xb0 [mdc] [19224.101504] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19224.101856] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19224.102103] after_reply+0xe2c/0x1780 [ptlrpc] [19224.102318] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19224.102513] ? __next_timer_interrupt+0x160/0x160 [19224.102523] ptlrpcd+0xa82/0xe40 [ptlrpc] [19224.103143] ? do_wait_intr+0xf0/0xf0 [19224.103153] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19224.103353] kthread+0x1d1/0x200 [19224.103359] ? set_kthread_struct+0x70/0x70 [19224.103363] ret_from_fork+0x1f/0x30 [19224.602274] ? do_wait_intr+0xf0/0xf0 [19224.610059] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19224.614788] mdc_close+0x53b/0xd70 [mdc] [19224.616858] lmv_close+0x18d/0x490 [lmv] [19224.618995] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [19224.621864] ll_close_inode_openhandle+0x493/0xea0 [lustre] [19224.624327] ll_md_real_close+0xa5/0x2a0 [lustre] [19224.626714] ll_md_close+0x274/0x800 [lustre] [19224.628933] ll_file_release+0x128/0xc50 [lustre] [19224.631297] __fput+0x10d/0x440 [19224.632589] ____fput+0x12/0x20 [19224.633423] task_work_run+0xc6/0x110 [19224.635335] exit_to_usermode_loop+0x1dd/0x1f0 [19224.637241] do_syscall_64+0x3d6/0x440 [19224.639047] entry_SYSCALL_64_after_hwframe+0x49/0xae [19224.641355] RIP: 0033:0x7f151bee3cd8 [19224.643107] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [19224.650364] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [19224.654217] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [19224.657360] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [19224.660226] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [19224.663339] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [19224.666188] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [19240.056342] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [writemany:36051] [19240.060302] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19240.080816] CPU: 0 PID: 36051 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19240.085558] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19240.089662] RIP: 0010:native_safe_halt+0x2e/0x50 [19240.091660] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19240.117171] RSP: 0018:ffffb2cb80d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19240.120691] RAX: 0000000000000001 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [19240.133699] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ee582034154 [19240.143229] RBP: ffff8ee582034140 R08: 0000000000000000 R09: 0000000000000000 [19240.151571] R10: 00000000039effe8 R11: ffff8ee54723f000 R12: 0000000000040000 [19240.158519] R13: ffff8ee5820b4140 R14: 000000000001e480 R15: ffff8ee58201e480 [19240.161250] FS: 00007f151c5b4740(0000) GS:ffff8ee582000000(0000) knlGS:0000000000000000 [19240.173216] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19240.180481] CR2: 000055690d6f1140 CR3: 000000010826d005 CR4: 0000000000170ef0 [19240.187998] Call Trace: [19240.189206] [19240.198705] ? show_regs.cold.9+0x22/0x2f [19240.201155] ? watchdog_timer_fn.cold.6+0x62/0x10a [19240.206898] ? __hrtimer_run_queues+0x1c2/0x4d0 [19240.209029] ? watchdog+0x40/0x40 [19240.211270] ? hrtimer_interrupt+0x128/0x2c0 [19240.214662] ? smp_apic_timer_interrupt+0xa6/0x270 [19240.223095] ? apic_timer_interrupt+0xf/0x20 [19240.229694] [19240.235625] ? native_safe_halt+0x2e/0x50 [19240.238859] kvm_wait+0xcd/0xe0 [19240.240813] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19240.243570] do_raw_spin_lock+0x177/0x190 [19240.245707] _raw_spin_lock+0x21/0x30 [19240.247400] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19240.250369] ? do_raw_spin_unlock+0x75/0x190 [19240.256495] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19240.260313] ? do_wait_intr+0xf0/0xf0 [19240.263126] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19240.264950] mdc_reint+0x2d/0x110 [mdc] [19240.275415] mdc_unlink+0x658/0x870 [mdc] [19240.280178] lmv_unlink+0x34d/0xbd0 [lmv] [19240.284273] ll_unlink+0x24d/0x7f0 [lustre] [19240.289328] vfs_unlink+0x202/0x330 [19240.291520] do_unlinkat+0x2fa/0x460 [19240.294060] __x64_sys_unlink+0x37/0x50 [19240.295638] do_syscall_64+0xc1/0x440 [19240.297305] entry_SYSCALL_64_after_hwframe+0x49/0xae [19240.300628] RIP: 0033:0x7f151bdee10b [19240.302663] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [19240.320495] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [19240.326337] RAX: ffffffffffffffda RBX: 0000000000001458 RCX: 00007f151bdee10b [19240.331886] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 00007fff812d9b60 [19240.335454] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [19240.341574] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [19240.347059] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [19244.515645] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17313s! [19244.520063] Showing busy workqueues and worker pools: [19244.522641] workqueue events: flags=0x0 [19244.525736] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19244.529944] pending: cpu_latency_work [ptlrpc], push_to_pool, sec_gc_main [ptlrpc] [19244.535309] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19244.538969] pending: kfree_rcu_monitor [19244.541523] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19244.544900] pending: nfs4_renew_state [nfsv4], key_garbage_collector, osc_grant_work_handler [osc] [19244.549582] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19244.552579] pending: ldlm_pools_recalc_task [ptlrpc], vmstat_shepherd [19244.556748] workqueue events_long: flags=0x0 [19244.559640] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19244.562423] pending: lu_objects_flush [obdclass] [19244.566482] workqueue events_unbound: flags=0x2 [19244.568980] pwq 8: cpus=0-3 flags=0x4 nice=0 active=6/512 refcnt=8 [19244.573258] pending: flush_memcg_stats_dwork, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn, idle_cull_fn [19244.580150] workqueue events_power_efficient: flags=0x80 [19244.583487] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19244.586413] pending: do_cache_clean, neigh_periodic_work, check_lifetime [19244.590667] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19244.594148] pending: neigh_periodic_work [19244.596291] workqueue mm_percpu_wq: flags=0x8 [19244.598224] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19244.601970] pending: vmstat_update, lru_add_drain_per_cpu [19244.608257] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19244.616943] pending: vmstat_update [19244.624413] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19244.632086] pending: lru_add_drain_per_cpu [19244.636112] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [19244.646980] pending: lru_add_drain_per_cpu BAR(48) [19244.653250] workqueue writeback: flags=0x4a [19244.655346] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [19244.661301] pending: wb_workfn [19244.671209] workqueue xprtiod: flags=0xa [19244.673327] pwq 8: cpus=0-3 flags=0x4 nice=0 active=3/256 refcnt=5 [19244.676615] pending: xs_stream_data_receive_workfn, xprt_autoclose, xs_error_handle [19244.686038] workqueue ipv6_addrconf: flags=0x40008 [19244.694423] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19244.706017] pending: addrconf_verify_work [19244.711782] workqueue knbd0-recv: flags=0x1a [19244.714981] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19244.720952] in-flight: 658:recv_work [19244.723447] workqueue ptlrpc_pinger: flags=0x2 [19244.727133] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19244.734346] in-flight: 38:ptlrpc_pinger_main [ptlrpc] [19244.740282] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17314s workers=4 idle: 32984 37 8 [19244.749311] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19229s workers=2 idle: 661 [19248.056062] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:3:38] [19248.061122] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19248.108800] CPU: 2 PID: 38 Comm: kworker/u8:3 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19248.115814] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19248.122994] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19248.130068] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19248.140127] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19248.160754] RSP: 0018:ffffb2cb80dabdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19248.168169] RAX: 0000000000007285 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [19248.175293] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee582134154 [19248.188774] RBP: ffff8ee582134140 R08: 0000000000000000 R09: 0000000000000000 [19248.198269] R10: 00000000039f28ac R11: ffff8ee545869000 R12: 00000000000c0000 [19248.202988] R13: ffff8ee5821b4140 R14: 000000000001e480 R15: ffff8ee58211e480 [19248.214176] FS: 0000000000000000(0000) GS:ffff8ee582100000(0000) knlGS:0000000000000000 [19248.223623] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19248.227229] CR2: 0000556e58d4bde8 CR3: 0000000093c16004 CR4: 0000000000170ee0 [19248.233206] Call Trace: [19248.235319] [19248.237267] ? show_regs.cold.9+0x22/0x2f [19248.241104] ? watchdog_timer_fn.cold.6+0x62/0x10a [19248.245297] ? __hrtimer_run_queues+0x1c2/0x4d0 [19248.250106] ? watchdog+0x40/0x40 [19248.252303] ? hrtimer_interrupt+0x128/0x2c0 [19248.260057] ? smp_apic_timer_interrupt+0xa6/0x270 [19248.265330] ? apic_timer_interrupt+0xf/0x20 [19248.266808] [19248.267602] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19248.277202] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19248.283616] do_raw_spin_lock+0x177/0x190 [19248.284828] _raw_spin_lock+0x21/0x30 [19248.293659] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19248.298963] process_one_work+0x2c8/0x700 [19248.302104] worker_thread+0x296/0x6e0 [19248.305006] ? rescuer_thread+0x570/0x570 [19248.310136] kthread+0x1d1/0x200 [19248.314018] ? set_kthread_struct+0x70/0x70 [19248.316208] ret_from_fork+0x1f/0x30 [19252.056571] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_rcv:2351] [19252.057661] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36052] [19252.062182] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) [19252.065438] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19252.065518] CPU: 3 PID: 36052 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19252.065526] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19252.065529] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19252.065544] Code: 4c 03 2c c5 80 58 e8 b2 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19252.065548] RSP: 0018:ffffb2cb81c13af0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19252.065553] RAX: 00000000000052b3 RBX: ffff8ee5497bb2e0 RCX: 0000000000000001 [19252.065555] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8ee5821b4154 [19252.065557] RBP: ffff8ee5821b4140 R08: 0000000000000000 R09: 0000000000000000 [19252.065559] R10: 00000000039f008e R11: ffff8ee545667000 R12: 0000000000100000 [19252.065561] R13: ffff8ee582034140 R14: 000000000001e480 R15: ffff8ee58219e480 [19252.065564] FS: 00007f151c5b4740(0000) GS:ffff8ee582180000(0000) knlGS:0000000000000000 [19252.065567] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19252.065569] CR2: 00007fc10f0dffd0 CR3: 0000000105681006 CR4: 0000000000170ee0 [19252.065579] Call Trace: [19252.065584] [19252.065590] ? show_regs.cold.9+0x22/0x2f [19252.065599] ? watchdog_timer_fn.cold.6+0x62/0x10a [19252.065605] ? __hrtimer_run_queues+0x1c2/0x4d0 [19252.065609] ? watchdog+0x40/0x40 [19252.065613] ? hrtimer_interrupt+0x128/0x2c0 [19252.065617] ? smp_apic_timer_interrupt+0xa6/0x270 [19252.065624] ? apic_timer_interrupt+0xf/0x20 [19252.065629] [19252.065632] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19252.065639] do_raw_spin_lock+0x177/0x190 [19252.065644] _raw_spin_lock+0x21/0x30 [19252.065649] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19252.066114] ? do_raw_spin_unlock+0x75/0x190 [19252.066120] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19252.066326] ? do_wait_intr+0xf0/0xf0 [19252.066334] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19252.066550] mdc_close+0x53b/0xd70 [mdc] [19252.066588] lmv_close+0x18d/0x490 [lmv] [19252.066615] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [19252.067006] ll_close_inode_openhandle+0x493/0xea0 [lustre] [19252.067124] ll_md_real_close+0xa5/0x2a0 [lustre] [19252.067208] ll_md_close+0x274/0x800 [lustre] [19252.067347] ll_file_release+0x128/0xc50 [lustre] [19252.067441] __fput+0x10d/0x440 [19252.067450] ____fput+0x12/0x20 [19252.067454] task_work_run+0xc6/0x110 [19252.067462] exit_to_usermode_loop+0x1dd/0x1f0 [19252.067468] do_syscall_64+0x3d6/0x440 [19252.067472] entry_SYSCALL_64_after_hwframe+0x49/0xae [19252.067479] RIP: 0033:0x7f151bee3cd8 [19252.067487] Code: 02 02 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 7a 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec [19252.067491] RSP: 002b:00007fff812d9af8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [19252.067496] RAX: 0000000000000000 RBX: 000000000000a950 RCX: 00007f151bee3cd8 [19252.067498] RDX: 0000000000000038 RSI: 00007fff812d9f60 RDI: 0000000000000003 [19252.067500] RBP: 00007fff812d9f60 R08: 00007f151c186230 R09: 00007f151c186260 [19252.067502] R10: 00007fff812d98a2 R11: 0000000000000246 R12: 0000000000000003 [19252.067504] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [19252.487717] mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19252.526843] CPU: 1 PID: 2351 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19252.555165] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19252.573843] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [19252.582750] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a a7 4e c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [19252.597389] RSP: 0018:ffffb2cb81937bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19252.608633] RAX: 0000000000000003 RBX: ffff8ee5497bb2e0 RCX: ffff8ee58209e480 [19252.617540] RDX: 00000000000074e3 RSI: 0000000000000003 RDI: ffff8ee5497bb2e0 [19252.622254] RBP: ffff8ee5820b4140 R08: 0000000000000008 R09: 00000000000000e8 [19252.637235] R10: ffffffffb3008b80 R11: ffffb2cb819378f0 R12: 0000000000080000 [19252.651003] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19252.653548] FS: 0000000000000000(0000) GS:ffff8ee582080000(0000) knlGS:0000000000000000 [19252.665173] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19252.670170] CR2: 000055b5b411df44 CR3: 0000000093c16004 CR4: 0000000000170ee0 [19252.675125] Call Trace: [19252.677037] [19252.681656] ? show_regs.cold.9+0x22/0x2f [19252.684140] ? watchdog_timer_fn.cold.6+0x62/0x10a [19252.688343] ? __hrtimer_run_queues+0x1c2/0x4d0 [19252.694921] ? watchdog+0x40/0x40 [19252.698179] ? hrtimer_interrupt+0x128/0x2c0 [19252.701757] ? smp_apic_timer_interrupt+0xa6/0x270 [19252.706765] ? apic_timer_interrupt+0xf/0x20 [19252.712023] [19252.719063] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [19252.721253] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19252.732540] do_raw_spin_lock+0x177/0x190 [19252.734064] _raw_spin_lock+0x21/0x30 [19252.739216] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19252.745744] obd_mod_free+0x46/0x140 [ptlrpc] [19252.749647] mdc_commit_open+0x91/0xb0 [mdc] [19252.757780] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19252.766427] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19252.768570] after_reply+0xe2c/0x1780 [ptlrpc] [19252.775711] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19252.780044] ? __next_timer_interrupt+0x160/0x160 [19252.783809] ptlrpcd+0xa82/0xe40 [ptlrpc] [19252.792737] ? do_wait_intr+0xf0/0xf0 [19252.804193] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19252.811808] kthread+0x1d1/0x200 [19252.815522] ? set_kthread_struct+0x70/0x70 [19252.823886] ret_from_fork+0x1f/0x30